Senior QA / ML Tester

Responsibilities

• Design and implement a functional test suite for the AWS AgentCore Evaluation API using pytest, covering known-good / known-bad session pair validation, multi-evaluator execution correctness, and edge case handling
• Develop and maintain integration tests for on-demand evaluation mode, integrated into the CI/CD pipeline with automated execution on each build
• Validate online mode sampling accuracy, design test scenarios, define acceptance criteria, and report deviations with reproducible evidence
• Conduct and document a feasibility assessment for non-AgentCore runtime evaluation: analyze alternative runtimes, define evaluation methodology, and deliver a structured findings report
• Test OpenTelemetry trace-based evaluation inputs and validate ADOT trace ingestion, trace structure correctness, and evaluator input integrity
• Collaborate with platform engineers to clarify evaluation contracts, reproduce defects, and align on quality gates
• Maintain test documentation, including test plans, test reports, defect logs, and evaluation feasibility artifacts in Confluence/Jira
• Participate in Agile ceremonies, including sprint planning, daily standups, demos, and retrospectives
• Contribute to EngX practices such as code review of test scripts, CI/CD pipeline integration, and test coverage reporting

Languages