Agent Evaluation Engineer - Build-Time Framework & Deployment Gates

Responsibilities

• Design and implement build-time evaluation frameworks for agentic workflows using LangGraph or comparable orchestration frameworks
• Create deterministic and LLM-as-judge grading pipelines covering reasoning, trajectory accuracy, and output quality
• Develop test harnesses for multi-turn conversational simulations and context-retention scoring
• Define reliability assessment methods including multi-trial metrics (pass@k, pass^k)
• Implement CI/CD deployment gates that enforce quality thresholds and block releases not meeting standards
• Integrate staging validation, shadow-mode traffic comparison, and A/B rollout control in deployment pipelines
• Leverage AWS AgentCore Evaluations for on-demand and online scoring components connected to production feedback
• Convert production incidents into reusable regression cases for continuous quality improvement
• Collaborate with engineering and DevOps teams to embed evaluation gates into automated workflows

Languages