Senior ML / Evaluation Engineer

Responsibilities

• Design and implement evaluation frameworks for LLM-based applications and AI agents
• Develop LLM-as-a-Judge and deterministic, code-based evaluators
• Build custom Python evaluators for quality and behavioral checks
• Define evaluation criteria, metrics, thresholds, and acceptance rules
• Evaluate agent behavior across individual responses, tool calls, and complete workflows
• Work with OpenTelemetry traces and spans as evaluation data
• Integrate evaluations into CI/CD pipelines and automated deployment gates
• Enable continuous quality monitoring of solutions in production
• Establish reusable evaluation patterns and engineering standards
• Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process

Languages

Required

• 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering
• Strong Python development experience
• Hands-on experience with LLM/GenAI evaluation
• Experience integrating AI/ML quality checks into CI/CD

Nice to have

• Hands-on experience with AWS AgentCore Evaluation
• Knowledge of CloudWatch metrics and production monitoring

Other open roles here

• Senior/Lead Power BI Software EngineerRemote · Ukraine
• Delivery Manager (AI Native)Remote · Ukraine
• Senior Financial Manager (m/f/d)Remote · Germany
• Principal, Business ConsultingRemote · United States
• Senior Delivery Manager - .NET/JavaIndia · On-site · relocation