Data Scientist - Agent Evaluations & Quality

Responsibilities

• Architect and maintain automated evaluation pipelines that measure agent quality across product surfaces.
• Translate agent capabilities into explicit pass, partial-pass, and failure criteria for complex multi-step tasks.
• Build representative gold datasets and regression suites covering real workflows, edge cases, and adversarial scenarios.
• Define meaningful metrics — task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability.
• Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and track grader agreement.
• Compare models, prompts, and implementations using rigorous offline experiments and production evidence.
• Analyze traces and production outcomes to identify root causes and build a practical failure taxonomy.
• Turn production failures into regression cases and continuously close gaps in evaluation coverage.
• Build dashboards and release-quality signals that make results actionable for engineering, product, and leadership.
• Recommend improvements to capability engineers and verify that fixes raise quality without unacceptable regressions.

Languages