Lead Engineer, AI Platform

Responsibilities

• Build and own our evaluation infrastructure.
• Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation.
• Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering…
• Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone.
• Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity.
• Lead and grow the AI Quality engineering team.
• Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself.
• Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.

Languages