Agent Evaluation and Self-Improvement Engineer

Responsibilities

• Build evaluation that works the same across harnesses, with automated checks on every change that catch regressions before users do.
• Mine production traces for failures, new usage patterns, missing skills and tools, cost and latency hot spots, and successful runs. Turn them into eval cases and improvement signals.
• Evaluate the eval datasets themselves: coverage of real usage, duplicate cases, and how closely they match production.
• Build the self-improvement loop for evolving harnesses: agents propose changes to their own skills, plugins, and harness configuration, and sandboxed evals test each one. Only measurable gains ship, with human sign-off and limits agents can't change.
• Create realistic enterprise environments: sandboxed replicas of issue trackers, code repositories, chat, and internal APIs, filled with synthetic or anonymized data, where agents safely run long, end-to-end tasks.
• Measure multi-agent swarms as a team: how agents split work, whether handoffs succeed, and whether permissions hold at every hop. Record how long each task takes and how many tokens it uses.
• Keep evals honest: calibrate model and agent judges against human labels. Catch agents that cheat, such as working backward from expected answers or reaching data and tools they shouldn't access. We work closely with product, platform, and hardware teams throughout NVIDIA, and we help shape open evaluation standards…
• What we need to see
• BS or MS in Computer Science, Engineering, or related field (or equivalent experience)
• 12+ years building large-scale systems, including evaluation systems for ML or agents and the data pipelines that feed them

Languages