Senior Product Manager - AI Inference Performance

Responsibilities

• Own the inference performance roadmap.
• Set direction across the stack: how models are represented, how memory and state are managed, how requests are scheduled and served, and how tokens get generated. The techniques change fast. Judge which ones matter, then decide what we build, what we adopt, and what we retire.
• Build platforms, not one-offs.
• Deliver capabilities that generalize across model families, deployment topologies, and customer sizes.
• Build for easy adoption, sane defaults, and extensibility.
• Agentic and Multi-Turn Workloads: Define the performance strategy for agentic applications, where long-running sessions, tool-call stalls, and unexpected output lengths break the assumptions built into single-turn serving.
• Drive capabilities around cross-turn cache reuse, request prioritization, and efficient handling of idle time in agent loops.
• Framework & Ecosystem Strategy: Define how our optimizations land across TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo.
• Partner with open-source communities and internal engineering teams so customers get great performance on NVIDIA hardware.
• Benchmarking & Performance Claims

Languages