Senior Software Engineer - AI Inference Performance

Responsibilities

• Lead end-to-end analysis of LLM/VLM inference processes. Define representative prefill and decode workloads. Optimize time to first token, inter-token latency, P99 end-to-end latency, processing efficiency, and key-value (KV) cache capacity. For multimodal models, isolate preprocessing, encoder, and decoder costs.
• Build speed-of-light and roofline models to quantify performance headroom. Connect arithmetic intensity, bandwidth, occupancy, memory hierarchy, and communication costs to clear optimization hypotheses.
• Profile workloads using NVIDIA Nsight Systems, Nsight Compute, PyTorch Profiler, and custom instrumentation. Eliminate bottlenecks in host code, CUDA kernels, memory, communication, and scheduling.
• Tune serving hyperparameters and techniques such as batching, KV-cache management, quantization, speculative decoding, CUDA Graphs, and model parallelism. Choose them based on workload, hardware, model quality, and service-level objectives.
• Build and optimize performance-critical kernels, including attention, matrix multiplication, mixture-of-experts routing, quantization, and data movement. Use CUDA, CUTLASS, Triton, or related technologies.
• Establish repeatable benchmarks, canonical run records, and performance regression gates.
• Manage aspects such as model, precision, hardware, topology, software, features, and workload; Balance between performance and accuracy.
• Collaborate across with various teams and contribute high-quality upgrades to TensorRT-LLM, vLLM, SGLang, or associated projects.
• What we need to see
• More than 6 years of experience in full-stack LLM/VLM inference performance involving models, serving, distributed runtimes, kernels, and hardware. Your efforts result in measurable gains in production or production-representative environments.

Languages