Senior System Software Engineer - LocalAI

Responsibilities

• Partner with NVIDIA’s software, research, architecture, and product teams to align technical requirements and strategic priorities, fostering the AI ecosystem on RTX and DGX PCs.
• Build and optimize the local AI inference stack for RTX, RTX Pro, and DGX GPUs, with a focus on performance, stability, and scalability across diverse hardware architectures.
• Design and develop modern inference runtimes and execution stacks using frameworks such as llama.cpp, vLLM, PyTorch, WinML, DXCGC, and TensorRT-RTX, supporting LLM, vision-language, TTS, ASR, and diffusion-based AI workloads.
• Perform end-to-end optimization of AI models, data pipelines, and inference runtimes to maximize performance on current and next-generation GPU architectures. Apply model optimization techniques, including quantization, pruning, sparsity, and distillation, to enable efficient deployment of large models on local and…
• Conduct system-level debugging, performance tuning, and performance-accuracy trade-off analysis; develop infrastructure for performance and accuracy sweeps; analyse results to identify gaps and drive fixes; and establish engineering guidelines to accelerate bring-up and ensure production readiness of new models and…
• What we need to see
• 5+ Years of experience with Bachelor’s, Master’s, or PhD in Computer Science, Software Engineering, Mathematics, or a related field, or equivalent experience.
• Excellent C++ programming and debugging skills, with a strong foundation in data structures, algorithms, and machine learning.
• Proven experience developing and optimizing AI inference pipelines and applications using ML/DL frameworks such as Llama.cpp, vLLM, PyTorch, Windows ML, DXCGC, and TensorRT.
• Deep understanding of inference backends and runtime internals, including scheduling, memory management, KV-cache behaviour, graph execution, quantization, and hardware-aware optimization.

Languages