Machine Learning Engineer - Inference Optimization

Responsibilities

• Optimize inference latency, throughput, and cost for large-scale ML models in production
• Profile and bottleneck GPU/CPU inference pipelines (memory, kernels, batching, IO)
• Implement and tune techniques such as
• Quantization (fp16, bf16, int8, fp8)
• KV-cache optimization & reuse
• Speculative decoding, batching, and streaming
• Model pruning or architectural simplifications for inference
• Collaborate with research engineers to productionize new model architectures
• Build and maintain inference-serving systems (e.g. Triton, custom runtimes, or bespoke stacks)
• Benchmark performance across hardware (NVIDIA / AMD GPUs, CPUs) and cloud setups

Languages