HPC Infrastructure Engineer - GPU Clusters

Responsibilities

• Operate and improve our GPU fleet end to end: provisioning, scheduling, monitoring, upgrades, capacity planning
• Build automation that keeps the fleet healthy without human intervention — node health checks, automated draining and remediation, burn-in pipelines for new capacity
• Own the stack beneath the training code: OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, high-speed networking (InfiniBand/RoCE)
• Run and tune job scheduling (Slurm or similar) so researchers get compute fairly and fast
• Build and maintain high-performance storage for datasets and checkpoints
• Hunt down performance problems: stragglers, degraded links, thermal issues, flaky GPUs — and fix the class of problem, not just the instance
• Evaluate rented GPU capacity: benchmark it, validate it, hold providers to their SLAs
• Hands-on hardware work when it's needed: racking, cabling, diagnostics, coordinating with datacenter staff and vendors
• Keep clusters secure by default: access control, network isolation, secrets

Languages