NCX Senior Engineer

Responsibilities

• Lead NCP Day 2 operational readiness efforts.
• Collaborate directly with NVIDIA Cloud Partners to set up the systems, procedures, automation, and operational methods necessary to consistently manage NVIDIA accelerated infrastructure following initial deployment and activation.
• Build continuous infrastructure validation.
• Develop and implement methods to continuously validate GPU, CPU, storage, and network health. Do this across large-scale AI clusters to identify degraded infrastructure before it impacts critical training or inference workloads.
• Establish observability and operational telemetry. Help NCPs implement comprehensive telemetry, monitoring, alerting, dashboards, and operational signals across compute, GPU, InfiniBand/RoCE networking, storage, Kubernetes, and AI workloads.
• Develop automated detection and remediation.
• Build workflows to detect, isolate, drain, repair, validate, and return unhealthy infrastructure to service while minimizing disruption to customer workloads.
• Refine fleet lifecycle administration.
• Implement scalable strategies for managing sizable GPU fleets, including NVIDIA driver and firmware lifecycle administration, Kubernetes node maintenance, OS patching, configuration management, upgrades, and configuration drift identification.
• Operationalize NVIDIA reference architectures.

Languages