Senior AI Reliability Engineer

Responsibilities

• Add AI telemetry to production LLM, RAG, and agentic applications using OpenTelemetry and APM-native AI monitoring tools
• Deploy distributed tracing across multi-model chains, agent workflows, and retrieval-augmented generation pipelines to identify systemic latency and failure points
• Establish and track AI-native SLIs and SLOs, including time to first token (TTFT), throughput, error and refusal rates, cost per request, semantic drift, hallucination boundaries, and contextual accuracy
• Implement structured semantic logging and prompt/response monitoring to support quality analysis
• Create and maintain evaluation loops for output quality and safety using golden sets, LLM-as-a-judge methods, and Ragas/DeepEval-style frameworks, integrating them into CI/CD and runtime environments
• Monitor token-based cloud spend, model API rate limits, and quota usage while driving AI cost optimization efforts
• Set up AI gateways to manage API load balancing, failover, and fallback models across multiple LLM providers
• Build guardrails covering prompt injection and jailbreak filtering, output compliance, and bias and safety constraints
• Develop detection, triage, restoration, and problem management workflows for AI incidents, incorporating autonomous AI agents into root cause analysis to parse logs, generate hypotheses, and correlate state changes
• Enable rollback, canary, and fail-safe patterns for model, prompt, and configuration releases, while maintaining reproducibility through versioning of data, code, prompts, and models

Languages