Senior AI Reliability Engineer

Responsibilities

• Instrument production LLM, RAG and agentic applications with AI telemetry based on OpenTelemetry and APM-native AI monitoring
• Implement distributed tracing across multi-model chains, agent workflows and retrieval-augmented generation pipelines to profile systemic latency and failure points
• Define and measure AI-native SLIs and SLOs: time to first token (TTFT), throughput, error and refusal rates, cost per request, semantic drift, hallucination boundaries, contextual accuracy
• Set up structured semantic logging and prompt/response monitoring for quality analysis
• Build and run evaluation loops for output quality and safety — golden sets, LLM-as-a-judge, Ragas/DeepEval-style frameworks — and wire them into CI/CD and runtime
• Track token-based cloud spend, model API rate limits and quota consumption; drive AI cost optimization
• Configure AI gateways for API load balancing, failover and fallback models across multiple LLM providers
• Implement guardrails: prompt injection and jailbreak filtering, output compliance, bias and safety constraints
• Design detection, triage, restore and problem management workflows for AI incidents; integrate autonomous AI agents into RCA to parse logs, form hypotheses and correlate state changes
• Support rollback, canary and fail-safe patterns for model, prompt and configuration releases; maintain reproducibility through versioning of data, code, prompts and models

Languages