AI Architect

Responsibilities

• Define the reference architecture for FourKites agents: the agent loop, tool execution, state and checkpointing, retries and recovery, guardrails, permissions, and human-in-the-loop handoffs
• Design for long-running, multi-step work that spans many context windows and recovers cleanly when a tool, carrier API or model call fails
• Decide where each problem is best solved (frontier LLM, open-weight model, fine-tuned encoder, or plain code) and own model routing, cost-per-task and latency budgets across the fleet
• Integrate agents with FourKites' event-driven platform (Kafka, microservices, Postgres) and with customer systems through APIs and MCP servers
• Set the standard for what goes into an agent's context window (system prompts, tool definitions, retrieved documents, memory, prior turns) within an explicit token budget
• Build shared components for retrieval, compaction and summarization, sub-agent isolation, prompt caching and persistent memory so teams stop reinventing them
• Diagnose context failures (lost instructions, tool confusion, quality degradation as the window fills) and turn fixes into reusable harness patterns
• Build the offline eval framework: golden datasets from production traces, code-based checks, LLM-as-judge graders calibrated against human labels, and CI regression suites that gate every prompt, model or tool change
• Build online evaluation: scoring sampled production traffic, trace- and task-level quality metrics, drift detection, shadow and A/B rollouts, user feedback signals, and quality alerts
• Connect eval metrics to the business outcomes customers pay for, such as tracking accuracy, exceptions resolved without a human, and appointment fill rate

Languages