AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash
DoorDash's GenAI platform team transformed evaluation from an engineering-centric task into a cross-functional workflow. By adopting an API-first strategy, they empowered non-engineers like Strategy & Operations to "vibe code" their own annotation UIs using coding agents and self-serve calibrate LLM judge prompts. This approach dramatically reduced annotation costs, accelerated iteration, and fostered broader organizational ownership of AI quality.
The DoorDash GenAI platform team, a horizontal entity supporting product teams, aims to balance accuracy, latency, and cost using primitives like an LLM gateway (for model switching), an agent gateway (for tool/agent connection, auth, identity), and open-weight model hosting. Originally focused on these three forces, the team identified evaluation (evals) as a crucial fourth pillar due to varying needs across internal product teams, ranging from session-level quality judgments for consumer discovery to trajectory-based evals for multi-agent systems.
To address these diverse and evolving needs, the team shifted its approach from “UI-first” (providing UIs for non-engineers) to “API-first” (enabling engineers to build custom systems), and ultimately to “workflow-first” by leveraging coding agents. This evolution empowers Strategy & Operations (S&O) and Product Managers (PMs) to navigate the platform and run operations, rather than solely relying on central engineering for every UI.
Evals as a Cross-Functional Endeavor
Nachiket Paranjape emphasized that evals are not merely an engineering harness but a “team sport” and a cross-functional effort vital for integrating domain-specific knowledge into AI quality. This reframing involves several key groups:
- Strategy & Operations: Set the overall quality bar and priorities.
- Product Managers: Translate quality requirements into concrete rubrics and workflows.
- Operations Teams: Execute annotations.
- Engineering Teams: Provide the foundational APIs, telemetry, datasets, and judges.
The ownership of judge prompts is flexible, with S&O, PMs, or engineering teams sometimes owning them. This variation is viewed as a sign of organizational learning rather than a problem to be standardized, highlighting the platform’s adaptability.
The Continuous Evaluation Loop
The team has established a continuous iteration loop for AI quality:
- Tracing: Capture and view agent/LLM outputs and session traces.
- Sampling: Reduce traces to a manageable subset for human review.
- Annotating: Add human judgment and domain expertise to sampled data.
- Reviewing & Golden Data Sets: Review annotations and promote a subset to “golden data sets” for calibration.
- Calibrating: Use golden data sets to calibrate LLM judges.
- Monitoring: Continuously monitor and iterate through the loop.
Platform Architecture: Telemetry and Workflow Surfaces
The DoorDash platform provides two main surfaces:
- Telemetry Layer: Consolidates traces, scores, and observations, accessible via internal tools (like an “MCP”), SDKs, and APIs.
- Workflow Layer: Designed for S&O and product teams, enabling them to set up annotation tasks, review golden data sets, and create/calibrate judges.
API-First and “Vibe Coding” Annotation UIs
A cornerstone of their strategy is the API-first approach, providing stable APIs for scores, datasets, and platform access. This empowered S&O teams to utilize coding agents (e.g., CodeX, Claude) to “vibe code” their own custom annotation UIs. This was crucial because the platform team recognized it couldn’t anticipate every specific UI need, such as grading restaurant menus or reviewing images. By decentralizing UI creation, operators could quickly build interfaces tailored to their unique use cases, drastically improving flexibility and speed.
Self-Serve Judge Prompt Calibration
To further reduce friction and empower non-engineers, DoorDash developed a self-serve UI for calibrating LLM judge prompts. The process involves:
- Starting with a baseline judge prompt.
- Running LLM judges on traces to get baseline scores.
- Optimizing the prompt using libraries like JPEA (a common prompt optimization library).
- Elevating the optimized prompt as the new LLM judge.
The UI is designed for product managers and operators, allowing them to configure and run calibration loops using various models (Gemini, Claude, OpenAI). A critical feature is the side-by-side display of the original and calibrated prompts, building trust and transparency by showing users what changes were made during optimization.
Impact and Results
This comprehensive strategy has yielded significant benefits:
- Cost Reduction: A substantial decrease in per-annotation cost, especially vital given thousands of rows annotated weekly at DoorDash’s scale.
- Increased Velocity: Faster iteration cycles for teams.
- Self-Serve Empowerment: Teams can calibrate their own judges and build annotation UIs independently.
- Reuse of Infrastructure: Leveraging existing DoorDash infrastructure to accelerate development.
The continuous loop and API-first philosophy have enabled DoorDash to ship reliable, high-quality AI products with remarkable speed and efficiency, making evals a truly cross-functional asset.