AI Evals for Cross-Functional Teams at DoorDash

AI evals for cross-functional teams started as an engineering harness at DoorDash and turned into an ops workflow. That shift is the core of a new talk by its GenAI platform team, shared via AI Engineer.

DoorDash (NASDAQ:DASH) runs a horizontal GenAI platform. Product teams build on its primitives, and the platform team says its job is to balance accuracy, latency and cost.

The four primitives

Three pillars are familiar. An LLM gateway lets teams swap models. An agent gateway centralizes tool and agent connectivity, auth and identity. Open-weights hosting attacks cost.

The fourth is evals. The team frames it as the control plane for quality. Models change, agents chain tools, costs drift. Evals decide what actually shipped.

Why one platform has to serve many masters

Needs diverge fast inside DoorDash. Consumer discovery and shopping needs session-level quality judgments. Personalization ML needs human judgment at scale. Multi-agent systems need trajectory-based evals.

The answer was not one more dashboard for engineers. Guidance from co-founder Andy Fang pushed the team to be UI-first so strategy and operations, PMs and labeling partners could contribute directly.

Then it went API-first so engineers could build without waiting on the central team. Now it is workflow-first, where ops and PMs run the whole eval operation themselves.

How the loop actually runs

DoorDash describes an eight-step continuous loop. Trace sessions, sample down to a manageable set, annotate with domain knowledge, review, build golden datasets, calibrate judges, monitor, repeat.

Two surfaces power it. A telemetry layer holds traces, scores and observations, accessible via MCP, SDK and APIs. A workflow layer hosts annotation tasks, golden datasets and judge calibration.

Tracing and sampling is step one. Capture what agents and LLMs actually output and make it browsable. Everything sits on stable table APIs that also power the UIs.

Annotation without bottlenecks

The hard part is annotation at scale. Different teams need different UIs: image annotation, manual testing, menu parses from restaurants. Building bespoke UIs for each does not scale.

DoorDash doubled down on API-first. Strategy and ops staff use coding agents like Codex and Claude Code to vibe-code their own annotation UIs on top of the same APIs.

The example shown is deliberately boring. A clean menu parser that does the job. The point is ownership: operators build what they need and keep the workflow moving.

Calibrating judges without engineering tickets

With golden data in hand, teams calibrate LLM-as-a-judge. Start with a judge prompt, run baseline scores on traces, then optimize.

The team uses the GEPA library, described in the talk as JPEA, for prompt optimization. The bigger change is process. Calibration is a self-serve UI.

PMs or operators pick a model, Gemini, Claude or OpenAI, set configs and run the loop themselves. A visualization shows the original prompt versus the calibrated prompt and the lift. That transparency builds trust and lets different teams assign prompt ownership differently, sometimes strat-ops, sometimes product, sometimes engineering.

Why this matters

Most eval startups sell to engineers. DoorDash built for the people who actually know if an answer was good: ops and domain experts. That is a different buyer and a different muscle.

It mirrors where the market is going. LangSmith, Braintrust and Langfuse all raced to add tracing, datasets and evals in 2024 and 2025. Humanloop and Evidently lean into domain-specific judges. The frontier is no longer just scoring outputs. It is trajectory evals for agents, cost-aware routing via gateways, and self-serve calibration for non-technical owners.

DoorDash also ties evals to dollars. The team reported material drops in per-annotation cost at thousands of rows per week and faster iteration after moving to self-serve annotation and judge calibration. It did not disclose absolute spend, which leaves the ROI hard to benchmark. StartupHub.ai data shows DoorDash’s GenAI platform sits at 71/100 with VERIFIED financials of $80M raised at Series A in 2023, a useful anchor for how heavily horizontal platform bets are now funded relative to point-tool eval vendors.

What is missing will matter more than what was shown. No disclosed reliability numbers for calibrated judges, no latency or cost tradeoffs for open-weights versus frontier models, and no detail on how trajectory failures get auto-sampled. Those are the exact gaps that break eval platforms in production.

DoorDash’s lesson is blunt. If only engineers can eval, you will not eval enough.