Lyft's Nick Ung on Building Better AI Evals

Nick Ung, a data science manager at Lyft, shared crucial insights into the process of building effective evaluation systems for AI agents, emphasizing the need for "evals that actually matter." Speaking alongside Ashe, Ung detailed Lyft's approach to developing and scaling their customer support AI agents, highlighting the importance of moving beyond superficial metrics to create evaluations that are both consequential and actionable.

The AI Evaluation Flywheel: Development to Production

Ung outlined Lyft's AI agent evaluation system, which operates in two distinct phases: development and production. During the development phase, key activities include agent engineering, managing context, building retrieval pipelines, defining tools, constructing agent graphs, and writing system prompts. Crucially, before an AI agent is deployed to live users, it undergoes a rigorous offline evaluation process. Ung drew parallels between AI agent development and traditional machine learning model building, stressing the necessity of robust offline evaluations to ensure sufficient performance.

For AI agents, particularly those in multi-turn conversational use cases like customer support, offline evaluation involves simulating conversations. This requires a synthetic dataset representative of production traffic. An LLM is used to play out these simulated conversations, and another LLM acts as a judge to evaluate the quality of the interactions. A "launch gate" mechanism is employed, ensuring that predefined criteria are met before deployment. Ung emphasized that the goal is to avoid using live users as test subjects, a practice he considers poor data science.

Addressing the "LLM User" Problem and Data Realism

A significant challenge identified by Lyft was ensuring their simulated users in offline evaluations were realistic. Initially, they used off-the-shelf LLMs, which resulted in overly polite and patient user interactions, leading to optimistic and misleading performance estimates (e.g., a 90%+ pass rate). To counter this, Lyft fine-tuned their LLM user simulator with actual customer verbatim data. This approach made the evaluation more challenging but also more reflective of real-world user behavior, which often involves impatience and frustration.

Lyft also focused on defining distinct user personas, such as "Bypasser," "Refund Seeker," and "AI Skeptics," to ground the LLM user simulations more closely to real user profiles. This persona-driven approach, inspired by research from Microsoft, aims to create more accurate and challenging test scenarios.

The Pitfalls of Generic Metrics and the Rise of actionable evals

Akshay Sharma, also from Lyft's team, discussed the common pitfalls in AI evaluation. He noted that relying solely on pre-built metrics, such as those found in frameworks like DeepEval, can be problematic because they often lack actionability. Metrics like "tool usage appropriateness," "response helpfulness," "conversation naturalness," and "conversation completeness" provide scores but don't offer clear guidance on how to improve the agent. Sharma stressed the importance of developing metrics that are actionable and directly tied to business outcomes.

The team shifted from scalar metrics to a simpler pass/fail binary outcome. They collaborate closely with operations and quality teams to develop rubrics that assess the quality of LLM responses based on specific failure modes. These rubrics define scenarios where an AI agent should fail, such as trying too many times to educate a user, escalating too soon, or making factual errors.

Statistical Rigor in Evaluation

To ensure the validity and reliability of evaluation numbers, statistical rigor is essential. This includes using confidence intervals to report uncertainty in judge agreement and pass rates, rather than relying solely on point estimates. Calibration checks are also crucial to ensure that a judge's stated confidence aligns with its observed accuracy. Furthermore, effective sampling, including slicing and stratifying data, is necessary to confirm that the numbers truly reflect meaningful insights.

The presentation highlighted the importance of an explicit error-analysis loop. This process involves deep-diving into raw traces, pinpointing failure modes, prioritizing actionable metrics that drive decision-making, forming hypotheses, and iterating on the process. Tools like Langsmith tracing are vital for developers and agent builders to gain visibility into the entire execution flow, including reasoning, education articles used, and tool calls.