AI Evals: Broken But Essential, Use Them Anyway

In the rapidly evolving world of AI, the effectiveness of evaluations is a critical, yet often contentious, topic. Ara Khan and Cline, in their presentation titled 'Evals Are Broken, Use Them Anyway,' tackle this head-on, arguing that despite their inherent flaws, evaluations are indispensable for driving progress in AI development.

Khan and Cline's core thesis is that while current evaluation methods for AI models are imperfect, they are still essential for building, interpreting, and ultimately improving AI agents. They highlight a common sentiment that many people are 'wrong about evals,' suggesting a need for a more nuanced understanding and application of these metrics.

The Two Camps of Flawed Evals

The presentation identifies two primary groups that misunderstand or misapply AI evaluations:

• The 'Objective Metrics' Camp: This group places an over-reliance on quantitative benchmarks, believing that high scores on public metrics automatically translate to real-world utility. They showcase examples of models that excel on benchmarks but fail in practical applications, often due to 'benchmark overfitting.'
• The 'Taste is King' Camp: Conversely, this group dismisses objective metrics entirely, prioritizing subjective 'taste' or qualitative assessments. While subjective feedback is valuable, relying on it exclusively can lead to inconsistent and biased evaluations.

Khan and Cline argue that the truth lies in the middle. Evals are neither the be-all and end-all, nor are they completely useless. There are indeed right and wrong ways to use them.

A Three-Stage Approach to Using Evals

To navigate the complexities of AI evaluations, the speakers propose a three-stage framework for developers:

• Level 1: Leverage Evals from Outside Sources. This involves utilizing existing benchmarks and evaluation datasets created by others.
• Level 2: Use Evals to Improve Your Own Agents. This stage focuses on applying evaluation results to iteratively refine the performance of your AI models based on specific use cases.
• Level 3: Build Your Own Evals for Specific Use Cases. This advanced level involves creating custom evaluation frameworks tailored to the unique requirements of your AI application.

The presentation emphasizes that while Level 1 and Level 2 are crucial starting points, the ultimate goal is to develop bespoke evaluations that accurately reflect the desired performance in real-world scenarios.

Heuristics for Interpreting Evals

To help developers interpret evaluation results more effectively, Khan and Cline offer several heuristics:

• Heuristic 1: Don't Believe Model Lab Evals Blindly; Treat Them as Approximations. The scores and results reported by model labs are often optimized for specific conditions and may not reflect real-world performance. It's important to treat them as estimates rather than absolute truths.
• Heuristic 2: Stay Current, But Don't Be the Earliest Adopter. The AI field moves incredibly fast, with new models and benchmarks emerging constantly. While it's important to stay informed, rushing to adopt the absolute latest might mean using evaluations that are not yet thoroughly validated or are susceptible to 'overfitting.' It’s often beneficial to wait for more robust and tested evaluations.
• Heuristic 3: Look for Very Precise and New Evals. As benchmarks and evaluation methodologies evolve, newer and more precise evaluations are developed. These are more likely to capture the nuances of AI performance and provide more reliable insights than older, more standardized tests.

The Process: Get a Score, Then Improve

The core process for improving AI models through evaluation is presented as a cyclical approach:

1. Run the Eval: Obtain an original score using an existing benchmark.
2. Evaluate All Failures: Analyze the failures to identify common themes or patterns. For instance, does a particular tool blow up on massive files? Does inference consistently time out? Does the agent get distracted and go off track?
3. Bucket Your Failures: Group similar failures together to understand the root causes.
4. Make One Tiny Change: Implement a small, targeted improvement based on the failure analysis.
5. Re-run the Whole Eval: Assess the impact of the change by re-running the evaluation and repeating the process.

What You're Actually Testing

The presentation clarifies that when conducting evaluations, you are essentially testing three intertwined components:

• The Model Itself: The inherent capabilities and limitations of the AI model.
• The Harness: The scaffolding or framework used to run the evaluations, which can significantly influence the results.
• The Problem: Whether the problem being addressed is well-defined and "sane." If you're optimizing for a flawed or misaligned problem, even perfect scores will be misleading.

All three elements need to be in alignment, and honesty with oneself about these components is crucial for meaningful evaluation and improvement.

Conclusion

Khan and Cline conclude by reiterating the importance of a pragmatic approach to AI evaluations. They advise finding relevant benchmarks, building honest evaluations, and critically assessing the results. Benchmarks alone are insufficient; they must be complemented by a qualitative 'vibe check' and a commitment to continuous, iterative improvement. By understanding the limitations and applying the right heuristics, developers can harness the power of evaluations to build better, more reliable AI systems.