Google DeepMind's Philipp Schmid on Testing AI Agent Skills

In the rapidly evolving world of artificial intelligence, the reliability of AI agents hinges on the quality of their skills. Philipp Schmid, a Staff Engineer at Google DeepMind, delivered a critical message at an AI.engineer event: "Don't Ship Skills Without Evals." He argued that the current practice of shipping AI agent skills based on minimal manual testing is akin to merging untested code, a practice unheard of in traditional software development. Schmid's talk, "Don't Ship Skills Without Evals," laid out a clear case for implementing rigorous evaluation processes throughout the lifecycle of building AI agent skills.

Who Is Philipp Schmid

Philipp Schmid is a Staff Engineer at Google DeepMind, where he works on prominent AI models like Gemini and Gemma. His role emphasizes enabling developers to build and utilize AI responsibly. With his background at one of the leading AI research labs, Schmid brings a deep understanding of the challenges and best practices in AI development.

The Problem with Untested AI Skills

Schmid highlighted a significant gap in the development of AI agents: the lack of robust testing for agent skills. He described a common scenario where thousands of skills are developed, but "almost none of them are tested." The typical validation process involves a couple of manual runs and perhaps a quick review from a colleague, a method he likened to a "vibe check." This approach, he contended, is fundamentally flawed. Shipping AI skills without proper evaluation risks releasing buggy or unreliable functionality to users, eroding trust and potentially causing significant issues.

Defining an AI Skill

The presentation aimed to demystify what constitutes an AI skill and how to ensure its correct implementation. Schmid emphasized that a skill is not just a piece of code but a functional unit designed to perform a specific task or achieve a particular outcome. The challenge lies in ensuring these skills trigger accurately and perform as intended under various conditions. He stressed the importance of understanding the underlying logic and potential failure points of each skill.

Building Reliable Agent Skills: The Lifecycle

Schmid outlined a comprehensive approach to building reliable AI agent skills, focusing on the entire development lifecycle. This begins with clearly defining what a skill is and what it is not, setting precise expectations for its functionality. The next crucial step involves writing skills that are designed to trigger correctly in response to user inputs or environmental cues. This requires careful consideration of prompt engineering, context management, and potential edge cases.

The Necessity of Evaluation Harnesses

The core of Schmid's argument revolved around the indispensable role of evaluation. He advocated for the development and use of lightweight evaluation harnesses. These systems are designed to systematically test AI skills, identifying failures before they reach end-users. Such harnesses allow for automated, repeatable testing, providing objective data on skill performance. By catching errors early, developers can iterate and improve skills, ensuring higher quality and reliability. "You wouldn't merge code without tests so why are we shipping skills without evals?" Schmid questioned, drawing a parallel to established software engineering practices. This rhetorical question underscores the urgency of adopting similar rigor in AI development.

Catching Failures Before Users Do

The ultimate goal of implementing evaluation harnesses is to protect the user experience. When AI agents function reliably, users are more likely to trust and adopt them. Conversely, a poorly tested skill can lead to frustration, incorrect actions, and a negative perception of the AI system. Schmid's talk provided a blueprint for developers to move beyond ad-hoc testing and establish a systematic process for ensuring the quality and dependability of the AI skills they deploy.