Steven Willmott on Spec-Driven Testing for AI Agents

Steven Willmott, CEO of SafeIntelligence, delivered a compelling talk at AI Engineer Europe 2026 on the critical topic of "Spec-Driven Testing for Agents With A Brain the Size of A Planet." Willmott highlighted the growing need for robust validation methods for AI agents, especially as they become more complex and capable of performing a wider range of tasks.

Understanding the Need for Spec-Driven Validation

Willmott began by posing a fundamental question: "A Smarter Agent is a Better Agent, Right?" He then challenged this assumption by pointing out the potential pitfalls of simply increasing an AI model's intelligence. Larger models can be more susceptible to jailbreaks, have a broader surface area for exploitation, and often come with higher costs and slower speeds. This sets the stage for the importance of rigorous testing beyond traditional dataset-based evaluations.

The core of Willmott's presentation focused on the concept of "spec-driven validation." He explained that for AI agents, particularly those designed for complex tasks, simply having a dataset of examples is insufficient. The validation process needs to be more sophisticated, ensuring that agents not only perform their tasks but also do so within defined safety and behavioral boundaries.

Key Components of Agent Specifications

Willmott outlined several key components that are essential for defining and validating AI agents:

• Ground Truth Golden Test Sets: These are curated sets of examples that represent the desired behavior and outcomes for the agent.
• Ontologies / Dictionaries: These provide a structured understanding of the domain the agent operates in, defining terms, relationships, and constraints.
• Rules: Explicit, programmable rules that the agent must adhere to, such as "no discounts over 20%" or "always be polite."
• Domain Knowledge: The contextual information and understanding the agent needs to operate effectively within its specific domain, including scientific knowledge or general safety knowledge.
• Robustness Requirements: Specifications that ensure the agent can handle variations, perturbations, and unexpected inputs without failing or behaving erratically.
• Rights & Roles: Defining the permissions, data access, and autonomy levels for the agent, ensuring it operates within its intended scope.

Willmott emphasized that these components collectively form a "Task / Role Specific Benchmark or Integration Test." The goal is to move beyond simple performance metrics to a more comprehensive understanding of an agent's behavior and its alignment with specified requirements.

The Challenge of Defining "Good" and "Harm"

A significant challenge in AI agent validation, as highlighted by Willmott, is the difficulty in precisely defining what constitutes "good" behavior and what constitutes "harm." He noted that while it's relatively straightforward to define "good" with a dataset of correct inputs and outputs, defining "harm" is more complex. An agent might fail to perform its task correctly, or it might perform the task in a way that is detrimental or unintended, even if it technically fulfills the prompt.

Willmott illustrated this with an example of a customer support agent: "If you're building an airline chat bot, you might have specifications like 'never give a discount over 20%' or 'always be polite.' But how do you define what good looks like, and what harm looks like for that matter?" He elaborated that an agent's specifications need to be detailed enough to cover edge cases and potential unintended consequences.

Industry Progress and Future Directions

Willmott showcased examples of industry progress, including different prompt management platforms and "A2A Agent Cards," which are structured descriptions of an agent's capabilities and specifications. He pointed out that while many platforms allow for detailed prompt management, ensuring that these specifications translate into actual agent behavior is an ongoing challenge.

The presentation concluded with a call to action: "Start Thinking in Agent Specs." Willmott encouraged the audience to focus on specifying agent behavior, ensuring independence from implementation details, closing the development loop, and creating reusable behavioral specifications. He also outlined future directions, including work on formats, tools, and experience to better capture and test these agent specifications.