AI Agents Don't Always Follow Rules

In the rapidly evolving world of AI development, ensuring that AI agents reliably execute tasks is a critical challenge. Talha Sheikh from Checkout.com recently shared insights into this issue, highlighting that AI coding agents do not always adhere to the rules and instructions given to them. This talk, presented at AI Engineer Europe, delves into the practical difficulties of relying solely on AI for complex coding tasks and emphasizes the indispensable role of human oversight and verification.

The Unreliable Nature of Coding Agents

Sheikh illustrated a common scenario where a user might ask an AI agent to perform a coding task. While the agent might indicate completion with a "done" status, the actual output may be flawed or incomplete. This discrepancy arises because the agent's perceived completion does not guarantee adherence to the user's specific requirements or the desired outcome. Sheikh's personal experience involved trying to have an AI agent complete a coding task, only to find that a small, critical detail was missing, leading to failure.

The core of the problem, as presented, is that simply instructing an AI agent is not enough. The agent might be capable of understanding the request and generating code, but it lacks the inherent understanding or motivation to ensure the code functions exactly as intended. This places the burden of verification on the human operator, who must then "check anyway" to confirm the agent's output. This necessity for human intervention underscores the current limitations of AI agents in fully autonomous execution.

The Need for Deterministic Verification

Sheikh proposed that the solution lies in creating a deterministic verification layer. This means building a system that can reliably check if the AI agent's output matches the intended outcome. The idea is to move beyond simply trusting the agent's reported completion status. By implementing checks at every stage of the AI agent's workflow, from initial conversation to the final output, developers can build more dependable systems. This involves defining clear criteria for success and having a mechanism to automatically validate that these criteria have been met.

The talk showcased a conceptual approach to this problem, using a configuration file to define various checks. These checks could range from ensuring basic code syntax to verifying specific functional requirements and coverage metrics. The goal is to create a system where "every check is a shell command" that must pass, ensuring that the AI agent's work is not only completed but also correct according to predefined standards.

Capability vs. Reliability and the Role of Guardrails

A key distinction made during the presentation was between an AI model's capability and its reliability. While a model might be highly capable in terms of generating complex code or understanding nuanced instructions, this does not automatically translate to reliability. Reliability, in this context, refers to the consistency and correctness of the output over time and across different tasks. Sheikh argued that focusing solely on increasing capability without addressing reliability can lead to systems that appear to work but are fundamentally untrustworthy.

To achieve reliability, Sheikh highlighted the importance of guardrails. These are mechanisms designed to constrain the AI's behavior and ensure it operates within defined boundaries. The presentation suggested that guardrails can be more effective than simply increasing model size. A smaller model with well-defined guardrails can achieve higher reliability and more predictable outcomes compared to a larger model without such constraints. This is supported by a cost analysis showing that implementing guardrails, even with smaller models, can lead to comparable or better results with lower resource utilization.

The Industry Shift: Verification Over Generation

The discussion also touched upon the broader industry trend. Sheikh observed that many companies are now building their own AI agent frameworks and incorporating verification layers. This indicates a growing recognition that the true value in AI development is shifting from merely generating code or content to ensuring the accuracy and trustworthiness of that output.

The core question is evolving from "Can you code?" to "Can you verify?" This shift signifies a maturation of the field, moving towards more practical and dependable applications of AI. Companies like OpenAI, with their "Harness Engineering" approach, and others like WorkOS and Cudo, are focusing on building systems that incorporate robust verification mechanisms. These systems aim to ensure that AI agents not only perform tasks but do so reliably and predictably, aligning with human intent and industry standards.

Ultimately, the message from Talha Sheikh's presentation is clear: while AI agents are powerful tools, human oversight and a strong emphasis on verification are essential for building reliable and trustworthy AI systems. The future of AI development lies in creating robust "harnesses" that guide and validate the AI's output, rather than solely focusing on the underlying code generation capabilities.