AI Agents Need Verification, Says Sonar CEO Tariq Shaukat

At the AI Engineer World's Fair, Tariq Shaukat, CEO of Sonar, a company specializing in code verification, delivered a critical message: in the age of AI agents, verification is paramount. While acknowledging the incredible advancements in AI models, Shaukat highlighted the persistent challenges enterprises face in deriving real value from them, particularly in the realm of software development.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

SonarQube

$4.7B

Automated code quality and security analysis platform for developers and organizations.

KPMG

Global network of professional services firms providing audit, tax, and advisory services.

The AI 'Slop' Problem

Shaukat pointed to numerous instances of major firms like KPMG and EY having to retract reports due to AI-generated hallucinations, and law firms facing trouble for fabricated citations. These examples underscore a broader issue: AI models, while adept at generating plausible-sounding outputs, often struggle with factual correctness. This problem extends to software development, where AI agents can produce code that is functionally correct but riddled with bugs, security vulnerabilities, or excessive complexity.

Citing data from meter, Shaukat illustrated how AI coding agents are rapidly improving in task completion time, with the latest models capable of handling tasks that would take humans 16-18 hours. However, he emphasized the crucial caveat: this is at a mere 50% success rate. When the accuracy bar is raised to 80%, the progress slows dramatically, with tasks taking significantly longer. He quoted a CTO who noted that an 80% accurate employee would still likely be on a performance review, highlighting that current AI outputs are often not enterprise-grade.

Technical Debt and AI

A study from Carnegie Mellon University revealed that while AI coding agents can provide an initial 3-5x boost in productivity, this benefit dissipates within three months. The reason, according to the study, is a concurrent increase in security issues, maintainability problems, reliability concerns, and code complexity. Essentially, the speed gained is offset by the rapid accumulation of technical debt.

Shaukat argued that in a world where code is provable but large codebases are inherently complex and messy, verification cannot be an afterthought. It must be baked into the development process. He proposed an 'Agent Centric Development Cycle' (AC/DC) framework, which places verification-powered agentic loops at its core. This cycle involves guiding agents with context and constraints, verifying outcomes, and then solving problems, all integrated into the software development lifecycle.

Guiding and Verifying AI Agents

Sonar's approach to 'guide' involves providing agents with context, such as architectural awareness and semantic navigation maps of existing codebases, and constraints, like coding standards, guardrails, and intended architecture. This contextualization, Shaukat noted, leads to significant improvements in agent effectiveness and a reduction in token consumption by over 30%, as it makes the agent's task easier.

For verification, Sonar advocates for a 'zero trust, multi-layered' strategy. This means using diverse models and techniques to ensure code is safe, reliable, and secure. The fusion of algorithmic verification (analyzing data flows, control flows, patterns, secrets) with agentic verification (assessing intent, business logic, emergent anomalies) is crucial. Partners and customers using this multi-layered approach report a 44% reduction in AI-derived production outages.

The Path Forward: Verification as a System

Shaukat stressed the importance of viewing these processes as a system, not isolated tools. The agentic loop, the CI verification loop, and the code maintenance loop must work together. He noted that customers who have neglected verification and code quality have seen the benefits of AI tools dissipate quickly, falling into a downward spiral. Conversely, integrating verification into these loops creates a self-reinforcing system that compounds positive outcomes.

He concluded by emphasizing that by embedding verification at the center of the agent-centric development cycle, enterprises can unlock the true value of AI in software development, ensuring reliability, security, and maintainability. Sonar's own products, such as Sonar Vortex and SonarQube, are designed to facilitate this verification-powered approach.