Ankit Jain: Rethinking Code Reviews with AI

In a presentation titled "How to Kill the Code Review," Ankit Jain, co-founder of Aviator, a company building an AI code verification platform, discussed the evolving challenges and potential solutions for code review processes in the age of AI-assisted development. Jain, drawing from his previous work on a five-layer trust model, argued that the current methods of code review are failing to keep pace with the rapid increase in code volume and the rise of AI coding agents.

The Problem with Modern Code Reviews

Jain highlighted several concerning trends: an 861% increase in code churn, a rising incidents-to-PR ratio indicating diminishing review effectiveness, and a 4x increase in time spent waiting for reviews. Alarmingly, over 30% of changes are merged without any review at all. He pointed to the common scenario where AI agents write and review code, leading to a situation where humans merely skim the output before merging. "When AI reviews and nobody reads, we have configured the wrong thing," Jain stated, questioning the purpose of a UI in this automated workflow.

Code Review: Beyond Semantic Accuracy

While acknowledging that code reviews have historically focused on catching bugs, conventions, and security issues, Jain emphasized that a critical, often overlooked, aspect is 'alignment.' This includes knowledge sharing, mentorship, architectural feedback, and onboarding. He stressed that for teams, this alignment is paramount, and while tools can improve semantic accuracy, the collaborative aspect of code review must be preserved.

Intent-Driven Development and AI's Role

Jain critiqued the "spec-driven development" model, comparing it to the limitations of the 1970s waterfall model due to its lack of feedback loops. He argued that true intent doesn't solely reside in specifications but also in Jira tickets and, crucially, in the prompts used to interact with AI coding agents. "Most importantly, intent lives in your prompts today," he said, lamenting that this valuable information is often discarded after a pull request is created.

Introducing the AI Slop Register

To address the issue of recurring review findings, Jain introduced the concept of an "AI Slop Register." This system would codify common issues identified during reviews, essentially turning repetitive comments into automated guardrails. "Every recurring comment is now a guardrail that you don't have to review again," he explained. This approach allows the system to learn from human review experiences, leading to better overall code quality.

Building a Unified System: Alignment and Accuracy

Jain proposed a system that integrates both alignment and semantic accuracy. The process involves capturing user responses from AI sessions to form acceptance criteria, which are then combined with the AI Slop Register to create a test plan. This test plan is then used in a verification system that spins up previews to ensure the code functions end-to-end. The review surface shifts from examining code line-by-line to reviewing the evidence of intent, capability implementation, and adherence to requirements.

He elaborated on the practical application: capturing decisions made during AI coding sessions converts them into acceptance criteria. LLMs can assist in generating test plans from these criteria, streamlining the process. The value of human input remains in the governance and review of these test plans, rather than the code itself. This approach, he noted, aligns with behavior-driven development, making test plans shareable and understandable across different roles like product managers and designers.

Leveraging LLMs and Deterministic Checks

Jain advocated for a balanced approach: "Deterministic where you can. LLM where you must." While deterministic checks ensure reproducibility, LLMs can handle fuzzy behavior and subjective judgments. He used an example of verifying a new payment form, where AI agents could browse the application, capture screenshots, and compare them with database snapshots to confirm criteria fulfillment. This provides more robust evidence for reviewers.

The Future of Code Review: Intent over Diffs

The core message was a paradigm shift: "Reviewers review intent. Not diffs." By capturing and leveraging the intent and architectural decisions made during AI interactions, teams can move beyond superficial code checks to more meaningful architectural discussions. Jain encouraged attendees to mine their past review comments to build their own AI Slop Registers, noting that while this process requires initial investment, it compounds over time, reducing future review efforts.

Aviator is currently piloting its "Verify" product, which aims to integrate these concepts. "If you remember one thing from today," Jain concluded, "remember code review is not about code review, it is about getting the alignment."