Copilot Code Review: Better Tools, Worse Results?

Giving an AI better tools should, in theory, lead to better performance. That was the thinking when GitHub aimed to integrate shared, Unix-style code exploration utilities like grep, glob, and view into its GitHub Copilot code review system. The goal was to streamline infrastructure and improve consistency across various Copilot products. However, the reality proved more complex.

Instead of an upgrade, benchmarks revealed a significant increase in review costs and a decrease in detected issues. The problem wasn't the utility of the tools themselves, but how the AI was instructed to use them. The agent's behavior shifted from efficient analysis to a broad, inefficient 'browsing' pattern.

The Wrong Instincts

Prior to this change, Copilot code review utilized custom exploration tools. These were designed for earlier generative AI models, which made fewer tool calls and required more upfront context. The new, shared tools, however, were part of the GitHub Copilot CLI Agents Get Context harness, used by multiple Copilot applications.

The migration aimed to consolidate duplicated tool implementations and enable cross-product improvements. The mapping was seemingly straightforward: list_dir to glob, search_file/search_dir to grep, and read_code to view. These shared tools, while powerful, lacked the specific contextual padding that the older, bespoke tools provided. This padding, while adding token cost, was beneficial for older models needing more initial information.

The initial benchmarks showed the agent behaving like it was exploring an entire repository, not a specific pull request (PR). It would search widely, guess paths, read broadly, and accumulate context, a pattern understandable for general repo analysis but inefficient for targeted PR review.

A human reviewer typically starts with the PR diff, asking focused questions: Where is this function called? Is this configuration key used elsewhere? What is the minimal context needed to answer the question?

This narrow focus is crucial because every tool output becomes part of the agent's working context, increasing costs and potentially diluting the review's focus. The shared tools worked fine; the instructions given to the agent were misaligned with the task of code review.

Reshaping Agent Workflows

The breakthrough came when GitHub rewrote the agent's instructions. Instead of general-purpose coding assistant prompts, the instructions were tailored to mimic how a human reviewer analyzes a PR. This shift refocused the agent on the diff, seeking narrow, relevant evidence rather than broad exploration.

This adjustment flipped the regression into a significant win. The result was roughly a 20% reduction in average review cost, while maintaining the same quality of review. It demonstrates that the effectiveness of AI in complex tasks like code review hinges not just on the sophistication of the tools, but critically on the alignment of agentic workflows with the specific task's nature.

This case highlights the nuanced challenges in optimizing Generative AI in software development, showing that even seemingly straightforward tool upgrades require careful consideration of the underlying agentic workflows and instructions.