Giving an AI better tools should, in theory, lead to better performance. That was the thinking when GitHub aimed to integrate shared, Unix-style code exploration utilities like grep, glob, and view into its GitHub Copilot code review system. The goal was to streamline infrastructure and improve consistency across various Copilot products. However, the reality proved more complex.
Instead of an upgrade, benchmarks revealed a significant increase in review costs and a decrease in detected issues. The problem wasn't the utility of the tools themselves, but how the AI was instructed to use them. The agent's behavior shifted from efficient analysis to a broad, inefficient 'browsing' pattern.
The Wrong Instincts
Prior to this change, Copilot code review utilized custom exploration tools. These were designed for earlier generative AI models, which made fewer tool calls and required more upfront context. The new, shared tools, however, were part of the GitHub Copilot CLI Agents Get Context harness, used by multiple Copilot applications.
The migration aimed to consolidate duplicated tool implementations and enable cross-product improvements. The mapping was seemingly straightforward: list_dir to glob, search_file/search_dir to grep, and read_code to view. These shared tools, while powerful, lacked the specific contextual padding that the older, bespoke tools provided. This padding, while adding token cost, was beneficial for older models needing more initial information.
The initial benchmarks showed the agent behaving like it was exploring an entire repository, not a specific pull request (PR). It would search widely, guess paths, read broadly, and accumulate context, a pattern understandable for general repo analysis but inefficient for targeted PR review.