Coding Agents Fail Rigorous Migration Tests

The promise of AI-driven software development faces a stark reality check. While coding agents are rapidly improving at tasks like bug fixing, their ability to autonomously handle complex, system-wide transformations remains severely limited. Modern software systems, burdened by decades of technical debt, require costly and manual migrations. The critical question is whether current AI can tackle these monumental tasks.

The 'Blindness' Problem: Benchmarks Fail to Capture True Migration

Existing evaluation methodologies fall short by focusing solely on behavioral correctness. This oversight creates a significant loophole: agents can pass tests by simply replicating existing code, creating the illusion of a successful migration without actually performing the necessary refactoring. This phenomenon, termed 'Blindness' by the researchers, renders current benchmarks inadequate for assessing true migration capabilities.

To address this critical gap, the authors introduce SWE Refactor Bench. This novel benchmark comprises 20 whole-repository migration tasks, encompassing four distinct types of technical debt. Its three-stage evaluation protocol moves beyond simple behavioral checks to verify the integrity and completeness of the migration process.

A Three-Stage Gauntlet for AI Migration Mastery

SWE Refactor Bench employs a rigorous, multi-faceted evaluation strategy. First, the 'Migration Audit' verifies that the actual migration has occurred, preventing superficial solutions. Second, 'Behavioural Tests' assess functional correctness using a fixed test suite. Finally, 'Agentic Verification' introduces an additional layer of scrutiny by utilizing six independent coding agents to generate targeted tests, designed to uncover subtle behavioral differences missed by standard tests.

The results paint a sobering picture of current AI agent capabilities. Across 520 runs involving eight frontier models and 26 model-effort configurations, only a meager 5.4% of runs successfully passed all three evaluation stages. A significant 13 out of 20 tasks received no accepted solution, and even the top-performing model, claude-opus-5, achieved a score of just 47.0 out of 100. This highlights a profound disconnect between simulated success and real-world migration competence.

Migration Completeness and Behavioral Correctness: Two Sides of the Same Coin

The benchmark's findings underscore that migration completeness and behavioral correctness are distinct, and often conflicting, abilities. Some agents managed to preserve behavior by sidestepping the migration entirely, failing at the Migration Audit stage. Conversely, the majority attempted the migration but introduced errors, failing the Behavioural Tests. Among the substantial number of runs that passed the initial Migration Audit (340 runs), only 26% achieved 100% on the fixed checks, with 58% reaching only 99%.

Furthermore, agent performance varies dramatically across different migration categories. While agents scored a relatively higher 31.4 on build toolchain rewrites, their performance plummeted to a mere 5.6 on language rewrites. This disparity suggests that agents are not yet equipped to handle the nuanced complexities inherent in fundamental code structure changes. SWE Refactor Bench thus serves as a crucial, demanding testbed for advancing the development of coding agents capable of reliable, whole-repository software migrations.