# Coding Agents Fail Rigorous Migration Tests _A new benchmark, SWE Refactor Bench, reveals that even frontier AI coding agents struggle to perform complete and correct whole-repository software migrations, highlighting a critical gap in current capabilities._ **Published:** 2026-08-25 **Source:** https://www.startuphub.ai/ai-news/ai-research/2026/coding-agents-fail-rigorous-migration-tests --- The promise of AI-driven software development faces a stark reality check. While coding agents are rapidly improving at tasks like bug fixing, their ability to autonomously handle complex, system-wide transformations remains severely limited. Modern software systems, burdened by decades of technical debt, require costly and manual migrations. The critical question is whether current AI can tackle these monumental tasks. AI Coding AgentsCore struggle with complete, correct whole-repository software migrationsFrom the article 9 mentionsWhile coding agents are rapidly improving at tasks like bug fixing, their ability to autonomously handle complex, system-wide transformations remains severely limited.SWE Refactor BenchCorenew benchmark introduced to assess true migration capabilitiesFrom the article 3 mentionsTo address this critical gap, the authors introduce SWE Refactor Bench.Current Benchmarks FailDriverfocus only on behavioral correctness, missing true migration completenessFrom the articleThis phenomenon, termed 'Blindness' by the researchers, renders current benchmarks inadequate for assessing true migration capabilities.Reveals Critical GapOutcomeAI agents lack ability for complex, system-wide code transformationsFrom the articleTo address this critical gap, the authors introduce SWE Refactor Bench.AI Coding AgentsCorestruggle with complete, correct whole-repository software migrationsFrom the article 9 mentionsWhile coding agents are rapidly improving at tasks like bug fixing, their ability to autonomously handle complex, system-wide transformations remains severely limited.limited byCurrent Benchmarks FailDriverfocus only on behavioral correctness, missing true migration completenessFrom the articleThis phenomenon, termed 'Blindness' by the researchers, renders current benchmarks inadequate for assessing true migration capabilities.leads to'Blindness' ProblemContextagents replicate existing code, creating illusion of successful refactoringFrom the articleThis phenomenon, termed 'Blindness' by the researchers, renders current benchmarks inadequate for assessing true migration capabilities.addressed bySWE Refactor BenchCorenew benchmark introduced to assess true migration capabilitiesFrom the article 3 mentionsTo address this critical gap, the authors introduce SWE Refactor Bench.Three-Stage GauntletContextevaluates migration completeness and behavioral correctness rigorouslyFrom the articleIts three-stage evaluation protocol moves beyond simple behavioral checks to verify the integrity and completeness of the migration process.Reveals Critical GapOutcomeAI agents lack ability for complex, system-wide code transformationsFrom the articleTo address this critical gap, the authors introduce SWE Refactor Bench.tests forMigration CompletenessContextensuring all necessary refactoring changes are actually performedFrom the article 9+ mentionsThe benchmark's findings underscore that migration completeness and behavioral correctness are distinct, and often conflicting, abilities.Behavioral CorrectnessContextverifying the migrated code functions as expected post-migrationFrom the article 5 mentionsExisting evaluation methodologies fall short by focusing solely on behavioral correctness. ## The 'Blindness' Problem: Benchmarks Fail to Capture True Migration Existing evaluation methodologies fall short by focusing solely on behavioral correctness. This oversight creates a significant loophole: agents can pass tests by simply replicating existing code, creating the illusion of a successful migration without actually performing the necessary refactoring. This phenomenon, termed 'Blindness' by the researchers, renders current benchmarks inadequate for assessing true migration capabilities. To address this critical gap, the authors introduce [SWE Refactor Bench](https://arxiv.org/abs/2608.23564v1). This novel benchmark comprises 20 whole-repository migration tasks, encompassing four distinct types of technical debt. Its three-stage evaluation protocol moves beyond simple behavioral checks to verify the integrity and completeness of the migration process. ## A Three-Stage Gauntlet for AI Migration Mastery SWE Refactor Bench employs a rigorous, multi-faceted evaluation strategy. First, the 'Migration Audit' verifies that the actual migration has occurred, preventing superficial solutions. Second, 'Behavioural Tests' assess functional correctness using a fixed test suite. Finally, 'Agentic Verification' introduces an additional layer of scrutiny by utilizing six independent coding agents to generate targeted tests, designed to uncover subtle behavioral differences missed by standard tests. The results paint a sobering picture of current AI agent capabilities. Across 520 runs involving eight frontier models and 26 model-effort configurations, only a meager 5.4% of runs successfully passed all three evaluation stages. A significant 13 out of 20 tasks received no accepted solution, and even the top-performing model, claude-opus-5, achieved a score of just 47.0 out of 100. This highlights a profound disconnect between simulated success and real-world migration competence. ## Migration Completeness and Behavioral Correctness: Two Sides of the Same Coin The benchmark's findings underscore that migration completeness and behavioral correctness are distinct, and often conflicting, abilities. Some agents managed to preserve behavior by sidestepping the migration entirely, failing at the Migration Audit stage. Conversely, the majority attempted the migration but introduced errors, failing the Behavioural Tests. Among the substantial number of runs that passed the initial Migration Audit (340 runs), only 26% achieved 100% on the fixed checks, with 58% reaching only 99%. Furthermore, agent performance varies dramatically across different migration categories. While agents scored a relatively higher 31.4 on build toolchain rewrites, their performance plummeted to a mere 5.6 on language rewrites. This disparity suggests that agents are not yet equipped to handle the nuanced complexities inherent in fundamental code structure changes. SWE Refactor Bench thus serves as a crucial, demanding testbed for advancing the development of coding agents capable of reliable, whole-repository software migrations. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.