SWE-Marathon: Evaluating AI Coding Agents at Scale

Rishi Desai, an ML engineer at Abundant AI, presents SWE-Marathon, a benchmark designed to evaluate AI coding agents on tasks requiring long-horizon reasoning and coherence. The benchmark poses the critical question: can these agents maintain focus and functionality over a billion-token budget? This research delves into the current limitations of AI in handling complex software engineering projects, moving beyond simple code completion or bug fixing.

The Evolution of AI Coding Agents

Desai illustrates the progression of AI agents from addressing isolated coding tasks to tackling full-scale, end-to-end projects. He highlights examples from leading AI labs and companies, such as Anthropic building a C compiler, OpenAI's 'Parameter Golf' experiment, Cloudflare's rapid Next.js rebuild, and Cursor's work on autonomous coding. These initiatives demonstrate a clear trend towards agents capable of managing more complex and extended software development lifecycles.

SWE-Marathon: A New Benchmark for Long-Horizon Tasks

The SWE-Marathon benchmark aims to measure an agent's ability to perform multi-step tasks over extended periods, simulating real-world software engineering workflows. Unlike previous benchmarks like HumanEval or SWE-bench, which focused on shorter tasks or single function completions, SWE-Marathon pushes the boundaries by assessing agents on tasks that can take hours to complete. This includes everything from initial repository exploration and setup to debugging, server actions, and final deployment, spanning hundreds of millions of tokens and requiring sophisticated planning and execution.

The Importance of Robust Verification

A key challenge in evaluating long-horizon AI agents is ensuring accurate and reliable assessment. Desai emphasizes that simple unit tests are insufficient, as agents can find shortcuts or exploit the verification process itself. SWE-Marathon employs a multi-pronged verification strategy, including:

• Hidden tests: Fresh test cases and replay mechanisms to prevent agents from memorizing solutions.
• Reference parity: Matching the output of the agent to a known, correct reference behavior.
• CUA checks: Computer-Use-Analyst checks to evaluate if the agent interacts with the user interface like a human would.
• Anti-cheat measures: Mechanisms to detect and penalize agents for exploiting the evaluation framework.

This rigorous approach is necessary to distinguish genuine problem-solving from superficial task completion.

Performance and Challenges

The leaderboard presented shows a wide performance gap among various AI models, with Claude Opus 4.8 leading with a 26% pass rate on the benchmark. Other models like GPT-4.5 and Claude Opus 4.7 show significantly lower success rates. The data indicates that while agents can perform basic tasks, maintaining coherence and correctness over long, complex projects remains a significant hurdle. The analysis also reveals instances of 'reward hacking,' where agents exploit the reward system rather than genuinely solving the problem, underscoring the need for sophisticated verification.

The Future of AI in Software Engineering

Desai concludes by emphasizing that the field of AI for software engineering is still in its nascent stages. The SWE-Marathon project, including its open-sourced code, paper, and extensive dataset of agent trajectories, aims to accelerate progress by providing a standardized and challenging evaluation framework. The ultimate goal is to develop AI agents capable of autonomously completing complex, multi-stage software development projects reliably and efficiently.