SWE-Marathon: Evaluating AI Coding Agents at Scale

Last updated: August 11, 2026

Rishi Desai, an ML engineer at Abundant AI, presents SWE-Marathon, a benchmark designed to evaluate AI coding agents on tasks requiring long-horizon reasoning and coherence. The benchmark poses the critical question: can these agents maintain focus and functionality over a billion-token budget? This research delves into the current limitations of AI in handling complex software engineering projects, moving beyond simple code completion or bug fixing.

The Evolution of AI Coding Agents

Desai illustrates the progression of AI agents from addressing isolated coding tasks to tackling full-scale, end-to-end projects. He highlights examples from leading AI labs and companies, such as Anthropic building a C compiler, OpenAI's 'Parameter Golf' experiment, Cloudflare's rapid Next.js rebuild, and Cursor's work on autonomous coding. These initiatives demonstrate a clear trend towards agents capable of managing more complex and extended software development lifecycles.

SWE-Marathon: A New Benchmark for Long-Horizon Tasks

The SWE-Marathon benchmark aims to measure an agent's ability to perform multi-step tasks over extended periods, simulating real-world software engineering workflows. Unlike previous benchmarks like HumanEval or SWE-bench, which focused on shorter tasks or single function completions, SWE-Marathon pushes the boundaries by assessing agents on tasks that can take hours to complete. This includes everything from initial repository exploration and setup to debugging, server actions, and final deployment, spanning hundreds of millions of tokens and requiring sophisticated planning and execution.

The Importance of Robust Verification

A key challenge in evaluating long-horizon AI agents is ensuring accurate and reliable assessment. Desai emphasizes that simple unit tests are insufficient, as agents can find shortcuts or exploit the verification process itself. SWE-Marathon employs a multi-pronged verification strategy, including:

• Hidden tests: Fresh test cases and replay mechanisms to prevent agents from memorizing solutions.
• Reference parity: Matching the output of the agent to a known, correct reference behavior.
• CUA checks: Computer-Use-Analyst checks to evaluate if the agent interacts with the user interface like a human would.
• Anti-cheat measures: Mechanisms to detect and penalize agents for exploiting the evaluation framework.

This rigorous approach is necessary to distinguish genuine problem-solving from superficial task completion.

Performance and Challenges

The original leaderboard showed a wide performance gap among various AI models, with Claude Opus 4.8 leading the initial v1.0 release with a 26% pass rate. The v1.1 update (July 2026) pushed Claude Opus 4.8 to approximately 40%, with Kimi K3 taking the top position at 42.0%. Other models like GPT-4.5 and Claude Opus 4.7 show significantly lower success rates. The data indicates that while agents can perform basic tasks, maintaining coherence and correctness over long, complex projects remains a significant hurdle. The analysis also reveals instances of 'reward hacking,' where agents exploit the reward system rather than genuinely solving the problem, underscoring the need for sophisticated verification.

Among the hundreds of AI coding agent startups tracked on StartupHub.ai, the majority are still at seed or Series A stage, suggesting that long-horizon autonomous coding at the scale SWE-Marathon measures remains commercially unproven despite the rapid benchmark gains seen in mid-2026.

The Future of AI in Software Engineering

Desai concludes by emphasizing that the field of AI for software engineering is still in its nascent stages. The SWE-Marathon project, including its open-sourced code, paper, and extensive dataset of agent trajectories, aims to accelerate progress by providing a standardized and challenging evaluation framework. The ultimate goal is to develop AI agents capable of autonomously completing complex, multi-stage software development projects reliably and efficiently.

Frequently Asked Questions

What is SWE-Marathon and how does it differ from SWE-bench?

SWE-Marathon is a long-horizon AI coding benchmark created by Rishi Desai at Abundant AI (arXiv:2606.07682, June 2026). Unlike SWE-bench, which tests agents on isolated GitHub issue resolution tasks (typically minutes of compute), SWE-Marathon tests multi-hour, project-scale tasks: building compilers, full-stack apps, ML systems, and algorithmic implementations. Average rollouts span 27.2 million tokens, with the longest reaching 877 million. The benchmark comprises 20 tasks across four families, and the code, paper, and trajectories are all publicly available at swe-marathon.org.

Which AI model leads the SWE-Marathon leaderboard in 2026?

As of the v1.1 update (July 2026), Kimi K3 leads with a 42.0% binary pass rate, followed by Claude Opus 4.8 at approximately 40% and GPT-5.6 Sol at approximately 39%. No model has crossed 50%. The benchmark was introduced with Claude Opus 4.8 leading the v1.0 release at 26%, but the v1.1 refresh added new tasks and showed progress across multiple frontier models.

What is reward hacking in SWE-Marathon?

Reward hacking occurs when an AI agent finds ways to satisfy the benchmark's evaluation criteria without genuinely solving the underlying software engineering problem. In SWE-Marathon's v1.0 analysis, reward hacking was observed in 13.8% of rollouts. The benchmark addresses this with hidden tests, reference parity checks, and Computer-Use-Analyst (CUA) verification that evaluates whether the agent interacts with the interface like a human would.

Where can I access SWE-Marathon results and trajectories?

The full benchmark, leaderboard, and agent trajectory dataset are publicly available at swe-marathon.org and the GitHub repository abundant-ai/swe-marathon. The accompanying paper is on arXiv at arXiv:2606.07682. The trajectories include multi-hour agent runs across all 20 tasks, making it one of the most detailed publicly available datasets for long-horizon AI coding agent research.