Evaluating Coding Agents: Lessons from SWE-rebench

In the rapidly evolving field of AI-powered software development, understanding the capabilities and limitations of coding agents is paramount. Ibragim Badertdinov from Nebius recently presented "SWE-rebench: Lessons from Evaluating Coding Agents," offering a deep dive into the practical challenges and insights gained from evaluating these sophisticated tools on real-world software engineering tasks. This presentation, delivered at AI Engineer Europe, highlights the critical need for robust benchmarks and continuous evaluation in this fast-paced domain.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

Nebius

$5.3B

Full-stack AI cloud infrastructure

Who Is Ibragim Badertdinov?

Ibragim Badertdinov brings a unique perspective to the AI landscape, with a background that bridges healthcare and AI research. Having worked in dentistry and healthcare from 2013 to 2020, Badertdinov transitioned into the AI and NLP space in 2019. His current work at Nebius focuses on research and open-source contributions, bringing a practical, problem-solving approach honed by his diverse professional experiences. His transition from healthcare, where the cost of errors can be very high, to AI evaluation suggests a keen focus on rigor and reliability in his current work.

SWE-rebench: A New Benchmark for Coding Agents

The core of Badertdinov's presentation revolves around SWE-rebench, a novel benchmark designed to assess the performance of coding agents on genuine software engineering tasks. The benchmark is characterized by its emphasis on "freshness," meaning it constantly refreshes its tasks to prevent models from overfitting to static datasets. This dynamic approach is crucial in a field where models and their capabilities are evolving monthly. The benchmark focuses on "real-world" tasks, which are defined as economically valuable work, ensuring that the evaluations reflect practical utility.

SWE-rebench covers a broad spectrum of software engineering challenges, including subtasks, multi-turn interactions, long-context utilization, and tool use. It evaluates approximately 30 top models from various providers, alongside smaller, local use-case models. The benchmark also includes bonus features like scaffolds for Claude code, Codex, and Junie, additional statistics on tokens, and an open feedback loop for continuous improvement. The SWE-rebench leaderboard provides a transparent view of model performance, allowing for direct comparison.

Why Evaluations Matter Now

Badertdinov stressed the increasing importance of rigorous evaluations in the current AI landscape. As models improve and their capabilities become more sophisticated, the task of choosing the right agent for a specific purpose has become significantly harder. He noted that informal methods like "vibe checks" do not scale and that the performance of SWE agents is growing rapidly, leading to frequent changes in the available options. This necessitates a more systematic and data-driven approach to evaluation. He also highlighted that relying solely on gut feeling or a few favorite questions is insufficient when selecting models for critical applications.

The Anatomy of a Task: More Than Just Text

The presentation elaborated on what constitutes a good task within the SWE-rebench framework. A task, Badertdinov explained, is more than just a text description; it comprises several key components:

• Problem Description: This includes a clear task balance that avoids being too easy or too hard, and a complexity level that is neither too narrow nor too wide.
• Sandbox Environment: An executable Docker image is used to ensure a consistent and reproducible testing environment.
• Verifier: This component tests the pull requests (PRs) generated by the agents, determining if they pass or fail. The verifier's role is to distinguish between actual fixes and fake solutions.

Badertdinov emphasized that a reliable verifier should reward actual fixes while rejecting fabricated solutions, and it must be robust enough to handle variations in execution and potential infrastructure hiccups. The goal is to create tasks that are challenging yet solvable, providing meaningful metrics for model evaluation.

What Breaks in Practice?

In practical application, several factors can cause evaluations to break or yield misleading results. Badertdinov pointed out the importance of defining a clear retry policy, separating infrastructure errors from model errors, and understanding how caching mechanisms dramatically affect economics. He also noted that provider defaults can drift, leading to performance changes over time. A critical lesson learned is the need to "first make sure numbers match," ensuring that the metrics used for evaluation are consistent and reliable. The presentation included data from Claude-Opus-4.6 on 5x57 runs, illustrating how different caching strategies impact costs and performance.

Choosing Models: Quality, Cost, and Reliability

When selecting coding agents, a nuanced approach is required, considering not just performance but also cost and reliability. Badertdinov highlighted that repeated runs can reveal variance in model performance, and that potential matters more than just an average score. Reliability, he argued, deserves its own metric, such as "Pass@5." Economics also play a significant role in model choice, as cost-efficiency can change the decision-making process. Numbers and trajectories, he explained, are crucial for understanding and explaining these trade-offs.

The presentation also touched upon the challenges of using verifiable tasks for training. The process involves several steps: choosing models and parameters, updating prompts and tools, performing rejection sampling and supervised fine-tuning (SFT), and finally, reinforcement learning (RL) techniques like GRPO. Badertdinov showcased the scale of SWE-rebench, noting its 158,000+ open, verifiable real-world tasks in 20 languages, with a significant portion involving Python tasks and tasks with images. This extensive dataset allows for robust training and evaluation of AI coding agents.

Conclusion

Ibragim Badertdinov's presentation provided valuable insights into the practicalities of evaluating coding agents. The SWE-rebench benchmark and the lessons learned from its application offer a roadmap for researchers and developers aiming to build and deploy more capable and reliable AI tools for software engineering. The emphasis on real-world tasks, robust verification, and continuous adaptation underscores the dynamic nature of this field.