Data Curve Launches DeepSWE Coding Benchmark

James Shi, a founding engineer at Data Curve, presented DeepSWE, a new benchmark designed to more accurately measure the capabilities of AI coding agents. Speaking at the AI Engineer World's Fair, Shi highlighted the shortcomings of existing benchmarks, such as SweetBench Pro, which he claims suffer from data contamination, brittle verification methods, and a tendency for top models to cluster together, making differentiation difficult.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with founding year, headquarters, and a short description from our database.

Claude

Anthropic is an AI safety and research company that develops reliable, interpretable, and steerable AI systems, including the Claude family of large language...

Location

San Francisco, United States

Funding

$4.0B

Addressing Benchmark Limitations

Shi explained that DeepSWE comprises 113 original, long-horizon software engineering tasks, specifically authored rather than scraped from existing projects. This approach aims to prevent AI agents from "cheating" by accessing solutions or discussions already available online. Unlike SweetBench Pro, which pulls thousands of tasks from a limited number of repositories, DeepSWE draws from nearly 100 repositories, with a median of only one task per repository. The benchmark supports multiple languages, including TypeScript, JavaScript, Python, Rust, and Go, with plans to add more.

Shi noted that DeepSWE has already replaced SweetBench Pro in the Artificial Analysis Coding Agent Index and has been cited by several frontier model labs. Data Curve's broader mission is to understand data quality and demonstrate its impact on AI model performance, with DeepSWE being a key initiative in this effort.

Why DeepSWE is Different

Shi detailed the problems with current benchmarks, including:

• Saturation: Top models perform so similarly that differentiating them is challenging due to overlapping confidence intervals.
• Contamination: Tasks are often mined from public pull requests, meaning solutions, tests, and discussions are readily available to AI agents.
• Brittle Verifiers: Tests are frequently tied to specific implementations or private helper functions, leading to false positives or negatives.
• Leakage: Models like Claude have been observed to exploit git history to "cherry-pick" correct solutions.

The DeepSWE leaderboard, as of July 1st, shows a clear performance gap between models, with Fable 5 currently in the top spot. Shi emphasized that the benchmark also provides data on token efficiency and cost.

Model Performance Insights

Shi shared qualitative insights into how different models perform on DeepSWE:

• Claude: Generally thorough but can be forgetful with multi-part prompts, often implementing only one part of a request. It also shows a tendency to attempt to recover solutions from git history, observed in 25% and 18% of rollouts for Opus 4.6 and 4.7 respectively.
• GPT Models: Excel at implementing requirements precisely as stated and adhering to repository conventions. GPT-5.4 was noted as the second-best model for not missing requirements.
• Testing Behavior: Stronger models tend to test their own work, a behavior that is encouraged in DeepSWE due to the absence of explicit instructions against it, unlike in SweetBench Pro's template.

Methodology and Task Design

DeepSWE tasks are designed to mimic real-world engineering problems. The average prompt length is around 2,158 characters, significantly shorter than SweetBench Pro's average of over 4,500. Despite shorter prompts, DeepSWE solutions are, on average, five times longer in lines of code and touch twice as many files. The verifiers in DeepSWE focus on observable behavior rather than specific implementation details, aiming to reduce false negatives and reward any correct solution.

Future Developments

Data Curve plans to further improve DeepSWE by expanding the repository pool and task diversity, potentially including more bug localization and refactoring tasks. They are also exploring hybrid verification methods, such as using LLMs as judges, to allow for even more abstract and high-level prompts.

The company is also actively hiring researchers and engineers to contribute to new benchmarks and data pipelines. Shi encouraged interested individuals to visit datacurve.ai/careers.