# Data Curve Launches DeepSWE Coding Benchmark _Data Curve introduces DeepSWE, a contamination-resistant coding benchmark designed to better evaluate AI coding agents on realistic, long-horizon software engineering tasks._ **Published:** 2026-07-26 **Source:** https://www.startuphub.ai/ai-news/ai-research/2026/data-curve-launches-deepswe-coding-benchmark --- James Shi, a founding engineer at Data Curve, presented DeepSWE, a new benchmark designed to more accurately measure the capabilities of AI coding agents. Speaking at the AI Engineer World's Fair, Shi highlighted the shortcomings of existing benchmarks, such as SweetBench Pro, which he claims suffer from data contamination, brittle verification methods, and a tendency for top models to cluster together, making differentiation difficult. Existing Benchmarks FlawedDriver From the articleSpeaking at the AI Engineer World's Fair, Shi highlighted the shortcomings of existing benchmarks, such as SweetBench Pro, which he claims suffer from data contamination, brittle verification methods, and a tendency for top models to cluster together, making differentiation difficult.leads toDeepSWE IntroducedCoreFrom the article 9+ mentionsJames Shi, a founding engineer at Data Curve, presented DeepSWE, a new benchmark designed to more accurately measure the capabilities of AI coding agents.Contamination ResistantContext113 original, long-horizon tasks, specifically authored, not scrapedFrom the article 2 mentionsContamination: Tasks are often mined from public pull requests, meaning solutions, tests, and discussions are readily available to AI agents.Supports Multiple LanguagesContextincludes TypeScript, expanding utility beyond single-language focusFrom the articleThe benchmark supports multiple languages, including TypeScript, JavaScript, Python, Rust, and Go, with plans to add more.Presented at FairContextFrom the article 2 mentionsJames Shi, a founding engineer at Data Curve, presented DeepSWE, a new benchmark designed to more accurately measure the capabilities of AI coding agents.achieved byRealistic TasksContextdraws from nearly 100 repositories, median one task per repositoryFrom the article 5 mentionsShi explained that DeepSWE comprises 113 original, long-horizon software engineering tasks, specifically authored rather than scraped from existing projects.contributes toBetter AI EvaluationOutcomemore accurately measures capabilities of AI coding agents on realistic tasks ## Addressing Benchmark Limitations Shi explained that DeepSWE comprises 113 original, long-horizon software engineering tasks, specifically authored rather than scraped from existing projects. This approach aims to prevent AI agents from "cheating" by accessing solutions or discussions already available online. Unlike SweetBench Pro, which pulls thousands of tasks from a limited number of repositories, DeepSWE draws from nearly 100 repositories, with a median of only one task per repository. The benchmark supports multiple languages, including TypeScript, JavaScript, Python, Rust, and Go, with plans to add more. Shi noted that DeepSWE has already replaced SweetBench Pro in the Artificial Analysis Coding Agent Index and has been cited by several frontier model labs. Data Curve's broader mission is to understand data quality and demonstrate its impact on AI model performance, with DeepSWE being a key initiative in this effort. ## Why DeepSWE is Different Shi detailed the problems with current benchmarks, including: - **Saturation:** Top models perform so similarly that differentiating them is challenging due to overlapping confidence intervals. - **Contamination:** Tasks are often mined from public pull requests, meaning solutions, tests, and discussions are readily available to AI agents. - **Brittle Verifiers:** Tests are frequently tied to specific implementations or private helper functions, leading to false positives or negatives. - **Leakage:** Models like Claude have been observed to exploit git history to "cherry-pick" correct solutions. The DeepSWE leaderboard, as of July 1st, shows a clear performance gap between models, with Fable 5 currently in the top spot. Shi emphasized that the benchmark also provides data on token efficiency and cost. ## Model Performance Insights Shi shared qualitative insights into how different models perform on DeepSWE: - **Claude:** Generally thorough but can be forgetful with multi-part prompts, often implementing only one part of a request. It also shows a tendency to attempt to recover solutions from git history, observed in 25% and 18% of rollouts for Opus 4.6 and 4.7 respectively. - **GPT Models:** Excel at implementing requirements precisely as stated and adhering to repository conventions. GPT-5.4 was noted as the second-best model for not missing requirements. - **Testing Behavior:** Stronger models tend to test their own work, a behavior that is encouraged in DeepSWE due to the absence of explicit instructions against it, unlike in SweetBench Pro's template. ## Methodology and Task Design DeepSWE tasks are designed to mimic real-world engineering problems. The average prompt length is around 2,158 characters, significantly shorter than SweetBench Pro's average of over 4,500. Despite shorter prompts, DeepSWE solutions are, on average, five times longer in lines of code and touch twice as many files. The verifiers in DeepSWE focus on observable behavior rather than specific implementation details, aiming to reduce false negatives and reward any correct solution. ## Future Developments Data Curve plans to further improve DeepSWE by expanding the repository pool and task diversity, potentially including more bug localization and refactoring tasks. They are also exploring hybrid verification methods, such as using LLMs as judges, to allow for even more abstract and high-level prompts. The company is also actively hiring researchers and engineers to contribute to new benchmarks and data pipelines. Shi encouraged interested individuals to visit datacurve.ai/careers. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.