OpenAI Flags Major Flaws in SWE-Bench Pro

OpenAI has identified significant issues within SWE-Bench Pro, a prominent benchmark for evaluating AI coding agents, estimating that roughly 30% of its tasks are broken. This detailed audit, published by OpenAI News, underscores the challenges in creating reliable coding evaluations for advanced AI models.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

OpenAI

Private / $100B+ est

OpenAI is an AI research and deployment company dedicated to ensuring that artificial general intelligence benefits all of humanity.

The company previously encouraged the community to adopt SWE-Bench Pro after finding fundamental design and contamination problems in its predecessor, SWE-bench Verified. However, their latest investigation reveals similar concerns.

OpenAI's methodology for this SWE-Bench Pro audit involved a comprehensive datapoint analysis pipeline. This system reviewed model attempts, task metadata, and failure traces to flag potential evaluation flaws. Each flagged task then underwent scrutiny through multiple investigator-agent passes and independent review by five experienced software engineers.

Of the 731 public split tasks, the analysis pipeline flagged 200 (27.4%) as broken, while human annotation identified 249 (34.1%). These coding benchmark evaluation flaws fall into four primary categories:

• Overly strict tests: These enforce specific implementation details not explicitly stated in the prompt, invalidating functionally correct submissions.
• Underspecified prompts: Key requirements are omitted, enforced only by hidden tests and not reasonably inferable.
• Low-coverage tests: These inadequately check the requested feature, allowing incomplete fixes to pass.
• Misleading prompts: Models are directed toward incorrect behavior or given instructions that contradict test requirements.

These findings highlight the ongoing difficulty in curating fair yet challenging benchmarks, especially as AI model capabilities improve. The audit suggests a growing utility for agents in scalable data quality checks, helping to surface issues that were once impractical to find at scale.

Given the widespread SWE-Bench Pro task issues, OpenAI has retracted its earlier recommendation for the benchmark. The company advises developers to carefully examine results derived from SWE-Bench Pro, emphasizing that valid and informative evaluations are crucial for sound deployment and safety decisions under OpenAI’s Preparedness Framework.