OpenAI discovered that approximately 30 percent of tasks in SWE-Bench Pro, a leading benchmark for evaluating AI coding abilities, contain errors that make them unsuitable for accurate performance measurement. The company is withdrawing its previous endorsement of the test.
SWE-Bench Pro serves as a standard tool for the AI industry to assess how well language models handle real-world software engineering tasks. The benchmark presents models with actual GitHub issues and asks them to generate working code fixes. Given its widespread adoption, the benchmark influences how researchers and companies rank AI systems and make decisions about model development priorities.
OpenAI's findings undermine confidence in comparative claims based on this test. When roughly one third of evaluation tasks are broken, leaderboard rankings become unreliable. Models scoring higher may have succeeded at different subsets of tasks, making direct performance comparisons meaningless. This matters because companies and researchers use such benchmarks to justify claims about AI progress and to guide investment decisions.
The discovery highlights a persistent problem in AI evaluation. Benchmarks become outdated as models improve and exploit edge cases in poorly designed tests. Manual task creation, as used in SWE-Bench Pro, requires careful quality control that often falls short at scale. The benchmark's maintainers must validate that test cases have correct solutions, that environments work properly, and that task descriptions accurately reflect the problem statement.
OpenAI did not specify exactly which tasks were broken or what types of errors affected them. The company's withdrawal of support suggests the problems were systematic enough to compromise the benchmark's validity rather than isolated issues.
This incident places pressure on the SWE-Bench Pro team to conduct a thorough audit and rebuild trust. Other AI companies likely used this benchmark to track progress, meaning they now operate with uncertain performance data. The broader implication is that AI evaluation requires institutional rigor comparable to scientific peer review. Without it,