OpenAI Walks Back Its Backing For SWE-Bench Pro Coding Test
OpenAI says a third of tasks in a leading AI coding benchmark are broken, and retracts its own endorsement. For India's IT buyers, the leaderboard just lost its shine.
The News
OpenAI has publicly retracted its own endorsement of SWE-Bench Pro, one of the most closely watched benchmarks for measuring how well AI models write software. In an analysis titled "Separating signal from noise in coding evaluations", published in early July, the company said a large share of the benchmark's tasks are flawed enough to distort the scores built on top of them.
The numbers are stark. The public portion of SWE-Bench Pro holds 731 tasks. OpenAI's automated review pipeline flagged 200 of them, or 27.4 per cent, as problematic. When five experienced software engineers examined the same set by hand, they flagged 249 tasks, or 34.1 per cent. The faults fell into a handful of buckets: tests that were overly strict, prompts that were underspecified or outright misleading, and tests with poor coverage of the code they were meant to check.
SWE-Bench Pro was built by Scale AI as a tougher successor to SWE-bench Verified. OpenAI had earlier suggested the field adopt it as the new standard. It is now walking that recommendation back, and urging researchers to stop reading its scores as a clean measure of coding ability.
Why It Matters
Benchmarks are the scoreboard of the AI industry. A leaderboard position can shape a launch narrative, move a funding conversation and swing an enterprise contract. When roughly one task in three is broken, the gap between two models near the top may be measuring the benchmark's defects rather than genuine skill.
This is not an isolated wobble. OpenAI deprecated the older SWE-bench Verified back in February over data contamination and saturation, as frontier models began acing it. Two retreats from coding benchmarks inside five months points to a deeper problem: the evaluations are ageing faster than the models they judge. The pattern echoes the earlier retirement of language benchmarks such as GLUE, which were quietly shelved once systems learned to game them. Each time, the industry discovers that a number everyone trusted was never as solid as it looked.
Indian Angle
For India, this is more than an academic footnote. The country's IT services giants, TCS, Infosys, Wipro and HCLTech, along with hundreds of global capability centres, are in the middle of buying AI coding assistants at scale. Those procurement calls, often worth crores in annual licensing, frequently lean on public leaderboard rankings to justify picking one model over another. OpenAI's disclosure is a warning that a single headline score is a shaky basis for a multi-year, rupee-denominated commitment.
It also lands as the IndiaAI Mission and MeitY push to build home-grown evaluation and safety infrastructure. If the world's best-resourced labs cannot keep a coding benchmark clean, an indigenous programme has an opening to do it better, with tests that reflect Indian enterprise codebases, regional languages and the messy realities of legacy banking and government systems that Indian engineers maintain every day.
For the millions of Indian developers who now write code alongside tools like Cursor and GitHub Copilot, the practical lesson is simpler. Judge a model on your own repository and your own tickets, not on a leaderboard that may be measuring its own bugs.
FAQ
What is SWE-Bench Pro?
It is a benchmark of real software-engineering tasks used to test whether AI models can resolve genuine coding problems. Built by Scale AI, it was pitched as a harder replacement for the earlier SWE-bench Verified, which frontier models had begun to saturate.
How many tasks were found to be broken?
OpenAI's automated pipeline flagged 200 of the 731 public tasks, or 27.4 per cent. A separate manual review by five software engineers flagged 249 tasks, or 34.1 per cent, citing overly strict tests, misleading prompts and poor test coverage.
What should Indian IT buyers take from this?
Treat leaderboard scores as a starting point, not a verdict. Run shortlisted models against your own codebase, tickets and quality gates before signing licensing deals, since public benchmarks can carry hidden defects that skew rankings.
Where can I read the original announcement?
OpenAI published the full analysis on its own site, linked in the attribution below.
This story was reported by OpenAI. Read the full original coverage at OpenAI.