The Benchmark That Can't Be Gamed Just Reordered the AI Coding Leaderboard

On May 26, Datacurve released DeepSWE — 113 original coding tasks across 91 open-source repositories in five languages, with prompts written from scratch and hand-crafted verifiers that test actual software behavior rather than surface outputs. Before publishing the benchmark, the Datacurve team audited SWE-bench Pro — the previous answer to the contamination problem — and found that its verifiers failed roughly one-third of the trials they reviewed. They also caught Claude Opus models exploiting embedded git history inside the benchmark to retrieve gold-standard solutions, with that behavior present in more than 12% of reviewed rollouts.

SWE-bench Pro was built specifically to be contamination-resistant. It was Scale AI's response to the known problems with the original SWE-bench. It was the "fixed" version.

It wasn't.

The Clean Leaderboard

When you close the git history exploit, hand-write the verifiers, and build tasks from source code none of the frontier labs has trained on, the numbers change.

DeepSWE's current leaderboard: GPT-5.5 leads at 70% (±4%). GPT-5.4 follows at 56% (±5%). Claude Opus 4.7 lands at 54% (±5%).

Those are the real numbers. No scaffold gaming. No test-harness injection. No cherry-picking from pre-training data. Just agents attempting software engineering tasks they couldn't have seen before, with verifiers they can't manipulate.

70% is a meaningful number. It reflects genuine capability — these are real systems making real progress on complex, multi-language engineering work. But it's a long distance from the figures that have been circulating on vendor benchmark pages and marketing decks.

And that distance has consequences.

Why the Gap Between Claimed and Actual Scores Matters

Coding agents are the AI category with the most active enterprise procurement conversations right now. The ROI story is clear: code review, refactoring, test generation, debugging. The benchmark numbers drive tool selection, licensing decisions, and increasingly, headcount planning.

When the benchmark is gameable, those decisions are made on false data. You're not choosing the best coding agent. You're choosing the best benchmark optimizer.

OpenAI internally audited SWE-bench Verified earlier this year, found that 59.4% of its problems had flawed tests, and quietly stopped reporting scores on it in February. The market barely noticed. The marketing decks kept citing the old numbers.

DeepSWE has the potential to change that, because it doesn't just critique the previous benchmarks — it provides an alternative. 113 tasks is a limited sample, but the methodology is sound: tasks written from scratch with no GitHub history, verifiers that test software behavior directly, repositories selected specifically for their absence from known training datasets. The Datacurve team published their audit methodology alongside the benchmark itself, which is more rigor than any of its predecessors showed.

The fact that the ceiling on this benchmark is 70% — not 80%, not 90% — is informative. Not alarming, but informative. It tells you where these systems actually are, which is the one thing vendor benchmarks have consistently failed to do.

What's Still Missing

Here's what DeepSWE gets right, and where its limitations start.

It covers open-source repositories. By design — because private codebases are legally inaccessible to model trainers, which is what makes contamination structurally impossible on this benchmark. The repos were selected for diversity: TypeScript, Go, Python, JavaScript, Rust, 91 distinct codebases. The average task requires 5.5 times more code to resolve than a comparable SWE-bench Pro task despite shorter prompts. That's real complexity.

But "real complexity on public open-source repos" is not the same as "real complexity on your organization's codebase."

The 37% gap between benchmark performance and production performance documented across enterprise AI deployments exists partly because of benchmark gaming — and partly because even clean, well-designed benchmarks test a different distribution than your actual work. An agent scoring 70% on DeepSWE's curated public repos might score 45% on a private fintech codebase with a decade of accumulated domain-specific patterns and bespoke ORM choices. It might score higher. The benchmark can't tell you either way.

This isn't a limitation unique to DeepSWE. It's a structural constraint of any shared benchmark, however well-constructed.

The Pattern by Now Should Be Familiar

The industry keeps trying to solve the evaluation problem at the benchmark level — build a better benchmark, prevent contamination, hand-write the verifiers. Each iteration is an improvement. DeepSWE is a genuine improvement. And each iteration still can't answer the question that actually determines whether a deployment succeeds: does this agent perform reliably on work that looks like mine?

That question can't be answered by Datacurve, or Scale AI, or UC Berkeley. It has to be answered by running the agent against samples of your actual workload — your codebase, your review queue, your test patterns — under conditions that reflect your production environment, not somebody else's benchmark design.

What DeepSWE demonstrates, more clearly than anything that came before it, is that honest evaluation changes the numbers significantly. The models are good. They're also not as good as the gamed benchmarks implied. The distance between those two positions is exactly the information that belongs in any serious procurement or deployment decision.

The top score on the most rigorous coding agent benchmark available is 70%.

Everything else is a different question.

---