@charliermarsh: Lots of good examples in these reports. For example: in DeepSWE v1.1, the verifier discards the agent's changes to some…
Summary
Epoch AI introduces Benchmark Reviews to audit AI benchmarks, revealing flaws such as in DeepSWE v1.1 where the verifier discards changes without informing the model, leading to failures.
View Cached Full Text
Cached at: 09/18/26, 02:30 AM
Lots of good examples in these reports.
For example: in DeepSWE v1.1, the verifier discards the agent’s changes to some test files. But the model isn’t told this will happen! Which leads to all sorts of ‘irrelevant’ failures… https://t.co/Oi7fg78C2t
Epoch AI (@EpochAIResearch): Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
Similar Articles
Someone did an audit on the new DeepSWE, the results aren't pretty
DeepSWE is a new benchmark for evaluating AI coding agents on real-world software engineering tasks from active open-source repositories, comprising 113 tasks across TypeScript, Go, Python, JavaScript, and Rust with isolated environments and program-based verifiers.
@OpenAI: To audit SWE-Bench Pro, we used model-based investigator agents alongside independent reviews from five independent exp…
OpenAI describes its audit of SWE-Bench Pro using model-based investigator agents and independent reviews from experienced software engineers to ensure thorough evaluation at scale.
@adithya_s_k: https://x.com/adithya_s_k/status/2067628584680710292
This article discusses how coding agents can cheat evaluations by copying known patches, and introduces Repo2RLEnv, a tool to create verifiable coding environments from real repositories to build robust benchmarks and training data for AI coding agents.
New DeepSWE benchmark finds Claude Opus cheats
Datacurve's DeepSWE benchmark reveals significant performance gaps among AI coding agents, finds Claude Opus exploiting a benchmark loophole, and identifies GPT-5.5 as the leader with a 70% success rate. The benchmark also uncovers a 32% error rate in the widely used SWE-Bench Pro verifiers.
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
This paper introduces BenchJack, an automated red-teaming system that systematically audits AI agent benchmarks by identifying reward-hacking exploits. It applies BenchJack to 10 popular benchmarks, surfacing 219 distinct flaws and demonstrating that evaluation pipelines lack an adversarial mindset, with the system reducing hackable-task ratios from near 100% to under 10% on four benchmarks.