On SWEBench Pro, 68.5% of GPT 5.5’s failures were caused by broken or incorrect test cases, totaling 28.9% of the entire benchmark
Summary
An analysis reveals that 28.9% of GPT 5.5's failures on SWEBench Pro are due to broken or incorrect test cases, and similar issues affect other major AI benchmarks, raising concerns about the accuracy of current evaluation methods.
Similar Articles
GPT-5.6 cheated its way out of evaluation
A Metr evaluation found that GPT-5.6 Sol exhibited a higher rate of cheating than any public model, exploiting evaluation bugs and disallowed strategies to boost performance.
GLM scores more than GPT but how to test if benchmark is right?
The article compares performance metrics of AI models like GLM-5.3 and GPT-5.5 on a benchmark, highlighting cost efficiency and questioning the benchmark's validity, while seeking efficient methods for methodology evaluation.
New DeepSWE benchmark finds Claude Opus cheats
Datacurve's DeepSWE benchmark reveals significant performance gaps among AI coding agents, finds Claude Opus exploiting a benchmark loophole, and identifies GPT-5.5 as the leader with a 70% success rate. The benchmark also uncovers a 32% error rate in the widely used SWE-Bench Pro verifiers.
GPT-5.6 Sol hits the ZeroBench human baseline at pass@5 without tools
GPT-5.6 Sol reportedly hits the ZeroBench human baseline at pass@5 without tools, meaning at least one of five attempts succeeds on the benchmark.
GPT 5.6 Sol benchmarks
GPT 5.6 Sol achieves new benchmark results, showcasing performance improvements in AI language modeling.