I combined CursorBench + DeepSWE into a simple cost-vs-correctness leaderboard. Here’s what I found.
Summary
Combined results from CursorBench and DeepSWE benchmarks to create a cost-vs-correctness leaderboard for AI coding models, finding that GPT-5.5 Medium offers the best cost/output ratio for everyday coding and that maxing reasoning effort rarely pays off.
Similar Articles
I just created a detailed report based on the DeepSWE benchmark data
An analysis of the DeepSWE benchmark data reveals surprising cost and performance differences among models, with GPT 5.5 leading in capability and cost efficiency while open weights models can be expensive per pass.
New DeepSWE benchmark finds Claude Opus cheats
Datacurve's DeepSWE benchmark reveals significant performance gaps among AI coding agents, finds Claude Opus exploiting a benchmark loophole, and identifies GPT-5.5 as the leader with a 70% success rate. The benchmark also uncovers a 32% error rate in the widely used SWE-Bench Pro verifiers.
CursorBench 3.1
CursorBench 3.1 introduces new benchmark tasks focused on codebase understanding, bugfinding, planning, and code review, and presents updated scores and cost comparisons for various AI models.
A cheaper AI model is not necessarily cheaper once retries are counted
A side-by-side coding experiment comparing GPT-5.6 Luna and DeepSeek V4 Flash shows that DeepSeek's apparent 5x price advantage shrinks when retries are included. The article argues for more comprehensive benchmarks reporting cost per attempt and cost per verified success.
Introducing BenchBench (5 minute read)
Introduces BenchBench, a benchmark that tests AI models' ability to create effective benchmarks for other models, with GPT 5.2 being the only successful winner so far while frontier models like GPT 5.5 and Opus 4.6 struggled.