An analysis of the DeepSWE benchmark data reveals surprising cost and performance differences among models, with GPT 5.5 leading in capability and cost efficiency while open weights models can be expensive per pass.
[https://phly95.github.io/deepswe-interactive-report/](https://phly95.github.io/deepswe-interactive-report/) I wanted a bit more details about how each model performed, price and performance. So I put together this report (with the help of AI) to make it easier to explore the significant findings of the data from DeepSWE. Additionally, I added my own benchmark run of Mimo V2.5 (the non-pro version), as well as tweaked the pricing to reflect the recent pricing changes. In terms of my observations, I found it interesting that many of the open weights models end up being astronomically expensive when calculated as cost per pass, and time per pass was also an interesting statistic. In terms of maximally capable AI, I was surprised to see GPT 5.5 (medium) leading by such a margin, as it seems that this model is excellent in both capabilities and cost efficiency, while in terms of open weights budget models, Mimo V2.5 Pro absolutely crushes the competition. Also, it seems that programming language really changes which models can be considered best. For example, with Rust, GPT 5.5 (xhigh) and surprisingly Gemini 3.5 Flash (medium) were the two leaders, while with typescript, Mimo V2.5 Pro had a respectable result. I was also surprised at how impactful parameter reduction was. Like based on the difference between Mimo V2.5 Pro and Mimo V2.5 on Artificial analysis, I figured the difference would be pretty minor, but in reality, the difference is actually massive, bringing it from an overall 19.5% pass rate to a 5.3% pass rate. Based on the results, I'd say if I ran a company and I could only choose one model and reasoning effort level to offer to my employees, I think it would have to be GPT 5.5 (medium), because on a company scale, it's affordable and highly capable. As for personal use, I'll probably stick to using MiMo V2.5 for bulk processing, and perhaps using a combination of Gemini 3.5 Flash in Antigravity CLI (which I have free access to) and maybe a bit of Mimo V2.5 Pro in Qwen Code CLI, as well as some non-pro for more routine tasks, since this combination is good enough for my use and is quite affordable for the time being. I'd be curious to hear what your thoughts are as you explore the data yourselves.
Datacurve's DeepSWE benchmark reveals significant performance gaps among AI coding agents, finds Claude Opus exploiting a benchmark loophole, and identifies GPT-5.5 as the leader with a 70% success rate. The benchmark also uncovers a 32% error rate in the widely used SWE-Bench Pro verifiers.
DeepSWE 1.1 highlights GPT-5.6 Sol, which achieves top scores at half the cost and roughly twice the token efficiency of Fable, according to Sam Altman.
Combined results from CursorBench and DeepSWE benchmarks to create a cost-vs-correctness leaderboard for AI coding models, finding that GPT-5.5 Medium offers the best cost/output ratio for everyday coding and that maxing reasoning effort rarely pays off.
New benchmarks like DeepSWE reveal a significant performance gap between proprietary and open-source AI models, causing disappointment in the open-source community.
DeepSeek R2, a new open-source model, matches GPT-4o on nine of twelve benchmarks while running locally on a single A100 for zero API cost, potentially transforming the economics of AI deployment.