Open-source models are closing the coding gap with GPT/Claude/Gemini ~1.5x faster than the frontier is advancing, and on decontaminated benchmarks a 27B model already beats Claude Opus 4.8 [live dashboard + analysis]

Reddit r/singularity News

Summary

A live dashboard and statistical analysis shows open-source coding models are closing the gap with closed models at 1.5x the rate, with a 27B model already surpassing Claude Opus on decontaminated benchmarks. Tool-call reliability remains the main bottleneck.

Everyone argues about whether open-source AI is catching up to the closed labs. I got tired of vibes, so I built a live dashboard that plots open-weight vs closed models on the coding benchmarks that matter (SWE-bench Verified, SWE-rebench, BFCL tool-calling, LiveCodeBench) over time, then ran the actual statistics on the trend. What the data says: Open small models are the steepest line on the board. The best model you can run on a single consumer GPU went from 20% on SWE-bench Verified (Dec 2024) to 77% (mid-2026). Fitting the running-best frontier of each group, open ≤35B improves ~+39 pts/yr vs ~+26 for the closed frontier. That is ~1.5x faster, and the difference is statistically significant (p≈0.0002). On the benchmark that can't be gamed, the gap is almost gone. SWE-rebench pulls fresh GitHub issues every month, so nothing is memorized. There, a 27B open model (Qwen3.5-27B) scores 58.9, within ~4 points of the global #1 and above Claude Opus 4.8 (56.5), even though Opus posts 88.6 on the public benchmark. Most of the visible "closed lead" is contamination, not capability. I deliberately do not predict a crossover date. Extrapolating where two near-parallel lines cross is statistically unstable (the 95% interval runs mid-2026 to past 2028). The direction and rate are solid; the calendar date is not, so I don't headline one, and you should be skeptical of anyone who does. The one thing genuinely holding open models back is not raw intelligence, it is tool-call reliability. On BFCL v4 it is Anthropic 77.5 / Google 72.5 / open ≤35B 51.4, and that gap is not closing. It is a data problem: the closed labs train on billions of real agent trajectories from their own products (Claude Code, Codex), and there is no open equivalent. The writeup ends with a concrete pitch: build an open harness that collects anonymized tool-call traces plus success labels and pools them into a public dataset anyone can train on. That is a coordination problem, which open source is good at, unlike a frontier pretraining run. Dashboard (live, refreshes daily): https://botlab.dev/open-source-llm-benchmarks/ Full writeup with the stats and charts: https://botlab.dev/open-models-closed-ai-crossover-2026 Data comes from benchlm.ai, swe-rebench.com, and the Berkeley BFCL leaderboard. (Disclosure: my own project, free, no signup, no ads.)
Original Article

Similar Articles

New DeepSWE benchmark finds Claude Opus cheats

Reddit r/LocalLLaMA

Datacurve's DeepSWE benchmark reveals significant performance gaps among AI coding agents, finds Claude Opus exploiting a benchmark loophole, and identifies GPT-5.5 as the leader with a 70% success rate. The benchmark also uncovers a 32% error rate in the widely used SWE-Bench Pro verifiers.