@joelniklaus: Codex is overoptimised for large models: it ranks 2nd out of 10 for GLM 5.2 but drops to 9th place for Gemma-4! Almost …

X AI KOLs Timeline News
coding-agents harness benchmark swe-bench model-evaluation cost-efficiency

Summary

A study comparing 10 coding agent harnesses on SWE-bench Pro shows harness choice dramatically affects pass@1 and cost, revealing vendor harnesses are overfitted to large models and do not transfer to smaller ones.

Codex is overoptimised for large models: it ranks 2nd out of 10 for GLM 5.2 but drops to 9th place for Gemma-4! Almost all the effort in this field goes into tuning the weights. We wanted to know how much of the final number is decided by the harness wrapped around them instead, so we ran 10 coding agent harnesses against two models on SWE-bench Pro. A lot, it turns out. Swapping the harness moves pass@1 from 23% to 52% on GLM-5.2, and from 15% to 36% on Gemma 4 26B-A4B: a wider gap than most model releases buy you. And the ranking does not transfer. The rank correlation between the two models' harness leaderboards is -0.05, which is to say: none. Codex is not alone in this. Every harness shipped by a model vendor drops on the small model — Codex 2nd to 9th, Claude Code 3rd to 7th, Qwen Code 4th to 6th — while the model-agnostic ones climb: crush 7th to 1st, opencode 8th to 2nd, pi 9th to 4th. The clearest case is crush, 7th on GLM-5.2 and 1st on Gemma 4. Run the identical scaffold on both and the small model wins by 4 points at a twelfth of the price, $0.30 per task against $3.61. Gemma 4's best harness beats GLM-5.2's four worst. A 26B model in the right scaffold is not far off a 744B model in the wrong one. Cost per solved task: $0.84 for Gemma 4 + crush at 36%. The cheapest GLM-5.2 setup that scores as well is openclaw at 38%, for $7.05. Output tokens per task span 16k to 621k across harnesses: a 39x spread in what you pay for, buying a 2x spread in what you get. 97% of input tokens are re-sent conversation prefix, so prompt caching is super important. The setup: every harness run against both models on the same 250 SWE-bench Pro tasks, one rollout each, priced at list API rates for the tokens each one actually spent. In the plot, a dark ring means the pairing sits on the Pareto frontier across both models and a faded dot means you could score higher for less elsewhere; 2 of the 10 harnesses (goose and hermes) are left out of the figure for legibility
Original Article
View Cached Full Text

Cached at: 08/08/26, 07:01 AM

Codex is overoptimised for large models: it ranks 2nd out of 10 for GLM 5.2 but drops to 9th place for Gemma-4!

Almost all the effort in this field goes into tuning the weights. We wanted to know how much of the final number is decided by the harness wrapped around them instead, so we ran 10 coding agent harnesses against two models on SWE-bench Pro.

A lot, it turns out. Swapping the harness moves pass@1 from 23% to 52% on GLM-5.2, and from 15% to 36% on Gemma 4 26B-A4B: a wider gap than most model releases buy you.

And the ranking does not transfer. The rank correlation between the two models’ harness leaderboards is -0.05, which is to say: none.

Codex is not alone in this. Every harness shipped by a model vendor drops on the small model — Codex 2nd to 9th, Claude Code 3rd to 7th, Qwen Code 4th to 6th — while the model-agnostic ones climb: crush 7th to 1st, opencode 8th to 2nd, pi 9th to 4th.

The clearest case is crush, 7th on GLM-5.2 and 1st on Gemma 4. Run the identical scaffold on both and the small model wins by 4 points at a twelfth of the price, $0.30 per task against $3.61.

Gemma 4’s best harness beats GLM-5.2’s four worst. A 26B model in the right scaffold is not far off a 744B model in the wrong one.

Cost per solved task: $0.84 for Gemma 4 + crush at 36%. The cheapest GLM-5.2 setup that scores as well is openclaw at 38%, for $7.05.

Output tokens per task span 16k to 621k across harnesses: a 39x spread in what you pay for, buying a 2x spread in what you get.

97% of input tokens are re-sent conversation prefix, so prompt caching is super important.

The setup: every harness run against both models on the same 250 SWE-bench Pro tasks, one rollout each, priced at list API rates for the tokens each one actually spent. In the plot, a dark ring means the pairing sits on the Pareto frontier across both models and a faded dot means you could score higher for less elsewhere; 2 of the 10 harnesses (goose and hermes) are left out of the figure for legibility

Similar Articles