The untuned 27B beat the tuned 75B as an agent
Summary
A comparison showing that an untuned 27B parameter model outperforms a tuned 75B parameter model in agent tasks, highlighting potential inefficiencies in scaling and fine-tuning.
Similar Articles
A 35B model beat a 120B one on my coding agent, 95% vs 53%. Build your own benchmark.
A developer built a custom benchmark for coding AI agents and found that a 35B-parameter model outperformed a 120B-parameter one when the harness was optimized, highlighting the importance of tailored evaluation over generic specifications.
Released a model tuned for agent testing work that other models refuse. AgentDojo 97.5% utility.
A fine-tuned model based on GLM-5.2, abliterated and specialized for agent testing and red teaming, achieving 97.5% benign utility on AgentDojo and strong coding benchmarks.
Same model, same prompt, two agent harnesses: 45/50 vs 43/50
The article benchmarks two open-source coding agents on the same deepseek-v4-flash model, finding similar task success rates but significant differences in performance metrics and a critical bug in one agent's error handling.
@che_shr_cat: 1/ Parameter scale is a brute-force crutch. What if a 35B model could beat a 1,000B model simply by scaling its search …
Explores whether a 35B parameter model could surpass a 1000B model by scaling its search horizon at test time, using structured process feedback rather than brute-force parameter scaling.
AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
AutoTuneBench presents a benchmark and measurement protocol for trustworthy agent auto-tuning of LLM serving engines, addressing failure modes like strawman baselines and infrastructure defects to ensure reliable results.