Same model, same prompt, two agent harnesses: 45/50 vs 43/50
Summary
The article benchmarks two open-source coding agents on the same deepseek-v4-flash model, finding similar task success rates but significant differences in performance metrics and a critical bug in one agent's error handling.
Similar Articles
Same agent, same prompt, different runs. Which output do you ship?
The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.
Your harness is failing your agent but there's no benchmark to prove it
The article highlights a lack of benchmarks for evaluating the reliability of agent harnesses, specifically focusing on how MCP implementations handle tool calls and errors compared to the models themselves.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
This paper shows that the choice of agent harness (scaffold) can cause up to a 40x difference in tokens per solved task, while model pass rates vary only slightly, demonstrating that harness–model pairs, not model alone, should be compared for human-centered coding-agent evaluation.
Same model, same prompt, 4 different agents
Explores how different agent architectures yield varying outputs from the same underlying model and prompt, highlighting the impact of agent design on LLM behavior.
Same model, different harness: 30-50 point performance swing. But teams still pick agents by model name.
The article highlights that agent harnesses cause a 30-50 point performance swing compared to model selection, arguing that teams should focus on instance-level verification rather than just model names.