Same model, same prompt, two agent harnesses: 45/50 vs 43/50
Summary
The article benchmarks two open-source coding agents on the same deepseek-v4-flash model, finding similar task success rates but significant differences in performance metrics and a critical bug in one agent's error handling.
Similar Articles
Same Model, Different Harness: Different Coding-Agent Results
This paper investigates how changing the harness configuration in a coding agent impacts performance on coding benchmarks when the model remains fixed. The study shows that a treatment harness, which shortens older tool results to manage context, improves task completion rates, especially under tight context constraints.
Which model, which harness? I have data for you.
An independent benchmark site (airbench.ai) tests AI agent harness and model setups on coding, vision, math, and computer-use tasks, finding that local quantized models like swift-1.5-qwen3.8-27b-q6_k and qwen3.8-flash-next-strata can now match Claude Opus 5.5's quality on an RTX 5090, and that harness choice (opencode/omp vs pi/hermes/openclaw) matters as much as the model itself.
Same agent, same prompt, different runs. Which output do you ship?
The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.
Your harness is failing your agent but there's no benchmark to prove it
The article highlights a lack of benchmarks for evaluating the reliability of agent harnesses, specifically focusing on how MCP implementations handle tool calls and errors compared to the models themselves.
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
This paper shows that the choice of agent harness (scaffold) can cause up to a 40x difference in tokens per solved task, while model pass rates vary only slightly, demonstrating that harness–model pairs, not model alone, should be compared for human-centered coding-agent evaluation.