Same model, same prompt, two agent harnesses: 45/50 vs 43/50

Reddit r/AI_Agents News

Summary

The article benchmarks two open-source coding agents on the same deepseek-v4-flash model, finding similar task success rates but significant differences in performance metrics and a critical bug in one agent's error handling.

Ran 50 tasks rebuilt from merged PRs through two open source coding agents, both on deepseek-v4-flash. Same model, same system prompt written into both configs by a sync script that fails on drift, same sealed container, no web access for either. Graded by each project's own held-out tests plus a 3-model judge panel. 45/50 against 43/50, judge average 88.6 against 85.6. Cost was a wash, $1.59 against $1.53 for the whole run. Two cases out of fifty is not a margin I'd defend on its own. The more interesting split was wall clock, and it taught me not to trust my own averages. On the mean ours looked 2 minutes a case slower. On the median it's 5.5 against 7.0, and case by case ours is faster in 31 of 50. The entire mean gap came from a single case, a react hydration bug both agents failed, where ours ran 271 minutes and 1322 steps before giving up and the other quit at 63. Our no-progress detection plainly didn't fire. That's a real bug rather than a measurement artifact, and it's the worst single result in the run. Near-identical money also bought very different shapes of work: 1.5M output tokens against 573K, and 1.0M reasoning against 1.3M. Disclosure: ours is octomind, so this is our benchmark and our bias. Repo link in the comments per sub rules. Posting mostly because I haven't seen many same-model harness comparisons, and I'd like to know if anyone has run one on a bench we don't own.
Original Article

Similar Articles

Same Model, Different Harness: Different Coding-Agent Results

arXiv cs.AI

This paper investigates how changing the harness configuration in a coding agent impacts performance on coding benchmarks when the model remains fixed. The study shows that a treatment harness, which shortens older tool results to manage context, improves task completion rates, especially under tight context constraints.

Which model, which harness? I have data for you.

Reddit r/LocalLLaMA

An independent benchmark site (airbench.ai) tests AI agent harness and model setups on coding, vision, math, and computer-use tasks, finding that local quantized models like swift-1.5-qwen3.8-27b-q6_k and qwen3.8-flash-next-strata can now match Claude Opus 5.5's quality on an RTX 5090, and that harness choice (opencode/omp vs pi/hermes/openclaw) matters as much as the model itself.

Same agent, same prompt, different runs. Which output do you ship?

Reddit r/AI_Agents

The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.