Same model, same prompt, two agent harnesses: 45/50 vs 43/50

Reddit r/AI_Agents News

Summary

The article benchmarks two open-source coding agents on the same deepseek-v4-flash model, finding similar task success rates but significant differences in performance metrics and a critical bug in one agent's error handling.

Ran 50 tasks rebuilt from merged PRs through two open source coding agents, both on deepseek-v4-flash. Same model, same system prompt written into both configs by a sync script that fails on drift, same sealed container, no web access for either. Graded by each project's own held-out tests plus a 3-model judge panel. 45/50 against 43/50, judge average 88.6 against 85.6. Cost was a wash, $1.59 against $1.53 for the whole run. Two cases out of fifty is not a margin I'd defend on its own. The more interesting split was wall clock, and it taught me not to trust my own averages. On the mean ours looked 2 minutes a case slower. On the median it's 5.5 against 7.0, and case by case ours is faster in 31 of 50. The entire mean gap came from a single case, a react hydration bug both agents failed, where ours ran 271 minutes and 1322 steps before giving up and the other quit at 63. Our no-progress detection plainly didn't fire. That's a real bug rather than a measurement artifact, and it's the worst single result in the run. Near-identical money also bought very different shapes of work: 1.5M output tokens against 573K, and 1.0M reasoning against 1.3M. Disclosure: ours is octomind, so this is our benchmark and our bias. Repo link in the comments per sub rules. Posting mostly because I haven't seen many same-model harness comparisons, and I'd like to know if anyone has run one on a bench we don't own.
Original Article

Similar Articles

Same agent, same prompt, different runs. Which output do you ship?

Reddit r/AI_Agents

The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.

Same model, same prompt, 4 different agents

Reddit r/LocalLLaMA

Explores how different agent architectures yield varying outputs from the same underlying model and prompt, highlighting the impact of agent design on LLM behavior.