Same agent, same prompt, different runs. Which output do you ship?
Summary
The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.
Similar Articles
Same model, same prompt, 4 different agents
Explores how different agent architectures yield varying outputs from the same underlying model and prompt, highlighting the impact of agent design on LLM behavior.
Same model, same prompt, two agent harnesses: 45/50 vs 43/50
The article benchmarks two open-source coding agents on the same deepseek-v4-flash model, finding similar task success rates but significant differences in performance metrics and a critical bug in one agent's error handling.
The longer an agent runs, the less I care about the prompt
The author reflects on how long-running AI agents encounter failures unrelated to the initial prompt, arguing that environment design (tools, docs, validation, architecture rules) matters more. They discuss concepts like harness engineering, keeping AGENTS.md small, using linters, and evaluator agents, while noting the cost trade-offs.
What’s your agent up to?
We built an open-source retrospective reader for Claude Code to analyze agent behavior, using real execution history to inform runtime governance.
@dair_ai: If you maintain an AGENTS.md or a CLAUDE.md, this is worth a read. (bookmark it) 288 gold-test evaluated runs across Cl…
This paper presents a controlled ablation study across Claude Code and Codex, 17 real tasks, and 288 runs, finding that context files like AGENTS.md/CLAUDE.md do not measurably improve correctness; agents fail on implementation skill, not missing repository knowledge.