An open-source coding agent harness was stress-tested by using at least 100 AI agents to update its own documentation, demonstrating efficient parallel task handling and identifying real issues in the docs.
We open-sourced our coding agent harness this week and wanted a stress test that also did real work. So we gave it one prompt: update the docs for the release, and use at least 100 agents to do it. Here's what the orchestrator (GPT-6 Astra) built from that: one planner split our 24 doc pages into 100 review jobs, four per page (claims vs the Rust source, what the release changed, whether examples and flags still match the code, links and limits), plus four site-wide checks 100 reviewers on GPT-6 Luna, all running at once, told to read and report, not edit 25 editors on GPT-6 Sol, one file each, so none of them could step on another two more Sol agents checking the edits against the code at the end Honestly, 24 pages don't need 100 agents. A dozen would've done it. We wanted to know whether the harness holds at that width, and it did: all 100 came back with a report, and the slowest took just over two minutes. The fan-out took 4m 55s, and the agents' time added up to 1h 57m. Prompt to commit was just under 10 minutes: 29 files, +276 / -90. The catches were real. The SDK error example in our docs didn't compile (it returned a String where the tool wanted a ToolError). A page claimed a subagent depth limit of 5 that the code doesn't have. Three supported providers were missing from the provider page. The final check found two pages disagreeing about an MCP timeout. It also documented that our TypeScript REPL tool is broken on recent bun, which is true and not fun to read in your own docs. What it cost We ran this one through OpenRouter to get a clean per-model breakdown: $2.94 for Astra, $2.08 for Sol, and $0.73 for all 100 Luna reviewers, about $5.75 in total. All three also run on a ChatGPT subscription, so if you already pay for one, the same run adds nothing to your bill. Paying per token instead, the cheap tier carries the load: 100 reviewers at under a cent each. Either way, the cost stays under control by design: Each reviewer gets one page, one angle, at most 6 rounds of looking and at most 4 findings. No dumping the repo into context, no compiling. Findings go to a file, and the reviewer replies to the orchestrator in under 100 words. After the whole fan-out Astra's context was at 75K of 1.05M, 7%. The model list is enforced, not suggested. You set which models each tier may use in /subagents, and a worker call naming anything else is refused before it starts, so the planner can't quietly reach for the big model. Workers are bounded by default (24 steps or 5 minutes), so a stuck one ends instead of spending. It all runs as tasks inside one 6.5 MB Rust binary rather than a hundred copies of a CLI. If you run fan-outs like this, where do you put the line between reviewer and editor? Or do your workers edit directly?
Analysis of 246 open-source repositories and 57 papers on agent harnesses highlights best practices like human-written context, limiting tools, and incremental testing to improve AI agent performance.
An open-source agent harness with 40 capabilities behind a single function call, including persistent memory, Docker sandbox, auto-summarization, stuck-loop detection, budget caps, and live run forking for branching agent execution. Built on Pydantic AI and designed to replace the 2000 lines of glue code every production agent needs.
The author found that AI agent reliability issues were caused by the harness rather than the model, and improved it by separating context tracking, loading, and output checking, while working on versioning to manage code across projects.
The post describes an evaluation of GPT-6 Astra across six agent harnesses on 29 agentic tasks, showing similar success rates but token usage varying by 3–5x on failures.
The author discusses building six AI agent harnesses and emphasizes the need for a dedicated database to track agent execution, state, and learnings, beyond just observability tools.