Audited 1,228 human interventions in our AI agent setup. 91% weren't decisions, so we stopped letting agents say "done".

Reddit r/AI_Agents News

Summary

Audited 1,228 human interventions in AI agent workflows, revealing that 91% weren't decisions, prompting a new verification service with work contracts and shadow mode to improve reliability.

We run ~25 AI agent workspaces (PM → Dev → Tester → Critic → Release). Agents hand work off with markers in comments like PASS and RELEASE_READY. We audited 8 weeks of it: - Only ~9% of human interventions were real decisions. The rest was verifying, closing and re-routing work. - 59% of tasks were closed by a human by hand, even though an agent had already said they were done. - 8–11 per 100 tasks were false "done". For example, "30 files pushed" when nothing was on the remote, or a PASS below the critic's own threshold. - The board-sweeping agent was 76% of all tasks. 92–95% of its runs ended with "no action". What we're trying: keep every agent as-is, but add a small service they talk to over MCP. - Work contract per task type, listing what must be true: links resolve, UTM intact, the release package is on the remote branch, and so on. - Checks run by the service on the pushed branch. Agents can't see or edit them, and some are hidden. - Three states: PASS / FAIL / UNVERIFIED. Anything we can't prove is never faked as a pass. - "Done" can only be set by the service. A marker from the wrong agent is ignored. We're starting in shadow mode, where it only watches and logs. Questions: Has anyone measured how often their verifiers miss, e.g. by injecting known faults? How do you handle subjective checks without an LLM judge becoming the authority again? Is anyone doing continuous re-verification after "done"?
Original Article

Similar Articles

How do you verify your agent actually did what it says it did?

Reddit r/AI_Agents

A practitioner raises a common production problem: AI agents falsely reporting tasks as complete when the underlying action failed, and asks how others verify agent outputs in real deployments. The author is researching the issue and plans to share a summary of community answers.

20 AI agents, hundreds of thousands of test executions, and I'm still figuring out how to keep them accountable

Reddit r/AI_Agents

A developer shares an update on their multi-agent governance project with 20 agents and hundreds of thousands of test executions, highlighting that proving agents did what they claimed—not getting them to act—has been the core challenge. The latest iteration adds a hash-chained audit ledger, policy checks, human approvals, and evidence verification running in shadow mode before full enforcement.