The hardest production failures in agent systems seem to happen at the boundaries, not inside the agent

I recently asked where multi-agent workflows break after the demo stage: state, approvals, evaluation, deployment, rollback, etc. The replies changed how I’m thinking about the problem. A few production patterns came up repeatedly, and they seem less like isolated bugs and more like failures at the boundaries between agents, humans, and external systems. Here’s what I took away. 1. “Done” cannot be something an agent says about itself One person put it very clearly: the word done broke first. An agent would say a task was complete, but the only evidence was the agent’s own sentence. That seems like a fundamental mistake. For an external action: agent says success != success Completion should require evidence from the other side: provider acknowledgement returned object / state external verification durable receipt If the system sends a request and then loses the response, that is not necessarily FAILED. It is UNKNOWN. And UNKNOWN should not automatically become RETRY, because the external effect may already have happened. This becomes especially dangerous with payments, emails, deletes, deployments, or anything irreversible. 2. Approval as a boolean is not enough Another production lesson was that weeks after an incident, nobody could answer: what exactly the human saw what exact action they approved which evidence was available at the time whether the executed payload was still the same thing they approved approved = true proves very little. The approval seems to need to bind to an immutable execution intent: exact action version evidence snapshot approver timestamp payload / intent hash scope of authorization Then execution has to prove: executed intent == approved intent Otherwise there is an approval/execution gap. 3. But even perfect approval provenance does not solve rubber-stamping This was the most uncomfortable feedback. A clean decision card showing: recommendation risk expected impact confidence evidence may actually make approval easier to rubber-stamp. After twenty similar decisions, the reviewer stops independently evaluating the evidence and becomes a slow confirmation API. So there may be two separate approval problems: Approval integrity: can we prove what was approved? Judgment integrity: did a meaningful human review actually happen? That raises harder questions: Should approval workload be rate-limited? Should some approvals be sampled for an independent second reviewer? Should high-risk decisions use blind review, where the human sees the facts before seeing the agent recommendation? Should reviewer disagreement be measured? Is a 99.9% approval rate actually a warning signal? When does “human in the loop” become “human rubber stamp in the loop”? I’m especially interested in production teams that have dealt with approval fatigue. 4. Automated evals have the same independence problem Another useful observation: the same people who build the agent often build the eval. That means they can share the same assumptions and blind spots. A particularly interesting metric was: machine PASS / human REJECT That disagreement may be more valuable than the machine pass rate itself. If the automated evaluator says 95% success but independent human review only accepts 78%, the interesting number is probably the 17-point false-confidence gap. I’m starting to think disagreement needs to be a first-class production signal: machine vs human reviewer A vs reviewer B agent vs verifier expected effect vs observed external result Not something averaged away into one score. 5. Multi-agent chains seem to fail at the handoff Another person described the handoff itself as the breaking point. Agent A may produce a good recommendation, but Agent B receives only: What disappeared? Potentially: facts assumptions rejected alternatives uncertainty evidence decision criteria authority unresolved questions So the real failure may be: agent output != transferable state Every handoff becomes an information-loss boundary. A production handoff probably needs to be a structured contract, not just text inserted into the next prompt. Something closer to: source / destination objective facts assumptions claims evidence references unresolved questions decision granted authority expected next action One team also mentioned letting agents challenge each other’s assumptions before a recommendation reaches a human. That sounds useful, but only if “challenge” means something verifiable — e.g. challenge a specific claim and provide counter-evidence or identify an evidence gap — rather than just adding another agent opinion. 6. Recovery probably needs to operate on effects, not runs This was another expensive production lesson. Imagine: create payment record ✅ send $500 payout ✅ record provider response ❌ worker crashes Retrying the workflow may send another $500. A run-level retry is the wrong recovery abstraction once external side effects exist. Each external effect needs its own identity and recovery state. Something like: PLANNED → AUTHORIZED → DISPATCHED → CONFIRMED with exceptional states such as: UNKNOWN → RECONCILING → CONFIRMED / NOT_COMMITTED / MANUAL_REVIEW Idempotency therefore seems necessary per effect, not just per workflow run. 7. A reboot test may be more useful than another happy-path benchmark One production test someone mentioned was simply killing the system in the middle of a scheduled workflow. After restart: the approved instruction was unchanged the same job identity survived previous receipts were still valid execution continued from durable state That made me realize there is a difference between storing a checkpoint and preserving semantic continuity. A durable agent system should probably be able to prove after restart: same intent same authorization same effect identities same evidence same schedule same receipts same causal history Persistence alone is not enough if the meaning changes after recovery. The model I’m currently considering These comments are pushing me toward thinking about production agent systems as a set of integrity boundaries: Agent → Agent Can the next agent verify what it inherited? Agent → Human Can the human independently judge rather than rubber-stamp? Human → Execution Can we prove that what executed is exactly what was approved? Execution → External system Do we have external evidence of what actually happened? Failure → Recovery Do we know which effects are safe to retry? Machine evaluation → Human evaluation Can we detect when automated confidence is wrong? Runtime → Restart Does identity, authorization, and evidence survive reboot? I’m building Neura around some of these problems, so there is obviously product-builder bias here. I’m more interested in finding where this model is wrong than validating it. For people running agents in production: Which of these boundaries has caused the worst incident for you? And more importantly: What invariant or control did you eventually introduce that actually prevented it from happening again? Real failure cases, ugly workarounds, or controls that sounded good but failed in practice would be especially useful.