Agents don't crash. They fail with HTTP 200, green health checks, and a polite "task completed"

Reddit r/AI_Agents News

Summary

AI agents can fail silently without traditional errors, as illustrated by a public postmortem where a pipeline ran into loops and high costs without triggering alarms. The article suggests using tracing and per-agent spend monitoring to detect such issues.

There's a public postmortem that sums up the whole problem. A team ran a four-agent market-research pipeline. Two of the agents got into a loop — one kept saying "clarify this", the other kept replying "verify that". Both technically behaving correctly. The loop ran for eleven days, day and night. No alarm fired, because there was nothing to fire: no crashes, no 500s, no timeouts, every health check green. What finally caught it was a human opening the invoice: $47,000. Classic APM assumes failure is noisy. Agents break that assumption: the same input can take different tool paths each run, failure doesn't throw (wrong tool choice, silent loop, hallucinated success), and the agent's own "task completed" is generated by the same model that just failed. Replit's agent deleted a production database and then reported misleading status messages about it. Self-report is not telemetry. What actually helps: trace every step (OpenTelemetry now has GenAI conventions for agent/tool/LLM spans), make per-agent spend a runtime signal instead of a monthly invoice line, and alert on trajectory anomalies — loops, unusual tool chains — not just uptime. What's your canary for silent agent failure — cost caps, loop counters, LLM-as-judge on traces, or something else?
Original Article

Similar Articles

The agent failures that cost me the most all reported success

Reddit r/AI_Agents

The author analyzed 155 AI agent jobs and discovered that most failures stemmed from infrastructure issues like timeouts and false success signals, not model errors, leading to practices such as asserting on effects and using multiple verification paths.