My pipeline ran "successfully" for a week. Turned out my agent had been silently skipping failed API calls the whole time.
Summary
A developer recounts how their automated pipeline silently skipped failed API calls due to rate limiting, producing seemingly successful runs with empty data. They discuss the trade-off between retrying and hard-failing, and ask the community for best practices in agent error handling.
Similar Articles
Agents don't crash. They fail with HTTP 200, green health checks, and a polite "task completed"
AI agents can fail silently without traditional errors, as illustrated by a public postmortem where a pipeline ran into loops and high costs without triggering alarms. The article suggests using tracing and per-agent spend monitoring to detect such issues.
The agent failures that get you aren't crashes. They're clean runs that did the wrong thing.
The article discusses how AI agents often fail silently by completing tasks incorrectly without crashing, leading to undetected errors. It highlights common failure modes and explores potential detection strategies.
The agent failures that cost me the most all reported success
The author analyzed 155 AI agent jobs and discovered that most failures stemmed from infrastructure issues like timeouts and false success signals, not model errors, leading to practices such as asserting on effects and using multiple verification paths.
Your agent's retry logic dies when the agent does
The author shares lessons from putting an AI agent with write access into production, explaining that retry logic inside the agent loop fails when the process dies. They advocate for treating side-effectful tool calls as durable background jobs with idempotency keys.
All our scheduled agent jobs died for 19 hours on a usage limit while the plain scripts kept running
The article describes an incident where scheduled AI agent jobs failed for 19 hours due to a usage limit, while plain scripts continued running, and outlines solutions like implementing pre-checks and custom error handling to prevent similar issues.