A practitioner details the patterns built to keep fleets of LLM agents alive on multi-hour jobs (heartbeats, epoch fencing, intent/done records, parent audits) and explains why failures like credential delivery, workspace resets, and missing dead-man switches still cause silent agent deaths, asking the community for concrete fixes.
What I've built to keep agents alive on long-horizon tasks, and why it's still not enough. Looking for concrete suggestions I run a fleet of LLM agents on long jobs that last hours to days. The sample project: a pipeline that transcribes and summarizes a few hundred videos from several channels into a searchable knowledge base. It runs in batches, hits rate limits, gets interrupted, and has to resume without redoing or double-sending anything. Agents kept dying quietly. Nothing crashed loudly, they just stopped, and I found out hours later. Here is what I built so far. Heartbeats on one shared board. One row per job: last check, next expected check, deadline, current step, checkpoint link. Everyone writes the same six fields, so a single sheet shows the whole fleet. Deadlines at 2-3x the check interval. A missed beat is a flag, not an instant verdict. Bounded retries. Every retry loop has a cap and a total deadline. Standing watches leave the total deadline blank so they never "expire"; only finite jobs get one. A parent audit every 30 minutes. A separate agent reads the board and flags overdue rows. If more than half the jobs are flagged at once, it sends one grouped alert, because that is usually an outage or quota problem, not 10 dead agents. "Unknown" is never "dead". If the status check itself is throttled or the service is down, the result is unknown and nobody gets relaunched. Intent/done records for side effects. Before and after anything irreversible (send, commit, publish) the agent writes an operation key, so a relaunched agent can tell whether it already happened. Epoch fencing for relaunches. A relaunched agent gets a higher epoch and the old one stops itself if it wakes up and sees a newer epoch. Drive-first checkpoints and milestone commits. Progress goes to durable storage after each batch, with a pointer in the board row. A watchdog wake that checks the audit itself is still running. Why it's still not enough: - Credential delivery. Agents can't safely get a git token. Our vault can only fill browser inputs and never hands a value to a shell, so commits go through a slow browser route, small text files only. - Workspace resets wipe local state. Anything not checkpointed externally is gone, and the board only helps if the agent got to write to it. - No external dead-man switch. If the audit agent and the board go down together, nothing notices. Everything lives in the same ecosystem. - Step and context limits. Agents hit them mid-task and stop without a clean handoff. - Sheets quota throttling. With a dozen agents beating hourly plus audits, read quota runs out, and I've had to treat throttling as "unknown" everywhere. What I'd like suggestions on: - A cheap external dead-man switch that doesn't depend on the same platform (cron ping service? something else?). - A better place than a spreadsheet for heartbeats at this scale. - How you hand secrets to agents so they can commit without exposing the value in chat or logs. - Patterns for clean handoff when an agent is about to hit its step limit. - Anything you'd cut. I suspect some of this is over-built. Happy to share more detail on any piece.
The author shares five patterns that consistently kill AI agents: too many jobs per agent, no human-in-the-loop for destructive actions, unstructured outputs, no spend caps, and lack of uncertainty escalation paths. Practical guardrails and a checklist for reliable agent deployment are provided.
A community discussion on agent memory reveals that while various patches exist for what to write down (e.g., plain files, layered memory, post-mortems), the unsolved problem is what to keep—detecting failures is tractable, but deciding which lessons persist still needs human judgment.
The author reflects on how long-running AI agents encounter failures unrelated to the initial prompt, arguing that environment design (tools, docs, validation, architecture rules) matters more. They discuss concepts like harness engineering, keeping AGENTS.md small, using linters, and evaluator agents, while noting the cost trade-offs.
The author discusses the unglamorous but critical aspects of engineering reliable AI agents in production, including monitoring mid-flight runs, resuming failed runs, and providing UI status, and asks the community about common pain points and off-the-shelf solutions.
A developer reflects on moving AI agent workflows to a server and discovering that boring infrastructure concerns like systemd, logging, idempotency, and failure alerts matter more than the agent itself.