Got a 2am PagerDuty alert because a background job double-fired after a transient DB error. The workflow had been running fine for weeks. Tests passed, CI was green. Step 2 committed a DB write. CDC picked up that committed change, published an event, consumer processed it. Step 3 failed before the workflow hit a durable checkpoint. Orchestrator retried step 2. Another committed write went through, CDC published again, consumer processed again. Three customer records were in a weird state and one charge went through twice. A 200 alone doesn't tell you whether the final business state is correct or whether an effect ran once. This shows up in background jobs, CRUD workflows, anything where a retry on a committed step can produce a second effect. Agents make it worse because they generate and run this logic fast. Idempotency keys, outbox patterns, and dedup exist to prevent it. The verification question is whether they're actually holding under that failure sequence. I'm building FetchSandbox around deterministic checks for this. Team defines the exact rule, say one committed customer update. On supported paths, FetchSandbox injects a controlled failure, you observe the app retry, then read what the saved state actually shows. Results come back pass, fail, or unmeasured. Gaps are explicit. Run the same checks again when the workflow changes. The DB fault injection and CDC-specific stack we hit here is something we want to build out with design partners. If you're an eng lead or workflow builder dealing with cases like this, what does your team do today to catch this?
A developer describes how an AI agent wrote a Stripe handler that double-provisioned customers on webhook retries, and how using FetchSandbox MCP to simulate retries helped catch and fix the idempotency bug.
The post describes building FetchSandbox, a tool that provides realistic testing environments for AI agents by simulating actual service provider responses to prevent failures in live production.