A developer reflects on an AI automation project that achieved 95% accuracy but still required full human review, leading to a redesign that routes uncertain outputs to a review queue and saves time overall.
I stopped by a clients office in March to see how a build was doing. Dana, the office manager, had two monitors going, AI output on the left, original customer emails on the right and a legal pad in the middle where she was ticking off every line by hand. She looked up at me with the tired politeness of someone checking a strangers homework then I realised that the stranger was me. I have been building products for 8 years(MVPs for founders, full production builds, a lot of rebuilds of other peoples abandoned systems and automations for small businesses as well) and this client was a supplier doing about 80 quotes a week and I had built them the thing every owner asks me for, a system that reads incoming purchase orders and drafts the quotes automatically. It hit 95 % in testing and the demo got actual applause in the conference room. Irl 95 % meant 4 wrong quotes a week and no one could know WHICH 4. So Dana checked all 80, every morning because one bad quote to a big account costs more than the software ever saved. I’ll be honest with you… my first instinct was to defend the 95. Then I sat with her timesheet and felt my stomach drop a little. She used to spend about 8 hours a week writing quotes. She was now spending 6 and a half checking them.... and the errors that got through were stranger than her old mistakes ever were. Wrong units on items she would never fumble and a discount for the one customer who never gets one. I had automated roughly 90 minutes and made the mistakes weirder and I got paid for it. The fix wasn’t more accuracy. I taught the system to know when it was unsure. A repeat customer ordering their usual items sails straight through. Anything new or a little off pattern lands in a review queue instead of going out. About a dozen quotes hit that queue each week. Dana checks those 12 like a hawk and ignores the rest completely and the ignoring is the whole product. There is an audit that I run on every build now. I add up the human mins spent reviewing what the machine produced. If that number is anywhere near the original task time then the work never left the building. It just changed desks. Dana still keeps the legal pad in her drawer btw. Doesn’t fully trust me yet which tbh seems fair.
The author reflects on conversations with people running AI automations, noting a pattern where verification is dropped after initial audits, which may hide silent failures. They ask for concrete stories about automations that were wrong without anyone noticing.
A developer discovers that a mandated AI rollout for contract review fails because the human process relied on a worker's informal relationship and context, leading to poor escalations; the fix involves keeping the human in the loop as an interpretation step.
The article discusses how the AI engineering loop can be fully automated but argues that handing over the entire loop produces 'agent slop' due to imperfect evals. It recommends automating certain steps while keeping human judgment for nuance.
AI has made writing code cheaper but shifted the hard work to context-setting, review, and cleanup, requiring more skilled oversight. The article argues that teams often treat AI-generated code as finished work when it is actually a fast first draft.