The article warns against using model-generated narration as the authoritative audit log for AI agents, advocating for persisting raw tool call data instead, and suggests a simple diff check to catch discrepancies.
In most agent stacks the record of what the agent did is written by the agent. The run summary, the narrated trace your teammate skims in the trace UI, all of it is model output. That text is a claim about what happened. Whether it matches what happened is a separate question. The two drift apart in boring, predictable ways. A tool call errors, the model retries twice, and the final summary describes the first attempt. A step gets skipped and the narration includes it anyway, because the summary is written from the plan and the plan said it would happen. Partial completion gets rounded up to done. None of this needs the model to be especially bad. Summarizing is lossy, and the loss leans toward whatever makes the story coherent. The problem is who reads what. Raw tool results are noisy JSON, so humans read the narration instead. Which means the layer with the weakest guarantees ends up dressed as the authoritative account, and that's the version that flows into status dashboards and postmortems. The fix is mostly plumbing. The runtime already sees every tool call: name, args, raw result, status code, timestamp. Persist that at call time, from the harness, and treat it as the log. Model text becomes annotation on top. Derive success from response codes or a check of external state. The model saying it succeeded doesn't count as evidence. And a dumb diff between the actions the summary claims and the actions the runtime recorded, run before the summary reaches anyone, catches most of the divergence nearly for free. It's string matching over tool names, roughly. Caveats, because this isn't a cure. The runtime log proves the call happened. Whether it did the semantically right thing is a different problem, the email really went out, just to the wrong list. External state checks cost real calls. And once in a while the narration is correct while the raw result misleads, like an idempotent retry coming back 409 because the thing it wanted to create already exists. Does anyone diff claimed actions against recorded ones automatically? And what's the worst narration vs reality gap you've caught after trusting a summary?
The author discusses a failure mode encountered while building Sentience Governor, a Python library for Claude Code that monitors agent actions and produces audit reports. The AI sometimes reconstructed explanations from raw traces, blurring the line between measured facts and probabilistic interpretation.
Discusses the importance of proper audit logging for AI agents, emphasizing the need for append-only, hash-chained logs that prevent tampering, rather than storing logs in the same writable application database.
Discusses a common failure mode in AI agents where the model claims to have executed a tool call without actually firing it, and advocates for trusting execution receipts over agent narration to ensure reliability in production.
This article argues that AI agents making business recommendations must maintain complete audit logs to ensure trust and accountability for users, merchants, developers, and platforms, drawing parallels with traditional advertising systems.
A reflection on current practices for verifying AI coding agent output, noting that developers often skim diffs and merge without fully auditing the agent's session activity, raising concerns about code review culture in the age of AI.