The author describes how they make their AI growth agent Alice trustworthy by using deterministic code, fact-set verification, and scoped number matching, rather than relying on prompt rules. They share practical lessons for any agent that reports numbers to users.
I'm building an AI growth agent called Alice, for solo SaaS founders or small teams. Every morning she reads your GA4, Search Console and other sources, names the part of your funnel that's actually leaking, and gives you one action. The hard part was never making her sound smart. It was making her stop lying with real numbers... lol Three examples off my own dashboard: - "59 of your 106 sessions came from accounts.google.com." Both numbers real. The 59 belonged to a different channel. The true answer was 32. - "5 clicks this week, down from 4." That's up. - She was holding 25 rows of search query data and told me to go export the query data from Search Console. Homework she had already done. None of those look like hallucinations. That's what makes them dangerous. Real numbers in the wrong sentence read as authoritative. What actually worked, in order: Deterministic code on the critical path This is the whole thing, everything below is an application of it. Decide which parts of your output the product's promise depends on, and take those away from the model. We had prompt rules covering every failure above. Measured compliance was around 70%. Fine for tone, useless for facts. If a rule has to hold every time, it lives in code, not in the prompt. Verify against a fact set, not a vibe check Before the model sees anything, code builds every number that legitimately exists: each value, each total, each per-source subtotal, plus fair derivations. After it writes, every number gets matched back. The match has to be scoped: a number in a sentence about one traffic source must belong to that source. A plain "does this number appear somewhere" check would have passed the 59. Retry once, naming the exact violation Not "try again." The retry gets the offending sentence and the reason. Keep whichever version has fewer violations, log anything that survives. Code decides, the model writes Two refreshes on identical data gave me two different "top problems." Now code scores the funnel layers and picks the bottleneck, and Alice just writes that verdict in plain English. If the headline names a different layer, it fails and retries. Same data, same verdict, every time. Don't let it do math Every legitimate derived figure gets computed server-side and handed over. A model doing arithmetic in prose is a bug factory, and no checker can tell good mental math from a lucky-looking invention. Test the checker harder than the model My favorite bug: the checker read the word "directly" as the Direct traffic channel, decided that sentence's numbers were misattributed, and vetoed the single best briefing she has ever produced. The homework version shipped instead. False positives destroy good output as reliably as hallucinations ship bad output. Most of my test suite now exists to prove correct sentences pass. Give people a playbook, not a blank chat box Accuracy is only half of trust. The other half is that most founders don't know what to ask an analytics agent, so a chat box makes them feel stupid and they leave. Alice opens with the verdict and one action already chosen, and the follow-up questions are pre-written and clickable. The agent decides what's worth asking today. The user decides whether to act. An agent that waits for the perfect question is just a mirror. Alice is live and free to try if you want to point her at your own property and see what she says. Link in the comments. If you're building agents that report numbers to users, what are you doing to verify outputs? Every check I've added has found something, which makes me suspect most agents shipping right now are wrong more often than their users know. :-)
The author shares practical insights on building client trust in AI agent systems, emphasizing the importance of narrow scope, robust error handling, and clear communication of system status.
As AI agents become ubiquitous, the challenge shifts from comparing performance to establishing trust and reputation, requiring new discovery and verification systems.
Anthropic publishes a research post detailing how to build trustworthy AI agents in practice, outlining core safety principles and product implementations like Claude Code and Claude Cowork.
The author argues that the reliability of AI agents comes from deterministic code, not the LLM, and shares five key practices for building trustworthy agents on messy real-world data.
The article discusses the governance challenges that arise when AI agents interact with real company data and tools, highlighting the need for policy enforcement and audit trails, and mentions Trust3 AI as a potential solution.