The author built a deterministic verification layer that recalculates financial numbers produced by AI copilots to catch errors, and is seeking feedback from finance and AI practitioners.
Quick context: I've spent the last year+ building something in the "AI hallucination" space, specifically for finance, and I want honest feedback before I go further — not upvotes, actual criticism. The problem I'm trying to solve: AI copilots are increasingly drafting financial numbers — ratios, covenant checks, reconciliations, KPIs pulled from statements. The issue isn't that AI is bad at this, it's that it's confidently wrong sometimes, and in finance a confidently wrong number in a report or a covenant calculation isn't a minor bug, it's a real liability. What I built: A separate, deterministic verification layer (not another AI model) that sits behind the AI output. It: Extracts the actual source values from the underlying documents (PDFs, XLSX, DOCX) Independently recalculates the claimed number using exact rules/formulas, not vibes Compares the AI's claim against the recalculated value Flags mismatches with a full audit trail — what evidence was used, what rule was applied, where they diverged So instead of "trust the AI's math," it's "here's proof the math is right, or here's exactly where it's wrong and why." Where it stands right now: Working end-to-end on core financial ratios (net leverage, and a few others) Full evidence-to-conclusion traceability (nothing is asserted without a pointer back to source data) Not yet: broad rule coverage, tolerance-based matching (right now it's strict exact-match, which I know will cause false positives on rounding — actively working on this) What I'm NOT asking for: Money, beta signups, "check out my landing page." I genuinely want this torn apart before I put more time into the wrong thing. What I actually want to know: If you work in finance/accounting/audit/compliance — does "AI drafts it, a deterministic engine proves it" sound like something you'd actually want, or is this solving a problem nobody has? If you've built anything adjacent (fact-checking pipelines, agent guardrails, financial data extraction) — what broke when you tried something similar? What am I not seeing yet? Anyone dealt with the "AI + audit trail" requirement from a compliance angle — what would actually satisfy an auditor or regulator here, versus what sounds good but isn't enough? Happy to answer anything about how it works under the hood. Not trying to be cagey, just trying to keep this post from turning into a spec doc.
The author shares their experience building a prototype to verify AI-generated financial claims, focusing on systems and engineering challenges like evidence reconciliation and deterministic verification, and invites conversations with like-minded engineers.
This paper analyzes why enterprise AI deployments stall in regulated firms, proposing a production bar that includes accuracy, reproducibility, groundedness, and detectability. It measures the human review burden across model and tool configurations, showing that confidence signals and source citation can cut review from 100% to 49% but self-verification adds latency without improving error tolerance.
The article reports benchmarking results for a deterministic AI financial verification engine, showing perfect performance on structured claims (66/66) but poor performance when LLM-generated claims are used (19/66), indicating a translation gap between LLMs and formal systems.
The author is developing a deterministic verification engine for AI-generated claims, focusing on formal verification methods and seeking researchers and partners for collaboration.
A tool is being developed that introduces structured uncertainty to AI models to evaluate their ability to recover from errors, and review feedback is requested.