The Checking Problem: What must be true before AI ships in a regulated firm

# The Checking Problem What must be true before AI ships in a regulated firm
Source: [https://arxiv.org/html/2607.28666](https://arxiv.org/html/2607.28666)
\(July 2026\)

###### Abstract

Enterprise AI programmes stall at a rate that is widely quoted and poorly explained\. This paper measures the mechanism\. Six document\-heavy workflows of the kind performed daily in regulated financial services were run across four model families and three tool configurations, three times each, producing 5,093 scored output elements across 72 configurations\. Each configuration was assessed twice: against a demonstration bar, being a single correct run on a single case, and against a production bar requiring sustained accuracy, reproducibility across repeats, verifiable attribution, and a confidence signal that carries information\. 57 of 72 configurations cleared the demonstration bar and 32 cleared the production bar, a survival rate of 56\.1 %\. The paper then computes the review burden each configuration imposes, estimated out of sample rather than with hindsight\. A tool that states no confidence requires review of 100 % of its output, because it offers a reviewer no basis for triage\. Requiring the tool to cite its sources and state a confidence reduces that to 49 % while holding the residual error tolerance in 17 of 20 configurations\. Adding a self\-verification pass costs 2\.3 times the latency of the plain configuration, reaches 44 %, and is the only configuration that fails to hold the error tolerance\. The practical implication is that the value of an AI workflow is set less by how often it is right than by how much of it a human must still check, and that the second property is measurable and rarely measured\.

## 1\. The problem

The statistics on enterprise AI attrition are quoted more often than they are explained\. The standing explanations are model capability, data quality, integration difficulty and change management\. Each is plausible and each is asserted\.

This paper proposes a different mechanism and measures it\. The bar a pilot is judged against and the bar a regulated deployment must clear are different instruments, and the second tests properties the first cannot see\. A tool can be shown once, on one document, to a receptive audience, and produce a correct answer\. Nothing in that exercise establishes whether it produces the same answer twice, whether its cited evidence exists, or whether it knows when it is wrong\. Those properties are invisible at a demonstration and decisive in production\.

The contribution is threefold: an instrument, a measurement, and a costing\. The instrument is a pair of acceptance bars, published in full so that a reader may dispute or apply them\. The measurement is what happens when the same tools are assessed against both\. The costing is what it takes to close the difference, expressed as the share of output a human must still review\.

## 2\. The two bars

### 2\.1Demonstration bar

One run, one case, all elements correct\. This is what a steering committee observes: the tool is exercised once on one document and the output is right\.

### 2\.2Production bar

Four criteria, all of which must hold\.

- •P1 Accuracy\.Element\-level correctness of at least 0\.99 across every case and every repeat, not on a favourable single run\.
- •P2 Reproducibility\.Identical input returns identical output across repeated runs at fixed settings\.
- •P3 Groundedness\.Where the tool cites a source, that passage occurs in the source document\. Threshold 0\.95\.
- •P4 Detectability\.Stated confidence separates correct from incorrect elements, measured as the area under the receiver operating characteristic curve, threshold 0\.70\. A value of 0\.5 means the confidence signal is worthless\.

Three further production requirements are described but not measured here: recorded human attestation, behavioural stability across model versions, and data\-boundary compliance\. These are properties of the control environment rather than of the output, and are noted so that the instrument is not mistaken for a complete control framework\.

The asymmetry is the point\.The demonstration bar tests P1 alone, once, on a friendly input\.P2, P3 and P4 are structurally invisible at pilot stage, which is why programmes pass the steering committee and then fail to ship\.

## 3\. Method

### 3\.1Workflows

Six workflows spanning the range from zero error tolerance to high judgement, each specified generically enough to apply to any document\-heavy regulated operation: fixed\-field extraction from fund terms \(W1\); entity screening against a reference list \(W2\); cross\-document reconciliation \(W3\); classification against a published regulatory taxonomy \(W4\); attributed summarisation of a long document \(W5\); and multi\-document comparison \(W6\)\. Full briefs are published with the corpus\.

### 3\.2Corpus, and why W1 to W4 are synthesised

W1 to W4 run on synthetic documents generated deterministically from a fixed seed by a standard\-library script with no language model involved at any point\. This is a deliberate methodological choice rather than a convenience\.

Ground truth must be exact, and a wrong answer key is the one error class that no downstream verification can detect\. If a real filing is used, the key must be established by hand: a person reads the document and records what the management fee is\. Any error in that reading is invisible thereafter, because every verification step compares the paper against the data and the data is already wrong\. Synthesis removes the failure mode: the generator selects the values first, writes the document around them, and emits them as the key\. The ground truth is the input to the document, not a reading of it\.

The synthetic documents are modelled on real filings in structure, section ordering, defined\-term convention and legal phrasing, and use real legal forms and domiciles\. Six genuine fund offering documents from six distinct fund families, retrieved from SEC EDGAR with full provenance, are published alongside as the structural reference, so that the claim of realism can be checked rather than taken on trust\. W5 and W6 use real filings directly as task inputs, so the battery is not entirely synthetic\.

The bias direction is stated plainly: synthetic documents are*cleaner*than real ones, having no scanning artefacts, no reflowed tables and no inconsistent drafting across sections\. Measured performance here is therefore an upper bound, and any production gap observed is wider on real filings\.

### 3\.3Tool configurations

Three configurations forming a ladder of engineering effort\.

- •C1 Plain\.Perform the task\. How most pilots are built\.
- •C2 Governed\.Cite the source passage for each element, state a confidence for each, and decline explicitly where the document does not support an answer\.
- •C3 Verified\.C2, followed by a second pass in which the tool re\-reads its own output against the source and issues a correction\.

These are a proxy for tool variation, not a census of it\. Real deployed tools differ in retrieval quality, chunking, orchestration, guardrails and human\-in\-the\-loop design, none of which is controlled here\. What the ladder does capture is the axis the paper’s claim implicates directly: how much verification apparatus surrounds the model\.

### 3\.4Models, repeats and scoring

Four families, so that no finding rests on one vendor: a frontier closed model, a mid\-tier closed model, and two open\-weights models\. Three repeats per cell at temperature zero\. Repeats are what make P2 measurable and are the reason this battery can report a statistic to which single\-run evaluation is structurally blind\.

W1 to W4 are scored deterministically against the answer key with no judge involved, removing judge variance from the majority of the battery\. W5 and W6 are scored mechanically on groundedness, which requires no judgement because the quoted passage either occurs in the filing or does not\.

### 3\.5Numeric discipline

Every figure in this paper is produced by script and substituted programmatically; none is typed by hand\. Four layers guard the arithmetic\. The answer keys are re\-derived independently from the rendered document text\. The statistics are computed once, then recomputed by a second implementation that shares no code and uses different algorithms\. Every numeral in the source is audited against the register of verified claims, so that a number cannot appear without a corresponding check\. Coverage extends to arithmetic performed in prose and to ordinal claims, not only to table cells\.

This architecture is airtight against arithmetic error and silent against ground\-truth error\. During this study it caught two implementation defects in the review\-burden computation\. It did not catch an error in an answer key; that was found by noticing an implausible result and investigating\. Section 8 records both\.

## 4\. Results

### 4\.1The gap

57 of 72 configurations cleared the demonstration bar\. 32 cleared the production bar\. Survival rate 56\.1 %, so43\.9 % of configurations that pass a demonstration cannot be placed in production\.

Table 1:Criteria failed among the 25 configurations that cleared the demonstration bar and failed the production bar\. A configuration may fail more than one criterion\.Accuracy is implicated in 22 of 25 failures, so it would be wrong to claim that accuracy is not the dominant constraint\. The defensible claim is narrower and still consequential:accuracy alone does not predict production readiness\. 3 configurations failed with no accuracy problem whatsoever, and reproducibility was the sole cause of failure more often than accuracy was\.

### 4\.2Accuracy by workflow and configuration

Table 2:Pooled element accuracy\. Reproducibility is the share of case\-element groups whose repeated runs agree exactly\.The gap is not uniform\. Fixed\-field extraction is solved: accuracy 1\.000 across every model and configuration\. Entity screening is close behind at 0\.983\. Reconciliation is the weakest workflow at 0\.799 and regulatory categorisation sits at 0\.889\. A blanket claim that document work is or is not automatable is unsupported in either direction; the answer is workflow\-specific, and the discriminating property is whether the task requires arithmetic over the document or the application of a rule to it\.

### 4\.3Real filings

W5 and W6 run on genuine filings retrieved from public registers\. Accuracy is 0\.964 and 0\.881 respectively, and mean groundedness, the share of cited passages that occur in the filing they are attributed to, is 0\.931 and 0\.919\. Neither workflow contributed a configuration that cleared the production bar on all applicable criteria, and detectability is the usual reason: mean area under the curve of 0\.520 on W5, which is close enough to 0\.5 to mean the confidence signal carries almost no information about correctness\.

Reproducibility is*not*assessed for W5 and the paper does not report a figure for it\. Two runs of a summarisation task never produce identical prose, and claims are reordered between runs, so a position\-wise comparison is meaningless; any set\-overlap measure would require inventing a threshold purely to produce a number\. W5 is therefore assessed against three production criteria rather than four, which makes its results more generous than the rest of the battery, not less\.

### 4\.4The demonstration trap

W3 illustrates the mechanism most sharply\. Pooled accuracy by model:

Table 3:Reconciliation accuracy by model, all configurations pooled\.Open\-weights model A scores 0\.319 on reconciliation against 0\.927 across the battery\. Broken out by element, it answers the binary question, whether the statement reconciles, at approximately chance, while returning the correct discrepancy amount far less often\. A demonstration that asks whether the tool can spot a reconciliation break observes a tool that works\. A process that needs the amount observes a tool that is guessing\. The same tool passes one bar and fails the other, and the difference is not subtle\.

## 5\. The verification burden

### 5\.1Definition

The commercially relevant quantity is not accuracy but the share of output a human must still inspect\. A reviewer triages by stated confidence, inspecting the least confident first\. Where a tool states no confidence, there is nothing to triage on\.

Two estimates are reported and the distinction matters\. The*oracle*figure selects the confidence threshold knowing the errors that actually occurred; it is a lower bound that no reviewer can achieve\. The*operational*figure calibrates the threshold on half the cases and applies it to the held\-out half, which is what a deployment actually does: calibrate on a pilot sample, then run on unseen work\.

Table 4:Review burden and configuration cost\. Residual error tolerance 1 % of elements\. “Holds” counts configurations whose out\-of\-sample residual error stayed inside tolerance\.Tolerance held in 20 of 20 configurations for C1, 17 of 20 for C2, and 16 of 20 for C3\.

### 5\.2What the numbers say

The oracle understates burden by roughly 2\.7 times\.Any evaluation that reports review burden with hindsight materially underestimates what a deployment costs\.

A plain tool requires review of 100 % of its output\.This is not a modelling artefact\. A tool that emits no confidence signal gives a reviewer no basis on which to check less than everything, so the calibration step finds no usable threshold\.

Governance instrumentation halves the burden to 49 %, holding the error tolerance out of sample in 17 of 20 configurations\. This is the measured value of requiring a tool to cite and to state confidence\. It does not make the tool trustworthy\. It lets a reviewer safely skip half the output, which is a different and more useful thing\.

Self\-verification costs 2\.3 times the latency, reaches 44 %, and is the only configuration that breaks the error bar\.Buying the most engineering did not buy the best outcome\.

### 5\.3What the ladder does and does not buy

Moving from C1 to C3 did not improve accuracy: 4 configurations improved and 11 degraded across 15 decisive pairs, mean change \-0\.0168, sign testp=p=0\.118\. It did not improve reproducibility either: 2 improved and 5 degraded across 7 pairs,p=p=0\.453\. Neither result is significant at conventional levels and neither is claimed as one; what can be said is that no accuracy benefit was detected in exchange for a 2\.3\-fold cost increase\.

Where C3 does help is attribution, raising mean groundedness from 0\.933 to 0\.952\. The practical reading is that self\-verification is worth its cost when the exposure is an untraceable audit trail, and is not when the exposure is inconsistent output or throughput\.

## 6\. Related work

This paper sits between three literatures that rarely meet\.

Productivity evidence\.Controlled studies establish large gains on tasks inside a model’s competence and little or none outside it\. Peng et al\. report a 55\.8 % reduction in completion time on a coding task\[[1](https://arxiv.org/html/2607.28666#bib.bib1)\]; Noy and Zhang report a 40 % time reduction with a quality gain on professional writing\[[2](https://arxiv.org/html/2607.28666#bib.bib2)\]; Brynjolfsson, Li and Raymond report output gains concentrated among less experienced workers\[[3](https://arxiv.org/html/2607.28666#bib.bib3)\]; and Dell’Acqua et al\. find gains inside a capability frontier and degradation outside it\[[4](https://arxiv.org/html/2607.28666#bib.bib4)\]\. These studies measure what a tool produces\. None measures what it costs to check\.

Confidence and selective prediction\.That the effective review rate is a coverage decision under a risk constraint is not novel: it is the risk–coverage tradeoff of selective classification, in which a model abstains on inputs it cannot answer within a target error rate\[[7](https://arxiv.org/html/2607.28666#bib.bib7)\]\. The question of whether stated confidence is informative is the calibration question\[[5](https://arxiv.org/html/2607.28666#bib.bib5)\], and the specific question of whether language models can report their own reliability has been studied directly\[[6](https://arxiv.org/html/2607.28666#bib.bib6)\]\. The contribution here is not the framework but its application: coverage is treated as a human review budget in a regulated operation, calibrated out of sample rather than reported as an oracle bound\.

Attribution and evaluation\.Measuring whether a generated claim is supported by its cited source is an established problem\[[8](https://arxiv.org/html/2607.28666#bib.bib8)\], adjacent to the hallucination literature\[[9](https://arxiv.org/html/2607.28666#bib.bib9)\]and to retrieval\-augmented evaluation frameworks\[[10](https://arxiv.org/html/2607.28666#bib.bib10)\]\. This paper’s groundedness criterion is a deliberately austere version: whether the quoted passage occurs in the filing at all, which requires no judgement and no model\. Where judged evaluation is used elsewhere, the reliability of the judge is itself contested\[[11](https://arxiv.org/html/2607.28666#bib.bib11)\], which is why the headline claims here rest on deterministic scoring\.

Governance of deployed systems\.Szpruch et al\. argue that in agentic financial\-services systems many material failures are process failures arising at runtime rather than defects visible in a model evaluation\[[12](https://arxiv.org/html/2607.28666#bib.bib12)\]\. The production bar proposed here is a concrete, measurable instance of that argument: three of its four criteria cannot be observed from a single output at all\.

What is not established\.The stall rates widely quoted in industry commentary are survey findings rather than controlled measurements, and are not relied upon here beyond motivating the question\. No prior study, to the author’s knowledge, measures the difference between demonstration acceptance and production acceptance directly, nor reports the review burden that difference implies\.

## 7\. Implications

If the binding constraint on deployment is the cost of checking rather than the frequency of error, three things follow\.

The question to put to a vendor changes\.Not how accurate is it, but does it know when it is wrong, and can its citations be traced\. Accuracy is reported by every pilot\. Detectability and groundedness are reported by almost none, and they are what determine review burden\.

Pilot acceptance criteria change\.A pilot that demonstrates a correct output on a representative document has tested one of four properties\. Running the same input three times, and checking whether cited passages exist, costs almost nothing and tests two more\.

The economic case is set by the review ratio\.Where checking an output is much cheaper than producing it, as in field extraction, a tool pays even at full review\. Where checking costs nearly as much as deciding, as in judgement\-heavy work, it does not\. This reproduces the familiar document\-heavy versus judgement\-heavy distinction, but derives it from measured review burden rather than asserting it\.

## 8\. Limitations

The acceptance bars are practitioner\-elicited, not derived\.They represent one operator’s view of what a regulated deployment requires\. They are published in full precisely so that they can be disputed, and no claim is made that they constitute a standard\.

The corpus is cleaner than production data\.W1 to W4 are synthetic and W5 and W6 are well\-formed electronic filings\. Real operational documents are messier, so the measured gap is a lower bound\.

Configurations proxy for tools\.Three configurations are not a census of deployment design\.

Sample size is modest\.Six workflows, four models, three repeats\. This is a first measurement of a quantity that has not been measured, not a definitive estimate, and the configuration comparisons in particular are underpowered\.

Ground\-truth risk is real and residual\.During this study one answer key was found to be wrong: a categorisation case whose figures fell below every statutory threshold was keyed as meeting them\. Two models were being marked incorrect for being correct, and a third appeared perfect because it agreed with the error\. Every verification layer passed throughout\. It was caught by noticing an implausible result\. The answer keys are now re\-derived independently from the rendered documents wherever they are mechanically derivable; where a key encodes a legal judgement rather than an arithmetic fact, no independent route exists and the risk remains\.

Two measurement defects were found and corrected, both in the direction of overstating performance\. Groundedness was initially measured against a converted rendering of each filing rather than the filing itself, and 20 % of apparent citation failures proved to be conversion artefacts\. The review\-burden computation initially selected its threshold with hindsight and, where no confidence existed, exploited arbitrary ordering\. Both are corrected in the figures reported here\. Both were caught by internal checks rather than by review, which is the argument for building them\.

## 9\. Reproducibility

The corpus generator, task prompts, scoring code, analysis, verification and audit scripts are published with the paper at[https://github\.com/dsauce/checking\-problem](https://github.com/dsauce/checking-problem)\. The synthetic corpus regenerates byte\-identically from its seed\. Real documents are identified by EDGAR accession number and canonical URL\.

## Appendix AWorkflow briefs

Each brief states the task, the input, and the basis on which the answer is scored\. The full text of every prompt, in all three configurations, is published with the code\.

W1 Fixed\-field extractionExtract twelve defined fields from a fund offering document into a fixed schema\. Scored by exact match per field against the generated key, with stated normalisation for currency and percentage formats\. Document intensity high, judgement low, error tolerance near zero\.

W2 Entity screeningScreen every disclosed party in a subscription document against a reference list of 36 entries, returning matches and non\-matches\. The list contains deliberate near\-misses: legal\-suffix variants \(Ltd against Limited, BV against NV\) and transliteration variants of surnames\. Scored exactly\. A false negative is a regulatory incident; a false positive is an operational cost\.

W3 Cross\-document reconciliationDetermine whether a stated closing balance reconciles to an opening balance as adjusted by a transaction log of 15 to 24 entries, and if not, report the discrepancy\. Half the cases carry a planted break\. Scored exactly on both the binary and the amount\.

W4 Regulatory categorisationCategorise a client under the MiFID II taxonomy and state the criterion relied upon\. One case is a trap: all three Annex II\.1\(2\) size tests fall just below their thresholds, so the correct answer is retail\. It separates a tool that applies the rule from one that recognises the shape of a large undertaking\.

W5 Attributed summarisationSummarise a filing in under 400 words and attribute every material claim to a part of the source\. Scored mechanically on whether each cited passage occurs in the filing\. Reproducibility not assessed\.

W6 Multi\-document comparisonCompare fee and liquidity terms across four filings and flag material differences\. Scored on whether every numeral asserted in a comparison cell occurs in the filing it is attributed to, rather than on verbatim phrasing, because a cell for a fund with tiered minimums or waived fees is legitimately a composed answer\.

## Appendix BCorpus provenance

W1 to W4 are generated bygen\_corpus\.pyfrom a fixed seed using only the Python standard library, with no language model involved\. The corpus regenerates byte\-identically\. Every document declares on its face that it is a fictional example; no real institution is named\.

The documents below are genuine filings retrieved from SEC EDGAR\. Those marked*ref*are held as the structural reference for the synthetic templates and are not scored; they exist so that the claim that the synthetic documents reflect real filings can be checked rather than taken on trust\. Those marked W5 and W6 are used directly as task inputs\. Each is retrievable by accession number\.

Table 5:Real filings used, by role, issuer, form type and EDGAR accession number\.
## Appendix CVerification architecture

Four independent layers\. No figure in this paper was typed by hand; all are substituted programmatically from the verified results file\.

1. 1\.Answer keysare re\-derived from the rendered document text, which is what the model sees, rather than from the generator’s internal state\. The reconciliation key is re\-established by re\-summing the printed transaction log\.
2. 2\.Statisticsare computed once, then recomputed by a second implementation sharing no code and using different algorithms: rank\-sum rather than pairwise for the area under the curve, threshold sweep rather than forward scan for the review rate\.
3. 3\.Every numeralin the source is audited against the register of verified claims\. A number cannot appear in the paper without a corresponding check; exemptions are limited to structural values and each carries a stated reason\.
4. 4\.Coverageextends to arithmetic performed in prose and to ordinal claims, not only to table cells\.

## References

- \[1\]Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer \(2023\)\. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot\.*arXiv \(Cornell University\)*\. arXiv:2302\.06590\.
- \[2\]Shakked Noy, Whitney Zhang \(2023\)\. Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence\.*SSRN Electronic Journal*\. DOI: 10\.2139/ssrn\.4375283\.
- \[3\]Erik Brynjolfsson, Danielle Li, Lindsey Raymond \(2025\)\. Generative AI at Work\.*The Quarterly Journal of Economics*\. arXiv:2304\.11771\.
- \[4\]Fabrizio Dell’Acqua, Edward McFowland, Ethan R\. Mollick, Hila Lifshitz\-Assaf, Katherine Kellogg, Saran Rajendran et al\. \(2023\)\. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality\.*SSRN Electronic Journal*\. DOI: 10\.2139/ssrn\.4573321\.
- \[5\]Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q\. Weinberger \(2017\)\. On Calibration of Modern Neural Networks\.*arXiv \(Cornell University\)*\. arXiv:1706\.04599\.
- \[6\]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez et al\. \(2022\)\. Language Models \(Mostly\) Know What They Know\.*arXiv*\. arXiv:2207\.05221\.
- \[7\]Yonatan Geifman, Ran El\-Yaniv \(2017\)\. Selective Classification for Deep Neural Networks\.*arXiv*\. arXiv:1705\.08500\.
- \[8\]Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das et al\. \(2023\)\. Measuring Attribution in Natural Language Generation Models\.*Computational Linguistics*\. DOI: 10\.1162/coli\_a\_00490\.
- \[9\]Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu et al\. \(2022\)\. Survey of Hallucination in Natural Language Generation\.*arXiv*\. arXiv:2202\.03629\.
- \[10\]Shahul Es, Jithin James, Luis Espinosa Anke, Steven Schockaert \(2024\)\. RAGAs: Automated Evaluation of Retrieval Augmented Generation\.*Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations*\. arXiv:2309\.15217\.
- \[11\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang et al\. \(2023\)\. Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.*Advances in Neural Information Processing Systems 36*\. arXiv:2306\.05685\.
- \[12\]Lukasz Szpruch, Agus Sudjianto, Tanveer Bhatti, Gary Ang \(2026\)\. Scalable Runtime Governance for Agentic AI in Financial Services\. DOI: 10\.2139/ssrn\.6567199\.