Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

# Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays
Source: [https://arxiv.org/html/2608.14639](https://arxiv.org/html/2608.14639)
###### Abstract

Per\-field accept/review with selective risk at mostα\\alpha— accept a field only if the error rate among accepted fields is controlled — is the trust contract document\-extraction systems need, and the natural procedure \(threshold a confidence score via an add\-one bound on a calibration split\) silently violates it on real documents\. On 13,859 genuineclaude\-sonnet\-5fields from 800 CORD receipts \(49\.0% field correctness\) we diagnose and quantify three failure modes:document clustering\(a design effect of 1\.84–2\.45 that roughly halves the effective calibration size\),score\-refit leakage\(fitting a learned score and its threshold on the same fields: coverage 0\.416 at risk 0\.127, violating nominalα=0\.10\\alpha\{=\}0\.10in 95% of splits\), and atie\-mass pathology\(a degenerate score distribution collapses the threshold grid; a counterfactual zeroing of a single signal reproduces a0\.030→0\.0010\.030\\\!\\to\\\!0\.001collapse of certified coverage\)\. We organize the fixes as avalidity ladderwith the guarantee form stated per tier\. A fit/val*split protocol*restores expected\-selective\-risk control for a learned fusion score: coverage0\.318at achieved risk 0\.096 at nominalα=0\.10\\alpha\{=\}0\.10with no tolerance band \(production variant 0\.326 at 0\.097\) — an on\-average operating point whose realized risk exceedsα\\alphain 47\.5% of resplits, not a certificate\. Mondrian Learn\-then\-Test with exact binomial tails yields per\-group PAC*certificates*: field\-iid0\.171coverage at risk 0\.068 \(violations 0\.03\), cluster\-corrected 0\.140 at 0\.051, and doc\-iid0\.060at 0\.020 — the only tier whose assumptions match documents, and honestly near\-vacuous today\. Support\-bin, the pre\-specified grounding\-derived*provenance*taxonomy, wins every rigor tier on the sonnet CORD capture \(p<10−4p<10^\{\-4\}, sign\-flip over 40 document\-level resplits; Bonferroni\-corrected over taxonomies\) — a win that does*not*replicate on the same documents under haiku or qwen \(§[7](https://arxiv.org/html/2608.14639#S7)\) — while on higher\-accuracy corpora pooled thresholds win: conditioning rescues certification exactly where a pooled threshold cannot certify, and a learned score subsumes it elsewhere\. A frozen\-configuration confirmation on a selection\-untouchedclaude\-haiku\-4\-5capture held at both risk levels \(0\.167 at 0\.093; 0\.068 at 0\.037\), and a blind three\-annotator human\-gold audit verifies the practical tier’s accepted\-set risk at*1\.3%1\.3\\%*against its10%10\\%budget \(Fleiss’κ=0\.83\\kappa\{=\}0\.83; automatic calibration labels err one\-sidedly pessimistic\)\. Released Apache\-2\.0 with seed\-pinned, regression\-gated procedures\.

## 1Introduction

Document parsers now read pages fluently and still emit silently\-wrong structured values\(Ferguson and others,[2026](https://arxiv.org/html/2608.14639#bib.bib1)\)\. The remedy the field converges on is a per\-field*trust contract*: each extracted field carries a confidence and an accept/review decision, and the system promises that the*selective risk*— the error rate among accepted fields — stays below a targetα\\alpha\. The natural implementation is folklore: fit a confidence score, hold out a calibration split, and pick the smallest threshold whose add\-one\-smoothed empirical selective risk is≤α\\leq\\alpha\(Geifman and El\-Yaniv,[2019](https://arxiv.org/html/2608.14639#bib.bib15); Angelopouloset al\.,[2024](https://arxiv.org/html/2608.14639#bib.bib17)\)\. This paper shows, on genuine frontier\-LLM output at scale, that the folklore procedure silently violates the contract on real documents — and shows what to run instead, at three explicit levels of rigor\.

Our testbed is deliberately hard and deliberately real: 13,859 per\-field predictions captured fromclaude\-sonnet\-5on 800 CORD receipts, of which only 49\.0% are correct, plus FUNSD and XFUND\-de captures spanning the difficulty spectrum \(Table[1](https://arxiv.org/html/2608.14639#S3.T1)\); construction, labels, and measurement findings are in the companion benchmark paper\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\.111Companion papers: this paper owns the procedures, guarantees, diagnoses, and characterization; VerifyDocBench\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)owns the datasets, labeling protocol and reliability audit, and the model/language measurement study\. Numbers here are reproducible from the released harness \(seed\-pinned splits, regression\-gated procedures\)\.

#### Contributions\.

\(C1\) Diagnosis\.Three quantified failure modes of naive per\-field selective guarantees on documents — document clustering \(design effect 1\.84–2\.45\), score\-refit leakage \(risk 0\.127 at nominal 0\.10, 95% of splits violating\), and tie\-mass pathology \(a degenerate score collapses the threshold grid\) — each pinned by a counterfactual experiment \(§[4](https://arxiv.org/html/2608.14639#S4)\)\. None had been quantified for document extraction\.\(C2\) Protocol\.A*validity ladder*as a reporting standard \(§[5](https://arxiv.org/html/2608.14639#S5)\): practical tier \(fit/val split protocol \+ add\-one; controls*expected*selective risk\), rigorous field\-iid PAC tier \(Mondrian Learn\-then\-Test with exact binomial tails, per\-groupP​\(risk\>α\)≤δP\(\\text\{risk\}\>\\alpha\)\\leq\\delta\), and rigorous doc\-iid PAC tier \(per\-document bound; the exchangeability unit that matches documents\), each row annotated with its estimand, assumption, and violation fraction\.\(C3\) Result\.The first held\-at\-nominal operating points and PAC certificates on genuine frontier\-LLM extraction at scale \(§[6](https://arxiv.org/html/2608.14639#S6)\), with*provenance*\(grounding\-derived support bins\) the pre\-specified conditioning taxonomy that wins every rigor tier on the hard regime’s sonnet capture \(p<10−4p<10^\{\-4\}\) — a model\-scoped result, not a corpus\-general one: §[7](https://arxiv.org/html/2608.14639#S7)shows the same win collapses on the identical documents under a weaker\-signal model\.\(C4\) Characterization\.A two\-regime empirical law with a stated mechanism \(§[7](https://arxiv.org/html/2608.14639#S7)\): with a learned accept score, covariates belong*in the score*\(Mondrian conditioning is subsumed and can hurt\); with a frozen or weak score, they belong*in the taxonomy*— and taxonomy\-side conditioning pays exactly where the pooled threshold cannot certify at the targetα\\alpha\.\(C5\) Artifact\.An open, decoupled harness: 40 fixed document\-level splits \(seed 7\), bit\-exact sanity gates chained across experiments, a 463/624\-cells\-bit\-identical regression check, and full disclosure of a capture\-stage data defect and its forensics \(Appendix[B](https://arxiv.org/html/2608.14639#A2)\)\.

We claim no new conformal theory: the machinery is classical\(Vovket al\.,[2003](https://arxiv.org/html/2608.14639#bib.bib24); Angelopouloset al\.,[2021](https://arxiv.org/html/2608.14639#bib.bib18);[2024](https://arxiv.org/html/2608.14639#bib.bib17)\)\. The contributions are the diagnoses, the protocol, the certified application, and the characterization\.

## 2Related Work

Confidence for extraction\.*Beyond Logprobs*\(Preprint authors,[2026a](https://arxiv.org/html/2608.14639#bib.bib8)\)fuses logprobs and consistency for document field confidence with ECE/AUROC/selective\-risk reporting; Cleanlab TLM\(Cleanlab,[2025](https://arxiv.org/html/2608.14639#bib.bib10)\)sells model\-agnostic per\-field trust scores; real\-time trustworthiness scoring\(Preprint authors,[2026b](https://arxiv.org/html/2608.14639#bib.bib9)\)is similar\. None attaches grounding, and none provides a risk\-controlled accept/review guarantee — the axes added here and in the companion benchmark\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\. Grounding for KIE is itself established \(DocILE’s KILE task\(Šimša and others,[2023](https://arxiv.org/html/2608.14639#bib.bib4)\), SROIE\(Huang and others,[2019](https://arxiv.org/html/2608.14639#bib.bib5)\), OCRBench v2\(Fu and others,[2025](https://arxiv.org/html/2608.14639#bib.bib6)\), BoundingDocs\(Preprint authors,[2025a](https://arxiv.org/html/2608.14639#bib.bib29)\)\); we use provenance as a*conditioning covariate for risk control*, not as an output format\. Uncertainty signals such as semantic entropy\(Farquharet al\.,[2024](https://arxiv.org/html/2608.14639#bib.bib11)\)and consistency/verbalized fusion\(Chen and Mueller,[2023](https://arxiv.org/html/2608.14639#bib.bib12)\)share a blind spot \(self\-consistent errors\), motivating the external verification signal — grounding support — our taxonomy uses\.

Selective prediction and risk control\.The add\-one selective\-risk threshold is the selective\-classification bound ofGeifman and El\-Yaniv \([2019](https://arxiv.org/html/2608.14639#bib.bib15)\); conformal risk control\(Angelopouloset al\.,[2024](https://arxiv.org/html/2608.14639#bib.bib17)\), conformal factuality\(Mohri and Hashimoto,[2024](https://arxiv.org/html/2608.14639#bib.bib22)\), and selective CRC\(Preprint authors,[2025b](https://arxiv.org/html/2608.14639#bib.bib26)\)control*marginal*risk\. Learn\-then\-Test\(Angelopouloset al\.,[2021](https://arxiv.org/html/2608.14639#bib.bib18)\)converts risk control into multiple testing — our rigorous tiers instantiate it with exact binomial tails\(Clopper and Pearson,[1934](https://arxiv.org/html/2608.14639#bib.bib21)\)and Holm step\-down\(Holm,[1979](https://arxiv.org/html/2608.14639#bib.bib20)\)\. Conditional\-coverage theory\(Gibbset al\.,[2025](https://arxiv.org/html/2608.14639#bib.bib23)\)shows exact per\-instance conditioning is impossible while exact*group*\-conditional control is attainable \(Mondrian CP\(Vovket al\.,[2003](https://arxiv.org/html/2608.14639#bib.bib24)\)\); risk\-controlling prediction sets\(Bateset al\.,[2021](https://arxiv.org/html/2608.14639#bib.bib19)\)give the PAC form we state\. CRC\-certify\(Preprint authors,[2026d](https://arxiv.org/html/2608.14639#bib.bib25)\)defines field\-level JSON losses and abstention bounds but does not condition on provenance and does not diagnose the document\-specific failure modes that are this paper’s subject\. Risk\-controlled generative OCR\(Preprint authors,[2026c](https://arxiv.org/html/2608.14639#bib.bib27)\)and VISA\(Preprint authors,[2024](https://arxiv.org/html/2608.14639#bib.bib28)\)establish visual attribution;Traub and Kirchhof \([2024](https://arxiv.org/html/2608.14639#bib.bib16)\)the evaluation side of selective prediction\.

## 3Setup

### 3\.1Task and trust contract

Given a documentDDand a JSON schemaSS, an extractor outputs leaf fields with values; a trust layer attaches to each field a confidencec∈\[0,1\]c\\in\[0,1\], a grounding \(page/bbox/char\-span with a support score\), and a decision in\{accept,review\}\\\{\\text\{accept\},\\text\{review\}\\\}\. The contract: maximize*coverage*\(fraction accepted\) subject to selective risk \(error rate among accepted\)≤α\\leq\\alpha\. Correctness labels are schema\-typed \(exact/numeric/semantic per leaf\); omission and hallucination are scored separately\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\.

### 3\.2Data: genuine frontier\-LLM captures

All headline experiments run on*genuine*per\-field output ofclaude\-sonnet\-5\(k=3k\{=\}3self\-consistency\), captured once and frozen \(Table[1](https://arxiv.org/html/2608.14639#S3.T1)\)\.These captures are text\-layer prompted: the model reads the document’s OCR text layer, not the page image — despite “VLM” being the natural shorthand for a closed frontier API model, no vision capability is exercised anywhere in this paper’s headline results \(the companion benchmark paper’s cross\-vendorgpt\-4orow is the one genuinely vision\-based capture in either paper; see\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\)\. CORD is the hard regime: only 49\.0% of asserted fields are correct, yet grounding is strongly discriminative \(grounded\-vs\-ungrounded correctness gap\+0\.352\+0\.352, 95% document\-clustered CI\[0\.33,0\.37\]\[0\.33,0\.37\]; verbalized AUROC0\.8450\.845\[0\.83,0\.86\]\[0\.83,0\.86\]\)\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\. Theclaude\-haiku\-4\-5capture is reserved for the frozen\-configuration confirmation \(§[6\.4](https://arxiv.org/html/2608.14639#S6.SS4)\) and touched by no selection step\.

Table 1:Genuine per\-field captures used in this paper\. “correct” = fraction of asserted fields scored correct; “grounded” = fraction with a located source\. The haiku capture is selection\-untouched \(confirmation only\)\.
### 3\.3Signals and accept scores

Per field we compute five signals: verbalized self\-report,kk\-sample self\-consistency, a grounded flag, an entailment\-NLI score of the value against its source region, and*ambiguity\-penalized grounding support*\. Support is the trust\-relevant provenance signal: when a predicted value matchesmmequally\-good page locations \(a bare “2” matches many tokens\), we retainsupport=score/m\\text\{support\}=\\text\{score\}/m\. Under a uniform prior over themmequally\-good matches with exactly one true source,1/m1/mis the chance a given match is the source — a well\-behaved heuristic \(not a claim of optimality\) that quarantines coincidental short\-value matches out of the well\-grounded group\. Table[2](https://arxiv.org/html/2608.14639#S3.T2)ablates the penalty form; softer forms \(1/m1/\\sqrt\{m\},1/\(1\+ln⁡m\)1/\(1\{\+\}\\ln m\)\) under\-penalize and leave coincidental matches \(and their error\) in the grounded group\.

Table 2:Ablation of the ambiguity\-penalty form: support retained for a unit\-score match found atmmequally\-good locations\. The uniform\-prior1/m1/mdemotes ambiguous matches most aggressively; softer forms under\-penalize\.Two accept scores are used throughout\. Theshared low\-capacity fusionis a 5\-signal logistic regression \(the black\-box default; we do not call it “fixed” because it is refit per split — see the caveat in §[6\.1](https://arxiv.org/html/2608.14639#S6.SS1)\)\. Thelearned fusion\(hgb\_split\) is a depth\-3 histogram gradient\-boosted tree over the five signals plus engineered features \(value length, digit fraction, is\-numeric, signal interactions, and field\-type one\-hots\), with document\-level early stopping; test AUROC on CORD 0\.925 vs 0\.871 for the LR\.\_noentvariants drop the NLI signal\.

### 3\.4Conditioning taxonomies

Mondrian conditioning applies the identical threshold rule within each group of a taxonomy\(Vovket al\.,[2003](https://arxiv.org/html/2608.14639#bib.bib24)\)\. Candidates:pooled\(one group\);support\-bin\(terciles of ambiguity\-penalized support at calibration quantiles0\.34/0\.670\.34/0\.67\) — the*pre\-specified provenance taxonomy*, declared as this project’s thesis in pre\-campaign drafts before any scale experiment ran;fieldtype\-freq\(leaf\-key vocabulary, keys with≥\\geq25 calibration fields\);fieldtype\-rule\(fixed keyword map, data\-independent\)\. All data\-dependent taxonomy parts \(bin edges, vocabularies\) are computed from threshold\-fitting rows only\.

### 3\.5Evaluation protocol: what “held” means

Every number is a mean over40 fixed document\-level 50/50 calibration/test splits\(numpygenerator seed 7; fields of one document never straddle a split\), atnominalα\\alphawithno tolerance band\. A configuration is*held*iff its mean achieved test selective risk is≤α\\leq\\alpha; we always co\-report the coverage standard deviation across splits andviol, the fraction of splits whose realized risk exceedsα\\alpha\. Paired differences use two\-sided sign\-flip permutation tests \(20,000 flips\): their floor is1/20,0011/20\{,\}001, so we writep<10−4p<10^\{\-4\}, never smaller; because the 40 resplits share documents, thesepp\-values measure*split\-resampling stability on this corpus*, not population\-level significance\. Zero\-coverage splits contribute risk 0 \(disclosed wherever it matters, §[6\.1](https://arxiv.org/html/2608.14639#S6.SS1)\)\.

## 4Why naive per\-field selective guarantees fail on documents

The add\-one rule picks the smallest thresholdτ\\tauwhose smoothed empirical selective risk on calibration is≤α\\leq\\alpha:

τ=min⁡\{t:1\+\#​\{i:ci≥t,erri\}1\+\#​\{i:ci≥t\}≤α\},τ=∞​\(review everything\) if none qualifies\.\\tau=\\min\\Big\\\{\\,t:\\ \\tfrac\{1\+\\\#\\\{i:\\,c\_\{i\}\\geq t,\\ \\text\{err\}\_\{i\}\\\}\}\{1\+\\\#\\\{i:\\,c\_\{i\}\\geq t\\\}\}\\ \\leq\\ \\alpha\\,\\Big\\\},\\qquad\\tau=\\infty\\text\{ \(review everything\) if none qualifies\.\}Under exchangeability of calibration and test fields this controls the*expected*selective risk atα\\alpha\(Geifman and El\-Yaniv,[2019](https://arxiv.org/html/2608.14639#bib.bib15); Angelopouloset al\.,[2024](https://arxiv.org/html/2608.14639#bib.bib17); Preprint authors,[2025b](https://arxiv.org/html/2608.14639#bib.bib26)\)\. On real documents, three separate mechanisms break it \(Table[3](https://arxiv.org/html/2608.14639#S4.T3)\)\. The “before” exhibit is the operating point an earlier draft of this work headlined: coverage 0\.114 at achieved risk0\.122\(78% of splits violating\) at nominalα=0\.10\\alpha\{=\}0\.10— produced by exactly these mechanisms plus taxonomy selection\.

Table 3:Three diagnosed failure modes, each pinned by a counterfactual\. All at nominalα=0\.10\\alpha\{=\}0\.10, 40 document\-level splits\.#### Failure 1: document clustering*invalidates*the marginal bound\.

Fields cluster within documents, and calibration/test splits are \(and must be\) document\-level\. At the add\-one threshold the estimated design effect is 2\.15 \(CORD\), 1\.84 \(FUNSD\), 2\.04 \(XFUND\-de\) — up to 2\.45 at other thresholds — so the effective calibration sample is roughly*half*its nominal size\. On the 6,901\-field CORD dump the pooled add\-one rule lands at achieved risk 0\.105, violating in 50% of splits \(FUNSD and XFUND\-de likewise, Table[3](https://arxiv.org/html/2608.14639#S4.T3)\)\. The mean overshoot is a hair — clustering’s main effect is per\-split*variance*— but the many\-group grounded×\\timessupport taxonomy shows the badly\-broken case: risk 0\.122 with 78% of splits violating\. A clean\-fit control \(score fit on held\-out documents, threshold on the rest\) still overshoots \(0\.105, 50%\), isolating clustering from refit leakage\.

#### Failure 2: score\-refit leakage\.

Fitting a high\-capacity score and its threshold on the*same*calibration fields transfers the score’s optimism into the threshold\. The depth\-3 gradient\-boosted fusion under the same\-half protocol posts coverage 0\.416 at risk0\.127, violating in 95% of splits — an invalid operating point that looks spectacular\. A 5\-parameter logistic fusion barely overfits \(risk 0\.105\), which is why the flaw goes unnoticed until score capacity grows\. The fix is protocol, not prose \(§[5\.1](https://arxiv.org/html/2608.14639#S5.SS1)\)\.

#### Failure 3: tie\-mass pathology \(discrete scores break threshold grids\)\.

Our first 13,859\-field capture silently shipped an all\-zero entailment column \(the NLI stage was skipped mid\-capture; Appendix[B](https://arxiv.org/html/2608.14639#A2)\)\. With only coarse discrete signals left, the fused score collapsed from 1,702 distinct calibration values \(intact 6\.9k dump\) to 257, with tie masses of 221 and 183 fields at the acceptance head\. A threshold accepts a tie mass whole or not at all: the smallest reachable candidate acceptedn=245n\{=\}245fields at empirical risk 0\.114\>α\>\\alpha, so no certificate existed at any confidence level — rigorous certified coverage collapsed0\.030→0\.0010\.030\\\!\\to\\\!0\.001and the doc\-level add\-one to exactly 0\. Causality is pinned counterfactually: zeroing entailment on the*intact*dump reproduces the collapse \(0\.0302→0\.00110\.0302\\\!\\to\\\!0\.0011\), and nothing else changed\. Two harness lessons generalize to any discrete or heavily\-tied score: \(i\) snap candidate thresholds to distinct\-value boundaries \(label\-independent, hence free of validity cost; regression\-gated — 463/624 cells bit\-identical, max headline drift 0\.0088, and the grid fix alone moves certified coverage by≤\\leq0\.004, i\.e\. the pathology was the score, not the grid\); \(ii\) argsort\-based diagnostics silently cherry\-pick inside tie masses — the*threshold\-realizable*top\-1% error was 0\.071 vs the argsort illusion of 0\.046\.

## 5Procedures: a validity ladder

### 5\.1Tier 1–2 fix: the fit/val split protocol

Split the calibration half \(by document\) into a*fit*half and a*val*half\. Fit the score model and every data\-dependent transform \(vocabularies, standardization, early stopping\) on the fit half only; compute the add\-one threshold*and*all Mondrian bin edges on the untouched val half; never touch test\. This restores score–threshold independence — it does*not*restore exchangeability, so document clustering remains and tiers 1–2 stay marginal, on\-average guarantees\. On the 6\.9k CORD dump the same learned score moves from an invalid 0\.416/0\.127 \(95% violating\) to0\.266 at 0\.092\(held\)\. Residual disclosure: support\-bin edges and fieldtype vocabularies are computed on the same val half as the threshold; both are label\-independent and second\-order\.

### 5\.2Tier 3–4 fix: Mondrian Learn\-then\-Test with exact binomial tails

For each taxonomy groupggand candidate thresholdtt, we testH0:selective risk of​t​in​g\>αH\_\{0\}:\\ \\text\{selective risk of \}t\\text\{ in \}g\>\\alphawith an exact binomial tailpp\-value\(Clopper and Pearson,[1934](https://arxiv.org/html/2608.14639#bib.bib21)\)on the calibration errors among accepted fields, then select thresholds by family\-wise\-error\-controlled multiple testing\(Angelopouloset al\.,[2021](https://arxiv.org/html/2608.14639#bib.bib18)\): half the budgetδ\\deltato Holm step\-down\(Holm,[1979](https://arxiv.org/html/2608.14639#bib.bib20)\), half to a fixed\-sequence pass from the most conservative candidate \(the “mix” rule; a union bound keeps it valid atδ\\delta, and it is the only variant that never collapses across our three corpora\)\. Candidates are 15 geometric acceptance\-fraction quantiles \(1%–100%\) of the group’s calibration scores, snapped to the nearest distinct\-value boundary — label\-independent, so multiplicity is paid only over 15 points; a naive fine grid destroys certification \(pooled certified coverage 0\.0009 vs 0\.0055 on the 6\.9k dump\)\. Budgets: per\-group \(δ\\deltaper group, three separate statements\) or simultaneous \(δ/G\\delta/G\)\.*Guarantee \(tier 3, per group\):*with probability≥1−δ\\geq 1\-\\deltaover the calibration draw, the true selective risk among accepted fields in that group is≤α\\leq\\alpha,ifwithin\-group accepted\-field errors are iid\. The iid premise is load\-bearing, not decorative: the measured design effect≈\\approx2 means the binomialnnoverstates evidence about twofold; we therefore co\-report a cluster\-corrected variant \(ltt\.neff: binomialnndeflated by the plug\-in design effect\) at every headline\.*Guarantee \(tier 4\):*replace the field unit by the document — with probability≥1−δ\\geq 1\-\\delta, the*mean per\-document error rate among accepting documents*is≤α\\leq\\alpha\(finite\-sample bound, documents iid\)\. Tier 4 is the only tier whose assumptions match the data\-generating process; note it bounds a*macro*per\-document functional, not field\-level \(micro\) selective risk\.

*Disclosure \(score refit\)\.*As implemented, tiers 2–4 refit the 5\-signal LR on the calibration half on which the LTTpp\-values are computed — formally the same premise violation as Failure 2\. The clean\-fit control bounds the effect at≈\\approx0 for this 5\-parameter score, and the split\-protocol twin of tier 2 \(0\.212 vs 0\.218, §[6\.1](https://arxiv.org/html/2608.14639#S6.SS1)\) confirms it empirically; running tiers 3–4 on the frozen fit\-half score is protocol hygiene we adopt for the camera\-ready harness\.

### 5\.3Taxonomy multiplicity discipline

Choosing the conditioning taxonomy amongKKcandidates to maximize coverage is a selection problem; “best held cell” tables silently reintroduce it\. Our discipline: \(i\)support\-bin is pre\-specifiedas the provenance thesis of this project, declared before the scale campaign ran; every headline table prints the pre\-specified configuration, never a per\-cell winner; \(ii\) the support\-bin\-vs\-pooled lift survives a Bonferroni correction over the four candidate taxonomies \(three comparisons at the Monte\-Carlo floor: correctedp<1\.5×10−4p<1\.5\\times 10^\{\-4\}\); \(iii\) an FWER\-valid selection variant \(LTT with aδ/5\\delta/5selection budget\) picks the support family in 37/40 and 39/40 splits atα≥0\.15\\alpha\\geq 0\.15but is underpowered atα=0\.10\\alpha\{=\}0\.10, where it picks pooled in 29/40 splits — we state this rather than oversell; \(iv\) the scale dump is a superset of the 400\-document dump on which support\-bin was originally selected, so scale re\-measurement is confirmation on overlapping data,*not*independent replication — the independent check is the frozen\-config run of §[6\.4](https://arxiv.org/html/2608.14639#S6.SS4)\.

## 6Main results

### 6\.1The validity ladder on CORD

Table[4](https://arxiv.org/html/2608.14639#S6.T4)is the paper’s central result: four pre\-specified operating points on the same 13,859 genuineclaude\-sonnet\-5fields, one per guarantee class\. Read it with its annotations: the four rows bound*different functionals*under*different assumptions*, and the score differs across tiers \(learned fusion at tier 1; shared 5\-signal LR at tiers 2–4\), so the column is a menu of guarantee classes, not a price curve for one score\.

Table 4:The validity ladder\(CORD, 13,859 genuineclaude\-sonnet\-5fields, 800 docs, 49\.0% correct; nominalα=0\.10\\alpha\{=\}0\.10,δ=0\.10\\delta\{=\}0\.10for PAC tiers; 40 document\-level splits, seed 7; no tolerance band\)\. Coverage is mean±\\pmsd; viol = fraction of splits with realized risk\>α\>\\alpha\. Every grouped row beats its pooled counterpart atp<10−4p<10^\{\-4\}\(sign\-flip; in\-corpus stability, §[3\.5](https://arxiv.org/html/2608.14639#S3.SS5)\)\. Vocabulary: tiers 1–2*control expected selective risk*; only tiers 3–4*certify*\.∗Tier 4 certifies nothing in 47\.5% of splits; 0\.020 is the zero\-filled mean — conditional on certifying anything, achieved risk is 0\.038\. Report both\.

Tier 1 \(practical\)\.The pre\-specified recipe \(learned fusion with all five signals, split\-protocol add\-one, pooled\) attains coverage0\.318at achieved risk 0\.096 \(SE≈\\approx0\.003 over the 40 resplits\); the production variant that drops the NLI stage attains 0\.326 at 0\.097\. This*controls expected selective risk*: it is an on\-average operating point, not a per\-deployment certificate — realized risk exceeded 0\.10 in 47\.5% \(45% for the variant\) of resplits, which is what a mean\-controlled bound sitting near its boundary looks like\. No per\-deployment or cross\-corpus statement is licensed at this tier; a reader who needsP​\(violation\)≤δP\(\\text\{violation\}\)\\leq\\deltabuys tier 3 at0\.318→0\.1710\.318\\\!\\to\\\!0\.171\. Against the 5\-signal LR control \(pooled add\-one, 0\.134\) the learned score is worth\+0\.184\+0\.184coverage \(p<10−4p<10^\{\-4\}\); the split\-protocol LR twin attains 0\.128\.

Tier 2 \(shared low\-capacity fusion\)\.The 5\-signal LR with add\-one×\\timessupport\-bin reaches0\.2120\.212at0\.0950\.095under the split protocol\. \(The same cell with the LR refit on the threshold half is 0\.218 at 0\.096 — statistically indistinguishable, consistent with the clean\-fit control; we headline the split\-protocol number\.\)

Tier 3 \(rigorous field\-iid PAC,δ=0\.10\\delta\{=\}0\.10\)\.Mondrian LTT with exact binomial tails certifies0\.171coverage at achieved risk 0\.068, violating in 1/40 splits and never returning an empty acceptance set\. Label it precisely: PAC under a field\-iid idealization that is*violated here*\(design effect≈\\approx2\) and empirically absorbed by the procedure’s conservatism \(achieved risk≈0\.7​α\\approx\\\!0\.7\\alpha, violations≤δ\\leq\\delta\); the cluster\-corrected rung, which deflates the binomial evidence by the measured design effect, certifies 0\.140 at 0\.051 with zero violations — the defensible field\-level number, at a cost of 0\.031 coverage\. Budget semantics: per\-group rows make three separateδ=0\.10\\delta\{=\}0\.10statements; the simultaneous version certifies 0\.160, still far above pooled at equal budget \(0\.091\), so the support\-bin lift is not a budget artifact\.

Tier 4 \(rigorous doc\-iid PAC\)\.The only assumption\-honest certificate today, and we lead with its weakness rather than bury it: 0\.060 mean coverage, certifying*nothing*in 19/40 splits \(coverage≈\\approx0\.11 when it fires\), risk 0\.020 zero\-filled / 0\.038 conditional, and it bounds the macro per\-document functional\. This is the honest price of document\-level exchangeability at 800 documents; powered doc\-level procedures \(clustered/variance\-adaptive bounds\) are the paper’s named open problem\. The 0\.060 already reflects the disclosed calibration\-refit of §[5\.2](https://arxiv.org/html/2608.14639#S5.SS2); re\-deriving it under a fully frozen fit/val split \(score fit on half the calibration documents, thresholds on the untouched other half\) is markedly worse,0\.006atα=0\.10\\alpha\{=\}0\.10— the threshold\-data halving costs an order of magnitude more coverage here than the refit leakage it removes \(leakage alone is≤\\leq0\.006 coverage at every tier, confirming the “≈\\approx0” claim of §[5\.2](https://arxiv.org/html/2608.14639#S5.SS2)precisely\)\. We report 0\.060 as the tier’s headline \(consistent with tiers 1–3, which report the same disclosed\-refit protocol\) and 0\.006 as the fully\-rigorous floor\.

Atα=0\.05\\alpha\{=\}0\.05the ladder compresses but survives: practical \(production variant\)0\.124±0\.0830\.124\\pm 0\.083at 0\.047 \(viol 0\.55\); add\-one×\\timessupport\-bin0\.128±0\.0240\.128\\pm 0\.024at 0\.046 \(viol 0\.35; this cell is under the shared\-calibration refit — its split\-protocol twin is 0\.119 at 0\.044\); LTT mix per\-group×\\timessupport\-bin certifies0\.047±0\.0510\.047\\pm 0\.051at 0\.020 with zero violations\.

### 6\.2Provenance wins every rigor tier on the sonnet CORD capture

Table[5](https://arxiv.org/html/2608.14639#S6.T5)crosses tiers with taxonomies on the corrected dump\. With the shared fusion, support\-bin — the pre\-specified provenance taxonomy — is the best taxonomy at*every*tier: add\-one 0\.218 vs 0\.144/0\.134/0\.127 for fieldtype\-freq/pooled/fieldtype\-rule; LTT 0\.171 vs 0\.097 \(fieldtype\-rule\) and 0\.091–0\.098 \(pooled\); doc\-LTT 0\.060 vs≤\\leq0\.006 for everything else \(all liftsp<10−4p<10^\{\-4\}, Bonferroni\-corrected per §[5\.3](https://arxiv.org/html/2608.14639#S5.SS3)\)\. This is the first evidence that provenance*certifiably*pays — at the rigorous tiers — for selective risk control in document extraction; scoped, per §[7](https://arxiv.org/html/2608.14639#S7), to the hard regime: on FUNSD and XFUND the ordering reverses\. A forensic footnote that strengthens the tie\-mass diagnosis: on the defective no\-NLI dump, fieldtype taxonomies won instead; restoring the continuous entailment signal \(which un\-ties the fused score\) restores support\-bin at every tier\.The win is also model\-scoped, not just corpus\-scoped:re\-running the full ladder on the same CORD documents underclaude\-haiku\-4\-5and Qwen2\.5\-14B \(weaker verbalized\-confidence signal, AUROC 0\.610/0\.684 vs sonnet’s 0\.845\) collapses every PAC\-tier cell to≤\\leq0\.005 coverage regardless of taxonomy, and the practical\-tier money\-table winners there are fieldtype\-rule/\-freq, not support\-bin \(sb\-vs\-pooled liftsp=0\.12p\{=\}0\.12–1\.01\.0\); PAC validity itself still transfers with zero violations\. Support\-bin’s advantage is established on the frontier\-LLM \(sonnet\) capture; we do not claim it generalizes across extractors of differing signal quality\.

Table 5:Taxonomy×\\timestier on CORD atα=0\.10\\alpha\{=\}0\.10\(shared 5\-signal fusion; coverage \(risk\) \[viol\]\)\. Support\-bin, pre\-specified, wins every tier\. Pooled LTT with the full Holm budget reaches 0\.098 \(0\.060\) \[0\.00\]\.
### 6\.3Generalization: FUNSD and XFUND\-de

Table[6](https://arxiv.org/html/2608.14639#S6.T6)runs the same pre\-specified machinery on the easier corpora\. On FUNSD \(1,999 fields, 175 docs, 67\.6% correct\) the practical tier reaches0\.491at 0\.093, and rigorous certification is attainable — but*pooled*LTT \(0\.280\) beats every taxonomy, and the per\-group fieldtype\-rule certificate \(0\.128 at 0\.034, zero violations\) shows the machinery holds without a taxonomy lift \(lift not significant at this tier\)\. On XFUND\-de \(523 fields, 42 documents\) the learned fusion has no advantage and the split protocol is expensive; the small\-data recipe — the 4\-signal LR under the same split protocol, pooled — is the practitioner deliverable below∼50\{\\sim\}50documents \(0\.427 at 0\.087; rigorous LTT certifies 0\.080 with 82\.5% zero\-coverage splits, not significant\)\. Nothing here contradicts the ladder; it locates its value: rigor is cheap where data is easy or plentiful, and provenance conditioning is the hard\-regime tool\.

Table 6:Generalization at nominalα\\alpha\(coverage±\\pmsd \(risk\) \[viol\]\)\. FUNSD taxonomy lifts at the LTT tier are not significant; XFUND\-de is small\-nn\(42 docs\)\.
### 6\.4Pre\-registered frozen\-configuration confirmation

Every selection step above touched one corpus family\. To break that loop, the production configuration \(learned no\-NLI fusion, pooled split\-protocol add\-one; frozen*before*the run\) was executedonce, with no tuning, on selection\-untouched genuine captures\. Onclaude\-haiku\-4\-5CORD \(5,341 fields, 400 docs, 56\.7% correct\) it attained coverage0\.167at achieved risk 0\.093 \(viol 0\.38\) atα=0\.10\\alpha\{=\}0\.10and0\.068at 0\.037 atα=0\.05\\alpha\{=\}0\.05— both held at nominal — against a near\-zero 4\-signal LR baseline \(0\.011\)\. The transfer is non\-trivial: haiku’s verbalized self\-report is far weaker than sonnet’s \(raw AUROC 0\.610 vs 0\.845; 0\.621 vs 0\.691 on matched fields — about 70% of the raw gap is assertion\-policy composition; haiku emits only 13 distinct confidence values\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\), so the held result is evidence for the*protocol*, not for one model’s confidence quality\. The same one\-shot protocol on an*open\-weights*capture — Qwen2\.5\-14B served locally via vLLM, 6,168 CORD fields, 398 docs, 53\.4% correct — again held at both budgets: coverage 0\.149 at achieved risk 0\.099 \(viol 0\.47\) atα=0\.10\\alpha\{=\}0\.10\(no\-NLI variant 0\.138 at 0\.088\) and 0\.048 at 0\.036 atα=0\.05\\alpha\{=\}0\.05, a∼\\sim6×\\timesgain over the 4\-signal LR baseline \(0\.022\)\. Across both untouched captures the risk contract never failed; what varies is*coverage*, which tracks the model’s signal quality \(verbalized AUROC 0\.845 / 0\.684 / 0\.610 for sonnet / qwen / haiku→\\rightarrowpractical\-tier coverage 0\.326 / 0\.149 / 0\.167\) — a dependence that broadly tracks signal quality \(sonnet≫\\ggqwen≈\\approxhaiku\), consistent with the two\-regime characterization \(§[7](https://arxiv.org/html/2608.14639#S7)\), and the quantity the companion benchmark measures per model\. Finally, the practical tier’s guarantee was audited against*human*gold\. Three independent annotators \(blind, Fleiss’κ=0\.83\\kappa=0\.83\) re\-judged 149 fields sampled from the production configuration’s accepted set atα=0\.10\\alpha\{=\}0\.10: thehuman\-verified selective risk is2/149=1\.3%2/149=1\.3\\%,95%95\\%CI\[0\.002,0\.048\]\[0\.002,0\.048\]— an order of magnitude under budget\. The audit also explains the margin: against human gold the automatic correctness labels used for calibration err one\-sidedly*pessimistic*\(21%21\\%of auto\-flagged CORD errors are actually correct;0%0\\%false\-optimism\), so thresholds fit against them are conservative\. The guarantee the ladder delivers is not only valid on untouched data; it is robust to the label noise it was computed under\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\.

## 7When does conditioning pay? A two\-regime characterization

The campaign’s ablations sharpen the old “conditioning helps when provenance is discriminative” intuition into a two\-regime empirical law\. We state it with its mechanism and its scope, and formalize the law’s idealized boundary cases in §[7\.1](https://arxiv.org/html/2608.14639#S7.SS1)\(Assumption[1](https://arxiv.org/html/2608.14639#Thmassumption1), Propositions[1](https://arxiv.org/html/2608.14639#Thmproposition1)–[3](https://arxiv.org/html/2608.14639#Thmproposition3)\); the general law away from that boundary remains a pattern over three corpora and two score families, not a proved rate\.

#### Regime 1: with a learned score, covariates belong in the score\.

On the corrected CORD dump, every Mondrian taxonomy stacked on the learned fusion either violates nominal risk or loses coverage \(pooled 0\.318 held; support\-bin 0\.307 at 0\.103, fieldtype\-freq 0\.272 at 0\.101, fieldtype\-rule 0\.295 at 0\.105 — all violated\)\. The controlled ablation \(on the 4\-signal scale run\) makes it causal: field\-type features are worth\+0\.069\+0\.069coverage inside the score \(p<10−4p<10^\{\-4\}\); removing them and handing field\-type to the taxonomy instead recovers essentially none of it \(0\.250 vs 0\.326,−0\.076\-0\.076,p<10−4p<10^\{\-4\}\); and fieldtype\-Mondrian on top of the full score is actively harmful \(−0\.062\-0\.062freq /−0\.036\-0\.036rule, bothp<10−4p<10^\{\-4\}; corrected dump:−0\.046\-0\.046,p<10−4p<10^\{\-4\}/−0\.023\-0\.023,p=0\.021p\{=\}0\.021\)\. Mechanism: a tree fusion splits on the covariate internally and equalizes per\-group score scales, so external conditioning only fragments the threshold sample — the per\-group finite\-sample penalty \(which scales likelog⁡\(1/δ\)/ng\\sqrt\{\\log\(1/\\delta\)/n\_\{g\}\}for the PAC tiers\) buys nothing\. Permutation importance confirms the score already consumes the signals \(verbalized 0\.130, support 0\.068, consistency×\\timesverbalized 0\.066\)\.

#### Regime 2: with a weak or frozen score, covariates belong in the taxonomy — where pooled cannot certify\.

For the shared LR the same conditioning is the single biggest lever: pooled 0\.134→\\tosupport\-bin 0\.218 \(\+0\.084\+0\.084,p<10−4p<10^\{\-4\}; split\-protocol twins 0\.128→\\to0\.212\), and on the 4\-signal run pooled 0\.042→\\tofieldtype 0\.133–0\.135 \(a∼3×\{\\sim\}3\\timeslift,p<10−4p<10^\{\-4\}\)\. At the rigorous tier the scoping is sharp:*conditioning rescues certification where the pooled score head cannot certify at the targetα\\alpha*\(CORD, base correctness 0\.490: support\-bin 0\.171 vs pooled 0\.091–0\.098\),*and fragmentation costs coverage where pooled certifies fine*\(FUNSD, base correctness 0\.676: pooled LTT 0\.280 vs 0\.149 for support\-bin per\-group; XFUND likewise favors pooled\)\. The counterexample is printed, not hidden — it is the boundary of the claim\.

#### Where the NLI signal lives\.

Entailment is droppable as a*feature*of the learned score \(no\-NLI 0\.326≥\\geqfull 0\.318 on CORD, a\+0\.008\+0\.008difference within split noise,p=0\.40p\{=\}0\.40; FUNSD differences not significant; on XFUND\-de dropping it helps,\+0\.074\+0\.074atα=0\.05\\alpha\{=\}0\.05,p=0\.0045p\{=\}0\.0045\) but is load\-bearing for the*shared fusion*: it is the only fine\-grained continuous signal, so it un\-ties the score distribution that the threshold grid needs \(backfilled: 34\.4% nonzero entailment, 2,016 distinct fused values; the LR’s add\-one coverage on the corrected dump is 0\.134 vs 0\.037 with the dead column\)\. Practical rule: keep the NLI stage for the shared\-fusion and rigorous tiers, optional for the learned tier\.

### 7\.1Formal statement of the two\-regime law

#### Setup and notation\.

Calibration fieldsi=1,…,ni=1,\\dots,nare exchangeable draws from a joint distribution over\(E,S,G\)\(E,S,G\): an error indicatorE∈\{0,1\}E\\in\\\{0,1\\\}, an accept scoreS∈\[0,1\]S\\in\[0,1\]\(frozen for this section\), and a group labelG∈\{1,…,K\}G\\in\\\{1,\\dots,K\\\}withπg=P​\(G=g\)\\pi\_\{g\}=P\(G=g\)andngn\_\{g\}the calibration count of groupgg\. For a thresholdt∈\[0,1\]t\\in\[0,1\]letA​\(t\)=\{S≥t\}A\(t\)=\\\{S\\geq t\\\}be the acceptance event,c​\(t\)=P​\(S≥t\)c\(t\)=P\(S\\geq t\)the population coverage, andR​\(t\)=P​\(E=1∣S≥t\)R\(t\)=P\(E=1\\mid S\\geq t\)the selective risk; writecg​\(t\)=P​\(S≥t∣G=g\)c\_\{g\}\(t\)=P\(S\\geq t\\mid G=g\)andRg\(t\)=P\(E=1∣S≥t,G=g\)R\_\{g\}\(t\)=P\(E=1\\mid S\\geq t,G=g\)for the group\-conditional analogues\. The pooled add\-one rule chooses the smallest threshold whose add\-one\-smoothed empirical selective risk is≤α\\leq\\alpha; Mondrian applies the same rule within each group\(Geifman and El\-Yaniv,[2019](https://arxiv.org/html/2608.14639#bib.bib15); Vovket al\.,[2003](https://arxiv.org/html/2608.14639#bib.bib24)\)\.

### 7\.2Regime 1: when conditioning pays

###### Assumption 1\(Weak within\-group score\)\.

There existsρ∈\[0,1\)\\rho\\in\[0,1\)such that for every groupggand everytt,Cov​\(E,𝟏​\{S≥t\}∣G=g\)≤ρ⋅Var​\(E∣g\)​Var​\(𝟏​\{S≥t\}∣g\)\\mathrm\{Cov\}\(E,\\mathbf\{1\}\\\{S\\geq t\\\}\\mid G=g\)\\leq\\rho\\cdot\\sqrt\{\\mathrm\{Var\}\(E\\mid g\)\\,\\mathrm\{Var\}\(\\mathbf\{1\}\\\{S\\geq t\\\}\\mid g\)\}\. The boundary caseρ=0\\rho=0\(S⟂E∣GS\\perp E\\mid G\) is exact; the results degrade linearly inρ\\rho\.

###### Proposition 1\(Conditioning dominates under heterogeneity and a weak score\)\.

Lett⋆t^\{\\star\}be the pooled threshold withR​\(t⋆\)=αR\(t^\{\\star\}\)=\\alpha, and for each group lettg⋆t\_\{g\}^\{\\star\}solveRg​\(tg⋆\)=αR\_\{g\}\(t\_\{g\}^\{\\star\}\)=\\alpha\(taketg⋆=1t\_\{g\}^\{\\star\}=1if no solution exists, i\.e\. groupggcannot be certified\)\. Then, under Assumption 1 withρ=0\\rho=0,

∑gπg​cg​\(tg⋆\)≥c​\(t⋆\),\\sum\_\{g\}\\pi\_\{g\}\\,c\_\{g\}\(t\_\{g\}^\{\\star\}\)\\;\\geq\\;c\(t^\{\\star\}\),with strict inequality whenever the group error ratesRg​\(t⋆\)R\_\{g\}\(t^\{\\star\}\)at the pooled threshold are not all equal toα\\alphaand at least one group withRg​\(t⋆\)<αR\_\{g\}\(t^\{\\star\}\)<\\alphahasπg\>0\\pi\_\{g\}\>0\. Equality holds iff all groups share the same selective\-risk curve\.

###### Proof sketch\.

Underρ=0\\rho=0, within each group the threshold does not change the conditional risk:Rg​\(t\)=egR\_\{g\}\(t\)=e\_\{g\}for alltt, whereeg=P​\(E=1∣G=g\)e\_\{g\}=P\(E=1\\mid G=g\)\. The pooled risk is the coverage\-weighted averageR​\(t\)=∑gwg​\(t\)​egR\(t\)=\\sum\_\{g\}w\_\{g\}\(t\)\\,e\_\{g\}withwg​\(t\)=πg​cg​\(t\)/c​\(t\)w\_\{g\}\(t\)=\\pi\_\{g\}c\_\{g\}\(t\)/c\(t\); att⋆t^\{\\star\}this equalsα\\alpha, so groups witheg<αe\_\{g\}<\\alphasubsidize groups witheg\>αe\_\{g\}\>\\alpha\. Groupggwitheg<αe\_\{g\}<\\alphacan drop its threshold totg⋆=inf\{t:cg​\(t\)\>0\}t\_\{g\}^\{\\star\}=\\inf\\\{t:c\_\{g\}\(t\)\>0\\\}and certify its entire group at riskeg≤αe\_\{g\}\\leq\\alpha; groups witheg\>αe\_\{g\}\>\\alphakeeptg⋆=1t\_\{g\}^\{\\star\}=1\. The pooled rule, by contrast, must raise the threshold until the average risk falls toα\\alpha, forfeiting acceptance mass in the low\-error groups\. Sincecg​\(tg⋆\)≥cg​\(t⋆\)c\_\{g\}\(t\_\{g\}^\{\\star\}\)\\geq c\_\{g\}\(t^\{\\star\}\)for everyggwith at least one strict foreg<αe\_\{g\}<\\alpha, the coverage comparison follows; strictness follows fromcg​\(tg⋆\)=cg​\(inf\{⋅\}\)\>cg​\(t⋆\)c\_\{g\}\(t\_\{g\}^\{\\star\}\)=c\_\{g\}\(\\inf\\\{\\cdot\\\}\)\>c\_\{g\}\(t^\{\\star\}\)wheneverRg​\(t⋆\)=eg<α=R​\(t⋆\)R\_\{g\}\(t^\{\\star\}\)=e\_\{g\}<\\alpha=R\(t^\{\\star\}\)and the group’s score distribution has mass belowt⋆t^\{\\star\}\. ∎

###### Proposition 2\(Finite\-sample price of conditioning\)\.

With probability≥1−δ\\geq 1\-\\deltaover the calibration draw, the add\-one bound in groupggcontrols the group’s true selective risk atα\\alphaup to an additive slack

εg≤1α​ng\+O​\(log⁡\(K/δ\)ng\),\\varepsilon\_\{g\}\\;\\leq\\;\\frac\{1\}\{\\alpha\\,n\_\{g\}\}\+O\\\!\\left\(\\sqrt\{\\tfrac\{\\log\(K/\\delta\)\}\{n\_\{g\}\}\}\\right\),so Mondrian strictly dominates pooled at the same nominal risk whenever the heterogeneity gain of Proposition 1 exceeds the total penalty∑gπg​εg\\sum\_\{g\}\\pi\_\{g\}\\,\\varepsilon\_\{g\}\. The sign of the gain is determined by calibration\-measurable quantities only: group sizesngn\_\{g\}, the group error gaps\|eg−α\|\|\\,e\_\{g\}\-\\alpha\\,\|, and the within\-group score\-error associationρ\\rho\.

###### Proof sketch\.

The add\-one bound for a group of sizengn\_\{g\}certifies risk≤α\\leq\\alphawhenever the empirical estimate errs by less than the add\-one slack1/\(α​ng\)1/\(\\alpha n\_\{g\}\)plus a Hoeffding fluctuation termlog⁡\(1/δg\)/\(2​ng\)\\sqrt\{\\log\(1/\\delta\_\{g\}\)/\(2n\_\{g\}\)\}; a union bound overg=1,…,Kg=1,\\dots,Kwithδg=δ/K\\delta\_\{g\}=\\delta/Kyields the statedεg\\varepsilon\_\{g\}\. Proposition 1’s gain decomposes as∑gπg​\[cg​\(tg⋆\)−cg​\(t⋆\)\]\\sum\_\{g\}\\pi\_\{g\}\\,\[c\_\{g\}\(t\_\{g\}^\{\\star\}\)\-c\_\{g\}\(t^\{\\star\}\)\], which is calibration\-measurable; comparing the two quantities gives the stated domination condition\. ∎

### 7\.3Regime 2: subsumption by a learned score

###### Proposition 3\(Subsumption\)\.

If the scoreSSsatisfiesP\(E=1∣S=s,G=g\)=r\(s\)P\(E=1\\mid S=s,G=g\)=r\(s\)for some functionrrindependent ofgg— i\.e\.SSis a sufficient statistic forEEgiven the group covariate — thenRg​\(t\)=R​\(t\)R\_\{g\}\(t\)=R\(t\)for allg,tg,t, so pooled and group\-conditional threshold rules coincide in the population, and in finite samples Mondrian is strictly worse: it pays the Proposition 2 penalty with zero gain\.

###### Proof\.

IfP​\(E=1∣S,G\)=r​\(S\)P\(E=1\\mid S,G\)=r\(S\)does not depend onGG, thenRg​\(t\)=E​\[r​\(S\)∣S≥t,G=g\]R\_\{g\}\(t\)=E\[r\(S\)\\mid S\\geq t,G=g\]; but the conditioning onGGadds nothing givenSS, soRg​\(t\)=E​\[r​\(S\)∣S≥t\]=R​\(t\)R\_\{g\}\(t\)=E\[r\(S\)\\mid S\\geq t\]=R\(t\)for everytt\. Hence the population\-optimal thresholds coincide,tg⋆=t⋆t\_\{g\}^\{\\star\}=t^\{\\star\}, the coverage gain in Proposition 1 is zero, and only the finite\-sample penalty of Proposition 2 remains\. ∎

###### Corollary 1\(The two regimes\)\.

With training data, a learned score that encodes the covariate removes the Regime\-1 gain \(Prop\. 3\): put the covariate*in the score*\. With a frozen, weak, or black\-box score that cannot encode it, conditioning*in the taxonomy*recovers exactly the between\-group signal the score lacks \(Prop\. 1–2\), and pays precisely where the pooled threshold cannot certify at the targetα\\alpha\.

### 7\.4Empirical verification

We evaluate the domination condition of Proposition 2 on the three genuine claude\-sonnet\-5 corpora of the benchmark\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\), using the paper’s frozen LR fusion score \(fit on a calibration half\) and the pre\-specified support\-bin taxonomy \(K=3K=3quantile bins of the ambiguity\-penalized support\)\. Group error gaps are the max\|eg−α\|\|e\_\{g\}\-\\alpha\|at the pooled operating point; the penalty uses the finite\-ngn\_\{g\}add\-one/Hoeffding slack; the sign prediction of Proposition 2 \(gain vs penalty\) is compared to the measured outcome over 40 document\-level calibration/test splits atα=0\.10\\alpha=0\.10\.

Table 7:Predicted vs measured sign of the conditioning gain \(Proposition 2\)\. CORD’s large error gaps dominate its group\-size penalty; FUNSD/XFUND’s small gaps do not — the theorem predicts the outcome on all three corpora\.

## 8The price of validity

What does a real certificate cost relative to the invalid folklore procedure? On the 6\.9k CORD dump with support\-bin conditioning \(Table[8](https://arxiv.org/html/2608.14639#S8.T8)\), the per\-group PAC certificate retains 28% of the invalid add\-one coverage atα=0\.10\\alpha\{=\}0\.10, 42% at 0\.15, and 84% at 0\.20 — and 0% at 0\.05, where∼200\{\\sim\}200calibration documents simply cannot support a 90%\-confidence certificate\. Rigor is a knob: the certificate’s cost collapses as the risk budget grows, which is exactly the operating guidance a practitioner needs\.

Table 8:Certified coverage retained vs the invalid add\-one baseline \(6\.9k CORD dump, support\-bin\)\.∗mean achieved risk exceedsα\\alpha\.
## 9Limitations

One corpus certifies non\-trivially\.The full rigorous ladder certifies useful coverage on CORD×\\timessonnet only; FUNSD certifies 0\.128 rigorously but its taxonomy lift is not significant, and XFUND\-de is small\-nn\. The frozen\-config haiku confirmation \(§[6\.4](https://arxiv.org/html/2608.14639#S6.SS4)\) is the first out\-of\-selection replication; full ladders on haiku and an open\-weights extractor are queued\.The doc\-iid tier is honest but near\-vacuous\(0\.060 coverage, 47\.5% zero\-coverage splits\)\. Powered document\-level PAC procedures are the named open problem\.The theorem covers idealized boundary cases, not the general regime\.§[7\.1](https://arxiv.org/html/2608.14639#S7.SS1)proves the conditioning\-dominates and subsumption directions at theρ=0\\rho=0boundary \(Assumption[1](https://arxiv.org/html/2608.14639#Thmassumption1)\) and states that results degrade linearly inρ\\rho, but does not prove a quantitative rate forρ\>0\\rho\>0; the finite\-sample penalty \(Proposition[2](https://arxiv.org/html/2608.14639#Thmproposition2)\) is a worst\-case Hoeffding/union bound, not a tight one\. The sign prediction is verified on all three corpora \(§[7\.4](https://arxiv.org/html/2608.14639#S7.SS4)\), stilln=3n\{=\}3\.Inference is in\-corpus\.All 40 resplits share documents; sign\-flippp\-values are split\-stability statements with a Monte\-Carlo floor of10−410^\{\-4\}, and the scale dump overlaps the dump on which support\-bin was first selected \(§[5\.3](https://arxiv.org/html/2608.14639#S5.SS3)\)\.Score refit on calibrationat tiers 2–4 is disclosed in §[5\.2](https://arxiv.org/html/2608.14639#S5.SS2); the measured effect is≈\\approx0 for the 5\-parameter LR, and the harness adopts fit\-half freezing as hygiene\.Label quality\.FUNSD free\-text correctness is protocol\-dependent \(automatic\-protocol agreementκ=0\.10\\kappa\{=\}0\.10vs 0\.78 for structured CORD fields; Appendix[C](https://arxiv.org/html/2608.14639#A3)\); a certificate against noisy labels is a certificate about those labels\. The companion’s human\-gold audit bounds the practical impact: automatic labels err one\-sidedly pessimistic, and the human\-verified accepted\-set risk is1\.3%1\.3\\%against the10%10\\%budget \(§[6\.4](https://arxiv.org/html/2608.14639#S6.SS4)\); scaling human gold to FUNSD/XFUND is future work\.

## 10Release

VerifyDoc is released Apache\-2\.0: apip\-installable library, CLI, review UI, an MCP server, and the evaluation harness that regenerates every number in this paper from configuration \(fixed seeds, pinned splits, chained bit\-exact sanity gates, and the 463/624\-cells regression check of Appendix[B](https://arxiv.org/html/2608.14639#A2)\)\. The library also ships a trust\-gated agentic layer \(repair/adjudicate/ adaptive\-kk\) that routesreviewfields through escalation tiers; we release it as an artifact and leave its empirical evaluation to future work — the companion ensemble study\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)shows naive multi\-extractor adjudication inherits a dominant model’s hallucinations, so the gate, not the ensemble, carries the trust contract\.

## References

- A\. N\. Angelopoulos, S\. Bates, E\. J\. Candès, M\. I\. Jordan, and L\. Lei \(2021\)Learn then test: calibrating predictive algorithms to achieve risk control\.arXiv:2110\.01052\.Cited by:[§1](https://arxiv.org/html/2608.14639#S1.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2608.14639#S2.p2.1),[§5\.2](https://arxiv.org/html/2608.14639#S5.SS2.p1.15)\.
- A\. N\. Angelopoulos, S\. Bates, A\. Fisch, L\. Lei, and T\. Schuster \(2024\)Conformal risk control\.InICLR,Note:arXiv:2208\.02814Cited by:[§1](https://arxiv.org/html/2608.14639#S1.SS0.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2608.14639#S1.p1.2),[§2](https://arxiv.org/html/2608.14639#S2.p2.1),[§4](https://arxiv.org/html/2608.14639#S4.p1.4)\.
- S\. Bates, A\. N\. Angelopoulos, L\. Lei, J\. Malik, and M\. I\. Jordan \(2021\)Distribution\-free, risk\-controlling prediction sets\.Journal of the ACM68\(6\)\.Note:arXiv:2101\.02703Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1)\.
- J\. Chen and J\. Mueller \(2023\)Quantifying uncertainty in answers from any language model and enhancing their trustworthiness\.arXiv:2308\.16175\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- Cleanlab \(2025\)Trustworthy language model \(tlm\) for data extraction\.Note:help\.cleanlab\.aiCited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- C\. J\. Clopper and E\. S\. Pearson \(1934\)The use of confidence or fiducial limits illustrated in the case of the binomial\.Biometrika26\(4\),pp\. 404–413\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1),[§5\.2](https://arxiv.org/html/2608.14639#S5.SS2.p1.15)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- Fergusonet al\.\(2026\)ExtractBench: a benchmark and evaluation methodology for complex structured extraction\.Note:arXiv:2602\.12247Contextual AICited by:[§1](https://arxiv.org/html/2608.14639#S1.p1.2)\.
- Fuet al\.\(2025\)OCRBench v2: an improved benchmark for evaluating large multimodal models on visual text localization and reasoning\.InNeurIPS Datasets and Benchmarks,Note:arXiv:2501\.00321Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2019\)Bias\-reduced uncertainty estimation for deep neural classifiers\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2608.14639#A1.p1.18),[§1](https://arxiv.org/html/2608.14639#S1.p1.2),[§2](https://arxiv.org/html/2608.14639#S2.p2.1),[§4](https://arxiv.org/html/2608.14639#S4.p1.4),[§7\.1](https://arxiv.org/html/2608.14639#S7.SS1.SSS0.Px1.p1.15)\.
- I\. Gibbs, J\. J\. Cherian, and E\. J\. Candès \(2025\)Conformal prediction with conditional guarantees\.Journal of the Royal Statistical Society B87\.Note:arXiv:2305\.12616Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1)\.
- B\. Gurram \(2026\)VerifyDocBench: benchmarking per\-field calibration, selective risk, and grounding for document extraction\.Note:companion paperCited by:[Table 9](https://arxiv.org/html/2608.14639#A3.T9),[§1](https://arxiv.org/html/2608.14639#S1.p2.1),[§10](https://arxiv.org/html/2608.14639#S10.p1.1),[§2](https://arxiv.org/html/2608.14639#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.14639#S3.SS1.p1.5),[§3\.2](https://arxiv.org/html/2608.14639#S3.SS2.p1.5),[§6\.4](https://arxiv.org/html/2608.14639#S6.SS4.p1.16),[§7\.4](https://arxiv.org/html/2608.14639#S7.SS4.p1.4),[footnote 1](https://arxiv.org/html/2608.14639#footnote1)\.
- S\. Holm \(1979\)A simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1),[§5\.2](https://arxiv.org/html/2608.14639#S5.SS2.p1.15)\.
- Z\. Huanget al\.\(2019\)ICDAR2019 competition on scanned receipt OCR and information extraction \(SROIE\)\.InICDAR,Note:arXiv:2103\.10213Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- C\. Mohri and T\. Hashimoto \(2024\)Language models with conformal factuality guarantees\.InICML,Note:arXiv:2402\.10978Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1)\.
- Preprint authors \(2024\)VISA: retrieval\-augmented generation with visual source attribution\.arXiv:2412\.14457\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1)\.
- Preprint authors \(2025a\)BoundingDocs: a unified dataset for document question answering with spatial annotations\.arXiv:2501\.03403\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- Preprint authors \(2025b\)Selective conformal risk control\.arXiv:2512\.12844\.Cited by:[Appendix A](https://arxiv.org/html/2608.14639#A1.p1.18),[§2](https://arxiv.org/html/2608.14639#S2.p2.1),[§4](https://arxiv.org/html/2608.14639#S4.p1.4)\.
- Preprint authors \(2026a\)Beyond logprobs: a multi\-signal confidence engine for llm\-based document field extraction\.arXiv:2606\.24420\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- Preprint authors \(2026b\)Real\-time trustworthiness scoring for llm structured outputs\.arXiv:2603\.18014\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- Preprint authors \(2026c\)Risk\-controlled generative ocr\.arXiv:2603\.19790\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1)\.
- Preprint authors \(2026d\)When can conformal risk control certify llm outputs? bounds, impossibility, and adaptation for structured generation\.arXiv:2606\.29054\.Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1)\.
- Š\. Šimšaet al\.\(2023\)DocILE benchmark for document information localization and extraction\.InICDAR,Note:arXiv:2302\.05658Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p1.1)\.
- J\. Traub and o\. Kirchhof \(2024\)Overcoming common flaws in the evaluation of selective classification systems\.InNeurIPS,Note:arXiv:2407\.01032 \(AUGRC\)Cited by:[§2](https://arxiv.org/html/2608.14639#S2.p2.1)\.
- V\. Vovk, D\. Lindsay, I\. Nouretdinov, and A\. Gammerman \(2003\)Self\-calibrating probability forecasting \(mondrian conformal prediction\)\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2608.14639#S1.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2608.14639#S2.p2.1),[§3\.4](https://arxiv.org/html/2608.14639#S3.SS4.p1.2),[§7\.1](https://arxiv.org/html/2608.14639#S7.SS1.SSS0.Px1.p1.15)\.

## Appendix AGuarantee statements and procedure details

Tiers 1–2 \(add\-one, marginal\)\.With calibration and test fields exchangeable, the add\-one threshold of §[4](https://arxiv.org/html/2608.14639#S4)controls𝔼​\[selective risk\]≤α\\mathbb\{E\}\[\\text\{selective risk\}\]\\leq\\alphaover calibration/test draws\(Geifman and El\-Yaniv,[2019](https://arxiv.org/html/2608.14639#bib.bib15); Preprint authors,[2025b](https://arxiv.org/html/2608.14639#bib.bib26)\); the field is the exchangeability unit; no per\-draw statement is made\. Document clustering violates exchangeability at the document level; the split protocol removes score–threshold dependence only\.Tier 3 \(Mondrian LTT\)\.For groupgg, candidatett, letntn\_\{t\}be accepted calibration fields andktk\_\{t\}the errors among them\. The exact binomial tailpt=Pr⁡\[Bin​\(nt,α\)≤kt\]p\_\{t\}=\\Pr\[\\mathrm\{Bin\}\(n\_\{t\},\\alpha\)\\leq k\_\{t\}\]testsH0:Rg​\(t\)\>αH\_\{0\}:R\_\{g\}\(t\)\>\\alpha\. Reject over the 15\-candidate grid withδ/2\\delta/2Holm step\-down plusδ/2\\delta/2fixed\-sequence \(most conservative first, stop at first non\-rejection\); by the union bound the pair is FWER\-valid atδ\\delta; the chosen threshold is the certified candidate with the largest calibration acceptance\. Per\-group budgets give each group its ownδ\\delta\(three statements\); the simultaneous form spendsδ/G\\delta/G\. Exact binomial dominates Hoeffding\-style tails for binary losses \(measured: 0\.022 exact\-binomial vs 0\.013 Hoeffding coverage on the 6\.9k dump\)\. Pure fixed\-sequence with many groups can silently fail \(lucky\-zero tiny\-nnbins: the grounded×\\timessupport taxonomy violated on mean, risk 0\.103, 25% of splits\); the mix rule is the variant that never collapsed across corpora\.Cluster correction\.ltt\.neffreplacesntn\_\{t\}bynt/deff^n\_\{t\}/\\widehat\{\\mathrm\{deff\}\}with the plug\-in design effect estimated from per\-document error clustering — approximate \(the deff is estimated\), and uniformly more conservative\.Tier 4 \(doc\-iid\)\.Per accepting documentdd, the loss is its within\-document error rate among accepted fields; a finite\-sample Hoeffding bound over documents tests whether the mean per\-document loss exceedsα\\alpha, with the same Holm/fixed\-sequence machinery\. Documents iid; bounds the macro functional\.

## Appendix BFull disclosure: the entailment\-capture defect and its forensics

The first 13,859\-field CORD capture shippedentailment≡\{\}\\equiv0\.0 for all fields: a mid\-capturetorchswap made the NLI stage skip silently \(logged per field asentailment skipped, with a torch\-compile indexing error\)\. The defect surfaced as two anomalies: rigorous certified coverage*shrank*∼4×\{\\sim\}4\\timeswhen the data doubled, and the doc\-level add\-one returned exactly 0\. Root cause: with the only fine\-grained continuous signal dead, the fused score collapsed to 257 distinct values \(tie masses of 221/183 fields at the acceptance head\), so the smallest realizable candidate accepted 245 fields at empirical risk 0\.114 — uncertifiable at any confidence\. Counterfactual reproduction: zeroing entailment on the intact 6\.9k dump reproduces the collapse \(0\.0302→0\.00110\.0302\\to 0\.0011certified coverage;0\.0371→0\.00000\.0371\\to 0\.0000doc add\-one\)\. Hardening: candidate thresholds are snapped to the nearest distinct\-value boundary \(ties toward the smaller, more conservative count; label\-independent\); a regression gate re\-runs the full pre\-fix grid — 463/624 cells bit\-identical, maximum headline\-alpha drift 0\.0088 \(≤0\.16\\leq 0\.16of one split\-sd\) — and a grid\-only ablation shows the fix moves certified coverage by at most 0\.004: the binding pathology was the degenerate score, not the grid\. After the NLI backfill \(34\.4% nonzero entailment, 2,016 distinct fused values\) both grids were re\-run to produce this paper’s numbers\. Diagnostic lesson: “the tail looks cleaner at scale” was a stable\-argsort illusion under ties; the threshold\-realizable top\-1% error was 0\.071 \(vs 0\.045 intact\)\.

## Appendix CLabeling reliability

Table[9](https://arxiv.org/html/2608.14639#A3.T9)quantifies correctness\-label stability by scoring the same predictions under two automatic protocols; every FUNSD free\-text number in this paper inherits theκ=0\.10\\kappa\{=\}0\.10caveat\.

Table 9:Agreement between two automatic scoring protocols \(strict exact\-match vs schema\-typed\), an honest lower\-bound proxy for human IAA\(Gurram,[2026](https://arxiv.org/html/2608.14639#bib.bib31)\)\. Structured/numeric labels are robust; free\-text correctness is protocol\-dependent\.
## Appendix DSupplementary studies \(simulated and floor\-extractor\)

These studies are kept out of the main text by design: neither may sit next to a genuine\-model number\. Both concern the same mechanism the main text certifies \(per\-group thresholds recover the well\-grounded fraction a pooled threshold forfeits\)\.

#### Simulated controlled study\.

With a*simulated*extractor and an uninformative accept score \(α=0\.05\\alpha\{=\}0\.05, 200 trials/condition\), pooled conformal accepts≈0%\{\\approx\}0\\%while grounding\-conditioned conformal accepts 33–67% with risk held in every condition \(Table[10](https://arxiv.org/html/2608.14639#A4.T10)\)\.

Table 10:*Simulated*controlled study \(α=0\.05\\alpha\{=\}0\.05\): mean coverage lift\+0\.50\+0\.50from provenance\-conditioning at held risk\.
#### Floor\-extractor at\-scale check\.

With the low\-recall label\-search extractor \(not a frontier LLM; conservative operating points\), the add\-one guarantee holds tightly at largeNNand the conditioning lift appears where the two\-regime characterization of §[7](https://arxiv.org/html/2608.14639#S7)expects it: FUNSD0\.24→0\.840\.24\\to 0\.84coverage at an achieved 2% risk; CORD — enabled by the ambiguity penalty of §[3\.3](https://arxiv.org/html/2608.14639#S3.SS3)—0\.01→0\.720\.01\\to 0\.72at a held 10% risk \(Table[11](https://arxiv.org/html/2608.14639#A4.T11)\)\. Without the1/m1/mpenalty the grounded group was uncertifiable\.

Table 11:Floor label\-search extractor, 40\-split means: grounding\-conditioned vs pooled conformal at largeNN\(supporting evidence only; no frontier LLM involved\)\.