Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

# 1. Introduction
Source: [https://arxiv.org/html/2606.24420](https://arxiv.org/html/2606.24420)
Beyond Logprobs: A Multi\-Signal Confidence Engine for LLM\-Based Document Field Extraction111Extended version of a paper accepted \(Oral\) at the RobustifAI Workshop, IJCAI\-ECAI 2026, Bremen, Germany\.

Nitesh Kumar1

1Perfios Software Solutions Pvt\. Ltd\.

nitesh\.kumar@perfios\.com

Abstract

In high\-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent\. The central challenge is not extraction accuracy alone but*reliable confidence estimation*: knowing, field by field, whether a given extraction can be trusted for automation or must be deferred to human review\. Existing approaches fall short: token\-level log\-probabilities, verbalized confidence, and multi\-sample self\-consistency all collapse toward all\-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions\.

We presentExtractConf, a cross\-domain, field\-agnostic confidence engine that grounds reliable confidence estimation in two structurally different readings of the same document\. A field\-guidedHuntercall extracts each field independently under schema\-slot completion prompt; a document\-guidedMappercall scans the document holistically and surfaces candidate values grounded in what the document actually contains, without being anchored to any single field\. This output\-structure asymmetry gives them different failure modes:Huntertends to hallucinate values for ambiguous or absent fields, whileMappertends to miss fields that are visually non\-salient\. Their disagreement is independently informative\.ExtractConffuses signals derived from this cross\-call disagreement, LLM\-internal uncertainty, OCR, image quality, and spatial layout into a binary classifier that operates without any domain\-specific rules or retraining for new document types or field sets\. On DocILE \(55\-field invoices, 26% natural failure rate\),ExtractConfachieves 0\.928 ROC AUC and reduces selective prediction risk by 70% over logprob\-mean\. In the top confidence band, accuracy reaches 99\.1%, a 25\.8\-point improvement over the base rate, enabling a practical human\-in\-the\-loop workflow: automate high\-confidence predictions, route the rest\. Zero\-shot transfer to CORD receipts achieves 0\.858 AUC without any retraining; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the reliability signals generalise across document domains\.

Automated extraction of structured fields from business documents is a high\-value production task where incorrect extractions carry direct financial consequences\. Recent large multimodal LLMs have substantially advanced document information extraction\[[1](https://arxiv.org/html/2606.24420#bib.bib1),[2](https://arxiv.org/html/2606.24420#bib.bib2)\], achieving strong accuracy on structured benchmarks spanning invoices, receipts, and forms\. Yet production deployments require more than accuracy: they require*knowing when to trust an extraction*\. A calibrated confidence score enables workflows to route low\-confidence predictions to human review, reducing costly downstream errors while maximising automation throughput\. This selective prediction problem has been well studied in classification settings\[[4](https://arxiv.org/html/2606.24420#bib.bib4),[5](https://arxiv.org/html/2606.24420#bib.bib5)\], but remains largely unsolved for LLM\-based document extraction\.

The standard approach is to use token\-level log\-probabilities as a proxy for correctness\. We show this fails substantially: on DocILE, a 55\-field invoice benchmark where frontier LLMs fail on 26% of fields, logprob\-mean achieves only 0\.705 ROC AUC and degrades to an all\-positive classifier at any practical threshold\. Verbalized self\-assessed confidence fares similarly \(0\.692 AUC\)\. Even self\-consistency across five sampled calls achieves only 0\.744 AUC, marginally better, but with AURC 0\.138, barely below logprob\-mean’s 0\.145, and at 5×\\timesthe single\-call API cost\.

The failure is structural\. These methods measure confidence in the*generated token sequence*, but extraction errors in document processing are frequently caused by failures the model cannot observe: unreadable source material, ambiguous layouts, OCR noise\. A frontier LLM confidently transcribing OCR noise produces high log\-probabilities for a wrong answer\.

ExtractConftakes a different approach: construct a reliability signal from multiple heterogeneous sources, namely what the model generated, what the document’s image quality reveals, where the extraction was spatially located, what the document OCR says and whether two structurally distinct extraction calls agree\. The design question that motivates the architecture is:*what if we read the same document twice, in two ways that have different failure modes?*

#### Contributions

1. 1\.We introduce a dual\-callHunter–Mapperdesign where two structurally asymmetric LLM calls approach the same document from opposite output\-structure orientations\.\(§[3](https://arxiv.org/html/2606.24420#S3)\)\.
2. 2\.We fuse this cross\-call signal with LLM\-internal uncertainty, document quality, and spatial layout into a gradient\-boosted confidence classifier, achieving 0\.928 ROC AUC and reducing selective prediction risk by 70% over logprob\-mean \(§[5](https://arxiv.org/html/2606.24420#S5)\)\.
3. 3\.We show that all three baselines fail as routing signals; self\-consistency costs 5×\\timesper\-field yet yields negligible AURC improvement; and document quality alone outperforms LLM\-internal uncertainty alone\. \(§[5](https://arxiv.org/html/2606.24420#S5)\)\.
4. 4\.Post\-hoc recalibration \(M7/M8\) reduces ECE by up to 83%, establishing three deployment configurations for discrimination, calibration, and routing quality \(§[5](https://arxiv.org/html/2606.24420#S5)\)\.
5. 5\.Zero\-shot Lasso recalibration on CORD achieves ECE−\-89% and Brier−\-43%, confirming the reliability signals generalise across document domains \(§[5\.1](https://arxiv.org/html/2606.24420#S5.SS1)\)\.

## 2\. Related Work

#### Multi\-signal KIE confidence

HYCEDIS\[[13](https://arxiv.org/html/2606.24420#bib.bib13)\]is the closest predecessor toExtractConf: it fuses CRNN OCR logits, LSTM lingual features, Graph\-KV structural logits, and a VAE image anomaly score through a learned binary classifier to predict whether each KIE extraction is correct, reporting AUC and ECE on SROIE and CORD\.ExtractConfinherits the multi\-signal fusion paradigm but extends it to the LLM era in three directions: LLM logprobs and entropy replace CRNN logits; the asymmetricHunter–Mapperdesign adds a cross\-call signal structurally absent from HYCEDIS; and evaluation uses DocILE, a 55\-field benchmark with 26% natural failure rate vs\. SROIE’s 4%, providing richer negative supervision\.

#### Selective prediction and abstention

ExtractConfis fundamentally a selective prediction system: it decides whether to act on an extraction or defer to human review, and is evaluated with AURC, a selective prediction metric\. Selective prediction with neural networks was formalised by Geifman and El\-Yaniv\[[4](https://arxiv.org/html/2606.24420#bib.bib4)\], who showed that a coverage\-accuracy trade\-off is achievable by abstaining on low\-confidence predictions\. Corbière et al\.\[[5](https://arxiv.org/html/2606.24420#bib.bib5)\]proposed learning a separate confidence score for abstention rather than relying on softmax probability, motivating our CatBoost meta\-classifier design\. The risk\-coverage curve and AURC used throughout this paper follow the selective prediction evaluation framework of Geifman and El\-Yaniv\[[6](https://arxiv.org/html/2606.24420#bib.bib6)\]\. Conformal prediction\[[14](https://arxiv.org/html/2606.24420#bib.bib14)\]provides coverage guarantees post\-hoc but produces set\-valued outputs rather than calibrated scalar scores;ExtractConfis complementary, providing the graded probability estimates that enable continuous routing decisions\.

#### Ensemble disagreement and epistemic uncertainty

TheHunter–Mapperdesign belongs to a broader family of disagreement\-based uncertainty estimators\. Query\-by\-committee\[[7](https://arxiv.org/html/2606.24420#bib.bib7)\]uses disagreement among ensemble members as an active learning signal; co\-training\[[8](https://arxiv.org/html/2606.24420#bib.bib8)\]exploits disagreement between two views of the same instance to propagate labels\. Lakshminarayanan et al\.\[[9](https://arxiv.org/html/2606.24420#bib.bib9)\]show that deep ensembles produce better\-calibrated uncertainty than single models via prediction disagreement\.ExtractConf’sHunter–Mapperpair differs from these in a critical respect: the two calls are not trained differently nor independently initialised; they are the same model receiving*structurally asymmetric prompts*with different failure modes\. The disagreement is therefore epistemic in origin, measuring whether the document provides sufficient grounding for both readings to converge, rather than reflecting parametric variance across ensemble members\.

#### LLM uncertainty quantification

Self\-consistency\[[28](https://arxiv.org/html/2606.24420#bib.bib28)\]samples multiple generations and uses majority agreement as confidence; we adopt this as baseline B3\. Its routing utility is limited by score granularity: with 5 calls\. BSDetector\[[17](https://arxiv.org/html/2606.24420#bib.bib17)\]and CONSTRUCT\[[18](https://arxiv.org/html/2606.24420#bib.bib18)\]extend consistency\-based UQ to structured extraction outputs but are document\-unaware; they use no OCR, spatial, or image quality signals\. Semantic entropy\[[19](https://arxiv.org/html/2606.24420#bib.bib19)\]clusters semantically equivalent generations and measures cluster entropy; P\(True\)\[[20](https://arxiv.org/html/2606.24420#bib.bib20)\]uses LLM self\-evaluation; and verbalized confidence\[[21](https://arxiv.org/html/2606.24420#bib.bib21)\]elicits numeric scores\. These methods are included as model\-only baselines \(B1, B2\)\. Semantic entropy is not directly applicable to structured field extraction, where correctness is determined by exact or near\-exact string match against ground truth, not semantic equivalence between free\-form generations\.

#### Multimodal grounding uncertainty

ExtractConf’s core insight, that extraction uncertainty should arise from document\-side grounding instability rather than just token probability, connects to the emerging literature on multimodal hallucination\. Li et al\.\[[10](https://arxiv.org/html/2606.24420#bib.bib10)\]show that vision\-language models hallucinate objects not present in images with high confidence, exactly the failure modeHunterexhibits on absent fields\. The document AI\-specific variant of this problem, where OCR confidence serves as a proxy for grounding quality, is validated by ConfBERT\[[15](https://arxiv.org/html/2606.24420#bib.bib15)\], which injects OCR confidence into BERT for post\-OCR error detection\.

#### Calibration for structured prediction

Jagannatha and Yu\[[22](https://arxiv.org/html/2606.24420#bib.bib22)\]benchmark temperature scaling, MC\-Dropout, and isotonic regression for NER and relation extraction calibration, establishing isotonic regression as a strong post\-hoc calibrator for structured NLP tasks, motivating our M7 design\. Nixon et al\.\[[23](https://arxiv.org/html/2606.24420#bib.bib23)\]provide the multi\-bin ECE framework; Kull et al\.\[[24](https://arxiv.org/html/2606.24420#bib.bib24)\]provide the Brier score decomposition into calibration and refinement components used to interpret our mapper contribution\.

## 3\. TheExtractConfArchitecture

![Refer to caption](https://arxiv.org/html/2606.24420v1/x1.png)Figure 1:ExtractConfarchitecture\. A document feeds two asymmetric LLM calls in parallel: a field\-guidedHunter\(blue\) and a document\-guidedMapper\(orange\)\. Both report extracted values, token logprobs, and the neighbouring text they attended to\. Value agreement, neighbourhood text overlap, and spatial centroid divergence \(from OCR bounding boxes\) form the cross\-call reliability signals\. OCR confidence and image quality feed the feature table via dashed arrows\. All 40 features are fused by a CatBoost classifier; optional post\-hoc recalibration \(M7/M8\) produces calibrated probabilities\. The scorec∈\[0,1\]c\\in\[0,1\]routes to automation or human review at thresholdτ\\tau\.### 3\.1 Dual\-Call Hunter–Mapper Design

Hunterreceives a*field\-guided*prompt: given the document, extract the value of fieldff\. It returns the extracted value, per\-token log\-probabilities, and the neighbouring text it attended to in the document\.

Mapperreceives a*document\-guided*prompt: given the document, holistically scan and identify the most salient candidate values for each target field, anchored to what the document actually contains rather than to any single field in isolation\. It returns a flat list of field\-value pairs grounded in document content, logprobs, and the neighbouring text each extraction attended to\.

The design rationale is the*asymmetry of output structure*\.Hunterfills a fixed schema slot for every field under completion pressure; it will produce an answer regardless of whether the field is clearly present in the document, making it prone to confabulating plausible values for absent or ambiguous fields\.Mapperproduces a free evidence list of whatever it finds most salient in the document; it reports only what is visually grounded, making it reliable when it does surface a value\. Their disagreement is therefore independently informative: value clashes signal genuine extraction ambiguity, whileHunterreporting whenMapperis silent signals a field that required completion pressure to surface, warranting lower confidence regardless of logprob quality\. Neither failure mode can be observed by resampling either call, and the two\-call cost is fixed per document regardless of schema size\.

### 3\.2 Feature Groups

#### Group 1\. LLM Internal Uncertainty

For bothHunterandMappercalls independently, we extract token\-level statistics from the logprob sequence returned by the API\. For log\-probabilities: mean, minimum, 10th\-percentile \(P10\), and standard deviation\. For Shannon token entropy: mean, maximum, and 90th\-percentile \(P90\)\. These statistics summarise how confident the model was over the entire decoding trajectory, while Shannon entropy specifically captures how dispersed the probability mass was at each step, providing a complementary view of uncertainty beyond raw logprobs\. This yields 14 features \(7 per call\), keeping the two sets separate sinceHunterandMapperhave structurally different output formats and their uncertainty distributions are not directly comparable\.

#### Group 2\. OCR Grounding and Value Confidence

The LLM returns the extracted value and field label as it read them from the document\. We ground these against the document OCR token list to compute region\-level confidence scores\.

Value\-in\-OCR matchis a binary indicator of whether the extracted value appears exactly anywhere in the raw OCR text of the document\. This is computed for both calls, detecting hallucinated values that have no grounding in any OCR token regardless of field assignment\.

Candidate competition featurescapture grounding evidence without requiring a single exact match\. It tries to capture how much alternative OCR evidence also looks compatible\. For theHunterlabel region, we compute the fraction of OCR candidates whose similarity to the extracted label exceeds three thresholds: 50%, 75%, and 90%, giving three features, giving six candidate competition features in total\. These encode how much competing OCR evidence supports the extraction and are computed forHunteronly, as the primary extraction call\.

Label and value OCR confidenceFor each call \(bothHunterandMapper\), we fuzzily match the LLM\-generated label text, extracted value, and reported nearby tokens against the document OCR token list\. The matched OCR tokens are converted to bounding boxes in document coordinates, and we take the minimal axis\-aligned bounding box that encloses all of them; this union box defines the attended region for that field and call\. Within this region, we computelabel and value OCR confidenceby matching the LLM\-generated label text and extracted value against the OCR tokens using fuzzy matching \(Levenshtein ratio or FuzzyWuzzy, depending on field type\), and taking the highest match confidence as the label confidence and value confidence, respectively\. This is computed for bothHunterandMapper, giving four region\-level confidence scores\.

#### Group 3\. Cross\-Call Value Agreement

Exact match: binary indicator of whether theHunterandMapperextracted values are identical after normalisation\.

Similarity match: Levenshtein ratio for numeric, ID, and limited\-text fields; FuzzyWuzzy partial ratio for free\-text fields\. Field type governs the matching function because numeric fields require near\-exact agreement while free\-text fields tolerate surface variation\.

![Refer to caption](https://arxiv.org/html/2606.24420v1/x2.png)Figure 2:Group 4 spatial feature extraction pipeline\. Nearby tokens from each call are matched against OCR tokens to construct per\-call bounding boxes, from which label neighbourhood match scores, bounding box OCR confidence, and Laplacian sharpness are derived\. Centroid distance between the two boxes measures spatial divergence betweenHunterandMappergroundings\.
#### Group 4\. Spatial Neighbourhood and Centroid

Each LLM call returns the neighbouring tokens it attended to when generating the extraction\. We use these self\-reported neighbours to construct a spatially grounded representation of where each call grounded its attention in the document\.

Bounding box construction: the matched OCR tokens from each call’s neighbourhood are used to construct a minimal bounding box enclosing the spatial region each call attended to in document coordinate space\.

Label neighbourhood match: the LLM\-reported nearby tokens for each call are matched against the OCR token list within the bounding box constructed\. The fraction of nearby tokens that can be matched to OCR tokens in that region gives a 0–1 score forHunterandMapperindependently, reflecting how well each call’s attended region is grounded in the document’s actual text\.

Bounding box OCR confidence: the cumulative OCR confidence of all tokens falling within each call’s bounding box region\. This gives a spatially localised readability score for the specific document area each call attended to, distinct from the value\-level OCR confidence in Group 2, which measures match quality against the extracted string itself\. A low score indicates the bounding box region is poorly recognised by the OCR engine, corroborating extraction unreliability at the spatial level\.

Centroid distance: the Euclidean distance between the centroid ofHunter’s bounding box and the centroid ofMapper’s bounding box, normalised by document dimensions\. A large centroid distance indicates the two calls grounded their extraction in physically different regions of the document, a strong reliability signal even when extracted values agree, since spatial divergence suggests the agreement may be coincidental rather than convergent\.

Normalised Laplacian Score: for the image region enclosed by each call’s bounding box, we compute the Laplacian variance of the pixel intensities, normalised across the document\. We use the Laplacian specifically because it is a local, second\-order measure of image sharpness; it responds to edges and fine print within the bounding box without being influenced by the quality of the rest of the document\. Unlike global image quality metrics, this gives a field\-specific readability signal: a low Laplacian score on theHunterbounding box means the region the model was reading was blurry or degraded, directly explaining why the extraction may be unreliable independently of what the logprobs report\.

#### Group 5\. Field Type Encoding

Five one\-hot features encoding inferred field type:numeric,date,limited\_text,free\_text,alphanumeric\. Field type is inferred from the field name and extracted value pattern\. These features enable the classifier to learn type\-conditional calibration without requiring separate models per field type\.

Full feature names and descriptions are given in Appendix[9](https://arxiv.org/html/2606.24420#S9)\.

### 3\.3 Classifier and Post\-Processing

The 40 features are fused by a CatBoost\[[27](https://arxiv.org/html/2606.24420#bib.bib27)\]binary classifier, selected for its native handling of missing values, calibrated probability outputs, and strong performance in low\-data regimes\. Hyperparameters are tuned per model via grid search over depth, learning rate, and L2 regularisation, optimising validation ROC AUC with StratifiedGroupKFold to prevent document\-level leakage\. Best M6 configuration: depth 8, lr 0\.1, l2 7, 1000 iterations\.

M7applies isotonic regression to M6 output scores on a held\-out calibration set\.M8applies Lasso\-regularised logistic regression as a calibration layer\. Neither modifies CatBoost model weights\.

## 4\. Experimental Setup

### 4\.1 Datasets

#### Implementation

AllHunterandMappercalls use GPT\-4o \(temperature=0\.0,logprobs=True,top\_logprobs=5\)\. OCR uses AWS Textract, providing per\-word confidence scores and bounding box coordinates fed directly into the feature pipeline\.

#### DocILE

DocILE\[[2](https://arxiv.org/html/2606.24420#bib.bib2)\]comprises 6,680 real business invoices with 55 KILE field categories spanning tax identifiers \(IBAN, BIC, VAT\), addresses, dates, and line\-item sub\-fields\. We process 1339 fields, yielding 1,072 train and 267 test field\-level samples \(73\.4% positive rate; 26\.6% natural LLM failure rate\)\.

#### CORD \(zero\-shot evaluation only\)

CORD\[[25](https://arxiv.org/html/2606.24420#bib.bib25)\]comprises 800 training and 100 test receipts with 30 hierarchical field categories\. We evaluate the DocILE\-trainedExtractConfon 828 CORD field\-level samples with no retraining, no feature modification, and with CORD specific prompt\. Recalibration \(M7/M8\) on CORD uses 5\-fold out\-of\-fold cross\-validation on the CORD test set only; no DocILE labels used\.

### 4\.2 Baselines

B1\.Logprob mean: mean log\-probability of the extracted value tokens\.B2\.Verbalized confidence: LLM self\-assessed 0–100 integer score with anchored band descriptions\.B3\.Self\-consistency: 5Huntersamples at temperatures sampled uniformly from\[0,1\]\[0,1\]; agreement fraction as confidence score\. All three collapse to near\-all\-positive classifiers at threshold 0\.5 \(B1/B2: recall 1\.000; B3: recall 0\.994\)\.

### 4\.3 Evaluation Metrics

ROC AUC \(threshold\-free discrimination\); PR AUC; ECE \(Expected Calibration Error, 10\-bin uniform\); Pos\-ECE \(ECE on predicted\-positive samples\); Brier score \(proper scoring rule\); AURC \(area under risk\-coverage curve, primary routing metric; lower is better\); Precision, Recall, F1 at threshold 0\.5\.

## 5\. Main Results

Table 1:Full results on DocILE\. B1–B3: model\-free baselines\. M1–M8: trained CatBoost classifier \(M7/M8 add recalibration\)\. All methods at threshold 0\.5\. Best per column inbold\.![Refer to caption](https://arxiv.org/html/2606.24420v1/x3.png)Figure 3:Calibration vs\. selective prediction quality across all methods\(DocILE test set, threshold 0\.5\)\. Lower\-left is better on both axes\. The green shaded region marks the production\-viable zone \(ECE<<0\.08, AURC<<0\.055\)\. Baselines B1–B3 cluster in the upper\-right: well\-separated from the trained models, they provide no reliable routing signal despite B3 costing 5×\\timesmore API calls\. Single\-group models M1–M4 reduce AURC substantially over baselines but remain outside the production\-viable zone\. M5 crosses into viable AURC territory but is miscalibrated\. M7 and M8 are the only configurations achieving simultaneously low ECE and low AURC, lying on the Pareto frontier of this calibration–routing trade\-off\. The dotted line connects the threeExtractConfdeployment configurations \(M6, M7, M8\)\.#### \(1\) Self\-consistency adds AUC, not routing quality

B3 improves AUC by 3\.9 pp over B1 but AURC by only 0\.007\. The cheapest trained model \(M1\) achieves AURC 0\.076, 45% below B3, confirming that a learned continuous function over 40 features outperforms repeated sampling of a coarse discrete signal\.

#### \(2\) Mapper agreement is the single largest gain

M5→\\toM6:\+\+1\.7 pp AUC,−\-34% AURC \(0\.065→\\to0\.043\),\+\+2\.4 pp F1\. The cross\-call agreement signal adds both discriminative and routing quality, the largest single step in the ablation\.

#### \(3\) OCR outperforms logprobs as a single signal group

M4 \(OCR only, 0\.896 AUC\) outperforms M1 \(logprobs\+entropy, 0\.880 AUC\) by 1\.6 pp\. Extraction errors are document\-caused, not model\-caused: OCR confidence directly measures the failure source; logprobs measure a consequence orthogonal to it\.

#### \(4\) Spatial features require OCR context

M2 \(Spatial only, 0\.814 AUC\) is the weakest single\-group model, 8\.2 pp below M4\. Centroid divergence and neighbourhood overlap are informative only when paired with the quality signal that gives them meaning\.

#### \(5\) Post\-hoc recalibration achieves substantially improved ECE

M7 reduces ECE from 0\.199 to 0\.034 \(−\-83%\) and Pos\-ECE to 0\.028 at only−\-0\.2 pp AUC\. M8 achieves the best AURC \(0\.042\) with ECE 0\.048\. As visible in[Figure˜3](https://arxiv.org/html/2606.24420#S5.F3), M7 and M8 are the only configurations in the production\-viable region\.

### 5\.1 Zero\-Shot Transfer to CORD

We apply the DocILE\-trainedExtractConfzero\-shot to 828 CORD receipt samples, a different document type with 30 fields vs\. 55 and an 88\.6% positive rate, without any retraining\. Recalibration uses 5\-fold OOF cross\-validation on CORD only; no DocILE supervision is transferred\.

Table 2:Zero\-shot transfer to CORD \(828 samples\)\. Trained on DocILE only\. Recalibration: 5\-fold OOF on CORD test set\. Best per column inbold\.#### Reliability signals are document\-agnostic

Lasso recalibration \(B3\) achieves ECE 0\.025 on CORD, 89% below B1’s ECE 0\.237, and Brier 0\.078 \(−\-43%\), without any DocILE calibration labels\. The multi\-signal score distribution is structurally compatible across document types because OCR confidence, spatial centroid divergence, and logprob entropy bear a document\-agnostic monotone relationship to extraction correctness\.

#### Lasso is the recommended zero\-shot configuration

B3 achieves the best ECE \(0\.025\), Brier \(0\.078\), and AURC \(0\.029\) at only−\-0\.4 pp AUC vs\. the uncalibrated base\.

#### Discrimination is preserved

B1 achieves 0\.858 AUC and 95\.4% precision zero\-shot\. For deployments prioritising discrimination over calibration, the uncalibrated base model is the optimal configuration\.

## 6\. Analysis

![Refer to caption](https://arxiv.org/html/2606.24420v1/plot_feature_importance.png)\(a\)Top\-15 CatBoost feature importances \(M6\)\. HV\>90 and HV\>75 dominate by a wide margin, confirming OCR value\-region confidence as the primary correctness signal\. M\_NLS \(Mapper lexical score, rank 4\) and CENT\_DIST \(centroid divergence, rank 6\) validate the cross\-call and spatial contributions\.
![Refer to caption](https://arxiv.org/html/2606.24420v1/x4.png)\(b\)Cumulative accuracy vs\. coverage \(M6, DocILE test set\)\. At 80% coverage, routing the bottom 20% of predictions to human review yields 99\.1% automated accuracy, a 25\.8\-point improvement over the 73\.3% base rate\.

Figure 4:\(a\)Feature importances confirm that document quality signals \(HV\>90, HV\>75, H\_OCR\_LBL\) dominate, followed by the Mapper cross\-call signal \(M\_NLS\), spatial centroid divergence \(CENT\_DIST\), and logprob features\. No single group is sufficient; the full fusion is required\.\(b\)The accuracy–coverage curve operationalises the AURC metric: deferring even a modest fraction of low\-confidence predictions yields near\-perfect automated accuracy on the retained set\.### 6\.1 Feature Importance

[Figure˜4](https://arxiv.org/html/2606.24420#S6.F4)a shows the top\-15 CatBoost importances from M6\.HV\>90andHV\>75dominate, with combined importance nearly double the next\-ranked feature\. These threshold indicators for OCR confidence on theHunter’s extracted value region explain why M4 \(OCR alone, 0\.896 AUC\) outperforms M1 \(logprobs alone, 0\.880 AUC\): value\-region readability is the primary correctness signal, not model\-internal uncertainty\.

M\_NLS\(rank 4\) is the highest\-rankedMapperfeature, the normalised Laplacian score of the spatial regionMapperattended to, reflecting local image quality at theMapperextraction site\. Its high rank confirms that document\-side readability in theMapper’s attended region is independently predictive of extraction correctness, separate fromHunter’s OCR confidence features\.

CENT\_DIST\(rank 6\) confirms spatial centroid divergence carries independent signal: where in the documentHunterandMapperlocated the field is informative beyond OCR quality alone\.

Logprob and entropy features appear five times in the top 15, confirming their group contribution \(M1, 0\.880 AUC\) is real but secondary to OCR and cross\-call signals\.

### 6\.2 Confidence–Coverage Trade\-off

[Figure˜4](https://arxiv.org/html/2606.24420#S6.F4)b shows that at 80% coverage,ExtractConfachieves 99\.1% automated accuracy, a 25\.8 pp improvement over the 73\.3% base rate\. This operationalises the AURC metric: a system with AURC 0\.042 delivers near\-perfect accuracy at the coverage level that matters for production automation\.

### 6\.3 Calibration by Field Type

Field type features \(ft\_limited\_text rank 5, ft\_free\_text rank 8\) appear in the top 10, confirming calibration quality varies across field types\. Numeric fields are well\-calibrated; free\-text fields show overconfidence at high predicted probabilities\. Type\-conditional routing thresholds are a direction for future work\.

## 7\. Discussion

#### Why document quality beats logprobs

Extraction errors in document processing are document\-caused, not model\-caused: a frontier LLM reading unreadable source material generates high\-logprob tokens about OCR noise\. OCR confidence directly measures the cause of failure; logprobs measure a consequence orthogonal to it\. This is a structural argument applicable to any task where LLM extraction is grounded in a physical document\.

#### Why self\-consistency fails for routing

Five binary calls produce only six distinct confidence values, making fine\-grained risk stratification impossible\. Increasing to 10 calls halves the granularity gap but costs 10×\\timesper field\.ExtractConfachieves AURC 0\.043 with 2 calls by learning a continuous reliability function over 40 heterogeneous features\.

#### Why spatial signals require OCR context

Centroid divergence and neighbourhood overlap are not independently informative \(M2, 0\.814 AUC\)\. They require OCR quality context: spatial divergence only signals a reliability problem when paired with evidence that the relevant document region was difficult to read\.

#### Why reliability signals transfer across domains

Lasso recalibration achieves ECE−\-89% on CORD without any DocILE calibration labels\. OCR confidence, spatial centroid divergence, and logprob entropy bear a document\-agnostic monotone relationship to extraction correctness\. A Lasso calibration layer fitted on∼\\sim165 in\-domain samples appears sufficient for a new deployment domain\.

#### Training data selection

*Negative sample quality dominates dataset size*for reliability model training: a dataset with 26% natural failure rate provides richer negative supervision than one with 4%, regardless of corpus size\.

#### Limitations

ExtractConfrequires explicit OCR pipeline outputs\. Evaluation covers English\-language business documents\. One LLM backbone was tested; multi\-backbone validation is future work\.

## 8\. Conclusion

We presentedExtractConf, a reliability engine for LLM\-based document field extraction that answers the question:*should this extraction be trusted for automation?*

Five findings define the contribution\.First, OCR value\-region confidence dominates feature importance, establishing document quality as the primary predictor of extraction correctness\.Second, theHunter–Mapperdual\-call design delivers the largest single ablation gain, providing reliability signal unavailable to any single\-call or resampling approach\.Third,ExtractConfachieves 0\.928 AUC and AURC 0\.042, which is 18\.4 pp above self\-consistency at lower API cost, and 70% lower AURC than logprob\-mean\.Fourth, post\-hoc recalibration reduces ECE by up to 83%, with three deployment configurations for different operational priorities\.Fifth, zero\-shot Lasso recalibration on CORD reduces ECE by 89% and Brier by 43% without any DocILE calibration data, confirming genuinely document\-agnostic reliability signals\.

At 80% coverage,ExtractConfachieves 99\.1% automated accuracy vs\. 73\.3% base rate, enabling practical human\-in\-the\-loop pipelines in high\-stakes document processing\.

## References

- \[1\]V\. Perot et al\.LMDX: Language Model\-based Document Information Extraction and Localization\.ACL Findings, 2024; arXiv:2309\.10952\.
- \[2\]S\. Šimsa et al\.DocILE Benchmark for Document Information Localization and Extraction\.ICDAR, 2023; arXiv:2302\.05658\.
- \[3\]\(Authors\)\.Reflect then Learn: Active Prompting for IE Guided by Introspective Confusion\.arXiv:2508\.10036, 2025\.
- \[4\]Y\. Geifman and R\. El\-Yaniv\.Selective Prediction in Deep Neural Networks\.NeurIPS, 2017\.
- \[5\]C\. Corbière et al\.Addressing Failure Prediction by Learning Model Confidence\.NeurIPS, 2019\.
- \[6\]Y\. Geifman and R\. El\-Yaniv\.SelectiveNet: A Deep Neural Network with an Integrated Reject Option\.ICML, 2019\.
- \[7\]H\. S\. Seung, M\. Opper, and H\. Sompolinsky\.Query by Committee\.COLT, 1992\.
- \[8\]A\. Blum and T\. Mitchell\.Combining Labeled and Unlabeled Data with Co\-training\.COLT, 1998\.
- \[9\]B\. Lakshminarayanan, A\. Pritzel, and C\. Blundell\.Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles\.NeurIPS, 2017\.
- \[10\]Y\. Li et al\.Evaluating Object Hallucination in Large Vision\-Language Models\.EMNLP, 2023\.
- \[11\]S\. Min et al\.FActScore: Fine\-grained Atomic Evaluation of Factual Precision in Long Form Text Generation\.EMNLP, 2023\.
- \[12\]K\. Gero et al\.Self\-Verification Improves Few\-Shot Clinical Information Extraction\.ACL Workshop on Clinical NLP, 2023\.
- \[13\]H\. T\. Nguyen et al\.HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System\.KDD Workshop on Document Intelligence, 2021; arXiv:2206\.02628\.
- \[14\]M\. Rombach and M\. Mehdiyev\.Beyond Accuracy: Understanding Model Confidence in Key Information Extraction with Conformal Prediction\.IJDAR, 2026; doi:10\.1007/s10032\-026\-00572\-y\.
- \[15\]A\. Hemmer et al\.Confidence\-Aware Document OCR Error Detection\.DAS/ICDAR, 2024; arXiv:2409\.04117\.
- \[16\]C\. L\. Willis et al\.Automated Information Extraction from Plant Specimen Labels using OCR and Large Language Models\.PMC12835874, 2025\.
- \[17\]J\. H\. Chen and J\. Mueller\.Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness\.ACL, 2024; arXiv:2308\.16175\.
- \[18\]J\. Goh and J\. Mueller\.Real\-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction\.Cleanlab, 2026; arXiv:2603\.18014\.
- \[19\]S\. Farquhar et al\.Detecting Hallucinations in Large Language Models using Semantic Entropy\.Nature, 630, 2024\.
- \[20\]S\. Kadavath et al\.Language Models \(Mostly\) Know What They Know\.arXiv:2207\.05221, 2022\.
- \[21\]K\. Tian et al\.Just Ask for Calibration\.EMNLP, 2023; arXiv:2305\.14975\.
- \[22\]A\. Jagannatha and H\. Yu\.Calibrating Structured Output Predictors for NLP\.ACL, 2020; arXiv:2004\.04361\.
- \[23\]J\. Nixon et al\.Measuring Calibration in Deep Learning\.CVPR Workshops, 2019\.
- \[24\]M\. Kull et al\.Beyond Temperature Scaling\.NeurIPS, 2019\.
- \[25\]S\. Park et al\.CORD: A Consolidated Receipt Dataset for Post\-OCR Parsing\.NeurIPS Workshop, 2019\.
- \[26\]Z\. Huang et al\.ICDAR 2019 Competition on Scanned Receipt OCR and Information Extraction\.ICDAR, 2019\.
- \[27\]L\. Prokhorenkova et al\.CatBoost: Unbiased Boosting with Categorical Features\.NeurIPS, 2018\.
- \[28\]X\. Wang et al\.Self\-Consistency Improves Chain of Thought Reasoning\.ICLR, 2023\.

## 9\. Feature Reference Table

TableLABEL:tab:featuresprovides a complete reference for all features used inExtractConf, grouped by signal category\. Feature names correspond to column headers in the pipeline output and to the feature importance plot in Figure[4](https://arxiv.org/html/2606.24420#S6.F4)a\.

Table 3:Complete feature reference forExtractConf\. H =Huntercall, M =Mappercall\.Feature NameGroupDescriptionGroup 1 — LLM Internal UncertaintyH\_LP\_meanG1Mean log\-probability ofHunteroutput tokensH\_LP\_minG1Minimum log\-probability ofHunteroutput tokensH\_LP\_p10G110th\-percentile log\-probability \(Hunter\)H\_LP\_stdG1Standard deviation ofHunterlog\-probabilitiesH\_Ent\_meanG1Mean Shannon entropy ofHunteroutput tokensH\_Ent\_maxG1Maximum Shannon entropy \(Hunter\)H\_Ent\_p90G190th\-percentile Shannon entropy \(Hunter\)MV\_LP\_meanG1Mean log\-probability ofMapperoutput tokensMV\_LP\_minG1Minimum log\-probability ofMapperoutput tokensMV\_LP\_p10G110th\-percentile log\-probability \(Mapper\)MV\_LP\_stdG1Standard deviation ofMapperlog\-probabilitiesMV\_Ent\_meanG1Mean Shannon entropy ofMapperoutput tokensMV\_Ent\_maxG1Maximum Shannon entropy \(Mapper\)MV\_Ent\_p90G190th\-percentile Shannon entropy \(Mapper\)Group 2 — OCR and Value ConfidenceH\_OCR\_LBLG2OCR confidence on theHunterfield label regionH\_OCR\_VALG2OCR confidence on theHunterextracted value regionM\_OCR\_LBLG2OCR confidence on theMapperfield label regionM\_OCR\_VALG2OCR confidence on theMapperextracted value regionHL\>50G2Fraction ofHunterlabel\-region OCR candidates with similarity≥\\geq50% to extracted labelHL\>75G2Fraction ofHunterlabel\-region OCR candidates with similarity≥\\geq75% to extracted labelHL\>90G2Fraction ofHunterlabel\-region OCR candidates with similarity≥\\geq90% to extracted labelHV\>50G2Fraction ofHuntervalue\-region OCR candidates with similarity≥\\geq50% to extracted valueHV\>75G2Fraction ofHuntervalue\-region OCR candidates with similarity≥\\geq75% to extracted valueHV\>90G2Fraction ofHuntervalue\-region OCR candidates with similarity≥\\geq90% to extracted valueH\_VAL\_EXG2Binary:Huntervalue found anywhere in raw OCR text of the documentM\_VAL\_EXG2Binary:Mappervalue found anywhere in raw OCR text of the documentGroup 3 — Cross\-Call Value AgreementEXG3Binary:HunterandMapperextracted values are identical after normalisationSIMG3Levenshtein ratio \(numeric/ID/limited\-text fields\) or FuzzyWuzzy partial ratio \(free\-text fields\) betweenHunterandMappervaluesGroup 4 — Spatial Neighbourhood and CentroidH\_LBLSG4Fraction ofHunternearby tokens matched to OCR tokens in the defined region of interest \(0–1\)M\_LBLSG4Fraction ofMappernearby tokens matched to OCR tokens in the defined region of interest \(0–1\)H\_OCR\_CONFG4Cumulative OCR confidence of all tokens within theHunterbounding box regionM\_OCR\_CONFG4Cumulative OCR confidence of all tokens within theMapperbounding box regionCENT\_DISTG4Euclidean distance betweenHunterandMapperbounding box centroids, normalised by document dimensionsH\_NLSG4Normalised Laplacian score of the image region enclosed byHunter’s bounding box; measures local sharpness at the extraction siteM\_NLSG4Normalised Laplacian score of the image region enclosed byMapper’s bounding box; measures local sharpness at the extraction siteGroup 5 — Field Type Encodingft\_numericG5One\-hot: field type inferred as numericft\_dateG5One\-hot: field type inferred as dateft\_limited\_textG5One\-hot: field type inferred as limited text \(short alphanumeric strings\)ft\_free\_textG5One\-hot: field type inferred as free text \(longer natural language values\)ft\_alphanumericG5One\-hot: field type inferred as alphanumeric \(codes, identifiers\)