Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding

# Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding
Source: [https://arxiv.org/html/2608.06532](https://arxiv.org/html/2608.06532)
###### Abstract\.

Large vision\-language models \(LVLMs\) are increasingly used to read financial charts, tables, and documents, where a single misread figure can move a decision and the most authoritative\-looking answer is sometimes one the model produced without reading the exhibit\. The operational question is therefore trust, not accuracy: which answers can be acted on, and which escalated to a reviewer\. We evaluate seven confidence estimators, three inference\-only and four trained internal probes, across five open\-weight LVLMs and four conditions from three financial visual question\-answering benchmarks, one bilingual; every probe is trained only on natural images and applied to finance without adaptation, so the results measure out\-of\-distribution transfer\. Three findings hold\. First, the scarce property is calibration, not ranking: the inference baselines rank correct above incorrect answers competitively but are badly overconfident, calibration error far above what a threshold can tolerate, and only the trained probes produce a thresholdable score\. Second, reliability is structured rather than global, along two axes a practitioner can read directly: the best estimator shifts with both model and task, none leading more than eight of twenty \(model, condition\) cells, and a controlled bilingual contrast exposes an apparent language robustness as a composition artifact that dissolves once models are read one at a time\. Third, cast as deferral under an error budget, how much can be safely automated is set first by the model’s competence and only narrowed by its confidence, so deferral clears a real share of the easiest condition and almost none of the hardest, near zero at a strict 5% budget\. Two trained probes carry the calibration a deferral policy needs, and among them only the grounding\-aware one lowers its confidence on answers a model gives without using the figure, separating detected non\-grounding from a fluent guess\.

confidence estimation, calibration, vision\-language models, financial document understanding, selective prediction

††copyright:none††conference:The 7th ACM International Conference on AI in Finance; November 14–17, 2026; Milan, Italy††ccs:Computing methodologies Neural networks## 1\.Introduction

In financial automation the binding question is when to trust an answer, not how often the model is right\.Reading and summarizing charts, tables, and documents is a large and repetitive part of financial analysis, and an LVLM can potentially clear many of the routine figure\-based questions and route the rest to an expert\. What makes that safe is not how often the model is right on average but whether its right answers can be told apart from its wrong ones\. Aggregate accuracy alone cannot do this: a model that is 80% accurate is wrong on twenty of every hundred answers, and if all hundred carry the same confidence, the expert cannot tell which twenty to catch and must re\-check every one, leaving no work actually automated\. And that is the observed behavior, since language models are systematically overconfident, reporting nearly the same confidence whether right or wrong\(Danget al\.,[2026](https://arxiv.org/html/2608.06532#bib.bib7)\)\. The operational question is therefore trust: which answers can be acted on, and which must be escalated to an expert\.

The harder problem in the multimodal setting is an answer the model gives without using the figure\.An LVLM prepends visual tokens to a language backbone, but architectural access to the exhibit does not make the answer depend on it: attention heads attend far more strongly to text than to visual tokens, and answer distributions are often nearly unchanged when the image is replaced by a blank frame\(Wooet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib1); Liet al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib2)\)\. A model can therefore answer a question about a price chart, an earnings table, or a filing from language priors alone, with the figure contributing nothing, a documented weakness of vision\-language models\(Liet al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib2)\)\. This is the worst case for a reviewer, because the answer is not faulty reasoning about the figure but fluent text that never consulted it, delivered with the same conviction as a grounded one, so nothing on the surface marks it as the answer to distrust\. We study open\-weight LVLMs for a related practical reason: financial data is frequently sensitive and processed on private infrastructure, and several of the estimators we examine read model internals that closed APIs do not expose\.

Confidence estimation for financial LVLMs has not been studied on its own\.Confidence estimators carry useful signal in the LVLM setting\(Liet al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib2)\), but they are typically evaluated for discrimination under roughly uniform error cost, whereas deployment imposes a different constraint: draw a principled boundary between what can be automated and what must be escalated while holding error on the automated fraction below a tolerated bound\(Kompaet al\.,[2021](https://arxiv.org/html/2608.06532#bib.bib8)\)\. No prior work, to our knowledge, studies confidence estimation for financial LVLMs as its own problem, across the range of figure types and task formats that financial analysis actually spans and across more than one language, under the transfer conditions a practitioner faces, where a signal learned on ordinary images must hold on a chart it has never seen\.

This study\.We evaluate seven confidence estimators across five open\-weight LVLMs on three financial VQA benchmarks spanning broad financial reasoning \(FinMME\), chart understanding \(FinChart\-Bench\), and bilingual financial\-document analysis \(MME\-Finance\), reported as four conditions\. Every trainable probe is trained only on general\-domain natural images and applied to finance without adaptation, so the results test transfer rather than in\-domain fitting\. We frame the comparison as automate\-or\-escalate under an error budget: a case is automated only when its confidence clears a threshold, and the rest are deferred to a reviewer\. The picture is consistent and cautionary\. Standard metrics are poor guides to which estimator to trust, neither a single estimator nor a single model is a safe default, and reliability resolves along two axes a practitioner can read directly, the task being performed and the language it is performed in, both tracking how capable the base model is to begin with\.

Contributions\.First, a systematic out\-of\-distribution study of confidence estimation for financial LVLMs: seven estimators across five open\-weight models and four financial conditions under one deployment\-oriented, transfer\-only protocol\. Second, evidence that confidence reliability in finance is structured along two observable axes rather than being a property of a model or an estimator alone: it varies with the task operation, from figure lookup to numerical reasoning to open\-ended captioning, and a controlled bilingual contrast on identical tasks separates a harder task from a weaker model, exposing an apparent language robustness as a composition effect\. Third, a deployment account in the language of bounded\-error automation: how much can be safely delegated is set first by base\-model competence, with the confidence layer adding deployable yield over a zero\-cost softmax baseline chiefly where the model is weak, so deferral recovers a real share of the easiest condition and almost none of the hardest\. The two trained internal probes are the only estimators whose scores a threshold can trust on calibration across conditions, and among them the grounding\-aware one adds a property the others lack: on the cases where the model answers without using the figure, it alone lowers its confidence, separating detected non\-grounding from a fluent guess\.

## 2\.Related Work

Calibration and discrimination are distinct, and only calibration supports deferral\.A confidence estimator attaches to each prediction a score meant to express how likely it is correct, and two distinct properties decide whether that score is usable\. A score is*calibrated*when its value matches empirical accuracy, so answers assigned 0\.8 confidence are correct about 80 percent of the time, and*discriminative*when higher scores are more often correct than lower ones, whatever the absolute scale\. A deferral policy needs calibration, because only a calibrated score lets a fixed threshold map to a controlled error rate; discrimination alone merely orders cases\. The two can come apart, and this study asks whether either, learned on ordinary images, survives the move to financial figures the probe has never seen\(Guoet al\.,[2017](https://arxiv.org/html/2608.06532#bib.bib9)\)\.

Confidence estimators differ by where they read the signal\.Prompt\-based methods read a self\-assessment off the model’s own outputs: P\(True\)\(Kadavathet al\.,[2022](https://arxiv.org/html/2608.06532#bib.bib3)\), Self\-Probing\(Xionget al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib4)\), and verbalized\-confidence prompting\(Tianet al\.,[2023](https://arxiv.org/html/2608.06532#bib.bib27)\)elicit a verbal or token\-level judgment, resting on the model’s capacity to introspect on its own computation, which is uneven for multi\-step numerical reasoning, while a prompt ensemble instead aggregates confidence across rephrasings of the query, a technique introduced for vision\-language\-action policies\(Zollo and Zemel,[2025](https://arxiv.org/html/2608.06532#bib.bib10)\)that we adapt to VQA\. Internal\-state probes train a lightweight classifier on hidden activations: SAPLMA\(Azaria and Mitchell,[2023](https://arxiv.org/html/2608.06532#bib.bib11)\)reads the post\-answer state, and InternalInspector\(Beigiet al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib12)\)learns contrastively across attention, feed\-forward, and activation states from all layers\. Internal\-stability methods read how a representation responds to controlled perturbation \(CCPS\(Khanmohammadiet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib6)\)\), and BICR\(Khanmohammadiet al\.,[2026b](https://arxiv.org/html/2608.06532#bib.bib14)\)additionally contrasts the representation under the real figure against one under a blanked figure, so its score reflects whether the answer used the image, the only estimator here trained with such a grounding signal; a related line contrasts chain\-of\-embedding trajectories to expose the same language\-prior reliance\(Longet al\.,[2026](https://arxiv.org/html/2608.06532#bib.bib30)\)\. All were developed for general\-domain discrimination under roughly uniform error cost, with a known tension between optimizing calibration and discrimination jointly\(Khanmohammadiet al\.,[2026a](https://arxiv.org/html/2608.06532#bib.bib13)\)\. General\-domain LVLM confidence has itself been studied, through sampling\-based uncertainty\(Zhanget al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib28)\), functionally\-equivalent input sampling\(Bhattacharyaet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib19)\), and decoupled calibration\(Xiaoet al\.,[2026](https://arxiv.org/html/2608.06532#bib.bib29)\), but on natural images under roughly uniform error cost; none asks whether such a signal survives the move to financial figures under a bounded\-error deferral constraint\.

Reliability, not capability, is the weak axis for financial models\.Strong models remain unreliable on financial tasks: open\-book question answering over filings leaves them wrong or refusing on most questions\(Islamet al\.,[2023](https://arxiv.org/html/2608.06532#bib.bib20)\), and a finance\-wide trustworthiness audit places truthfulness among the weakest axes\(Huet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib21)\)\. The problem carries into multimodal financial models\(Bhatiaet al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib22)\)and takes a sharp form on charts, where models answer confidently when the evidence is absent from or contradicts the figure\(Wanget al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib23)\), the ungrounded behavior our study is built to catch\. Capability gains do not remove the need to know which answers to trust, which is exactly what a confidence signal must supply\.

Selective deferral is the deployment frame, and in finance it has so far been text\-only\.Abstaining when uncertain rather than maximizing aggregate accuracy is the established route to safe automation under an error budget, formalized as selective classification with a risk\-coverage tradeoff\(El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2608.06532#bib.bib32); Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2608.06532#bib.bib31)\)and, in high\-stakes settings, as abstention with a second opinion\(Kompaet al\.,[2021](https://arxiv.org/html/2608.06532#bib.bib8)\), including in visual question answering, where a model does better to abstain than to answer wrong\(Whiteheadet al\.,[2022](https://arxiv.org/html/2608.06532#bib.bib26)\)\. In finance the framing has been applied only to text: selective prediction over financial text QA\(Singha,[2025](https://arxiv.org/html/2608.06532#bib.bib24)\), and financial hallucination detection over text and tables through retrieval and claim verification\(Guoet al\.,[2026](https://arxiv.org/html/2608.06532#bib.bib25)\)\. Ours is the multimodal, internal\-state counterpart, scoring a frozen LVLM over financial figures at no added inference cost and judging that score by calibration and bounded\-error deferral\. We evaluate on FinMME\(Luoet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib16)\), FinChart\-Bench\(Shuet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib17)\), and the bilingual MME\-Finance\(Ganet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib18)\), accuracy benchmarks on which we ask the complementary question of when an answer can be trusted\.

## 3\.Study Design

Three financial benchmarks, two observable axes of variation\.We evaluate on three financial VQA benchmarks chosen to span distinct figure types and distinct distances from the natural\-image data on which the probes are trained, and built so that confidence quality can be read along two axes a practitioner can see: the task being performed and, within one benchmark, the language it is performed in\. FinMME\(Luoet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib16)\)\(11,099 questions\) is a broad financial multimodal benchmark over charts and figures with single\-choice, multiple\-choice, and numerical formats, the largest and most heterogeneous of the three\. FinChart\-Bench\(Shuet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib17)\)\(7,019\) is a chart\-understanding benchmark with true/false, multiple\-choice, and open question\-answer items about plotted financial data\. MME\-Finance\(Ganet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib18)\)\(2,274\) is a multi\-task financial\-document benchmark spanning eleven task types \(OCR, entity recognition, spatial awareness, financial knowledge, numerical calculation, and others\), bilingual in English \(1,171\) and Chinese \(1,103\); we report the two languages as separate conditions, giving three benchmarks and four evaluation conditions in total\. Posing the same MME\-Finance tasks in two languages turns the bilingual split into a controlled contrast that holds task content fixed and varies only difficulty, which lets us separate a harder task from a weaker model rather than confounding the two\.

Out\-of\-distribution protocol\.Every trainable estimator is trained and validated only on general\-domain GQA\(Hudson and Manning,[2019](https://arxiv.org/html/2608.06532#bib.bib5)\)\(20,000 train, 5,000 validation\) and evaluated on each financial condition without adaptation, so all financial sets are unseen during training\. This isolates the question a practitioner faces: does a confidence signal learned on ordinary images still mean what it says on a chart it has never seen? On MME\-Finance\-EN we use the same 892\-sample English subset as\(Khanmohammadiet al\.,[2026b](https://arxiv.org/html/2608.06532#bib.bib14)\), so the new MME\-Finance contributions here are the Chinese condition and the English\-versus\-Chinese contrast, while FinMME and FinChart\-Bench are new evaluations in full\.

Models\.We evaluate five open\-weight, instruction\-tuned LVLMs spanning 8B to 27B parameters and three vision\-encoder lineages: Qwen3\-VL\-8B, LLaVA\-NeXT\-13B, InternVL3\.5\-14B, DeepSeek\-VL2, and Gemma\-3\-27B\. All run under identical generation conditions \(greedy decoding, 64 new tokens, images downscaled to a maximum of 2,048 pixels on the long edge\)\. Base VQA accuracy spans a wide, realistic range and, crucially for the deployment analysis, differs systematically across conditions: pooled over the five LVLMs it is 60\.8% on FinChart\-Bench, 44\.8% on MME\-Finance\-EN, 41\.8% on FinMME, and 34\.5% on MME\-Finance\-ZH, ordering the conditions by difficulty and by distance from the natural\-image training distribution\. In this regime no single LVLM is reliable on its own, so how much can be safely delegated depends jointly on the model’s competence on a condition and on the confidence layer that decides which answers to trust\.

Correctness labels\.Following standard practice in the confidence\-estimation literature, we assign every answer a binary correctness labely∈\{0,1\}y\\in\\\{0,1\\\}with a singlegpt\-5\-minijudge that sees the image, question, gold answer, and generated response and decides semantic equivalence, a protocol shown reliable against human annotators\(Kapooret al\.,[2024](https://arxiv.org/html/2608.06532#bib.bib15); Khanmohammadiet al\.,[2025](https://arxiv.org/html/2608.06532#bib.bib6),[2026a](https://arxiv.org/html/2608.06532#bib.bib13)\)\. Applying one judge uniformly across datasets and models is a deliberate control: differences in measured confidence quality then reflect model behavior rather than grading rules that vary across datasets\. To validate the judge on the slice where semantic\-equivalence grading is most likely to fail, one author adjudicated a 100\-item audit stratified toward numerically\-exact and OCR answers; agreement with the judge was 94\.8% \(Cohen’sκ=0\.89\\kappa=0\.89\), with no degradation on the hard numerical/OCR slice relative to the rest\.

Confidence estimators and metrics\.The seven estimators group into three inference\-only methods that treat the LVLM as a black box \(P\(True\), Self\-Probing, Prompt Ensemble\) and four trained probes that read or perturb internal representations \(SAPLMA, CCPS, InternalInspector, and BICR\)\. Each trained probe uses the architecture reported best in its source paper, is trained on GQA over five seeds\{23,42,137,2024,3407\}\\\{23,42,137,2024,3407\\\}, and is evaluated with seed\-averaged confidence\. We report calibration with Expected Calibration Error \(ECE, 10\-bin\) and Brier score, and discrimination with AUROC and average precision \(AUCPR\); the selective\-prediction quantities \(risk\-coverage and safe yield\) are defined in Section[5](https://arxiv.org/html/2608.06532#S5)\.

## 4\.Results Across Models, Tasks, and Languages

We report calibration \(ECE, Brier\) and discrimination \(AUROC, AUCPR\) with task accuracy on each condition\. Headline numbers are pooled over the five LVLMs \(every \(LVLM, question\) pair concatenated, the metric computed once on the shared intersection of questions common to all seven methods per model\); trained probes use seed\-averaged confidence\. Table[1](https://arxiv.org/html/2608.06532#S4.T1)is the central result\.

![Refer to caption](https://arxiv.org/html/2608.06532v1/x1.png)Figure 1\.Reliability diagrams pooled across the five LVLMs \(dashed line is perfect calibration\)\. Marker size is proportional to the number of samples in each confidence bin\. BICR tracks the diagonal closely; P\(True\) sits above it, issuing high confidence the empirical accuracy does not support, which makes a raw self\-report unsafe to threshold for automation\.Table 1\.Confidence estimation across four financial conditions under out\-of\-distribution evaluation \(probes trained on natural\-image GQA, applied to finance without adaptation\)\. Each metric is pooled over the five LVLMs on concatenated \(model, question\) pairs \(×100\\times 100\), so the sample size feeding each metric is about five times the per\-model question count reported in the block headers; trained probes use seed\-averaged confidence\. Calibration: ECE \(10\-bin\), Brier \(BS\)\. Discrimination: AUROC, AUCPR\. ConfErr is the fraction of a method’s errors answered with confidence above 0\.8 \(lower is better\)\. Best per column in bold\.What the trained probes buy is calibration, not ranking\.Across all four conditions the inference\-only baselines discriminate competitively but are badly overconfident, posting ECE from 0\.25 to 0\.58 and placing most of their probability mass far from empirical accuracy \(Figure[1](https://arxiv.org/html/2608.06532#S4.F1)\)\. The failure is starkest for the cheapest self\-report: Self\-Probing’s confident\-error rate, the share of its mistakes it still labels with confidence above 0\.8 \(ConfErr in Table[1](https://arxiv.org/html/2608.06532#S4.T1)\), reaches nearly nine in ten, confidence a deferral policy would read as safe\. The trained internal probes cut this error by half to an order of magnitude while staying competitive at discrimination, and between them BICR and InternalInspector hold the best calibration on every condition\. This is the deployment\-relevant property, because a zero\-cost logit can rank cases but only a calibrated score can be thresholded into a controlled error rate\.

Neither a single estimator nor a single model is a safe default\.In the pooled view the strongest method already shifts with the domain: BICR owns FinMME on every metric at once \(ECE 8\.6, BS 22\.4, ACC 64\.2, AUCPR 63\.1, AUROC 68\.5\); on FinChart\-Bench the roles split, with InternalInspector the best calibrated, a Prompt Ensemble the best discriminator \(AUROC 69\.5\), and SAPLMA the most accurate; and on the harder MME\-Finance benchmark no internal probe leads ranking at all, P\(True\) taking AUROC in both languages \(73\.7 EN, 77\.4 ZH\)\. Disaggregated to the twenty \(model, condition\) cells the lead scatters further: the highest\-AUROC method is P\(True\) in eight cells and BICR in five, with SAPLMA, Self\-Probing, and Prompt Ensemble in two each and InternalInspector in one, and no method wins more than eight of twenty\. A Friedman test over the seven methods across the cells is significant \(χ2=41\.1\\chi^\{2\}=41\.1,p<10−6p<10^\{\-6\}\): the methods are separable, but none dominates\. A practitioner reading only a pooled leaderboard would pick the wrong tool on most conditions, so confidence estimators for financial LVLMs must be reported per model and per condition rather than collapsed to one number \(Table[2](https://arxiv.org/html/2608.06532#S4.T2)\)\.

Table 2\.Significance of BICR’s discrimination \(DeLong test for correlated AUROCs on pooled samples\) on the two conditions where it is the top discriminator, and the cross\-cell Friedman test over the twenty \(LVLM, condition\) cells\.Whether a confidence score is trustworthy depends on the task operation, not the benchmark\.Pooling the twelve largest native task and format categories across the three benchmarks \(Figure[2](https://arxiv.org/html/2608.06532#S4.F2)\) shows that separability is organized by the kind of operation the model performs on the figure, while calibration in the same cells tracks the method rather than the task, the trained probes holding low ECE across nearly every operation and the inference\-only methods staying miscalibrated throughout\. Confidence is most trustworthy on lookup\-style tasks, where the best methods reach AUROC near 0\.80 on OCR and the high 0\.70s on single\- and multiple\-choice items, and collapses toward chance on open\-ended judgment: nearly every method sits at or below 0\.69 on image captioning, and most sit near 0\.50 on true/false verification, where a fluent guess is hard to tell from a grounded answer\. No method owns a row, with P\(True\) and the Prompt Ensemble taking the most categories while BICR is strongest on single\-choice, multiple\-choice, and OCR and weakest on captioning and true/false\. The operational reading is that a confidence\-gated pipeline should follow the task mix, trusting the score first on retrieval and structured\-answer operations and last on free\-form description\.

![Refer to caption](https://arxiv.org/html/2608.06532v1/x2.png)Figure 2\.Per financial task\-type confidence quality \(twelve largest native task and format categories pooled across the benchmarks\)\. Each cell is split diagonally: the upper\-right number and color are AUROC \(higher and greener is better\), the lower\-left number is ECE \(lower is better\)\. AUROC separability is organized by operation, highest on lookup tasks \(OCR, single\- and multiple\-choice\) and lowest on captioning and true/false, whereas calibration tracks the method rather than the task: the four trained probes hold low ECE across nearly every task while the inference\-only methods stay miscalibrated throughout\. No method is uniformly best across tasks\.A controlled bilingual contrast separates a harder task from a weaker model\.Because MME\-Finance poses the same tasks in English and Chinese, the two conditions form a natural experiment that isolates difficulty from competence\. The Chinese side is harder, pooled base accuracy 34\.5% against 44\.8% for English, a ten\-point gap concentrated in the weaker models\. Pooled method rankings look preserved across the two languages, P\(True\) the top pooled discriminator and BICR the best calibrated on each, but this stability is an artifact of aggregation rather than a property of any estimator: within individual models P\(True\)’s AUROC falls from English to Chinese in four of the five LVLMs, including ones whose base accuracy barely moves across language, so its steady per\-language pooled figures are a composition effect of mixing models with different base rates\. P\(True\) also stays poorly calibrated on Chinese \(ECE up to 0\.61\), so even where it ranks well its score is not thresholdable\. The language changes how much can be trusted, not which signal to trust, and the pooled number hides the within\-model decline that a deployment on Chinese documents would actually meet\.

Where BICR leads, it leads through calibration and a low confident\-error rate, not through ranking\.On FinMME, where BICR is the top discriminator, a DeLong test for correlated AUROCs places it significantly ahead of all six baselines \(Table[2](https://arxiv.org/html/2608.06532#S4.T2)\); on MME\-Finance\-EN its edge is partial, ahead of the other trained probes but behind the self\-report and the prompt ensemble, the document regime where no internal probe leads ranking\. Calibration is the more consistent evidence: the two trained probes hold the best calibration on every condition and rarely fire confidently on a wrong answer, with ConfErr of 2 to 12% for BICR and as low as 0\.7% for InternalInspector against 35 to 47% for P\(True\)\. A score confidently wrong on up to nearly half its errors cannot be thresholded into a safe policy, which is what the deployment analysis turns on\.

## 5\.Deployment: What Can Be Safely Delegated

The deployable question is automate\-or\-escalate under an error tolerance\.Aggregate calibration and discrimination do not answer the question an operation faces: how much of this workload can be cleared automatically, and which cases must reach a person? We adopt the standard bounded selective\-prediction framing\(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2608.06532#bib.bib31); El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2608.06532#bib.bib32); Kompaet al\.,[2021](https://arxiv.org/html/2608.06532#bib.bib8)\), prior apparatus we apply to financial VQA rather than a contribution of this work: automate a case only when its confidence clears a threshold, defer the rest, and judge a method by what it clears under an error budget a deploying desk would set from the cost of a wrong figure\. Per \(condition, method\) we trace the risk\-coverage curve, error among automated cases against the fraction automated \(Figure[3](https://arxiv.org/html/2608.06532#S5.F3), the five most informative methods shown\), and read off the*safe yield*, the largest fraction automatable while holding that error at or below a tolerance, which Table[3](https://arxiv.org/html/2608.06532#S5.T3)reports per model for the three estimators the analysis turns on\. Calibration is what makes the threshold meaningful, since a cutoff at 0\.8 maps to a controlled error rate only when 0\.8 means about 80% correct\.

What can be delegated is set by the model, and the confidence layer earns its yield only where competence runs out\.Safe yield tracks base\-model competence, which tracks each condition’s difficulty: across the twenty cells the best attainable confidence AUROC rises with base accuracy \(r=0\.68r=0\.68, Figure[4](https://arxiv.org/html/2608.06532#S5.F4)\), so the confidence signal is least reliable exactly where the model is weakest\. To ask what the confidence layer adds beyond competence, we compare each method’s yield against the model’s own answer softmax, a zero\-cost ranker that uses no trained layer\. The layer’s yield advantage is real but concentrated: on the harder conditions it adds a meaningful share \(BICR\+8\.7\+8\.7points on FinMME, P\(True\)\+17\.2\+17\.2on MME\-Finance\-EN over the softmax baseline at the 20% budget\), whereas on the easy FinChart\-Bench the free softmax already deploys 39\.2% and no method beats it\. Deployable yield under a tuned threshold, then, is set first by the model and improved by a confidence signal chiefly where the model is weak\. That comparison rewards ranking under a tuned threshold and is blind to calibration, the fixed\-cutoff safety the free softmax lacks and the trained probes supply \(Table[1](https://arxiv.org/html/2608.06532#S4.T1)\)\. The confidence layer’s contribution is therefore two\-part: domain\-specific yield where competence is low, and domain\-general calibration a zero\-cost baseline cannot provide\.

Table 3\.Per\-model safe yield under bounded selective prediction for the three deployment\-relevant estimators \(P\(True\), InternalInspector,BICR\), at the 5% and 20% error budgets\. Safe yield is the maximum automatable fraction while keeping error among automated cases at or below the budget \(↑\\uparrow\); “base acc” is task accuracy\. Each block reports the five LVLMs with model\-specific operating points, followed by an italic*Pooled*row under a shared threshold\. Bold marks the best of the five models per column \(excluding*Pooled*\)\. Pooling collapses per\-model yield whenever strong and weak models are mixed, with the 1\.0% versus 94\.4% gap on FinChart\-Bench the sharpest example\.![Refer to caption](https://arxiv.org/html/2608.06532v1/x3.png)Figure 3\.Risk\-coverage curves pooled across five LVLMs \(lower is better\)\. Dotted lines mark the 5% and 10% error budgets\. FinChart\-Bench admits real automation; FinMME admits little; MME\-Finance clears the budget only at negligible coverage\.![Refer to caption](https://arxiv.org/html/2608.06532v1/x4.png)Figure 4\.Attainable confidence separability rises with base VQA accuracy across the twenty \(model, condition\) cells \(r=0\.68r=0\.68\)\. Each marker is one cell, colored by condition\.The pooled verdict is an artifact of mixing models; per model, automation is real but narrow\.Pooled over the five LVLMs, safe yield at a strict 5% budget is near zero on every condition \(Table[3](https://arxiv.org/html/2608.06532#S5.T3)\), and useful automation appears only at a looser budget on the easier conditions\. But the pooled figure understates what a single deployed model can do, because pooling strong and weak models under one threshold erases the strong model’s reachable region: on FinChart\-Bench at the 20% budget InternalInspector pools to 1\.0% while clearing 94\.4% on Qwen3\-VL alone, and BICR pools to 0\.0% on MME\-Finance\-EN while clearing 20\.4% on Qwen3\-VL\. Automation is therefore real but narrow, confined to the strongest models on the easier conditions and vanishing on the weak models and the document tasks, where no operating point carves out a low\-error subset once the model is wrong on more than half of cases\. The five financial LVLMs we study are not ready to answer chart and document questions on their own at a deployment\-grade tolerance; the workable role is confidence\-gated automation with an expert signing off every escalation, and within that role the choice of estimator still matters\.

A thresholded BICR score is safe to act on because it backs off when the figure is ignored\.What makes a score actionable is that a threshold maps to a controlled error rate, which BICR most nearly achieves, and its grounding\-aware objective transfers to finance\. Replacing each chart with a deterministic random image and isolating the figure\-invariant subpopulation, the samples whose first generated token does not change under the swap and on which the model demonstrably ignores the figure, BICR is the best\- or tied\-best\-calibrated of the seven estimators on FinMME, MME\-Finance\-EN, and MME\-Finance\-ZH, with significantly lower per\-sample Brier than every baseline on the first two and than five of six on the third \(paired bootstrap, Table[4](https://arxiv.org/html/2608.06532#S5.T4)\)\. Because those subpopulations are majority incorrect \(42%, 45%, 27% correct\), the low, well\-calibrated confidence reflects detected non\-grounding rather than blanket under\-confidence\. The effect is absent on FinChart\-Bench, whose figure\-invariant items are majority correct \(61%\) and largely answerable from text, so first\-token invariance does not isolate grounding there; we report that null as a scope limit of the diagnostic rather than a failure of the method\.

Table 4\.Calibration on the figure\-invariant subpopulation \(first generated token unchanged when the chart is replaced with a deterministic random image\)\. On this population the model answers without using the figure, and BICR is best\- or tied\-best\-calibrated on genuinely figure\-dependent cases\. Significance is a paired bootstrap of per\-sample Brier against BICR \(∗\\ast= 95% CI excludes zero\)\. Pooled over five LVLMs\.
## 6\.Discussion

A deployment threshold consumes calibration, not ranking\.A score is actionable only if it can be thresholded: when 0\.8 means roughly 80% correct, a cutoff maps to a controlled error rate\(Kompaet al\.,[2021](https://arxiv.org/html/2608.06532#bib.bib8)\)\. Discrimination is necessary but not sufficient, because a perfectly ranked yet miscalibrated score yields no safe operating point, which is why the inference\-only baselines, competitive at AUROC yet overconfident at ECE 0\.25 to 0\.58, are non\-functional for deferral on the harder conditions\. The portable lesson across four financial conditions is that the trained internal probes earn their place on calibration: BICR and InternalInspector are the only methods whose scores a threshold can trust, and BICR’s margin is widest on the broad benchmark where the figure is most often necessary\.

No single estimator and no single model is a safe default\.The preferred method shifts with both domain and model: BICR owns FinMME on every metric, a prompt ensemble is the best discriminator on chart understanding, and a raw self\-report is both the strongest discriminator and the most automatable estimator on MME\-Finance\. A practitioner reading only a pooled leaderboard would pick the wrong tool on most conditions, so methods should be reported and selected per model and per task family; the two trained probes are the safe general choice for a thresholded score, and the grounding\-aware one is preferable where visual ungroundedness is the failure to guard against\.

The honest near\-term role is a confidence\-gated reviewer, not an autonomous one\.Selective deferral, not raw accuracy, is what permits safe automation: the near\-term opportunity is clearing routine figure questions and prioritizing the rest with a person on every escalation, and the risk is mistaking a score that is merely discriminative, or merely low, for one that is safe\. As financial LVLMs improve, the safe\-yield frontier will rise unevenly, fastest on the models and domains closest to general training data and slowest on the hardest document tasks, and a well\-calibrated estimator, grounding\-aware where ungroundedness is the risk, is the signal whose meaning a user can rely on at any point along that frontier\.

## 7\.Limitations

Our probes are trained on a single natural\-image source \(GQA\) and transferred zero\-shot, which is the transfer setting we set out to measure rather than the best achievable result; in\-domain financial training is left to future work and would test a different question\. Correctness labels come from a model judge, standard in this literature but imperfect for numerically exact answers\. The per\-task analysis pools categories across benchmarks for statistical power, and the deployment numbers use a single shared threshold in the pooled view and a model\-specific one per cell, whereas a real system would set the threshold on a held\-out slice of the target distribution\. The behavioral grounding test uses first\-token invariance as a proxy for full\-answer invariance, which isolates figure\-grounding only where the task is not answerable from text, as the FinChart\-Bench null shows\.

## 8\.Conclusion

We evaluated seven confidence estimators across five open\-weight LVLMs and four financial conditions under one out\-of\-distribution protocol, asking whether a confidence signal learned on ordinary images can tell, on financial figures it has never seen, when an answer should be trusted\. Three findings hold\. First, the trained probes earn their place on calibration, not ranking: the inference baselines discriminate competitively but are too overconfident to threshold, while only BICR and InternalInspector map to a controlled error rate\. Second, reliability is structured, not global: it shifts with model and task, no method winning more than eight of twenty cells, and a bilingual contrast exposes an apparent language robustness as a composition effect, so methods must be selected per model and per task family\. Third, how much can be safely delegated is set first by base\-model competence \(r=0\.68r=0\.68\); against the model’s own softmax the confidence layer improves yield chiefly where the model is weak, its domain\-general contribution being calibration, not yield\. Across all three, only the grounding\-aware probe lowers its confidence on answers the model gives without using the figure, separating detected non\-grounding from a fluent guess\. The five LVLMs we study cannot read charts and documents on their own at a deployment\-grade tolerance; the near\-term role is confidence\-gated automation with an expert on every escalation\.

## Acknowledgments

This work was supported by the JPMorgan Chase AI Research Faculty Research Award\. The authors are solely responsible for the contents of this paper; the opinions expressed do not necessarily reflect those of the funding organizations\. The authors also acknowledge the use of Large Language Models to assist in polishing the language and grammar of this manuscript\.

## Disclaimer

This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates \(“JP Morgan”\), and is not a product of the Research Department of JP Morgan\. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein\. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful\.

## References

- A\. Azaria and T\. Mitchell \(2023\)The internal state of an llm knows when it’s lying\.External Links:2304\.13734,[Link](https://arxiv.org/abs/2304.13734)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- M\. Beigi, Y\. Shen, R\. Yang, Z\. Lin, Q\. Wang, A\. Mohan, J\. He, M\. Jin, C\. Lu, and L\. Huang \(2024\)InternalInspectorI2I^\{2\}: robust confidence estimation in LLMs through internal states\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 12847–12865\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.751/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.751)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- G\. Bhatia, E\. M\. B\. Nagoudi, H\. Cavusoglu, and M\. Abdul\-Mageed \(2024\)FinTral: a family of GPT\-4 level multimodal financial large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 13064–13087\.External Links:[Link](https://aclanthology.org/2024.findings-acl.774/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.774)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p3.1)\.
- D\. Bhattacharya, A\. Kulkarni, and S\. Ganapathy \(2025\)FESTA: functionally equivalent sampling for trust assessment of multimodal LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 12277–12295\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.657/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.657),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- Y\. Dang, Y\. Jiang, Y\. Jiang, A\. Chen, W\. Li, and Y\. Gao \(2026\)Instinct vs\. reflection: unifying token and verbalized confidence in multimodal large models\.External Links:2604\.17274,[Link](https://arxiv.org/abs/2604.17274)Cited by:[§1](https://arxiv.org/html/2608.06532#S1.p1.1)\.
- R\. El\-Yaniv and Y\. Wiener \(2010\)On the foundations of noise\-free selective classification\.J\. Mach\. Learn\. Res\.11,pp\. 1605–1641\.External Links:ISSN 1532\-4435Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1),[§5](https://arxiv.org/html/2608.06532#S5.p1.1)\.
- Z\. Gan, D\. Zhang, H\. Li, Y\. Wu, X\. Lin, J\. Liu, H\. Wu, C\. Fu, Z\. Xu, R\. Zhang, and Y\. Dai \(2025\)MME\-finance: a multimodal finance benchmark for expert\-level understanding and reasoning\.InProceedings of the 33rd ACM International Conference on Multimedia,MM ’25,New York, NY, USA,pp\. 12867–12874\.External Links:ISBN 9798400720352,[Link](https://doi.org/10.1145/3746027.3758230),[Document](https://dx.doi.org/10.1145/3746027.3758230)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1),[§3](https://arxiv.org/html/2608.06532#S3.p1.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 4885–4894\.External Links:ISBN 9781510860964Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1),[§5](https://arxiv.org/html/2608.06532#S5.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \- Volume 70,ICML’17,pp\. 1321–1330\.Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p1.1)\.
- D\. Guo, J\. Wu, and S\. M\. Yiu \(2026\)FinGround: detecting and grounding financial hallucinations via atomic claim verification\.External Links:2604\.23588,[Link](https://arxiv.org/abs/2604.23588)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1)\.
- T\. Hu, T\. Hu, L\. Bai, Y\. Zhao, A\. Cohan, and C\. Zhao \(2025\)FinTrust: a comprehensive benchmark of trustworthiness evaluation in finance domain\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 10099–10128\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.512/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.512),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p3.1)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 6693–6702\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2019.00686)Cited by:[§3](https://arxiv.org/html/2608.06532#S3.p2.1)\.
- P\. Islam, A\. Kannappan, D\. Kiela, R\. Qian, N\. Scherrer, and B\. Vidgen \(2023\)FinanceBench: a new benchmark for financial question answering\.External Links:2311\.11944,[Link](https://arxiv.org/abs/2311.11944)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p3.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. Kaplan \(2022\)Language models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- S\. Kapoor, N\. Gruver, M\. Roberts, A\. Pal, S\. Dooley, M\. Goldblum, and A\. Wilson \(2024\)Calibration\-tuning: teaching large language models to know what they don’t know\.InProceedings of the 1st Workshop on Uncertainty\-Aware NLP \(UncertaiNLP 2024\),R\. Vázquez, H\. Celikkanat, D\. Ulmer, J\. Tiedemann, S\. Swayamdipta, W\. Aziz, B\. Plank, J\. Baan, and M\. de Marneffe \(Eds\.\),St Julians, Malta,pp\. 1–14\.External Links:[Link](https://aclanthology.org/2024.uncertainlp-1.1/)Cited by:[§3](https://arxiv.org/html/2608.06532#S3.p4.2)\.
- R\. Khanmohammadi, E\. Miahi, S\. Kaur, C\. Smiley, I\. Brugere, K\. S\. Thind, and M\. M\. Ghassemi \(2026a\)How reliable are confidence estimators for large reasoning models? a systematic benchmark on high\-stakes domains\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 1669–1754\.External Links:[Link](https://aclanthology.org/2026.eacl-long.78/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.78),ISBN 979\-8\-89176\-380\-7Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1),[§3](https://arxiv.org/html/2608.06532#S3.p4.2)\.
- R\. Khanmohammadi, E\. Miahi, S\. Kaur, C\. H\. Smiley, I\. Brugere, K\. Thind, and M\. M\. Ghassemi \(2026b\)Grounded or guessing? lvlm confidence estimation via blind\-image contrastive ranking\.External Links:2605\.10893,[Link](https://arxiv.org/abs/2605.10893)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1),[§3](https://arxiv.org/html/2608.06532#S3.p2.1)\.
- R\. Khanmohammadi, E\. Miahi, M\. Mardikoraem, S\. Kaur, I\. Brugere, C\. Smiley, K\. S\. Thind, and M\. M\. Ghassemi \(2025\)Calibrating LLM confidence by probing perturbed representation stability\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 10448–10514\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.530/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.530),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1),[§3](https://arxiv.org/html/2608.06532#S3.p4.2)\.
- B\. Kompa, J\. Snoek, and A\. L\. Beam \(2021\)Second opinion needed: communicating uncertainty in medical machine learning\.npj Digital Medicine4\(1\),pp\. 4\.External Links:[Document](https://dx.doi.org/10.1038/s41746-020-00367-3),[Link](https://doi.org/10.1038/s41746-020-00367-3),ISSN 2398\-6352Cited by:[§1](https://arxiv.org/html/2608.06532#S1.p3.1),[§2](https://arxiv.org/html/2608.06532#S2.p4.1),[§5](https://arxiv.org/html/2608.06532#S5.p1.1),[§6](https://arxiv.org/html/2608.06532#S6.p1.1)\.
- Q\. Li, J\. Geng, C\. Lyu, D\. Zhu, M\. Panov, and F\. Karray \(2024\)Reference\-free hallucination detection for large vision\-language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4542–4551\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.262/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.262)Cited by:[§1](https://arxiv.org/html/2608.06532#S1.p2.1),[§1](https://arxiv.org/html/2608.06532#S1.p3.1)\.
- L\. Long, C\. Oh, S\. Park, and S\. Li \(2026\)Understanding language prior of lvlms by contrasting chain\-of\-embedding\.External Links:2509\.23050,[Link](https://arxiv.org/abs/2509.23050)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- J\. Luo, Z\. Kou, L\. Yang, X\. Luo, J\. Huang, Z\. Xiao, J\. Peng, C\. Liu, J\. Ji, X\. Liu, S\. Han, M\. Zhang, and Y\. Guo \(2025\)FinMME: benchmark dataset for financial multi\-modal reasoning evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 29465–29489\.External Links:[Link](https://aclanthology.org/2025.acl-long.1426/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1426),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1),[§3](https://arxiv.org/html/2608.06532#S3.p1.1)\.
- D\. Shu, H\. Yuan, Y\. Wang, Y\. Liu, H\. Zhang, H\. Zhao, and M\. Du \(2025\)FinChart\-bench: benchmarking financial chart comprehension in vision\-language models\.External Links:2507\.14823,[Link](https://arxiv.org/abs/2507.14823)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1),[§3](https://arxiv.org/html/2608.06532#S3.p1.1)\.
- M\. Singha \(2025\)Detecting ai hallucinations in finance: an information\-theoretic method cuts hallucination rate by 92%\.External Links:2512\.03107,[Link](https://arxiv.org/abs/2512.03107)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5433–5442\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.330/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.330)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- X\. Wang, Y\. Cui, X\. Yao, S\. Wang, G\. Hu, and X\. Qin \(2025\)ChartHal: a fine\-grained framework evaluating hallucination of large vision language models in chart understanding\.External Links:2509\.17481,[Link](https://arxiv.org/abs/2509.17481)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p3.1)\.
- S\. Whitehead, S\. Petryk, V\. Shakib, J\. Gonzalez, T\. Darrell, A\. Rohrbach, and M\. Rohrbach \(2022\)Reliable visual question answering: abstain rather than answer incorrectly\.InComputer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI,Berlin, Heidelberg,pp\. 148–166\.External Links:ISBN 978\-3\-031\-20058\-8,[Link](https://doi.org/10.1007/978-3-031-20059-5_9),[Document](https://dx.doi.org/10.1007/978-3-031-20059-5%5F9)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p4.1)\.
- S\. Woo, D\. Kim, J\. Jang, Y\. Choi, and C\. Kim \(2025\)Don’t miss the forest for the trees: attentional vision calibration for large vision language models\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 1927–1951\.External Links:[Link](https://aclanthology.org/2025.findings-acl.99/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.99),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2608.06532#S1.p2.1)\.
- W\. Xiao, X\. XU, and L\. Gan \(2026\)VL\-calibration: decoupled confidence calibration for large vision\-language models reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 44791–44815\.External Links:[Link](https://aclanthology.org/2026.acl-long.2074/),ISBN 979\-8\-89176\-390\-6Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2024\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.External Links:2306\.13063,[Link](https://arxiv.org/abs/2306.13063)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- R\. Zhang, H\. Zhang, and Z\. Zheng \(2024\)VL\-uncertainty: detecting hallucination in large vision\-language model via uncertainty estimation\.External Links:2411\.11919,[Link](https://arxiv.org/abs/2411.11919)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.
- T\. P\. Zollo and R\. Zemel \(2025\)Confidence calibration in vision\-language\-action models\.External Links:2507\.17383,[Link](https://arxiv.org/abs/2507.17383)Cited by:[§2](https://arxiv.org/html/2608.06532#S2.p2.1)\.