Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

# Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
Source: [https://arxiv.org/html/2607.22034](https://arxiv.org/html/2607.22034)
###### Abstract

Vision\-language models \(VLMs\) are increasingly deployed on consumer hardware where input images are degraded by compression, camera shake, and poor lighting\. In such settings a reliable uncertainty signal matters more than raw accuracy, because it determines when a system should defer rather than answer\. We evaluate two small open\-weight VLMs – Qwen2\-VL\-2B\-Instruct and SmolVLM\-Instruct – across six realistic photographic degradations at three severity levels, comparing two confidence signals: the confidence the model*states*in natural language, and the model’s own mean token probability over its generated answer\. Across 3,800 predictions we find a large and consistent gap\. Verbalized confidence in Qwen2\-VL is almost constant \(mean 0\.87–0\.90 across all conditions\) and detects its own errors at chance level \(AUROC 0\.39–0\.75, typically≈\\approx0\.50\), while internal token probability from the same model separates correct from incorrect answers with AUROC 0\.92–0\.99\. In SmolVLM, verbalized confidence proved largely unobtainable: across three prompt templates only one of five pilot attempts produced a parseable confidence value, while internal probability again yielded above\-chance error detection \(AUROC 0\.54–0\.92\)\. Both models fail in the same place: under severe underexposure, accuracy collapses \(0\.99→0\.220\.99\\rightarrow 0\.22for Qwen2\-VL,0\.97→0\.420\.97\\rightarrow 0\.42for SmolVLM\) while both confidence signals barely move, and internal error\-detection falls to chance\. We conclude that small VLMs encode usable self\-knowledge that their verbalized output does not express, that internal probability is therefore the better deferral signal in constrained deployment, and that neither signal should be trusted under severe low\-light conditions\.

## 1Introduction

The practical deployment of vision\-language models increasingly happens at the small end of the parameter range\. Models of two to eight billion parameters run on consumer GPUs and edge devices, and are chosen for cost, latency, privacy, or the absence of reliable network connectivity\. In these settings the images reaching the model are rarely clean benchmark photographs: they are compressed by messaging applications, blurred by handheld capture, underexposed indoors, or washed out by direct light\.

Accuracy alone is an insufficient description of how such a system behaves\. What determines whether a deployment is safe is whether the model’s expressed uncertainty tracks its actual reliability – whether, when the model becomes unreliable, anything in its output indicates this\. A model that is wrong and signals doubt can defer to a human\. A model that is wrong and signals confidence cannot\.

Generative VLMs offer two distinct routes to an uncertainty estimate\. The first is to ask the model directly and read the number it states –*verbalized confidence*\. The second is to read the model’s own probability distribution over the tokens it generated –*internal confidence*\. These are not obviously the same quantity, and the relationship between them under input degradation has not been systematically examined for small open\-weight models\.

This paper asks three questions:

- •RQ1\.How does the accuracy of small VLMs degrade under realistic photographic corruption?
- •RQ2\.Does verbalized confidence track that degradation?
- •RQ3\.Does internal token probability track it better?

Our contributions are: \(i\) a calibration\-centred benchmark of two small open\-weight VLMs across six realistic degradations at graded severities, with bootstrap confidence intervals; \(ii\) evidence that verbalized and internal confidence diverge sharply, with internal probability providing near\-ceiling error detection where verbalized confidence performs at chance; and \(iii\) an honest characterisation of where both signals fail\.

## 2Related Work

### 2\.1Verbalized uncertainty in language and vision\-language models

Because generative models emit text rather than a probability vector, a natural approach to uncertainty estimation is to prompt the model to state its own confidence\.Xionget al\.\([2024](https://arxiv.org/html/2607.22034#bib.bib2)\)evaluated confidence\-elicitation strategies in LLMs and reported that models are overconfident, with the majority of confidence scores falling in the 80–100 range\.Tianet al\.\([2023](https://arxiv.org/html/2607.22034#bib.bib3)\)found that for models fine\-tuned with RLHF, verbalized confidence can be better calibrated than sampling\-based estimates of internal token probabilities – a result that has made verbalized uncertainty the default choice in much subsequent VLM work\.Groot and Valdenegro\-Toro \([2024](https://arxiv.org/html/2607.22034#bib.bib4)\)extended verbalized confidence estimation to visual question answering and found VLMs poorly calibrated and severely overconfident\.

### 2\.2Verbalized uncertainty under image corruption

The work closest to ours isBorszukovszkiet al\.\([2025](https://arxiv.org/html/2607.22034#bib.bib1)\), who evaluated three proprietary VLMs \(GPT\-4 Vision, Gemini Pro Vision, Claude 3 Opus\) on three corruption types \(Gaussian noise, defocus blur, JPEG compression\) at five severity levels, across easy VQA, hard VQA, and counting tasks\. They report that increasing corruption severity increases Expected Calibration Error, that models remain overconfident throughout, and that higher refusal rates improve calibration\.

We replicate their central findings on open\-weight models: our Qwen2\-VL\-2B states a near\-constant confidence of 0\.87–0\.90 regardless of accuracy, consistent with the 80–100 clustering they andXionget al\.\([2024](https://arxiv.org/html/2607.22034#bib.bib2)\)report, and calibration degrades sharply under severe corruption\.

Our contribution is orthogonal to theirs, and their paper states precisely why\. They motivate their use of verbalized confidence by noting that for the models they study, “since these models are proprietary, we don’t have access to these individual token probabilities,” and describe their contribution as extending the research “into VLMs where internal token probabilities are not available\.” Their design therefore*cannot*compare verbalized confidence against the model’s own token\-level distribution – the signal is inaccessible by construction\.

By evaluating open\-weight models, we recover exactly that signal\. This allows the comparison their setting excludes: verbalized against internal confidence, computed on the same predictions, under the same corruptions\. Our central finding – that internal token probability detects errors with AUROC up to 0\.99 while verbalized confidence from the same model performs at chance – is only measurable in the open\-weight setting\.

We differ in three further respects\. First,Borszukovszkiet al\.\([2025](https://arxiv.org/html/2607.22034#bib.bib1)\)evaluate frontier proprietary models; we evaluate small \(≈\\approx2B\) open\-weight models representative of constrained deployment\. Second, they use three synthetic corruption families fromMichaeliset al\.\([2019](https://arxiv.org/html/2607.22034#bib.bib5)\); we use six degradations selected to approximate real phone\-camera artifacts, including low light and glare, and find that low light produces a failure mode not visible in their corruption set\. Third, they report accuracy, confidence, and ECE; we add error\-detection AUROC with bootstrap confidence intervals, which we argue is the more diagnostic metric when a confidence signal has near\-zero variance \(Section[4](https://arxiv.org/html/2607.22034#S4)\)\.

Finally,Borszukovszkiet al\.\([2025](https://arxiv.org/html/2607.22034#bib.bib1)\)explicitly identify temperature scaling as future work, noting it “would be interesting to explore if this overconfidence in VLMs could be treated with temperature scaling\.” We test this directly and report a negative result: post\-hoc temperature scaling improves ECE under mild degradation but*worsens*it under severe low light, because rescaling a near\-constant signal cannot recover information the signal never contained\.

### 2\.3Corruption robustness and calibration under shift

The systematic study of corruption robustness was formalised byHendrycks and Dietterich \([2019](https://arxiv.org/html/2607.22034#bib.bib9)\), who introduced parameterised corruption families at calibrated severity levels;Michaeliset al\.\([2019](https://arxiv.org/html/2607.22034#bib.bib5)\)extended this taxonomy\. We follow their methodological template of graded severities reported as a function of severity rather than in aggregate\.

Separately,Ovadiaet al\.\([2019](https://arxiv.org/html/2607.22034#bib.bib8)\)established that predictive uncertainty degrades under dataset shift, andGuoet al\.\([2017](https://arxiv.org/html/2607.22034#bib.bib7)\)showed that post\-hoc temperature scaling corrects overconfidence in classifiers\. Our negative temperature\-scaling result is consistent with the mechanismGuoet al\.\([2017](https://arxiv.org/html/2607.22034#bib.bib7)\)describe: temperature scaling redistributes probability mass but cannot create discriminative signal where none exists\.

### 2\.4Positioning summary

To our knowledge, no prior work compares verbalized against internal confidence on the same predictions in vision\-language models under image degradation\. The closest work\(Borszukovszkiet al\.,[2025](https://arxiv.org/html/2607.22034#bib.bib1)\)identifies the absence of internal token probabilities as a defining constraint of its setting\. This paper occupies that gap\.

## 3Method

### 3\.1Models

We evaluate two small open\-weight instruction\-tuned VLMs:Qwen2\-VL\-2B\-Instruct\(Wanget al\.,[2024](https://arxiv.org/html/2607.22034#bib.bib11)\)andSmolVLM\-Instruct\(Hugging Face TB,[2024](https://arxiv.org/html/2607.22034#bib.bib12)\)\. Both run in fp16 on a single free\-tier NVIDIA T4 \(16 GB\)\. Generation is greedy \(do\_sample=False\) throughout for reproducibility\.

### 3\.2Dataset

A 100\-item subset of Food101\(Bossardet al\.,[2014](https://arxiv.org/html/2607.22034#bib.bib13)\)\(test split, streamed, seed 0\), posed as four\-option multiple choice: one correct label and three distractors sampled uniformly from the remaining 100 classes\. Food101 provides real photographs at usable resolution, appropriate for a corruption study\.

### 3\.3Degradations

Six degradation families were chosen to approximate real phone\-camera artifacts, each applied at three severity levels \(Table[1](https://arxiv.org/html/2607.22034#S3.T1)\)\.

Table 1:Degradation families and severity parameters\.With the clean baseline this yields 19 conditions per model, 100 items each: 1,900 predictions per model and 3,800 in total\.

### 3\.4Confidence signals

#### Verbalized confidence\.

The model is prompted to answer and state an integer confidence in\[0,100\]\[0,100\], normalised to\[0,1\]\[0,1\]\. The prompt template was selected empirically: three candidates were piloted on five items and the one with the highest parse rate retained\. For Qwen2\-VL a few\-shot template achieved 5/5 parseable outputs; for SmolVLM the best of three achieved only 1/5 \(Section[5\.3](https://arxiv.org/html/2607.22034#S5.SS3)\)\. Exact templates appear in Appendix[B](https://arxiv.org/html/2607.22034#A2)\.

#### Internal confidence\.

During generation we retain per\-step output distributions and compute the mean probability assigned to each token actually emitted in the answer span:

cint=1T​∑t=1Tpθ​\(yt∣y<t,x\),c\_\{\\text\{int\}\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}p\_\{\\theta\}\(y\_\{t\}\\mid y\_\{<t\},x\),\(1\)whereyty\_\{t\}is thett\-th generated token andxxthe multimodal input\. This is continuous in\(0,1\]\(0,1\]and requires no additional forward passes\.

### 3\.5Answer matching

Generated answers rarely match gold labels exactly\. We normalise case and punctuation, accept exact matches, and fall back to unique containment \(the prediction contains exactly one option string\)\. The match method is recorded per prediction for transparency\.

### 3\.6Metrics

We report accuracy, Expected Calibration Error\(Naeiniet al\.,[2015](https://arxiv.org/html/2607.22034#bib.bib14)\)with 10 bins, Brier score\(Brier,[1950](https://arxiv.org/html/2607.22034#bib.bib15)\), anderror\-detection AUROC– the probability that a randomly chosen correct answer receives higher confidence than a randomly chosen incorrect one\. AUROC is our primary metric because it directly measures whether a confidence signal is usable for deferral\. Confidence intervals are percentile bootstrap withB=2000B=2000resamples\.

## 4Results

### 4\.1Accuracy is robust except under severe low light

Qwen2\-VL achieves 0\.99 clean accuracy and remains at 0\.87–1\.00 across five of six degradation families at all severities\. The exception is low light:0\.99→0\.920\.99\\rightarrow 0\.92\(s2\)→\\rightarrow0\.22\(s3\), approaching the 0\.25 chance rate for four\-option multiple choice\. SmolVLM shows the same pattern at slightly lower absolute accuracy: 0\.97 clean, 0\.74–0\.97 across most conditions, collapsing to0\.42at low light s3\.

![Refer to caption](https://arxiv.org/html/2607.22034v1/fig2_accuracy_both.png)Figure 1:Accuracy as a function of corruption severity for both models\. Five of six degradation families produce little accuracy loss; low light \(green\) collapses toward the four\-way chance rate \(dotted red\) in both models\. The replication across two independent model families is the point of this figure\.#### Finding 1\.

*Accuracy in small VLMs is largely insensitive to compression, blur, glare, tilt and resampling, but collapses under severe underexposure\. This replicates across both models\.*

### 4\.2Verbalized confidence is constant and uninformative

For Qwen2\-VL, mean verbalized confidence ranges from 0\.87 to 0\.90 across all 19 conditions\. It does not respond to severity, degradation family, or accuracy\. At low light s3 – accuracy 0\.22 – the model still states 0\.87\. Table[2](https://arxiv.org/html/2607.22034#S4.T2)gives error\-detection AUROC with 95% bootstrap intervals\.

#### Finding 2\.

*Verbalized confidence in Qwen2\-VL is effectively a constant and carries almost no information about correctness\.*

### 4\.3Internal confidence detects errors

Table 2:Qwen2\-VL\-2B: error\-detection AUROC for verbalized versus internal confidence, with 95% bootstrap confidence intervals \(B=2000B=2000,n≈100n\\approx 100per condition\)\.In clean, low\_light s1, resample s1 and rotation s3 the two intervals do not overlap: the difference is statistically robust at this sample size\.

![Refer to caption](https://arxiv.org/html/2607.22034v1/fig1_auroc_comparison.png)Figure 2:Error\-detection AUROC for verbalized \(red\) versus internal \(blue\) confidence in Qwen2\-VL\-2B, across all conditions where AUROC is defined\. Dashed line marks chance \(0\.5\)\. Verbalized confidence sits at or near chance in most conditions and falls*below*it under low light s1; internal confidence from the same predictions ranges 0\.82–0\.99\. The two JPEG conditions where verbalized AUROC reaches 1\.00 have near\-perfect accuracy and correspondingly few errors, so those values rest on a handful of points and should not be read as evidence that verbalized confidence works there\.#### Finding 3 \(headline\)\.

*The same model that cannot state a useful confidence number nonetheless encodes a strong internal signal of its own correctness\.*

### 4\.4The finding replicates in SmolVLM, more weakly

SmolVLM internal AUROC is above 0\.5 in all 19 conditions, ranging 0\.54–0\.96: clean 0\.77 \[0\.58, 1\.00\], low\_light s1 0\.96 \[0\.92, 1\.00\], rotation s2 0\.92 \[0\.82, 1\.00\], resample s1 0\.81 \[0\.60, 0\.98\]\. The direction replicates but the effect is weaker and noisier than in Qwen2\-VL\. Several SmolVLM intervals include or approach 0\.5 – glare s1 0\.63 \[0\.32, 0\.94\], jpeg s1 0\.62 \[0\.33, 0\.98\] – and are thereforenot distinguishable from chance at this sample size\. We report these as inconclusive rather than positive\.

#### Finding 4\.

*Internal confidence provides above\-chance error detection in both models, but its strength is model\-dependent: strong and consistent in Qwen2\-VL \(0\.92–0\.99\), weaker and more variable in SmolVLM \(0\.54–0\.92\)\.*

### 4\.5Both signals fail under severe low light

Table 3:Both models fail together under severe underexposure\.For Qwen2\-VL, accuracy falls 0\.770 while verbalized confidence falls 0\.029 and internal confidence falls 0\.057\. Internal confidence is roughly twice as responsive, but both are negligible against the accuracy drop\. Both models’ internal AUROC intervals include 0\.5 at this severity\.

![Refer to caption](https://arxiv.org/html/2607.22034v1/fig3_divergence.png)Figure 3:Qwen2\-VL\-2B under increasing low light\. Accuracy \(black\) falls from 0\.99 to 0\.22 while verbalized confidence \(red\) moves 0\.029 and internal confidence \(blue\) moves 0\.057\. Internal confidence is the more responsive of the two, but the gap that opens between accuracy and both signals is the failure mode this paper cautions against\.#### Finding 5\.

*Neither signal tracks catastrophic failure\. Internal confidence is a useful deferral signal under mild\-to\-moderate degradation and loses that property precisely under the severe degradation where it would matter most\. This replicates across both models\.*

### 4\.6Calibration error hides the difference between the two signals

Expected Calibration Error is low for both signals in most conditions \(roughly 0\.02–0\.12\), which follows mechanically from the model being both accurate and confident nearly everywhere\. ECE rises sharply only at low light s3: 0\.650 for verbalized and 0\.606 for internal confidence\.

We report ECE for comparability with prior VLM calibration work, but in this setting it is the*less*informative metric\. Because verbalized confidence is nearly constant, its ECE is dominated by the accuracy of the condition rather than by any property of the confidence signal itself\. Figure[4](https://arxiv.org/html/2607.22034#S4.F4)makes this concrete: on clean images the two signals are visually indistinguishable in a reliability diagram, despite an error\-detection AUROC gap of more than 0\.4 on the very same predictions\.

![Refer to caption](https://arxiv.org/html/2607.22034v1/fig4_reliability.png)Figure 4:Reliability diagrams for both confidence signals on clean images \(Qwen2\-VL\-2B\)\. Both distributions are concentrated in\[0\.8,1\.0\]\[0\.8,1\.0\]and both sit near the diagonal, because the model is 99% accurate here and both signals are correspondingly high\.The reliability diagram cannot distinguish the two signals, even though their error\-detection AUROC differs by more than 0\.4 \(0\.49 versus 0\.92\)\. We include this as a methodological caution: calibration\-error measures assess whether confidence*magnitude*matches accuracy, not whether confidence*discriminates*correct from incorrect answers\. A signal with near\-zero variance can appear well calibrated while carrying no usable information\.Error\-detection AUROC is the more diagnostic measure in this regime, and we recommend it for future work on verbalized confidence in small VLMs\.

### 4\.7Post\-hoc temperature scaling

In a single\-signal run we fitted a temperature parameter on a held\-out clean split \(T=0\.50T=0\.50\) and applied it unchanged to all degraded conditions\. ECE improved substantially in most conditions \(clean0\.090→0\.0020\.090\\rightarrow 0\.002; jpeg s10\.107→0\.0020\.107\\rightarrow 0\.002; low\_light s10\.103→0\.0070\.103\\rightarrow 0\.007; rotation s30\.090→0\.0020\.090\\rightarrow 0\.002\) butworsenedat low light s3,0\.650→0\.7560\.650\\rightarrow 0\.756\. The interpretation is mechanical: rescaling a near\-constant signal redistributes a fixed value and cannot recover information the signal never contained\.

## 5Discussion

### 5\.1Why the two signals diverge

Verbalized confidence requires the model to introspect and then serialise that introspection into a number\. This is a learned surface behaviour, and in small instruction\-tuned models it appears to collapse to a stock response – 0\.90 in almost every case for Qwen2\-VL\. Internal token probability requires no introspection: it is a direct read of the model’s own distribution and cannot be produced by imitation of a training pattern in the same way\. The practical implication is that the model has more self\-knowledge than its text output reveals\.

### 5\.2What practitioners should do

For deployment on small open\-weight VLMs,do not use stated confidence for deferral\. It is at chance in one of our models and largely unobtainable in the other\. Use mean token probability instead: it is free to compute, requires no retraining, and provides usable error detection under mild\-to\-moderate degradation\. However,do not treat it as a safety guarantee under low light\. Both models retained high internal confidence while accuracy collapsed\. A deployment that may encounter severe underexposure needs an explicit image\-quality check upstream of the model, not a confidence threshold downstream of it\.

### 5\.3Verbalized confidence may be unobtainable, not merely poor

SmolVLM did not simply give poor confidence numbers – it largely refused to give them\. Across three prompt templates in a five\-item pilot, only one attempt produced a parseable value; the model returned bare answers \(“Takoyaki\.”, “Pho\.”\) regardless of instruction\. We report this as a qualitative observation rather than a measured rate, and note it as a limitation: a more extensive prompt search might succeed where ours did not\.

### 5\.4Limitations

1. 1\.Undefined AUROC cells\.Conditions at or near 1\.00 accuracy \(motion\_blur s1, rotation s1 for Qwen2\-VL\) contain too few errors for AUROC to be defined\. These are reported as n/a and excluded from aggregates\. They are*not*evidence of good calibration\.
2. 2\.Sample size\.n≈100n\\approx 100per condition\. Several SmolVLM intervals span 0\.3–0\.4 in width and are uninformative\. A largernnis the single highest\-value extension of this work\.
3. 3\.One dataset\.All results are on Food101 posed as four\-way multiple choice\. Generality to open\-ended VQA and other domains is untested\.
4. 4\.Prompt sensitivity\.Verbalized confidence was elicited with one template per model, chosen by pilot\. The near\-constant 0\.90 in Qwen2\-VL may be partly template\-induced\. An elicitation ablation is required before the claim can be stated in full generality\.
5. 5\.Synthetic degradations\.Our corruptions approximate phone\-camera artifacts programmatically; they are not photographs actually taken under those conditions\.
6. 6\.Two models, one size class\.Both models are≈\\approx2B parameters\. Whether the verbalized/internal gap narrows with scale is untested and is the most interesting open question this work raises\.
7. 7\.No refusal behaviour\.Our multiple\-choice format forces a response and therefore cannot capture refusal, whichBorszukovszkiet al\.\([2025](https://arxiv.org/html/2607.22034#bib.bib1)\)identify as a meaningful calibration mechanism: in their open\-ended setting, models that declined to answer under severe corruption achieved better calibration than those that guessed\. Whether small open\-weight models refuse under degradation, and whether refusal correlates with low internal confidence, is an open question our design cannot address\.

## 6Conclusion

Across 3,800 predictions from two small open\-weight vision\-language models under six realistic photographic degradations, we find that verbalized confidence is close to useless – constant and chance\-level in Qwen2\-VL\-2B, largely unobtainable in SmolVLM – while internal token probability from the same predictions detects errors with AUROC up to 0\.99\. Small VLMs appear to encode substantially more self\-knowledge than their natural\-language output expresses\. Both models nonetheless fail together under severe underexposure, where accuracy collapses and neither signal responds\. We recommend internal token probability over stated confidence as a deferral signal in constrained deployment, coupled with an explicit upstream image\-quality check\.

## 7Reproducibility

All code, configurations, per\-prediction CSVs, and figures are available at[https://github\.com/Asif\-Ferdous/vlm\-reliability](https://github.com/Asif-Ferdous/vlm-reliability)\. Experiments run on a single free\-tier NVIDIA T4; the full study reproduces in approximately 90 minutes of GPU time\. Random seeds are fixed throughout\.

## References

- M\. Borszukovszki, I\. P\. de Jong, and M\. Valdenegro\-Toro \(2025\)Know what you do not know: verbalized uncertainty estimation robustness on corrupted images in vision\-language models\.arXiv preprint arXiv:2504\.03440\.Cited by:[§2\.2](https://arxiv.org/html/2607.22034#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2607.22034#S2.SS2.p5.1),[§2\.2](https://arxiv.org/html/2607.22034#S2.SS2.p6.1),[§2\.4](https://arxiv.org/html/2607.22034#S2.SS4.p1.1),[item 7](https://arxiv.org/html/2607.22034#S5.I1.i7.p1.1)\.
- L\. Bossard, M\. Guillaumin, and L\. Van Gool \(2014\)Food\-101 – mining discriminative components with random forests\.InComputer Vision – ECCV 2014,Lecture Notes in Computer Science, Vol\.8694,pp\. 446–461\.Cited by:[§3\.2](https://arxiv.org/html/2607.22034#S3.SS2.p1.1)\.
- G\. W\. Brier \(1950\)Verification of forecasts expressed in terms of probability\.Monthly Weather Review78\(1\),pp\. 1–3\.Cited by:[§3\.6](https://arxiv.org/html/2607.22034#S3.SS6.p1.1)\.
- T\. Groot and M\. Valdenegro\-Toro \(2024\)Overconfidence is key: verbalized uncertainty evaluation in large language and vision\-language models\.arXiv preprint arXiv:2405\.02917\.Cited by:[§2\.1](https://arxiv.org/html/2607.22034#S2.SS1.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational Conference on Machine Learning \(ICML\),pp\. 1321–1330\.Cited by:[§2\.3](https://arxiv.org/html/2607.22034#S2.SS3.p2.1)\.
- D\. Hendrycks and T\. Dietterich \(2019\)Benchmarking neural network robustness to common corruptions and perturbations\.arXiv preprint arXiv:1903\.12261\.Cited by:[§2\.3](https://arxiv.org/html/2607.22034#S2.SS3.p1.1)\.
- Hugging Face TB \(2024\)SmolVLM\.Note:[https://huggingface\.co/HuggingFaceTB/SmolVLM\-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct)Cited by:[§3\.1](https://arxiv.org/html/2607.22034#S3.SS1.p1.1)\.
- C\. Michaelis, B\. Mitzkus, R\. Geirhos, E\. Rusak, O\. Bringmann, A\. S\. Ecker, M\. Bethge, and W\. Brendel \(2019\)Benchmarking robustness in object detection: autonomous driving when winter is coming\.arXiv preprint arXiv:1907\.07484\.Cited by:[§2\.2](https://arxiv.org/html/2607.22034#S2.SS2.p5.1),[§2\.3](https://arxiv.org/html/2607.22034#S2.SS3.p1.1)\.
- M\. P\. Naeini, G\. F\. Cooper, and M\. Hauskrecht \(2015\)Obtaining well calibrated probabilities using bayesian binning\.Proceedings of the AAAI Conference on Artificial Intelligence29\(1\),pp\. 2901–2907\.Cited by:[§3\.6](https://arxiv.org/html/2607.22034#S3.SS6.p1.1)\.
- Y\. Ovadia, E\. Fertig, J\. Ren, Z\. Nado, D\. Sculley, S\. Nowozin, J\. Dillon, B\. Lakshminarayanan, and J\. Snoek \(2019\)Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift\.Advances in Neural Information Processing Systems32\.Cited by:[§2\.3](https://arxiv.org/html/2607.22034#S2.SS3.p2.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.arXiv preprint arXiv:2305\.14975\.Cited by:[§2\.1](https://arxiv.org/html/2607.22034#S2.SS1.p1.1)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, Y\. Fan, K\. Dang, M\. Du, X\. Ren, R\. Men, D\. Liu, C\. Zhou, J\. Zhou, and J\. Lin \(2024\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§3\.1](https://arxiv.org/html/2607.22034#S3.SS1.p1.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2024\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2607.22034#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.22034#S2.SS2.p2.1)\.

## Appendix AFull results tables

Tables[4](https://arxiv.org/html/2607.22034#A1.T4)and[5](https://arxiv.org/html/2607.22034#A1.T5)give the complete per\-condition results for both models\. Values are transcribed fromsummary\_with\_CIs\.csvandsummary\_smolvlm\.csv; brackets give 95% percentile bootstrap intervals \(B=2000B=2000\)\. Cells marked n/a have insufficient errors for AUROC to be defined\.

Table 4:Qwen2\-VL\-2B\-Instruct: complete results, all 19 conditions\.Table 5:SmolVLM\-Instruct: complete results, all 19 conditions\. Verbalized confidence was not reliably obtainable \(Section[5\.3](https://arxiv.org/html/2607.22034#S5.SS3)\), so only internal confidence is reported\.Table 6:Post\-hoc temperature scaling on Qwen2\-VL verbalized confidence\. A single temperatureT=0\.50T=0\.50was fitted on a held\-out clean split and applied unchanged to all degraded conditions\. Note the failure at low light s3\.
## Appendix BPrompt templates

All templates are reproduced verbatim\.\{question\}and\{options\}are substituted at runtime; options are presented as a comma\-separated list\.

#### Qwen2\-VL\-2B\-Instruct \(selected, 5/5 parse rate\)\.

A few\-shot template with an explicit worked example:

```
You must reply in exactly two lines.

Example reply:
Answer: pizza
Confidence: 85

Now do the same for this image.
Question: {question}
Options: {options}
```

#### Qwen2\-VL rejected candidates\.

Two alternatives were piloted on the same five items and discarded\. Template A \(0/5 parse rate\) returned bare numbers such as100with no answer field:

```
Question: {question}
Options: {options}

Answer the question, then rate your confidence from 0 to 100.
Use exactly this format and nothing else:
Answer: <option>
Confidence: <number>
```

Template C \(0/5\) returned bare labels such asTakoyakiwith no confidence field:

```
{question}
Choose one: {options}

Reply with two lines only:
Answer: (your choice)
Confidence: (0-100)
```

#### SmolVLM\-Instruct candidates \(best 1/5\)\.

All three templates below were piloted; the model returned bare answers \(“Takoyaki\.”, “Cheese plate\.”, “Pho\.”\) without a confidence field in all but one of fifteen attempts\.

```
(A)  {question}
     Options: {options}

     Answer: <one option>
     Confidence: <0-100>

(B)  Look at the image.
     {question}
     Options: {options}

     Give your answer and how sure you are.
     Answer: pizza
     Confidence: 85

     Now yours:
     Answer:

(C)  {question}
     Choose from: {options}
     Then on a NEW line write Confidence: followed by a number 0-100.
     Answer:
```

#### SmolVLM final template \(internal confidence only\)\.

Because no template reliably elicited verbalized confidence, SmolVLM was run with a minimal prompt and evaluated on internal token probability alone:

```
{question}
Options: {options}
Answer:
```

#### Parsing\.

Answers were extracted with the regular expressionAnswer:\\s\*\(\.\+\)and confidences withConfidence:\\s\*\(\[0\-9\]\{1,3\}\), the latter normalised by dividing by 100\. Responses without a parseable confidence were recorded and counted toward the per\-condition parse\-failure rate reported in Section[4](https://arxiv.org/html/2607.22034#S4)\.