HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide
Summary
This paper introduces HalluPrism, a behavioral diagnostic method for multimodal large language models that uses visual perturbation probes to identify hallucination failure modes, improving failure-family classification over confidence-only methods.
View Cached Full Text
Cached at: 09/01/26, 01:10 PM
# 1Introduction
Source: [https://arxiv.org/html/2608.29193](https://arxiv.org/html/2608.29193)
\\papertitle
HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide\\papershorttitleHalluPrism\\papershortauthorsA\. Prakash, S\. Dasgupta, and T\. Chakraborty\\paperauthorsAman Prakash1Sourish Dasgupta2Tanmoy Chakraborty1\\paperaffil1LCS2 Lab, Indian Institute of Technology Delhi, New Delhi, India 2KDM Lab, Dhirubhai Ambani University, Gujarat, India\\paperemailsamanprakash\.connect@gmail\.com, sourish\_dasgupta@dau\.ac\.in, tanchak@iitd\.ac\.in\\paperkeywordsMultimodal Large Language Models, Multimodal Uncertainty, Hallucination Diagnosis, Behavioral Probing, Cross\-Modal Grounding, Selective Prediction\\paperabstractMultimodal Large Language Models \(MLLMs\) can assign similar confidence to answers that fail for different reasons\. We proposeHalluPrism, a behavioral diagnostic that re\-runs an answer after visual degradation, blank\-image replacement, and grounding or relation checks\. These targeted probes yield a signature over visual\-perturbation sensitivity\(V\)\(V\), image\-removal confidence retention\(L\)\(L\), and grounding/relation\-probe instability\(A\)\(A\)\. Across 58K\+ examples from four benchmarks and four MLLMs, image\-removal confidence retention is most prevalent, while grounding/relation\-probe instability better separates failure families\. Only 18 of 48 source\-target checks are diagonally aligned, so the coordinates should be interpreted jointly rather than as independent causal sources\. With the dataset fixed, the joint signature improves failure\-family AUROC from0\.6340\.634to0\.7690\.769on HallusionBench and from0\.7070\.707to0\.8170\.817on VizWiz, with smaller gains on POPE and VSR\. In pooled XGBoost analysis, AUROC rises from0\.780\.78with scalar confidence to0\.950\.95with\(V,L,A\)\(V,L,A\)and0\.970\.97when confidence is added\. The same signature does not automatically improve correctness ranking\. The three tested direct scalarizations can harm it\. These results separate failure diagnosis from abstention scoring: multimodal uncertainty should characterize failure structure before it is used to decide whether to abstain or correct\.\\appendixtocon\\appendixtocnameAppendix Contents\\makelabtitle
## 1Introduction
A visually\-grounded answer can be unsafe even when a Multimodal Large Language Model \(MLLM\) assigns high confidence to it\. Consider a street\-scene image where a traffic cone is to the right of a parked car, and the user asks, “Is the traffic cone to the left of the car?” The model may answer “yes” with high confidence\. This single confident answer can hide different risks\. The cone may be small, blurred, or partly occluded, suggesting weak visual evidence\. The model may still answer “yes” after the image is replaced with a blank canvas, showing answer persistence without the image, which may be consistent with language\-prior reliance on the question alone\. It may also recognize both the cone and the car while reversing the left\-right relation, suggesting cross\-modal binding failure\. In such cases, while scalar uncertainty can help decide whether the model should abstain, it cannot explain the failure source\.
Existing hallucination benchmarks already show that multimodal errors are heterogeneous, covering object hallucination, visual illusion, attribute error, spatial error, and answerability failure\[[29](https://arxiv.org/html/2608.29193#bib.bib4),[16](https://arxiv.org/html/2608.29193#bib.bib5),[40](https://arxiv.org/html/2608.29193#bib.bib6),[32](https://arxiv.org/html/2608.29193#bib.bib32)\]\. Confidence, calibration, and selective prediction methods usually rank likely correctness or decide whether the model should abstain\[[4](https://arxiv.org/html/2608.29193#bib.bib37),[22](https://arxiv.org/html/2608.29193#bib.bib9),[25](https://arxiv.org/html/2608.29193#bib.bib10),[12](https://arxiv.org/html/2608.29193#bib.bib11)\], while recent MLLM uncertainty work moves closer to hallucination detection through evidential conflict, hidden\-state reliability, and uncertain visual tokens\[[20](https://arxiv.org/html/2608.29193#bib.bib24),[34](https://arxiv.org/html/2608.29193#bib.bib25),[38](https://arxiv.org/html/2608.29193#bib.bib27)\]\. However, scalar abstention is not an internal model guarantee\. It is rather a deployment guardrail whose validity depends on designer\-chosen scores, thresholds, validation sets, and risk assumptions that may fail for new open\-world instances\. We, therefore, argue that diagnosis should be the core object of multimodal uncertainty — before answering or correcting, the system should characterizewhether behavior is consistent with visual sensitivity, image\-removal confidence retention, or cross\-modal binding instability\.
Figure 1:An overview of theHalluPrismperturbation\-response protocol\. HereLLdenotes image\-removal confidence retention; language\-prior reliance is only a possible interpretation of this response\. The scores are behavioral probes, not a causal decomposition\. Full prompt templates and probe parameters are given in Appendix[9](https://arxiv.org/html/2608.29193#S9)\.We proposeHalluPrism, a perturbation\-response diagnostic for MLLM uncertainty\. Given an image, a prompt, and a generated answer,HalluPrismcomputes scalar confidenceccand a perturbation\-response signatures\(x\)=\(V,L,A\)s\(x\)=\(V,L,A\)for visual\-perturbation sensitivity\(V\)\(V\), image\-removal confidence retention\(L\)\(L\), and grounding/relation\-probe instability\(A\)\(A\)\. In the traffic\-cone example,VVincreases when blur, crop, darkness, or noise changes the answer or its confidence,LLincreases when confidence is retained relative to the clean\-image prediction after blank\-image replacement, andAAincreases when grounding or relation perturbations expose instability in the left\-right binding\. We additionally use a stricter image\-removal control that requires the original answer itself to persist after image removal\. The signatures\(x\)s\(x\)is a perturbation\-response profile, not a decomposition of hidden mechanisms\. Hence, a highLLdoes not prove that the model never used the image, while highVVorAArecords sensitivity rather than an isolated internal cause\.
We useHalluPrismas ajointbehavioral signature for diagnosis and correction\-path selection\. It first acts as a diagnostic feature for failure\-family prediction\. It then selects a correction path for risky predictions: high visual\-perturbation sensitivity selectsvisual re\-groundingof the cone and car, high image\-removal confidence retention selectsanti\-prior promptingthat asks the model to check the image rather than rely on question\-only plausibility, and high grounding/relation\-probe instability selectsrelation checkingof whether the cone is to the left or right of the car\. We call this signature\-conditioned choice of correction promptintervention routing\.
Across58,44058\{,\}440examples from Gemma 4, LLaVA\-Mistral/Vicuna, and Qwen3\-VL on HallusionBench, POPE, VSR, and VizWiz\-VQA, we find four patterns\. First, raw prevalence does not have diagnostic value\. AlthoughLLdominates99\.7%99\.7\\%of samples,AAis seen to be the strongest tree\-based feature for failure\-family prediction\. Second, the scores are not independent labels\. IfVV,LL, andAAwere indeed source\-specific, then visual, language, and alignment\-ruined inputs would have mainly moved corresponding matching coordinates\. However, across4×4×34\\times 4\\times 3ruined\-condition checks, only18/4818/48such events pass\. Visual inputs pass in11/1611/16settings, while language and alignment inputs pass in2/162/16and5/165/16\. Thus,\(V,L,A\)\(V,L,A\)is a joint Perturbation\-Response Signature, not independent causes\. Third, this joint signature remains highly diagnostic\. With the dataset fixed,\(V,L,A\)\(V,L,A\)improves AUROC over scalar confidence from0\.6340\.634to0\.7690\.769on HallusionBench and from0\.7070\.707to0\.8170\.817on VizWiz, with smaller gains on POPE and VSR\. In the pooled XGBoost analysis, AUROC improves from0\.780\.78with scalar confidence to0\.950\.95with\(V,L,A\)\(V,L,A\), and to0\.970\.97with confidence added\.
Fourth, as a secondary application, the signature guides safer correction\. Matched routing selects visual re\-grounding, anti\-prior prompting, or relation checking, instead of one generic prompt for all examples\. On Qwen3\-VL/VSR, generic prompting drops accuracy from87\.4%87\.4\\%to76\.2%76\.2\\%and breaks15\.5%15\.5\\%of correct answers\. Matched routing maintains87\.6%87\.6\\%accuracy and lowers breakage to2\.7%2\.7\\%\.
Contributions\.\(i\) We argue that perturbation\-response diagnosis should be the core of multimodal uncertainty\. \(ii\) We proposeHalluPrism, a perturbation\-response signature over visual\-perturbation sensitivity, image\-removal confidence retention, and grounding/relation\-probe instability\. \(iii\) We show that controlled probes expose interventional entanglement rather than clean source separability\. \(iv\) We show that this entangled signature improves failure\-family diagnosis, while naive scalarization can hurt abstention ranking\.111The source code is available at[https://github\.com/gitgoap/HalluPrism](https://github.com/gitgoap/HalluPrism)\.
## 2Related Work
Hallucination Evaluation\.Vision\-language hallucination work shows that multimodal errors are heterogeneous\. Captioning and Visual Question Answering connect object hallucination and visual mis\-reasoning to language priors and weak grounding\[[36](https://arxiv.org/html/2608.29193#bib.bib2),[15](https://arxiv.org/html/2608.29193#bib.bib3)\]\. Benchmarks such as POPE, HallusionBench, AMBER, MME, and MMBench make this explicit across object presence, visual illusion, attributes, relations, perception, reasoning, and instruction following\[[29](https://arxiv.org/html/2608.29193#bib.bib4),[16](https://arxiv.org/html/2608.29193#bib.bib5),[40](https://arxiv.org/html/2608.29193#bib.bib6),[8](https://arxiv.org/html/2608.29193#bib.bib31),[32](https://arxiv.org/html/2608.29193#bib.bib32)\]\.HalluPrisminstead measures a behavioral Perturbation\-Response Signature over visual\-perturbation sensitivity, image\-removal confidence retention, and grounding/relation\-probe instability\.
##### Uncertainty and Abstention\.
Reject\-option and selective\-classification work treats uncertainty as a decision object\. A model answers when risk is low and abstains otherwise\[[4](https://arxiv.org/html/2608.29193#bib.bib37),[10](https://arxiv.org/html/2608.29193#bib.bib38),[11](https://arxiv.org/html/2608.29193#bib.bib39)\]\. Calibration asks whether confidence matches empirical correctness\[[17](https://arxiv.org/html/2608.29193#bib.bib36)\]\. Language\-model uncertainty uses self\-estimation, token probabilities, semantic equivalence, and sample inconsistency as reliability signals\[[22](https://arxiv.org/html/2608.29193#bib.bib9),[25](https://arxiv.org/html/2608.29193#bib.bib10),[12](https://arxiv.org/html/2608.29193#bib.bib11),[33](https://arxiv.org/html/2608.29193#bib.bib12)\]\. Multimodal uncertainty adds visual abstention, evidential conflict, hidden\-state reliability, and visual\-token uncertainty\[[5](https://arxiv.org/html/2608.29193#bib.bib13),[39](https://arxiv.org/html/2608.29193#bib.bib14),[20](https://arxiv.org/html/2608.29193#bib.bib24),[34](https://arxiv.org/html/2608.29193#bib.bib25),[38](https://arxiv.org/html/2608.29193#bib.bib27)\]\. Our critique is narrower\. Abstention has statistical value under its assumptions\. However, a thresholded score is warranted by validation transfer and need not diagnose risk in a new open\-world instance\.
##### Probing and Attribution\.
Behavioral testing studies model behavior through controlled input changes\. CheckList, contrast sets, and counterfactually augmented data expose local failures under small edits\[[35](https://arxiv.org/html/2608.29193#bib.bib40),[9](https://arxiv.org/html/2608.29193#bib.bib41),[23](https://arxiv.org/html/2608.29193#bib.bib42)\]\. Attribution methods instead use visual\-region localization, attention flow, or image\-token attention analysis\[[37](https://arxiv.org/html/2608.29193#bib.bib15),[21](https://arxiv.org/html/2608.29193#bib.bib16),[1](https://arxiv.org/html/2608.29193#bib.bib17),[13](https://arxiv.org/html/2608.29193#bib.bib18)\]\.HalluPrismfollows the behavioral\-testing view\. It asks how the output changes under pressure on visual evidence, question\-only priors, or cross\-modal binding\.
##### Mitigation and Routing\.
Hallucination mitigation often applies a fixed correction rule during decoding or after generation\. Visual Contrastive Decoding, OPERA, M3ID, Woodpecker, and Volcano reduce hallucination through visual contrast, decoding penalties, image\-grounded amplification, or post\-hoc revision\[[27](https://arxiv.org/html/2608.29193#bib.bib19),[19](https://arxiv.org/html/2608.29193#bib.bib20),[6](https://arxiv.org/html/2608.29193#bib.bib21),[41](https://arxiv.org/html/2608.29193#bib.bib22),[26](https://arxiv.org/html/2608.29193#bib.bib23)\]\. Recent inference\-time methods refine this direction through inter\-modality calibration, attention calibration, and visual\-aware sparsification\[[28](https://arxiv.org/html/2608.29193#bib.bib34),[7](https://arxiv.org/html/2608.29193#bib.bib26),[42](https://arxiv.org/html/2608.29193#bib.bib28)\]\.HalluPrismis diagnosis\-first\. It profiles a risky prediction and then selects a matched prompt\-based correction\.
## 3Methodology
We study multimodal uncertainty as apre\-decisiondiagnostic object\. The goal is not to build a better scalar abstention score\. We instead test whether a prediction carries a behavioral Perturbation\-Response Signature that can diagnose failure family and route correction without assuming clean causal separability\. The design therefore separates three objects that are often conflated: correctness ranking, failure\-source diagnosis, and correction\-path choice\. This separation is important because the same Perturbation\-Response Signature can be useful for explaining a failure, can be harmful under the direct scalarizations tested here, and useful again when choosing a safer correction path\. A diagnostic representation can therefore improve failure typing without necessarily improving an answer\-vs\.\-abstain score\.
All cited formal results appear in Appendix[7](https://arxiv.org/html/2608.29193#S7)\.
##### Prediction and Perturbation\-Response Signatures\.
Consider an imageIIwhere a traffic cone is to the right of a parked car, and a promptqqasks whether the cone is to the left of the car\. If an MLLMMManswers “yes” tox=\(I,q\)x=\(I,q\)while the reference answeryyis “no’’, the output is incorrect222We normalize answers before comparison, mapping surface variants such as “yes”, “Yes\.”, or “yes, it is” to the same canonical task label\.\. We write correctness asz\(x,M\)=𝟏\[norm\(y^\)=norm\(y\)\]z\(x,M\)=\\mathbf\{1\}\[\\mathrm\{norm\}\(\\hat\{y\}\)=\\mathrm\{norm\}\(y\)\]\. Incorrect outputs receive a coarse failure\-family labelf\(x,M\)∈ℱf\(x,M\)\\in\\mathcal\{F\}, such asobject\-hallucination,spatial/relation\-error,attribute\-error, oranswerability\-failure\. A scalar confidence estimator maps the input\-output instance toc\(x,M\)∈\[0,1\]c\(x,M\)\\in\[0,1\]\. For a tokenized answery^=\(t1,…,tN\)\\hat\{y\}=\(t\_\{1\},\\ldots,t\_\{N\}\), we use the length\-normalized confidence probability:
c\(y^∣I,q\)=exp\(1N∑i=1Nlogp\(ti∣t<i,I,q\)\)\.\\begin\{split\}&c\(\\hat\{y\}\\mid I,q\)=\\exp\\left\(\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log p\(t\_\{i\}\\mid t\_\{<i\},I,q\)\\right\)\.\\end\{split\}\(1\)For answerability\-focused datasets, we use the dataset\-specific normalized answer probability\. This scalar score ranks likely correctness\. However, it does not say whether the answer depends on weak visual evidence, survives without the image, or fails because the image\-text relation is misbound\. Hence, scalar confidence is a correctness\-ranking surrogate, not a grounded reliability certificate\. This non\-identifiability is formalized in Proposition[1](https://arxiv.org/html/2608.29193#Thmproposition1)\. We attach a\[0,1\]\[0,1\]\-bounded Perturbation\-Response Signature to the same prediction:
s\(x,M\)=\(V\(x,M\),L\(x,M\),A\(x,M\)\)\.\\begin\{split\}&s\(x,M\)=\(V\(x,M\),L\(x,M\),A\(x,M\)\)\.\\end\{split\}\(2\)HereVVmeasures visual\-perturbation sensitivity,LLmeasures image\-removal confidence retention, andAAmeasures grounding/relation\-probe instability\. These scores are behavioral probe responses, not hidden causal mechanisms\. They remain useful because failure families may occupy different regions of the joint\(V,L,A\)\(V,L,A\)space\.
##### Perturbation Probes\.
We measure each score by asking how the same prediction behaves under controlled input pressure\. The clean answery^\\hat\{y\}is the baseline\. A probe changes one part of the input and producesy~\\tilde\{y\}\. We compare answers withflip\(y^,y~\)\\mathrm\{flip\}\(\\hat\{y\},\\tilde\{y\}\), which equals11when the answer changes, and confidence shifts withΔc\(y^,y~\)=\|c\(y^\)−c\(y~\)\|\\Delta c\(\\hat\{y\},\\tilde\{y\}\)=\|c\(\\hat\{y\}\)\-c\(\\tilde\{y\}\)\|\. Flips capture discrete instability\. Confidence shifts capture softer changes\. Thevisual scoreVVmeasures sensitivity to degraded visual evidence\. We keep the prompt fixed and use Gaussian blur \(σ=15\\sigma\\\!=\\\!15\), center crop \(50%50\\%per dimension, resized back\), reduce brightness to30%30\\%of original image, and Gaussian noise \(μ=0,σ=25\\mu\\\!=\\\!0,\\sigma\\\!=\\\!25\)\. For thekk\-th visual perturbation answery~kv\\tilde\{y\}^\{v\}\_\{k\}:
V=0\.6⋅1K∑k=1K=4flip\(y^,y~vk\)\+0\.4⋅min\(2⋅1K∑k=1K=4Δc\(y^,y~vk\),1\)\.\\begin\{split\}&V=0\.6\\cdot\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K=4\}\\mathrm\{flip\}\(\\hat\{y\},\\tilde\{y\}^\{v\}\_\{k\}\)\\\\ &\\qquad\+0\.4\\cdot\\min\\left\(2\\cdot\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K=4\}\\Delta c\(\\hat\{y\},\\tilde\{y\}^\{v\}\_\{k\}\),1\\right\)\.\\end\{split\}\(3\)The larger flip weight makes changed answers the primary signal of visual fragility\. Theimage\-removal scoreLLis operationalized as confidence retention after removing image evidence\. We replace the image with a uniform blank imageIblankI\_\{\\mathrm\{blank\}\}and keep the prompt fixed\. Lety~ℓ\\tilde\{y\}^\{\\ell\}be the blank\-image answer\. We define:
L≡Lconf=min\(c\(y~ℓ\)c\(y^\)\+ϵ,1\)\.\\begin\{split\}&L\\equiv L\_\{\\mathrm\{conf\}\}=\\min\\left\(\\frac\{c\(\\tilde\{y\}^\{\\ell\}\)\}\{c\(\\hat\{y\}\)\+\\epsilon\},1\\right\)\.\\end\{split\}\(4\)
A highL=LconfL=L\_\{\\mathrm\{conf\}\}means that confidence is largely retained after image removal, even when the blank\-image answer differs from the clean answer\. It is a behavioral marker of confidence retention, not a causal claim that the model ignores the image\. Since Eq\.[4](https://arxiv.org/html/2608.29193#S3.E4)can saturate, we use the full\(V,L,A\)\(V,L,A\)vector for diagnosis\. Appendix[10\.1](https://arxiv.org/html/2608.29193#S10.SS1)reports answer\-persistence, match\-only, and denominator\-stable controls\.
Lpersist=𝟏\[match\(y^,y~ℓ\)\]⋅Lconf\.\\begin\{split\}&L\_\{\\mathrm\{persist\}\}=\\mathbf\{1\}\[\\mathrm\{match\}\(\\hat\{y\},\\tilde\{y\}^\{\\ell\}\)\]\\cdot L\_\{\\mathrm\{conf\}\}\.\\end\{split\}\(5\)
The stricter score is nonzero only when the original answer itself persists after image removal\. Neither score alone is a causal claim that the model ignores the image\. In the main results,LLdenotesLconfL\_\{\\mathrm\{conf\}\}unless stated otherwise\. Appendix[10\.1](https://arxiv.org/html/2608.29193#S10.SS1)reports the corresponding controls\.
Thealignment scoreAAmeasures sensitivity to image\-text binding pressure\. We use grounding re\-checking for all datasets and relation swapping when an explicit relation is available\. VSR supports relation swaps because its captions contain relations such asleft,right,above, andbehind\. In the cone\-car example, a left\-of question is swapped into the corresponding right\-of question\. A relation\-sensitive model should not preserve the same normalized answer under both prompts\. Lety~grounda\\tilde\{y\}^\{a\}\_\{\\mathrm\{ground\}\}be the grounding\-prompt answer andy~rela\\tilde\{y\}^\{a\}\_\{\\mathrm\{rel\}\}the relation\-swapped answer when available\. We define:
Pg=0\.5⋅flip\(y^,y~grounda\)\+min\(Δc\(y^,y~grounda\),0\.5\),Pr=𝟏\[available\]⋅𝟏\[norm\(y~arel\)=norm\(y^\)\],A=\{min\(\(1−λ\)Pg\+λPr,1\),ifavailable,min\(Pg,1\),otherwise,λ=0\.6\.\\begin\{split\}&P\_\{g\}=0\.5\\cdot\\mathrm\{flip\}\(\\hat\{y\},\\tilde\{y\}^\{a\}\_\{\\mathrm\{ground\}\}\)\\\\ &\\qquad\+\\min\(\\Delta c\(\\hat\{y\},\\tilde\{y\}^\{a\}\_\{\\mathrm\{ground\}\}\),0\.5\),\\\\ &P\_\{r\}=\\mathbf\{1\}\[\\mathrm\{available\}\]\\cdot\\mathbf\{1\}\[\\mathrm\{norm\}\(\\tilde\{y\}^\{a\}\_\{\\mathrm\{rel\}\}\)=\\mathrm\{norm\}\(\\hat\{y\}\)\],\\\\ &A=\\begin\{cases\}\\min\(\(1\-\\lambda\)P\_\{g\}\+\\lambda P\_\{r\},1\),&\\mathrm\{if\\ available\},\\\\ \\min\(P\_\{g\},1\),&\\mathrm\{otherwise\},\\end\{cases\}\\\\ &\\lambda=0\.6\.\\end\{split\}\(6\)The relation component receives larger weight because it gives a sharper binding stress\. The expected answer should change under an explicit relation swap\. The grounding component gives a weaker universal alignment probe\. We therefore avoid comparing rawAAmagnitudes across relation\-bearing and non\-relation\-bearing datasets\. Cross\-dataset diagnosis uses dataset and model controls, while routing uses within\-setting normalized dominance\. Appendix[10\.2](https://arxiv.org/html/2608.29193#S10.SS2)reports the no\-VSR control\.
##### Interpreting And Using Perturbation\-Response Signatures\.
Once the probes produce\(V,L,A\)\(V,L,A\), we separate three questions\. Which coordinate is largest? Does a targeted intervention mainly move its intended coordinate? Does the full signature help diagnosis, abstention, or routing? A large coordinate is not a causal assignment\. Raw dominance only counts the largest coordinate,sraw∗\(x,M\)=argmaxQ∈\{V,L,A\}Q\(x,M\)s^\{\*\}\_\{\\mathrm\{raw\}\}\(x,M\)=\\arg\\max\_\{Q\\in\\\{V,L,A\\\}\}Q\(x,M\)\. Because one coordinate can saturate, we use within\-setting normalized dominance for routing:snorm∗\(x,M\)=argmaxQ∈\{V,L,A\}zQ\(x,M\)s^\{\*\}\_\{\\mathrm\{norm\}\}\(x,M\)=\\arg\\max\_\{Q\\in\\\{V,L,A\\\}\}z\_\{Q\}\(x,M\), wherezQ\(x,M\)=\(Q\(x,M\)−μQ\)/\(σQ\+ϵ\)z\_\{Q\}\(x,M\)=\(Q\(x,M\)\-\\mu\_\{Q\}\)/\(\\sigma\_\{Q\}\+\\epsilon\)\. This prevents a saturated coordinate, such asLL, from dominating every routing decision\. We next test whether the scores behave as separable labels\. Let𝒯V\\mathcal\{T\}\_\{V\},𝒯L\\mathcal\{T\}\_\{L\}, and𝒯A\\mathcal\{T\}\_\{A\}denote visual\-ruined, language\-ruined, and alignment\-ruined conditions\. These are stronger than the local probes used to compute the scores\. For targetRRand coordinateQ∈\{V,L,A\}Q\\in\\\{V,L,A\\\}:
ΔR,Q\(x,M\)=Q\(𝒯R\(x\),M\)−Q\(x,M\)\.\\begin\{split\}&\\Delta\_\{R,Q\}\(x,M\)=Q\(\\mathcal\{T\}\_\{R\}\(x\),M\)\-Q\(x,M\)\.\\end\{split\}\(7\)A cleanly separable signature would satisfy magnitude\-based diagonal dominance:
\|ΔR,R\(x,M\)\|\>maxQ≠R\|ΔR,Q\(x,M\)\|\.\\begin\{split\}&\|\\Delta\_\{R,R\}\(x,M\)\|\>\\max\_\{Q\\neq R\}\|\\Delta\_\{R,Q\}\(x,M\)\|\.\\end\{split\}\(8\)We test this condition rather than assume it\. Definition[7\.1](https://arxiv.org/html/2608.29193#S7.Thmdefinition1)states the condition\. Proposition[2](https://arxiv.org/html/2608.29193#Thmproposition2)formalizes the key limitation\. If a source\-targeted intervention changes another coordinate at least as much as the intended coordinate, the intervention cannot be read as source\-specific\. We then evaluate three uses of Perturbation\-Response Signatures\. Failure\-family diagnosis asks whethers\(x,M\)s\(x,M\)predictsf\(x,M\)f\(x,M\)better than scalar confidence\. The role separation follows from Proposition[3](https://arxiv.org/html/2608.29193#Thmproposition3)and Theorem[1](https://arxiv.org/html/2608.29193#Thmtheorem1): a signature can help explain the kind of error without improving correctness ranking\. Abstention scalarization asks whether a source\-aware scorec~\(x,M\)=g\(c,V,L,A\)\\tilde\{c\}\(x,M\)=g\(c,V,L,A\)improves answer\-versus\-abstain ranking\. Proposition[4](https://arxiv.org/html/2608.29193#Thmproposition4)and Corollary[1](https://arxiv.org/html/2608.29193#Thmcorollary1)state the ranking risk when source penalties are not aligned with correctness\. Routing asks whether the signature helps choose a safer correction path\. Proposition[5](https://arxiv.org/html/2608.29193#Thmproposition5)and Corollary[2](https://arxiv.org/html/2608.29193#Thmcorollary2)motivate reporting fix rate and break rate with accuracy\. For abstention, we test three scalarizations:
cmax=c⋅\(1−max\(V,L,A\)\),cw=c⋅\(1−0\.3V−0\.3L−0\.4A\),cgeo=c⋅\(\(1−V\)\(1−L\)\(1−A\)\)1/3\.\\begin\{split\}&c\_\{\\max\}=c\\cdot\(1\-\\max\(V,L,A\)\),\\\\ &c\_\{w\}=c\\cdot\(1\-0\.3V\-0\.3L\-0\.4A\),\\\\ &c\_\{\\mathrm\{geo\}\}=c\\cdot\(\(1\-V\)\(1\-L\)\(1\-A\)\)^\{1/3\}\.\\end\{split\}\(9\)For intervention routing, the generic baseline applies the same reasoning prompt to every input\. The matched policy mapssnorm∗\(x,M\)s^\{\*\}\_\{\\mathrm\{norm\}\}\(x,M\)to visual re\-grounding, anti\-prior prompting, or relation checking\. For example, an alignment\-dominant cone\-car prediction is routed to relation checking rather than a generic reasoning prompt\. We report accuracy, fix rate, and break rate:
Fix=\#\{incorrect→correct\}\#\{incorrect\},Break=\#\{correct→incorrect\}\#\{correct\}\.\\begin\{split\}&\\mathrm\{Fix\}=\\frac\{\\\#\\\{\\mathrm\{incorrect\}\\rightarrow\\mathrm\{correct\}\\\}\}\{\\\#\\\{\\mathrm\{incorrect\}\\\}\},\\\\ &\\mathrm\{Break\}=\\frac\{\\\#\\\{\\mathrm\{correct\}\\rightarrow\\mathrm\{incorrect\}\\\}\}\{\\\#\\\{\\mathrm\{correct\}\\\}\}\.\\end\{split\}\(10\)The protocol uses one forward pass, four visual perturbation passes, one blank\-image pass, and 1\-2 alignment passes, depending on whether relation perturbation is available\. No model is retrained\.
Table 1:Evaluation suite and roles\. Evaluation roles focus on mixed failure\-family diagnosis for HallusionBench, object hallucination/abstention for POPE, relation error/routing breakdown for VSR, and answerability diagnosis for VizWiz\-VQA\. Each model\-dataset output receives scalar confidenceccand Perturbation\-Response Signature\(V,L,A\)\(V,L,A\)\. Full dataset roles, split details, probe availability, and failure\-family mappings are given in Appendix[8](https://arxiv.org/html/2608.29193#S8)\.
## 4Experimental Setup
We evaluate whether multimodal uncertainty should be studied as Perturbation\-Response diagnosis and correction\-path selection rather than only as answer\-versus\-abstain scoring\. For each model output, we compute scalar confidenceccand Perturbation\-Response Signatures\(x\)=\(V,L,A\)s\(x\)=\(V,L,A\)\. We test failure diagnosis, abstention, and intervention routing under matched perturbation protocols\.
Datasets and Models\.We evaluate Gemma 4, Qwen3\-VL, LLaVA\-Mistral, and LLaVA\-Vicuna on HallusionBench, POPE, Visual Spatial Reasoning \(VSR\), and VizWiz\-VQA\[[14](https://arxiv.org/html/2608.29193#bib.bib1),[2](https://arxiv.org/html/2608.29193#bib.bib29),[31](https://arxiv.org/html/2608.29193#bib.bib30),[16](https://arxiv.org/html/2608.29193#bib.bib5),[29](https://arxiv.org/html/2608.29193#bib.bib4),[30](https://arxiv.org/html/2608.29193#bib.bib8),[18](https://arxiv.org/html/2608.29193#bib.bib7)\]\. The suite covers visually grounded hallucination, object\-presence hallucination, spatial relation binding, and answerability\. We use the image\-paired HallusionBench subset, all POPE splits, the VSR zero\-shot development set, and the VizWiz\-VQA validation split\. We use greedy decoding with temperature00, disabled sampling, and a128128\-token cap\. Split, filtering, checkpoint, and license details appear in Appendices[8](https://arxiv.org/html/2608.29193#S8)and[13](https://arxiv.org/html/2608.29193#S13)\.
Diagnostic and Decision Tasks\.Each prediction receives correctness and coarse failure\-family labels\. Correct predictions map tono failure\. Incorrect predictions follow benchmark structure\. POPE maps to object\-presence failure, VSR to spatial or relation error, VizWiz\-VQA to answerability failure, and HallusionBench to visually grounded hallucination categories\. This tests broad failure\-family separation rather than an exhaustive taxonomy\. We train logistic regression, XGBoost, and LightGBM with stratified55\-fold cross\-validation over\[c\]\[c\],\[V,L,A\]\[V,L,A\], and\[V,L,A,c\]\[V,L,A,c\]\[[3](https://arxiv.org/html/2608.29193#bib.bib33),[24](https://arxiv.org/html/2608.29193#bib.bib35)\]\. We report diagnosis AUROC and Macro\-F1, source profiles, abstention AUROC and risk\-coverage, and routing accuracy, fix rate, and break rate\. These metrics keep failure typing, answer\-versus\-abstain ranking, and correction effects separate\. Appendix[8\.6](https://arxiv.org/html/2608.29193#S8.SS6)gives the full mapping\.
Controls, Abstention, and Routing\.Since failure families and datasets are partly aligned, we add dataset\-ID and within\-dataset controls\. We also test the two main score\-design concerns\. ForLL, Appendix[10\.1](https://arxiv.org/html/2608.29193#S10.SS1)compares the clipped\-ratio score with match\-only and denominator\-stable variants\. ForAA, Appendix[10\.2](https://arxiv.org/html/2608.29193#S10.SS2)removes VSR, the only relation\-swap\-eligible dataset\. Abstention compares scalar confidence with Eq\.[9](https://arxiv.org/html/2608.29193#S3.E9)\. Routing compares no intervention, one generic reasoning prompt, and matched intervention\. Matched routing selects image enhancement, anti\-prior instructions, or grounding/relation checking from normalized source dominance\. Exact templates and enhancement parameters appear in Appendix[9\.4](https://arxiv.org/html/2608.29193#S9.SS4)\. We summarize benchmark roles in Table[1](https://arxiv.org/html/2608.29193#S3.T1)\.
Human Audit\.We run a blind audit over200200examples, with5050from each dataset\. Three annotators assign visual evidence insufficiency, language\-prior reliance, alignment error, mixed, unclear, or no failure\. Annotators do not seeVV,LL,AA, scalar confidence, or predicted source\. We adjudicate3636disagreement cases\. Full\-label Fleiss’κ\\kappais0\.2360\.236, and source versus non\-source Fleiss’κ\\kappais0\.480\.48\. We use this audit only as a limited semantic plausibility check; it is not ground truth, training supervision, score calibration, or evidence of causal source recovery\. Appendix[11](https://arxiv.org/html/2608.29193#S11)gives the full protocol and tables\.
Evidence BlockSettingMain ValueMean image\-removal confidence retentionAll model\-dataset settingsLLranges from0\.910\.91to11Blank\-image matchAll model\-dataset settingsLP match ranges from37\.6%37\.6\\%to96\.5%96\.5\\%Highest visual\-perturbation sensitivityVizWiz settingsVVranges from0\.190\.19to0\.300\.30Highest alignment instabilityGemma 4/VSR, Qwen3\-VL/VSRA=0\.68A=0\.68andA=0\.61A=0\.61Raw dominant source58,44058\{,\}440pooled samplesL/A/V=58,287/151/2L/A/V=58\{,\}287/151/2Magnitude\-based diagonal pass countAll intervention checks18/4818/48passVisual\-ruined targetAcross1616settings11/1611/16passLanguage\-ruined targetAcross1616settings2/162/16passAlignment\-ruined targetAcross1616settings5/165/16passQwen3\-VL/HallusionBenchVisual\-ruined deltasΔV=−0\.09\\Delta V=\-0\.09,ΔL=\+0\.01\\Delta L=\+0\.01,ΔA=−0\.01\\Delta A=\-0\.01Qwen3\-VL/HallusionBenchLanguage\-ruined deltasΔV=−0\.05\\Delta V=\-0\.05,ΔL=−0\.09\\Delta L=\-0\.09,ΔA=\+0\.18\\Delta A=\+0\.18
Table 2:Source\-profile and Entanglement Summary\. LP match is exact\-answer agreement under blank\-image replacement\. Diagonal pass counts test whether the intended source has the largest absolute response under the corresponding ruined condition\. Full profiles and intervention deltas are given in Appendix Tables[13](https://arxiv.org/html/2608.29193#S10.T13)\-[19](https://arxiv.org/html/2608.29193#S10.T19)\.Tree\-Based Feature ImportanceFeatureXGBLGBMRoleAA0\.50\\mathbf\{0\.50\}0\.36\\mathbf\{0\.36\}AlignmentLL0\.200\.200\.150\.15Language priorcc0\.150\.150\.270\.27ConfidenceVV0\.150\.150\.220\.22Visual fragility
Table 3:Failure\-family diagnosis, feature importance, and dataset\-ID controls\. Classifier rows report pooled weighted one\-vs\-rest AUROC over58,44058\{,\}440examples\. Feature rows report pooled tree\-based feature importance\. Dataset\-ID controls use LightGBM and report gains after adding source features beyond dataset identityDD, model identityMM, and scalar confidencecc\. Full\-precision diagnosis and control results are given in Appendix Tables[21](https://arxiv.org/html/2608.29193#S10.T21)\-[25](https://arxiv.org/html/2608.29193#S10.T25)\.Table 4:Probe\-design controls\. The image\-removal rows replace only theLLcoordinate inside the pooled\(V,L,A\)\(V,L,A\)diagnosis feature set\. The no\-VSR control removes the only relation\-swap\-eligible dataset and compares scalar confidence with\(V,L,A\)\(V,L,A\)under the same non\-VSR scope\. Details are in Appendix Tables[10](https://arxiv.org/html/2608.29193#S10.T10)\-[11](https://arxiv.org/html/2608.29193#S10.T11)
Table 5:Decision\-facing behavior\. Abstention compares scalar confidence against the best source\-aware scalarization for each model family\. Routing compares no intervention, generic prompting, and matched routing\. TheHighest accuracyrow counts which policy obtains the best accuracy in each model\-dataset setting\. The pairwise rows compare generic prompting and matched routing across the same1616settings\. Full\-precision results appear in Appendix Tables[32](https://arxiv.org/html/2608.29193#S10.T32)\-[36](https://arxiv.org/html/2608.29193#S10.T36)\.
## 5Results
We evaluate\(V,L,A\)\(V,L,A\)for failure typing and matched correction, not as a scalar confidence replacement\. The supporting appendix evidence follows the same order: source profiles in Tables[13](https://arxiv.org/html/2608.29193#S10.T13)\-[14](https://arxiv.org/html/2608.29193#S10.T14), entanglement in Tables[15](https://arxiv.org/html/2608.29193#S10.T15)\-[19](https://arxiv.org/html/2608.29193#S10.T19), diagnosis in Tables[20](https://arxiv.org/html/2608.29193#S10.T20)\-[27](https://arxiv.org/html/2608.29193#S10.T27), human audit in Tables[28](https://arxiv.org/html/2608.29193#S10.T28)\-[31](https://arxiv.org/html/2608.29193#S10.T31), abstention in Tables[32](https://arxiv.org/html/2608.29193#S10.T32)\-[34](https://arxiv.org/html/2608.29193#S10.T34), and routing in Tables[35](https://arxiv.org/html/2608.29193#S10.T35)\-[36](https://arxiv.org/html/2608.29193#S10.T36)\.
##### Prevalence is Not Diagnostic Value\.
We observe from Table[2](https://arxiv.org/html/2608.29193#S4.T2)thatLLis high across settings, with means from0\.910\.91to11and image\-removal answer persistence from37\.6%37\.6\\%to96\.5%96\.5\\%\. VizWiz gives the highest visual scores \(0\.190\.19to0\.300\.30\), while Gemma 4/VSR and Qwen3\-VL/VSR give the highest alignment scores \(0\.680\.68and0\.610\.61\)\. The raw largest\-coordinate count is almost entirelyLL\-dominant\. Overall,58,287/58,44058\{,\}287/58\{,\}440samples areLL\-dominant, versus151151forAAand22forVV\. This99\.7%99\.7\\%dominance refers to the confidence\-retention score\. Under the stricter answer\-persistence scoreLpersistL\_\{\\mathrm\{persist\}\},LLdominance falls to51\.6%51\.6\\%, whileAAandVVdominate27\.9%27\.9\\%and20\.4%20\.4\\%of examples, respectively\. This is why we treatLLas a confidence\-retention marker and use the full vector, not rawLLdominance, for diagnosis\. Appendix Tables[13](https://arxiv.org/html/2608.29193#S10.T13)\-[14](https://arxiv.org/html/2608.29193#S10.T14)give the full source profiles and raw dominance counts\. Appendix Table[27](https://arxiv.org/html/2608.29193#S10.T27)gives matched source\-stress slices that support the same reading\.
##### Interventions Expose Entanglement\.
We test whether source\-targeted ruined inputs isolate their intended coordinate under Definition[7\.1](https://arxiv.org/html/2608.29193#S7.Thmdefinition1)\. We observe from Table[2](https://arxiv.org/html/2608.29193#S4.T2)that only18/4818/48checks pass magnitude\-based diagonal dominance: visual11/1611/16, language2/162/16, and alignment5/165/16\. Qwen3\-VL/HallusionBench gives the main intuition\. Visual ruin decreasesVVby0\.090\.09with smallL/AL/Achanges, while language ruin decreasesLLby0\.090\.09and increasesAAby0\.180\.18\. Appendix Table[15](https://arxiv.org/html/2608.29193#S10.T15)gives the pass counts, and Appendix Tables[16](https://arxiv.org/html/2608.29193#S10.T16)\-[19](https://arxiv.org/html/2608.29193#S10.T19)give the model\-specific signed deltas\. These failures match Proposition[2](https://arxiv.org/html/2608.29193#Thmproposition2)\. Hence,\(V,L,A\)\(V,L,A\)is a joint perturbation\-response signature rather than three causal labels\. This is, however, not a failure of the probe design\. It is the diagnostic setting we want to expose: a visually fragile answer can also be image\-removal sensitive or alignment\-sensitive\.
##### Signatures Diagnose Failure Families\.
Although the scores are entangled, their joint pattern remains diagnostic\. Since failure\-family labels and benchmark identity are partly aligned, we first test dataset and model controls\. We observe from Table[3](https://arxiv.org/html/2608.29193#S4.T3)that adding source features beyond these identifiers improves raw\-family Macro\-F1 from0\.710\.71to0\.780\.78, and the stricter outcome\-family setting from0\.310\.31to0\.350\.35\. Appendix Tables[23](https://arxiv.org/html/2608.29193#S10.T23)\-[24](https://arxiv.org/html/2608.29193#S10.T24)give these controls\. Fixed\-dataset results are strongest on HallusionBench and VizWiz, weaker on POPE, and nearly neutral on VSR, as reported in Appendix Table[25](https://arxiv.org/html/2608.29193#S10.T25)\. Specifically,\(V,L,A\)\(V,L,A\)improves AUROC over scalar confidence from0\.6340\.634to0\.7690\.769on HallusionBench, from0\.7070\.707to0\.8170\.817on VizWiz, from0\.8230\.823to0\.8390\.839on POPE, and from0\.7220\.722to0\.7240\.724on VSR\. These results bound the claim: Perturbation\-Response Signatures are most useful when multiple failure modes or answerability\-sensitive distinctions remain, not as a uniformly transferable rule across all benchmarks\. In an additional 10\-split dataset/model\-controlled analysis, adding\(V,L,A\)\(V,L,A\)beyond dataset ID, model ID, and scalar confidence improves benchmark\-family Macro\-F1 by0\.1700\.170, with a positive gain in all10/1010/10splits\. Leave\-one\-model\-out Macro\-F1 improves from0\.5580\.558to0\.7160\.716\.
We also test whether the result is carried by probe\-design choices\. We observe from Table[4](https://arxiv.org/html/2608.29193#S4.T4)that replacing the clipped\-ratioLLwith match\-only or denominator\-stable variants keeps diagnosis stable\. Removing VSR, the only relation\-swap\-eligible dataset, also preserves the gain of\(V,L,A\)\(V,L,A\)over scalar confidence\. Thus, the result is not explained by the clipped\-ratio form ofLLor by the VSR\-only relation\-swap branch\. In the main pooled tree\-based setting, XGBoost improves from0\.780\.78AUROC with scalar confidence to0\.950\.95with\(V,L,A\)\(V,L,A\), and to0\.970\.97with confidence added\. Appendix Table[21](https://arxiv.org/html/2608.29193#S10.T21)gives the full diagnosis results\. The gain is not merely prevalence\-driven: the alignment coordinate contributes0\.500\.50of pooled XGBoost importance, as detailed in Appendix Table[22](https://arxiv.org/html/2608.29193#S10.T22)\. Appendix Table[20](https://arxiv.org/html/2608.29193#S10.T20)reports source means by failure family\. A blind human audit gives bounded semantic support: clear\-source labels are mostly alignment \(71/10771/107\), and\(V,L,A\)\(V,L,A\)improves clear\-source AUROC over scalar confidence from0\.430\.43to0\.600\.60\. Appendix Tables[28](https://arxiv.org/html/2608.29193#S10.T28)\-[31](https://arxiv.org/html/2608.29193#S10.T31)give the audit protocol, agreement, confusion matrix, and audit\-side classifier\.
##### Visual\-Probe Robustness\.
We additionally test whether the diagnosis result depends on one visual perturbation or on the selected perturbation strength\. Removing blur, crop, brightness, or noise individually keeps diagnosis AUROC between0\.8220\.822and0\.8240\.824, with89\.8%89\.8\\%–97\.8%97\.8\\%normalized source\-assignment agreement\. On the same10,54010\{,\}540outputs, increasing perturbation strength from weak to default to strong increases meanVVfrom0\.1320\.132to0\.1960\.196to0\.2520\.252and the visual answer\-flip rate from0\.1740\.174to0\.2630\.263to0\.3420\.342\. UsingLpersistL\_\{\\mathrm\{persist\}\}, diagnosis AUROC remains0\.7850\.785,0\.7860\.786, and0\.7830\.783, compared with0\.6320\.632for scalar confidence\. All paired\-bootstrap intervals for these AUROC and Macro\-F1 gains exclude zero\. Full results are reported in Appendix[10\.3](https://arxiv.org/html/2608.29193#S10.SS3)\.
##### Naive Scalarization Can Hurt\.
The diagnostic gain does not mean that the three direct source\-aware scalarizations tested here should improve abstention\. We observe from Table[5](https://arxiv.org/html/2608.29193#S4.T5)that weighted, geometric, and max\-penalty scalarizations underperform in AUROC\. Qwen3\-VL drops from0\.810\.81scalar AUROC to0\.730\.73for the best source\-aware score\. Appendix Table[32](https://arxiv.org/html/2608.29193#S10.T32)gives aggregate scalarization results, Appendix Table[33](https://arxiv.org/html/2608.29193#S10.T33)gives per\-dataset comparisons, and Appendix Table[34](https://arxiv.org/html/2608.29193#S10.T34)gives risk\-coverage values\. This does not certify scalar confidence as grounded\. Rather, it shows that a diagnostic feature can be useful for explaining the kind of failure while still being misaligned with answer\-versus\-abstain ranking\. In other words, source\-aware uncertainty should not be naively scalarized for abstention\. These experiments do not rule out learned, calibrated, or validation\-trained scalarizers\. As formalized in Appendix Proposition[4](https://arxiv.org/html/2608.29193#Thmproposition4)and Corollary[1](https://arxiv.org/html/2608.29193#Thmcorollary1), source penalties and near\-saturated coordinates can reverse useful confidence orderings\.
##### Diagnosis\-Triggered Routing\.
We treat routing as a secondary downstream application rather than as validation of the source coordinates\. We test whether Perturbation\-Response Signatures help choose safer correction paths\. Generic prompting applies one reasoning prompt to every example, while matched routing selects visual re\-grounding, anti\-prior prompting, or relation checking from normalized source dominance\. We observe from Table[5](https://arxiv.org/html/2608.29193#S4.T5)that matched routing is not a universal accuracy booster\. It beats generic prompting in10/1610/16settings, with pooled accuracy75\.9%75\.9\\%versus75\.7%75\.7\\%for generic prompting and76\.6%76\.6\\%without intervention\. Its value is clearer in break\-rate control\. On Qwen3\-VL/VSR, matched routing avoids the generic accuracy drop from87\.4%87\.4\\%to76\.2%76\.2\\%and lowers breakage from15\.5%15\.5\\%to2\.7%2\.7\\%, while maintaining87\.6%87\.6\\%accuracy\. It also lowers breakage on Gemma 4/HallusionBench and LLaVA\-Mistral/VizWiz \(15\.1%→4\.9%15\.1\\%\\\!\\to\\\!4\.9\\%and21\.4%→17\.4%21\.4\\%\\\!\\to\\\!17\.4\\%\)\. Appendix Table[35](https://arxiv.org/html/2608.29193#S10.T35)gives the full routing results, and Appendix Table[36](https://arxiv.org/html/2608.29193#S10.T36)gives model\-level win counts\. This supports routing when generic correction risks damaging correct predictions\. The practical use is therefore not universal correction, but avoiding avoidable damage when a generic correction prompt is too blunt\. Since baseline accuracy exceeds0\.50\.5, Corollary[2](https://arxiv.org/html/2608.29193#Thmcorollary2)gives larger weight to avoiding broken correct answers than to an equal\-sized repair gain\. As an additional control, we evaluate a probe\-informed policy against generic CoT on6,5616\{,\}561model\-output instances\. Relative to generic CoT, the probe\-informed policy improves pooled accuracy by3\.543\.54percentage points \(95% CI\[2\.82,4\.25\]\[2\.82,\\,4\.25\]\) and reduces break rate by5\.125\.12points \(95% CI\[4\.12,6\.09\]\[4\.12,\\,6\.09\]\)\. It improves accuracy in all four evaluated settings, but is not universally better than no intervention; VizWiz remains a boundary case\.
## 6Conclusion
In this paper, we presentHalluPrismas a perturbation\-response diagnostic for multimodal uncertainty in MLLMs\. Across 58K\+ examples, perturbation\-response signatures improve failure\-family diagnosis, while matched routing reduces harmful breakage in selected high\-risk settings\. The same evidence shows a boundary: better failure typing does not imply better abstention scoring\. In our experiments, the three tested direct signature\-based scalarizations do not improve correctness ranking and can perform worse than scalar confidence; learned or calibrated scalarizers remain an open direction\. Perturbation\-response diagnosis should therefore characterize whether behavior is consistent with visual sensitivity, image\-removal confidence retention, or alignment instability, and help determine whether a generic correction is safe\.
## Limitations
HalluPrismis a behavioral diagnostic layer, not a causal explanation of hidden mechanisms inside a Multimodal Large Language Model \(MLLM\)\. The scores are defined using a specific set of probes and perturbation strengths\. Our ablation and severity analyses show that the main diagnostic trends remain stable across the settings tested here, while broader probe choices remain an avenue for future evaluation\. Because the evaluated benchmarks emphasize different failure families, we additionally report controlled and fixed\-dataset analyses\. The magnitude of the diagnostic gain varies across datasets, and cross\-dataset generalization remains an important direction for future study\. The evaluation covers four public benchmarks and four open\-weight MLLM families, not video, multi\-image reasoning, medical, or document\-heavy expert settings, or latency\-critical deployments\. The implementation requires77to88forward passes per example\. Routing uses lightweight prompts and gives conditional actionability, not universal mitigation\. The human audit is intentionally small and checks semantic plausibility, not causal source recovery\. Our abstention analysis focuses on three direct hand\-designed scalarizations; learned or calibrated scalarization strategies are outside the scope of the present study\. The no\-free\-lunch argument does not reject selective prediction, calibration, risk control, or conformal prediction for their stated statistical targets\. It states that these guarantees require validation\-to\-deployment assumptions and, by themselves, do not provide instance\-grounded reliability\.
## Ethics Statement
HalluPrismis intended for model auditing and diagnostic analysis, not for standalone safety certification or deployment approval\. Its scores are behavioral Perturbation\-Response Signatures from controlled probes and should not be treated as causal explanations of hidden model mechanisms\. The main ethical risk is over\-trust, where a compact source label is mistaken for a complete reliability guarantee\. We therefore keep diagnosis, scalarization, fix rate, and break rate separate\. The experiments use public multimodal benchmarks and publicly accessible model families\. We do not collect new user images or train a deployed model\. Since VizWiz\-VQA contains images and questions from blind or low\-vision users, any release should avoid raw\-image redistribution unless allowed by the original license\. The human audit uses benchmark examples only, and annotators do not see model\-derived source scores or routing decisions\.
## Acknowledgment
Tanmoy Chakraborty acknowledges the support of the Rajiv Khemani Young Faculty Chair Professorship in Artificial Intelligence and the ICMR Extramural Research \(Centre for Advanced Research\) Project Grant \(EM/DEV/CAR\-2025/00725\)\. We also thank Ayush Kumar, Kumari Shristee, and Aryan Raj for their valuable assistance with the data annotation process\.
## References
- \[1\]\(2020\)Quantifying attention flow in transformers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\.4190–4197\.External Links:[Link](https://aclanthology.org/2020.acl-main.385/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.385)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu\(2025\)Qwen3\-vl technical report\.External Links:2511\.21631,[Link](https://arxiv.org/abs/2511.21631)Cited by:[§4](https://arxiv.org/html/2608.29193#S4.p2.1)\.
- \[3\]T\. Chen and C\. Guestrin\(2016\)XGBoost: a scalable tree boosting system\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,KDD ’16,New York, NY, USA,pp\.785–794\.External Links:ISBN 9781450342322,[Link](https://doi.org/10.1145/2939672.2939785),[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[§4](https://arxiv.org/html/2608.29193#S4.p3.1)\.
- \[4\]C\. Chow\(1970\)On optimum recognition error and reject tradeoff\.IEEE Transactions on Information Theory16\(1\),pp\.41–46\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1970.1054406)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.29193#S7.p1.1)\.
- \[5\]J\. Eisenschlos, H\. Maina, G\. Ivetta, and L\. Benotti\(2024\)Selectively answering visual questions\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\.4219–4229\.External Links:[Link](https://aclanthology.org/2024.findings-acl.250/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.250)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]A\. Favero, L\. Zancato, M\. Trager, S\. Choudhary, P\. Perera, A\. Achille, A\. Swaminathan, and S\. Soatto\(2024\)Multi\-modal hallucination control by visual information grounding\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\.14303–14312\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01356)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
- \[7\]M\. Fazli, B\. Wei, A\. Sari, and Z\. Zhu\(2025\)Mitigating hallucination in large vision\-language models via adaptive attention calibration\.External Links:2505\.21472,[Link](https://arxiv.org/abs/2505.21472)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
- \[8\]C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun, Y\. Wu, R\. Ji, C\. Shan, and R\. He\(2025\)MME: a comprehensive evaluation benchmark for multimodal large language models\.External Links:2306\.13394,[Link](https://arxiv.org/abs/2306.13394)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.p1.1)\.
- \[9\]M\. Gardner, Y\. Artzi, V\. Basmov, J\. Berant, B\. Bogin, S\. Chen, P\. Dasigi, D\. Dua, Y\. Elazar, A\. Gottumukkala, N\. Gupta, H\. Hajishirzi, G\. Ilharco, D\. Khashabi, K\. Lin, J\. Liu, N\. F\. Liu, P\. Mulcaire, Q\. Ning, S\. Singh, N\. A\. Smith, S\. Subramanian, R\. Tsarfaty, E\. Wallace, A\. Zhang, and B\. Zhou\(2020\)Evaluating models’ local decision boundaries via contrast sets\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\.1307–1323\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.117/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.117)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]Y\. Geifman and R\. El\-Yaniv\(2017\)Selective classification for deep neural networks\.External Links:1705\.08500,[Link](https://arxiv.org/abs/1705.08500)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.29193#S7.p1.1)\.
- \[11\]Y\. Geifman and R\. El\-Yaniv\(2019\)SelectiveNet: a deep neural network with an integrated reject option\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\.2151–2159\.External Links:[Link](https://proceedings.mlr.press/v97/geifman19a.html)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.29193#S7.p1.1)\.
- \[12\]J\. Geng, F\. Cai, Y\. Wang, H\. Koeppl, P\. Nakov, and I\. Gurevych\(2024\)A survey of confidence estimation and calibration in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\.6577–6595\.External Links:[Link](https://aclanthology.org/2024.naacl-long.366/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.366)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]X\. Gong, T\. Ming, X\. Wang, and Z\. Wei\(2024\)DAMRO: dive into the attention mechanism of LVLM to reduce object hallucination\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\.7696–7712\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.439/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.439)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px2.p1.1)\.
- \[14\]Google DeepMind\(2026\)Gemma 4 Model Card\.Note:Accessed: 2026\-05\-20External Links:[Link](https://ai.google.dev/gemma/docs/core/model_card_4)Cited by:[§4](https://arxiv.org/html/2608.29193#S4.p2.1)\.
- \[15\]Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. Parikh\(2017\)Making the v in vqa matter: elevating the role of image understanding in visual question answering\.In2017 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\.6325–6334\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2017.670)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.p1.1)\.
- \[16\]T\. Guan, F\. Liu, X\. Wu, R\. Xian, Z\. Li, X\. Liu, X\. Wang, L\. Chen, F\. Huang, Y\. Yacoob, D\. Manocha, and T\. Zhou\(2024\)HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision\-language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\.14375–14385\.Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.p1.1),[§4](https://arxiv.org/html/2608.29193#S4.p2.1)\.
- \[17\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\.1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]D\. Gurari, Q\. Li, A\. J\. Stangl, A\. Guo, C\. Lin, K\. Grauman, J\. Luo, and J\. P\. Bigham\(2018\)VizWiz Grand Challenge: Answering Visual Questions from Blind People\.In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,Los Alamitos, CA, USA,pp\.3608–3617\.External Links:ISSN,[Document](https://dx.doi.org/10.1109/CVPR.2018.00380),[Link](https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00380)Cited by:[§4](https://arxiv.org/html/2608.29193#S4.p2.1)\.
- \[19\]Q\. Huang, X\. Dong, P\. Zhang, B\. Wang, C\. He, J\. Wang, D\. Lin, W\. Zhang, and N\. Yu\(2024\)OPERA: alleviating hallucination in multi\-modal large language models via over\-trust penalty and retrospection\-allocation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\.13418–13427\.Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
- \[20\]T\. Huang, Z\. Liu, R\. Wang, Y\. Zhang, and L\. Jing\(2025\)Visual hallucination detection in large vision\-language models via evidential conflict\.International Journal of Approximate Reasoning186,pp\.109507\.External Links:ISSN 0888\-613X,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.ijar.2025.109507),[Link](https://www.sciencedirect.com/science/article/pii/S0888613X25001483)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]S\. Jain and B\. C\. Wallace\(2019\)Attention is not Explanation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\.3543–3556\.External Links:[Link](https://aclanthology.org/N19-1357/),[Document](https://dx.doi.org/10.18653/v1/N19-1357)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px2.p1.1)\.
- \[22\]S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. Kaplan\(2022\)Language models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[23\]D\. Kaushik, E\. Hovy, and Z\. Lipton\(2020\)Learning the difference that makes a difference with counterfactually\-augmented data\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Sklgs0NFvr)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]G\. Ke, Q\. Meng, T\. Finley, T\. Wang, W\. Chen, W\. Ma, Q\. Ye, and T\. Liu\(2017\)LightGBM: a highly efficient gradient boosting decision tree\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf)Cited by:[§4](https://arxiv.org/html/2608.29193#S4.p3.1)\.
- \[25\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.External Links:2302\.09664,[Link](https://arxiv.org/abs/2302.09664)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[26\]S\. Lee, S\. H\. Park, Y\. Jo, and M\. Seo\(2024\)Volcano: mitigating multimodal hallucination through self\-feedback guided revision\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\.391–404\.External Links:[Link](https://aclanthology.org/2024.naacl-long.23/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.23)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
- \[27\]S\. Leng, H\. Zhang, G\. Chen, X\. Li, S\. Lu, C\. Miao, and L\. Bing\(2023\)Mitigating object hallucinations in large vision\-language models through visual contrastive decoding\.External Links:2311\.16922,[Link](https://arxiv.org/abs/2311.16922)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
- \[28\]J\. Li, J\. Zhang, Z\. Jie, L\. Ma, and G\. Li\(2026\)Cross\-modal attention calibration for lvlm hallucination mitigation\.External Links:2501\.01926,[Link](https://arxiv.org/abs/2501.01926)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
- \[29\]Y\. Li, Y\. Du, K\. Zhou, J\. Wang, X\. Zhao, and J\. Wen\(2023\)Evaluating object hallucination in large vision\-language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\.292–305\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.20/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.20)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.p1.1),[§4](https://arxiv.org/html/2608.29193#S4.p2.1)\.
- \[30\]F\. Liu, G\. Emerson, and N\. Collier\(2023\)Visual spatial reasoning\.Transactions of the Association for Computational Linguistics11,pp\.635–651\.External Links:[Link](https://aclanthology.org/2023.tacl-1.37/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00566)Cited by:[§4](https://arxiv.org/html/2608.29193#S4.p2.1)\.
- \[31\]H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee\(2024\)Improved baselines with visual instruction tuning\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\.26286–26296\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.02484)Cited by:[§4](https://arxiv.org/html/2608.29193#S4.p2.1)\.
- \[32\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu, K\. Chen, and D\. Lin\(2024\)MMBench: is your multi\-modal model an all\-around player?\.External Links:2307\.06281,[Link](https://arxiv.org/abs/2307.06281)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.p1.1)\.
- \[33\]P\. Manakul, A\. Liusie, and M\. Gales\(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\.9004–9017\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.557/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.557)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[34\]E\. Mushtaq, Z\. Fabian, Y\. F\. Bakman, A\. Ramakrishna, M\. Soltanolkotabi, and S\. Avestimehr\(2025\)HARMONY: hidden activation representations and model output\-aware uncertainty estimation for vision\-language models\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops \(CVPRW\),Vol\.,pp\.1654–1659\.External Links:[Document](https://dx.doi.org/10.1109/CVPRW67362.2025.00154)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[35\]M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh\(2020\)Beyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\.4902–4912\.External Links:[Link](https://aclanthology.org/2020.acl-main.442/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.442)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px2.p1.1)\.
- \[36\]A\. Rohrbach, L\. A\. Hendricks, K\. Burns, T\. Darrell, and K\. Saenko\(2018\)Object hallucination in image captioning\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\.4035–4045\.External Links:[Link](https://aclanthology.org/D18-1437/),[Document](https://dx.doi.org/10.18653/v1/D18-1437)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.p1.1)\.
- \[37\]R\. R\. Selvaraju, M\. Cogswell, A\. Das, R\. Vedantam, D\. Parikh, and D\. Batra\(2017\)Grad\-cam: visual explanations from deep networks via gradient\-based localization\.In2017 IEEE International Conference on Computer Vision \(ICCV\),Vol\.,pp\.618–626\.External Links:[Document](https://dx.doi.org/10.1109/ICCV.2017.74)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px2.p1.1)\.
- \[38\]H\. Seo, D\. U\. Kang, H\. Cho, J\. Lee, and S\. Y\. Chun\(2025\)On epistemic uncertainty of visual tokens for object hallucinations in large vision\-language models\.External Links:2510\.09008,[Link](https://arxiv.org/abs/2510.09008)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[39\]T\. Srinivasan, J\. Hessel, T\. Gupta, B\. Y\. Lin, Y\. Choi, J\. Thomason, and K\. Chandu\(2024\)Selective “selective prediction”: reducing unnecessary abstention in vision\-language reasoning\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\.12935–12948\.External Links:[Link](https://aclanthology.org/2024.findings-acl.767/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.767)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px1.p1.1)\.
- \[40\]J\. Wang, Y\. Wang, G\. Xu, J\. Zhang, Y\. Gu, H\. Jia, J\. Wang, H\. Xu, M\. Yan, J\. Zhang, and J\. Sang\(2024\)AMBER: an llm\-free multi\-dimensional benchmark for mllms hallucination evaluation\.External Links:2311\.07397,[Link](https://arxiv.org/abs/2311.07397)Cited by:[§1](https://arxiv.org/html/2608.29193#S1.p2.1),[§2](https://arxiv.org/html/2608.29193#S2.p1.1)\.
- \[41\]S\. Yin, C\. Fu, S\. Zhao, T\. Xu, H\. Wang, D\. Sui, Y\. Shen, K\. Li, X\. Sun, and E\. Chen\(2024\)Woodpecker: hallucination correction for multimodal large language models\.Science China Information Sciences67\(12\),pp\.220105\.Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
- \[42\]X\. Zhuang, Z\. Zhu, Y\. Xie, L\. Liang, and Y\. Zou\(2025\)VASparse: towards efficient visual hallucination mitigation via visual\-aware token sparsification\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\.4189–4199\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52734.2025.00396)Cited by:[§2](https://arxiv.org/html/2608.29193#S2.SS0.SSS0.Px3.p1.1)\.
\\beginappendix
## 7Theoretical Analysis
We provide formal results that support the Perturbation\-Response interpretation used in the main paper\. The results do not claim thatVV,LL, andAArecover hidden causal mechanisms inside a Multimodal Large Language Model \(MLLM\)\. They instead separate four roles: scalar confidence ranks correctness, Perturbation\-Response Signatures support failure\-family diagnosis, scalarization can distort abstention rankings, and routing must be evaluated through both fix and break rates\. The setup is related to classical reject\-option classification and selective prediction\[[4](https://arxiv.org/html/2608.29193#bib.bib37),[10](https://arxiv.org/html/2608.29193#bib.bib38),[11](https://arxiv.org/html/2608.29193#bib.bib39)\]\.
### 7\.1Scalar Confidence Does Not Identify Failure Source
LetXXdenote the input\-output instance,C=c\(X\)∈\[0,1\]C=c\(X\)\\in\[0,1\]denote scalar confidence,Z∈\{0,1\}Z\\in\\\{0,1\\\}denote correctness, andF∈ℱF\\in\\mathcal\{F\}denote the failure\-family label, withF=NoneF=\\mathrm\{None\}whenZ=1Z=1\. A source\-diagnosis rule based only on scalar confidence is any measurable maph:\[0,1\]→ℱ∪\{None\}h:\[0,1\]\\rightarrow\\mathcal\{F\}\\cup\\\{\\mathrm\{None\}\\\}\.
###### Proposition 1\(Scalar Non\-Identifiability of Failure Source\)\.
A zero\-error diagnosis ruleh\(C\)h\(C\)exists if and only ifFFis measurable with respect toCC\. Equivalently, for almost every confidence valuecc, the conditional distributionP\(F∣C=c\)P\(F\\mid C=c\)must place all mass on one label\. If there exists a setB⊆\[0,1\]B\\subseteq\[0,1\]withP\(C∈B\)\>0P\(C\\in B\)\>0such that
𝔼\[1−maxf∈ℱ∪\{None\}P\(F=f∣C\)∣C∈B\]\>0,\\begin\{split\}&\\mathbb\{E\}\\left\[1\-\\max\_\{f\\in\\mathcal\{F\}\\cup\\\{\\mathrm\{None\}\\\}\}P\(F=f\\mid C\)\\mid C\\in B\\right\]\>0,\\end\{split\}\(11\)then every scalar\-confidence\-only diagnosis rule has positive error onBB\.
###### Proof\.
IfFFis measurable with respect toCC, then there exists a measurable functionhhsuch thatF=h\(C\)F=h\(C\)almost surely\. This gives zero diagnosis error\. Conversely, if a zero\-error ruleh\(C\)h\(C\)exists, thenF=h\(C\)F=h\(C\)almost surely, and henceFFis measurable with respect toCC\. For the positive\-error statement, condition on each confidence value\. The best scalar rule at confidence valueC=cC=cselects the label with largest conditional probability, so its conditional error is1−maxfP\(F=f∣C=c\)1\-\\max\_\{f\}P\(F=f\\mid C=c\)\. Averaging this quantity overC∈BC\\in Bgives the displayed expression\. If this value is positive, every scalar\-confidence\-only diagnosis rule has positive error onBB\. ∎
Implications\.Proposition[1](https://arxiv.org/html/2608.29193#Thmproposition1)explains why scalar confidence is an incomplete object for multimodal hallucination analysis\. Even if scalar confidence is useful for abstention, it cannot diagnose whether an error is an object hallucination, a spatial error, an attribute error, or an answerability failure unless those labels are already determined by confidence\. This motivates the failure\-family diagnosis experiment, where\(V,L,A\)\(V,L,A\)is evaluated against scalar confidence for predicting the type of failure\.
### 7\.2Interventional Separability and Entanglement
LetS\(X\)=\(V\(X\),L\(X\),A\(X\)\)S\(X\)=\(V\(X\),L\(X\),A\(X\)\)denote the Perturbation\-Response Signature\. Let𝒯V\\mathcal\{T\}\_\{V\},𝒯L\\mathcal\{T\}\_\{L\}, and𝒯A\\mathcal\{T\}\_\{A\}denote visual, image\-removal, and grounding/relation interventions\. ForR,S∈\{V,L,A\}R,S\\in\\\{V,L,A\\\}, define the intervention response:
ΔR,S\(X\)=S\(𝒯R\(X\)\)−S\(X\)\.\\begin\{split\}&\\Delta\_\{R,S\}\(X\)=S\(\\mathcal\{T\}\_\{R\}\(X\)\)\-S\(X\)\.\\end\{split\}\(12\)The expected response matrix isD∈ℝ3×3D\\in\\mathbb\{R\}^\{3\\times 3\}, with entries:
DR,S=𝔼\[ΔR,S\(X\)\]\.\\begin\{split\}&D\_\{R,S\}=\\mathbb\{E\}\[\\Delta\_\{R,S\}\(X\)\]\.\\end\{split\}\(13\)
###### Definition 7\.1\(Diagonal Separability Under A Probe Family\)\.
The Perturbation\-Response Signature is diagonally separable under the probe family\{𝒯V,𝒯L,𝒯A\}\\\{\\mathcal\{T\}\_\{V\},\\mathcal\{T\}\_\{L\},\\mathcal\{T\}\_\{A\}\\\}if, for everyR∈\{V,L,A\}R\\in\\\{V,L,A\\\},
DR,R\>maxS≠RDR,S\.\\begin\{split\}&D\_\{R,R\}\>\\max\_\{S\\neq R\}D\_\{R,S\}\.\\end\{split\}\(14\)
###### Proposition 2\(Off\-Diagonal Magnitude Response Implies Probe\-Level Entanglement\)\.
If Definition[7\.1](https://arxiv.org/html/2608.29193#S7.Thmdefinition1)fails for some source\-targeting ruined conditionRR, then the probe family does not support a one\-to\-one interpretation in which conditionRRprimarily affects only sourceRR\. In particular, if\|DR,S\|≥\|DR,R\|\|D\_\{R,S\}\|\\geq\|D\_\{R,R\}\|for someS≠RS\\neq R, then the observed response to𝒯R\\mathcal\{T\}\_\{R\}is at least as large in magnitude on an off\-target source as on its intended source\.
###### Proof\.
The condition for a one\-to\-one source\-selective probe interpretation is magnitude\-based diagonal dominance under Definition[7\.1](https://arxiv.org/html/2608.29193#S7.Thmdefinition1)\. If\|DR,S\|≥\|DR,R\|\|D\_\{R,S\}\|\\geq\|D\_\{R,R\}\|for someS≠RS\\neq R, then the off\-target response has magnitude no smaller than the intended response\. Hence the intervention response cannot be assigned uniquely to the intended source under this probe family\. ∎
Implications\.Proposition[2](https://arxiv.org/html/2608.29193#Thmproposition2)gives the formal basis for the interventional entanglement matrix\. The empirical question is not whether every ruined condition validates a clean source label\. The question is whether each source\-targeting condition produces the largest absolute response on its intended coordinate or whether comparable off\-target responses appear\. When a visual\-ruined condition changes alignment or behavior consistent with language\-prior reliance, that response is informative rather than a nuisance\. It supports the interpretation of\(V,L,A\)\(V,L,A\)as a joint Perturbation\-Response Signature\.
### 7\.3Diagnosis And Abstention Are Different Prediction Problems
LetTTdenote the failure type conditional on being wrong, so thatT=FT=FwhenZ=0Z=0\. LetCCdenote scalar confidence and letS=\(V,L,A\)S=\(V,L,A\)denote the Perturbation\-Response Signature\. Abstention requires ranking examples byP\(Z=1∣⋅\)P\(Z=1\\mid\\cdot\)\. Failure diagnosis requires predictingTTorFF\.
###### Proposition 3\(Diagnostic Information Need Not Improve Correctness Ranking\)\.
There exist distributions over\(C,𝐒,Z,T\)\(C,\\mathbf\{S\},Z,T\)such that the Perturbation\-Response Signature𝐒\\mathbf\{S\}is strictly more informative than scalar confidenceCCfor failure\-type prediction among incorrect examples, whileCCis at least as informative as𝐒\\mathbf\{S\}for correctness ranking\. HereC∈\[0,1\]C\\in\[0,1\]denotes scalar confidence,𝐒=\(V,L,A\)\\mathbf\{S\}=\(V,L,A\)denotes the Perturbation\-Response Signature,Z∈\{0,1\}Z\\in\\\{0,1\\\}denotes correctness, andTTdenotes the failure type conditional onZ=0Z=0\. One sufficient condition is:
T⟂⟂𝐒\|Z=0,T⟂⟂C\|Z=0,P\(Z=1∣C,𝐒\)=P\(Z=1∣C\)\.\\begin\{split\}&T\\not\\\!\\perp\\\!\\\!\\\!\\perp\\mathbf\{S\}\\mid Z=0,\\\\ &T\\perp\\\!\\\!\\\!\\perp C\\mid Z=0,\\\\ &P\(Z=1\\mid C,\\mathbf\{S\}\)=P\(Z=1\\mid C\)\.\\end\{split\}\(15\)Under these conditions,𝐒\\mathbf\{S\}can improve failure\-type diagnosis overCCamong incorrect examples, although adding𝐒\\mathbf\{S\}cannot improve the Bayes\-optimal correctness score beyondCC\.
###### Proof\.
The first two conditions state thatSScarries information about the failure type among incorrect examples, whileCCdoes not\. Therefore a classifier with access toSScan improve failure\-type prediction over a classifier usingCCalone whenever the conditional distributions differ across failure types\. The third condition states that, after conditioning onCC, the Perturbation\-Response Signature provides no additional information about correctness\. Hence the Bayes\-optimal correctness ranking is a function ofCCalone\. AddingSScannot improve that ranking\. ∎
Implications\.Proposition[3](https://arxiv.org/html/2608.29193#Thmproposition3)explains why the failure\-diagnosis and abstention results can move in opposite directions\. The Perturbation\-Response Signature can substantially improve failure\-family AUROC while still failing to improve correctness ranking\. This supports the paper’s central distinction\. Perturbation\-Response Signatures are diagnostic representations\. They are not guaranteed scalar confidence improvements\.
###### Theorem 1\(Role Separation Between Perturbation\-Response Signatures And Scalar Confidence\)\.
Suppose the conditions in Proposition[3](https://arxiv.org/html/2608.29193#Thmproposition3)hold\. Then any Bayes\-optimal abstention policy based on\(C,S\)\(C,S\)can be matched by a policy based onCCalone, while a Bayes\-optimal failure\-family classifier based on\(C,S\)\(C,S\)can strictly outperform every classifier based onCCalone\.
###### Proof\.
For abstention, the Bayes\-optimal score isP\(Z=1∣C,S\)P\(Z=1\\mid C,S\)\. By the third condition in Eq\.[15](https://arxiv.org/html/2608.29193#S7.E15), this equalsP\(Z=1∣C\)P\(Z=1\\mid C\)\. Therefore a policy that uses onlyCCcan match the optimal policy that uses\(C,S\)\(C,S\)\. For failure\-family prediction, the first two conditions imply thatSSchanges the conditional distribution ofTTamong incorrect examples, whileCCdoes not\. Therefore the Bayes\-optimal classifier with access toSScan achieve lower expected classification error than any classifier restricted toCCwhen the conditional failure\-type distributions differ across Perturbation\-Response regions\. ∎
Implications\.Theorem[1](https://arxiv.org/html/2608.29193#Thmtheorem1)states that a Perturbation\-Response Signature may be valuable for explaining what kind of failure occurred without improving answer\-versus\-abstain decisions\. This justifies evaluating failure\-family diagnosis and abstention as separate tasks, rather than treating abstention as the only test of uncertainty usefulness\.
### 7\.4Why Naive Source\-Aware Scalarization Can Hurt Abstention
Many source\-aware confidence rules have the multiplicative form:
C~=C⋅ϕ\(S\),\\begin\{split\}&\\tilde\{C\}=C\\cdot\\phi\(S\),\\end\{split\}\(16\)whereϕ\(S\)∈\[0,1\]\\phi\(S\)\\in\[0,1\]is a source penalty\. The weighted, geometric, and max\-penalty scores in Section[3](https://arxiv.org/html/2608.29193#S3.SS0.SSS0.Px3)are instances of this pattern\.
###### Proposition 4\(Pairwise Inversion Under Source Penalties\)\.
Consider two examplesiiandjjwith scalar confidencesCi\>CjC\_\{i\}\>C\_\{j\}\. A multiplicative source\-aware scoreC~=Cϕ\(S\)\\tilde\{C\}=C\\phi\(S\)reverses their ranking if and only if:
ϕ\(Sj\)ϕ\(Si\)\>CiCj\.\\begin\{split\}&\\frac\{\\phi\(S\_\{j\}\)\}\{\\phi\(S\_\{i\}\)\}\>\\frac\{C\_\{i\}\}\{C\_\{j\}\}\.\\end\{split\}\(17\)Therefore, source penalties can hurt abstention whenever penalty variation is not aligned with correctness and is large enough to overcome scalar\-confidence margins\.
###### Proof\.
The source\-aware ranking reverses the scalar\-confidence ranking whenC~j\>C~i\\tilde\{C\}\_\{j\}\>\\tilde\{C\}\_\{i\}\. This is equivalent toCjϕ\(Sj\)\>Ciϕ\(Si\)C\_\{j\}\\phi\(S\_\{j\}\)\>C\_\{i\}\\phi\(S\_\{i\}\)\. Dividing both sides byCjϕ\(Si\)C\_\{j\}\\phi\(S\_\{i\}\)gives Eq\.[17](https://arxiv.org/html/2608.29193#S7.E17)\. ∎
Implications\.Proposition[4](https://arxiv.org/html/2608.29193#Thmproposition4)explains how source\-aware scalarization can reduce abstention quality\. A penalty based onVV,LL, andAAcan reorder examples that scalar confidence ranks correctly\. This matters especially when the penalty is weakly related to correctness\. The result supports the source\-aware confidence\-trap analysis in Section[5](https://arxiv.org/html/2608.29193#S5)\.
###### Corollary 1\(Saturated Source Scores Can Compress Useful Rankings\)\.
Suppose a source penaltyϕ\(S\)\\phi\(S\)is strongly affected by a nearly saturated coordinate such asL≈1L\\approx 1\. Ifϕ\(S\)\\phi\(S\)varies across examples due to small changes in saturated source scores and those variations are weakly correlated with correctness, then the multiplicative scoreC~=Cϕ\(S\)\\tilde\{C\}=C\\phi\(S\)can introduce pairwise inversions relative toCCwithout adding useful correctness information\.
###### Proof\.
By Proposition[4](https://arxiv.org/html/2608.29193#Thmproposition4), ranking inversions occur whenever the penalty ratio exceeds the scalar\-confidence ratio\. Near saturation, small absolute variations in a penalty function can still produce relative penalty differences across examples, especially for nonlinear or max\-based penalties\. If those penalty differences are weakly correlated with correctness, they can reverse confidence orderings without improving correctness ranking\. Therefore the scalarized score can compress or distort useful confidence rankings\. ∎
Implications\.Corollary[1](https://arxiv.org/html/2608.29193#Thmcorollary1)connects the formal ranking argument to the empirical source profiles\. The image\-removal confidence\-retention score is near saturation in many model\-dataset settings\. Multiplicative source penalties therefore risk flattening or distorting the confidence ranking\. This explains why Perturbation\-Response Signatures can improve failure diagnosis while source\-aware scalar confidence can underperform scalar confidence in selective prediction\.
### 7\.5Routing Depends On Fix And Break Rates
Letaabe the baseline accuracy before intervention\. For a policyπ\\pi, letFixπ\\mathrm\{Fix\}\_\{\\pi\}be the fraction of originally incorrect examples corrected by the intervention and letBreakπ\\mathrm\{Break\}\_\{\\pi\}be the fraction of originally correct examples made incorrect\.
###### Proposition 5\(Accuracy Decomposition For Intervention Routing\)\.
The post\-intervention accuracy of policyπ\\piis:
Accπ=a\+\(1−a\)Fixπ−aBreakπ\.\\begin\{split\}&\\mathrm\{Acc\}\_\{\\pi\}=a\+\(1\-a\)\\mathrm\{Fix\}\_\{\\pi\}\-a\\mathrm\{Break\}\_\{\\pi\}\.\\end\{split\}\(18\)For two policiesπ1\\pi\_\{1\}andπ2\\pi\_\{2\}, policyπ1\\pi\_\{1\}has higher post\-intervention accuracy thanπ2\\pi\_\{2\}if and only if:
\(1−a\)\(Fixπ1−Fixπ2\)\>a\(Breakπ1−Breakπ2\)\.\\begin\{split\}&\(1\-a\)\(\\mathrm\{Fix\}\_\{\\pi\_\{1\}\}\-\\mathrm\{Fix\}\_\{\\pi\_\{2\}\}\)\\\\ &\\hskip 18\.49988pt\>a\(\\mathrm\{Break\}\_\{\\pi\_\{1\}\}\-\\mathrm\{Break\}\_\{\\pi\_\{2\}\}\)\.\\end\{split\}\(19\)
###### Proof\.
Among the originally correct fractionaa, the intervention preserves a fraction1−Breakπ1\-\\mathrm\{Break\}\_\{\\pi\}\. Among the originally incorrect fraction1−a1\-a, it repairs a fractionFixπ\\mathrm\{Fix\}\_\{\\pi\}\. Thus:
Accπ=a\(1−Breakπ\)\+\(1−a\)Fixπ,Accπ=a\+\(1−a\)Fixπ−aBreakπ\.\\begin\{split\}&\\mathrm\{Acc\}\_\{\\pi\}=a\(1\-\\mathrm\{Break\}\_\{\\pi\}\)\+\(1\-a\)\\mathrm\{Fix\}\_\{\\pi\},\\\\ &\\mathrm\{Acc\}\_\{\\pi\}=a\+\(1\-a\)\\mathrm\{Fix\}\_\{\\pi\}\-a\\mathrm\{Break\}\_\{\\pi\}\.\\end\{split\}\(20\)Subtracting the expression forπ2\\pi\_\{2\}from the expression forπ1\\pi\_\{1\}gives Equation\.[19](https://arxiv.org/html/2608.29193#S7.E19)\. ∎
Implications\.Proposition[5](https://arxiv.org/html/2608.29193#Thmproposition5)explains why intervention routing should be evaluated with both fix rate and break rate\. A generic intervention can repair some failures and still reduce accuracy if it breaks many originally correct examples\. This is especially important in high\-accuracy settings, where theaBreakπa\\mathrm\{Break\}\_\{\\pi\}term is large\. The result supports the paper’s routing analysis, where matched routing is most meaningful when it avoids harmful generic overcorrection\.
###### Corollary 2\(Break\-Rate Reduction Can Matter More Than Fix\-Rate Gain\)\.
If baseline accuracya\>0\.5a\>0\.5, then a one\-point reduction in break rate contributes more to post\-intervention accuracy than a one\-point increase in fix rate\. Formally, the coefficient onBreakπ\\mathrm\{Break\}\_\{\\pi\}in Eq\.[18](https://arxiv.org/html/2608.29193#S7.E18)has magnitudeaa, while the coefficient onFixπ\\mathrm\{Fix\}\_\{\\pi\}is1−a1\-a\.
###### Proof\.
The claim follows directly from Eq\.[18](https://arxiv.org/html/2608.29193#S7.E18)\. The contribution of fix rate is weighted by1−a1\-a, while the contribution of break rate is weighted byaa\. Ifa\>0\.5a\>0\.5, thena\>1−aa\>1\-a, and break\-rate changes have larger absolute effect on final accuracy than equal\-sized fix\-rate changes\. ∎
Implications\.Corollary[2](https://arxiv.org/html/2608.29193#Thmcorollary2)clarifies why the break rate is emphasized alongside accuracy\. In settings such as Qwen3\-VL/VSR, the generic intervention harms many originally correct answers\. Reducing breakage can be more safety\-relevant than maximizing repair count alone\. This supports the conditional routing claim: Perturbation\-Response Signatures are useful when they prevent harmful interventions, not only when they increase pooled accuracy\.
### 7\.6Limits Of Scalar Abstention Under Open\-World Deployment
The preceding results imply a broader limitation of score\-based abstention\. Abstention is not an intrinsic inference\-time capability of a model\. It is a designer\-imposed guardrail over a score, threshold, prediction set, verifier, judge, or learned rejection rule\. Such a guardrail can support selective answering under validation assumptions\. However, it is not automatically an instance\-grounded verdict for a new input\. We separate two requirements\. First, the scalar score must be sufficient for the grounded safe\-answer label\. Second, the validation\-to\-deployment score\-risk relation must transfer to the inference distribution\. Without these requirements, a score\-based abstention rule can be executed, but it has no unconditional inference\-time guarantee\.
###### Theorem 2\(No Grounded Abstention From A Non\-Sufficient Scalar\)\.
LetX=\(I,q,y^\)X=\(I,q,\\hat\{y\}\)be a prediction instance\. LetY⋆∈\{0,1\}Y^\{\\star\}\\in\\\{0,1\\\}denote the grounded safe\-answer label, whereY⋆=1Y^\{\\star\}=1means that the model should answer under the target reliability criterion, andY⋆=0Y^\{\\star\}=0means that it should not answer\. LetR=r\(X\)∈ℝR=r\(X\)\\in\\mathbb\{R\}be any scalar abstention score, and let an abstention policy be any measurable functionh\(R\)∈\{0,1\}h\(R\)\\in\\\{0,1\\\}\.
A perfectly reliable scalar abstention policy exists if and only ifY⋆Y^\{\\star\}is measurable with respect toRR\. Equivalently, for almost every score value or score region used by the policy, all examples assigned the same scalar information must have the same grounded safe\-answer label\. If there exists a score regionBBwithP\(R∈B\)\>0P\(R\\in B\)\>0such that
P\(Y⋆=1∣R∈B\)\>0,P\(Y⋆=0∣R∈B\)\>0,\\begin\{split\}&P\(Y^\{\\star\}=1\\mid R\\in B\)\>0,\\\\ &P\(Y^\{\\star\}=0\\mid R\\in B\)\>0,\\end\{split\}\(21\)then every abstention policy based only onRRhas positive error onBB\. Therefore, unless the scalar score is sufficient for the grounded safe\-answer label, the question “should the model answer?” cannot be reliably answered from that scalar alone\.
###### Proof\.
IfY⋆Y^\{\\star\}is measurable with respect toRR, then there exists a measurable functionhhsuch thath\(R\)=Y⋆h\(R\)=Y^\{\\star\}almost surely\. This gives a perfectly reliable scalar abstention policy\. Conversely, suppose a perfectly reliable scalar abstention policyh\(R\)h\(R\)exists\. ThenY⋆=h\(R\)Y^\{\\star\}=h\(R\)almost surely, which meansY⋆Y^\{\\star\}is measurable with respect toRR\. For the positive\-error statement, condition onR∈BR\\in B\. Any policy based only onRRmust assign its decision using only the scalar information available inBB\. Since bothY⋆=1Y^\{\\star\}=1andY⋆=0Y^\{\\star\}=0have positive conditional mass inBB, the policy must be wrong on the mass assigned to at least one of these labels\. Hence, the error onBBis positive\. ∎
##### Implication\.
The theorem is score\-agnostic\. It applies to confidence, entropy, calibrated probability, semantic uncertainty, verifier scores, judge scores, conformal nonconformity scores, and learned rejection scores\. A scalar score can support reliable abstention only when it already contains the grounded safe\-answer state required by the deployment criterion\.
###### Proposition 6\(No Unconditional Inference\-Time Abstention Guarantee\)\.
LetR=r\(X\)R=r\(X\)be any scalar score used for abstention, and letτ\\taube a threshold chosen from validation data\. The inference\-time decision
answer\(X\)=𝟏\[R\(X\)≥τ\]\\begin\{split\}&\\mathrm\{answer\}\(X\)=\\mathbf\{1\}\[R\(X\)\\geq\\tau\]\\end\{split\}\(22\)is guaranteed to satisfy a target deployment risk only under a transfer condition connecting the validation and deployment distributions, for example
Pdeploy\(Y⋆=0∣R≥τ\)=Pval\(Y⋆=0∣R≥τ\)\.\\begin\{split\}&P\_\{\\mathrm\{deploy\}\}\(Y^\{\\star\}=0\\mid R\\geq\\tau\)\\\\ &\\hskip 18\.49988pt=P\_\{\\mathrm\{val\}\}\(Y^\{\\star\}=0\\mid R\\geq\\tau\)\.\\end\{split\}\(23\)Without such a condition, there exists a deployment distribution with the same score range and the same abstention rule for which the accepted region has arbitrarily high error\. Hence, score\-based abstention has no distribution\-free inference\-time guarantee over open\-world inputs\.
###### Proof\.
The thresholdτ\\tauis fixed after validation\. If no assumption linksPdeployP\_\{\\mathrm\{deploy\}\}toPvalP\_\{\\mathrm\{val\}\}, we can construct a deployment distribution that places positive, or even all, mass on examples withR≥τR\\geq\\tauandY⋆=0Y^\{\\star\}=0\. The abstention rule accepts these examples because it observes onlyRR\. Therefore, the accepted risk on deployment can be arbitrarily larger than the validation risk\. Any guarantee must therefore come from an external transfer assumption, not from the score\-threshold rule itself\. ∎
##### Implication\.
The proposition locates the open\-world weakness of score\-threshold abstention\. A validation risk curve, calibration curve, or prediction\-set guarantee depends on the validation support and the assumed transfer relation\. The accepted region is therefore warranted by the designer’s validation assumptions, not by an unconditional inference\-time guarantee for each new instance\.
###### Corollary 3\(No Source\-Complete Scalar Reliability\)\.
LetGGdenote a decision\-relevant grounding or failure\-source state, such as whether the answer is visually supported, image\-removal sensitive, or cross\-modally unstable\. LetA⋆A^\{\\star\}denote the safe correction action, withA⋆=noneA^\{\\star\}=\\textsc\{none\}when no correction is needed\. If eitherGGorA⋆A^\{\\star\}is not measurable with respect to a scalar abstention scoreRR, then no policy using onlyRRas an answer\-versus\-abstain interface is source\-complete\. Such a score may support selective answering under its assumptions, but it cannot identify the failure source or decide whether a generic correction is safe\.
###### Proof\.
A policy using onlyRRcan condition only on the information contained inRR\. IfGGis not measurable with respect toRR, then no function ofRRcan recover the grounding or failure\-source state\. IfA⋆A^\{\\star\}is not measurable with respect toRR, then no function ofRRcan determine the safe correction action\. Therefore, a scalar answer\-versus\-abstain interface can be selective without being source\-complete\. ∎
##### Implication\.
The corollary connects scalar abstention to source incompleteness\. A scalar score may support selective answering under its assumptions, although it cannot by itself identify the failure source, establish grounding, or decide whether a generic correction is safe\.
Table 6:Dataset details for theHalluPrismevaluation suite\. Counts are per model before pooling across the four evaluated model families\.Table 7:Failure\-family mapping used for diagnostic classification\. Correct predictions are mapped tono failure\. Incorrect predictions are mapped using each benchmark’s primary stress factor and available labels\.Table 8:Dataset composition and failure\-family coverage\. Counts are reported per model and after pooling across the four evaluated model families\.Table 9:Probe availability across datasets\. Relation\-swapped probing is used only when the input contains an explicit perturbable relation\.
## 8Dataset Details
We use four public multimodal benchmarks that stress different sources of prediction risk\. The goal is not to build a single pooled hallucination leaderboard\. We instead need a dataset suite in which visual fragility, image\-removal confidence retention, and grounding/relation\-probe instability can be observed under different task conditions\. HallusionBench contributes visual illusion and grounded hallucination cases, POPE contributes object\-presence hallucination probes, Visual Spatial Reasoning \(VSR\) contributes explicit relation\-binding examples, and VizWiz\-VQA contributes real\-user visual insufficiency and answerability cases\. This mixture supports the Perturbation\-Response view because the same scalar confidence value can occur in examples with different underlying diagnostic stresses\.
### 8\.1Dataset Roles
Table[6](https://arxiv.org/html/2608.29193#S7.T6)summarizes the datasets, splits, sample counts, and their roles in the paper\. The counts are reported per model before pooling across the four evaluated model families\. HallusionBench contains1,1291\{,\}129datapoints, out of which178178are text\-only\. SinceHalluPrismrequires visual perturbation, we evaluate the951951image\-paired examples\. POPE contributes9,0009\{,\}000object\-presence examples across random, popular, and adversarial splits\. VSR contributes340340zero\-shot development examples with explicit spatial relations\. VizWiz\-VQA contributes4,3194\{,\}319validation examples and is used for visual insufficiency and answerability analysis\.
### 8\.2HallusionBench
HallusionBench evaluates visually grounded hallucination and illusion cases\. We use the image\-paired subset because the Perturbation\-Response protocol requires image perturbations\. Text\-only examples are excluded since visual degradation, blank\-image replacement, and image\-grounded alignment probes would not be meaningful without an associated image\. The evaluated subset contains yes/no questions, which allows answer normalization into canonicalyesandnolabels\. We map the benchmark categories into attribute\-level and object\-level hallucination labels for failure\-family diagnosis\. HallusionBench is useful for the main story because it contains mixed visual and semantic pressures: a model may answer from a plausible prior, from partial visual evidence, or from unstable image\-text binding\.
### 8\.3POPE
POPE provides object\-presence hallucination probes\. We use all9,0009\{,\}000examples per model across the random, popular, and adversarial splits\. The task is binary, and outputs are normalized toyesandno\. POPE is primarily used to measure object\-hallucination behavior under strong object\-prior pressure\. It is also useful for the abstention boundary check because object\-presence examples often produce high scalar confidence even when image\-removal confidence retention is high\. Since POPE does not contain explicit spatial relations, the relation\-swap alignment probe is not used\. Alignment is measured through the grounding re\-check probe\.
### 8\.4Visual Spatial Reasoning
VSR evaluates whether an image satisfies a spatial caption\. We use340340zero\-shot development examples\. The base prompt asks the model whether the caption is true or false, and outputs are normalized totrueandfalse\. VSR is the clearest dataset for relation\-sensitive alignment probing because captions contain explicit spatial relations such asleft,right,above,below,in front of, andbehind\. For VSR, we use both alignment probes: \(i\) grounding re\-checking and \(ii\) relation\-swapped caption verification\. The relation\-swapped probe is central to testing whether the model preserves the same answer when the expected relation should change\. VSR therefore anchors the cross\-modal binding part of the Perturbation\-Response Signature\.
### 8\.5VizWiz\-VQA
VizWiz\-VQA contains visual questions from blind or low\-vision users\. We use the validation split with4,3194\{,\}319examples\. The dataset is important for two reasons\. First, many examples contain weak, blurry, poorly framed, or incomplete visual evidence, which directly stresses visual fragility\. Second, some questions are not answerable from the image, making the dataset useful for answerability diagnosis\. We instruct the model to returnunanswerablewhen the image is too unclear or the question cannot be answered from visual evidence\. We normalize answer variants before exact comparison\. Since VizWiz questions are open\-ended and do not usually contain controlled spatial relations, the relation\-swap probe is not used\. Alignment is measured through grounding re\-checking\.
### 8\.6Failure\-Family Mapping
Each model output receives a correctness label and a failure\-family label\. Correct predictions are mapped tono failure\. Incorrect predictions are mapped according to the benchmark stress factor and the available metadata\. This mapping is intentionally coarse \(see Table[7](https://arxiv.org/html/2608.29193#S7.T7)\)\. We use it to test whether Perturbation\-Response Signatures diagnose broad failure families, not to claim exhaustive error\-taxonomy coverage\.
### 8\.7Probe Availability By Dataset
We use visual degradation, blank\-image replacement, and grounding re\-checking for all image\-paired datasets\. We use relation\-swapped probing only when the input contains an explicit perturbable relation, which applies to VSR \(see Table[9](https://arxiv.org/html/2608.29193#S7.T9)\)\.
### 8\.8Pooled Evaluation Units
The basic evaluation unit is a model\-output instance\(x,M\)\(x,M\), wherexxis a dataset example andMMis an evaluated model family\. We use this pooled unit for source\-profile summaries, dominant\-source counts, failure\-family diagnosis, abstention scalarization, and routing analysis\. The per\-dataset and pooled counts are reported in Table[8](https://arxiv.org/html/2608.29193#S7.T8)\. We also report dataset\-identity controls and within\-dataset diagnosis because benchmark identity and failure family are partially aligned by design\.
## 9Complete Prompt Catalog
We provide the exact templates used for clean prediction, Perturbation\-Response probing, probe\-level entanglement tests, intervention routing, and auxiliary confidence reporting\. These templates define the reproducible prompting protocol used in the experiments\.
### 9\.1Base Dataset Prompts
Base prompts define the clean predictiony^\\hat\{y\}for each dataset\.
Yes/No Visual Reasoning: POPE and HallusionBenchRequired Input:•Question: \{question\}Prompt Template:\{question\}Answer with only ‘Yes’ or ‘No’\.
Spatial Caption Verification: VSRRequired Input:•Caption: \{caption\}Prompt Template:Look at the image carefully\. Is the following statement true or false?Statement: “\{caption\}”Answer with only ‘True’ or ‘False’\.
Open\-Ended VQA With Answerability: VizWizRequired Input:•Question: \{question\}Prompt Template:\{question\}If the image is too unclear to answer, or the question cannot be answered from the image, say ‘unanswerable’\. Otherwise, give a short answer \(one or two words\)\.
### 9\.2Diagnostic Perturbation\-Response Probes
These probes produce the perturbed answers used to computeVV,LL, andAA\.
#### 9\.2\.1Visual Fragility Probe
The visual probe keeps the base prompt fixed and applies image degradations\.
Visual Fragility ProbeRequired Input:•Image: \{image\}•Base Prompt: \{base\_prompt\}Image Perturbations:•Gaussian blur withσ=15\\sigma=15\.•Center crop covering50%50\\%of each image dimension, resized back to the original size\.•Brightness reduction to30%30\\%of the original image\.•Gaussian noise withμ=0\\mu=0andσ=25\\sigma=25\.Prompt Template:\{base\_prompt\}
#### 9\.2\.2Image\-Removal Confidence\-Retention Probe
The image\-removal probe replaces the original image with a blank canvas while keeping the base prompt fixed\.
Image\-Removal Confidence\-Retention ProbeRequired Input:•Blank Image: \{blank\_image\}•Base Prompt: \{base\_prompt\}Image Replacement:The original image is replaced with a uniform blank canvas of the same input size\.Prompt Template:\{base\_prompt\}
#### 9\.2\.3Cross\-Modal Alignment Probe
The alignment probe uses grounding re\-checking for all datasets and relation\-swapped prompting only for VSR\.
Alignment Probe: Grounding Re\-CheckRequired Input:•Original Prompt: \{original\_prompt\}Prompt Template:Look at the image carefully\. First, describe what you see in the image in one sentence\. Then answer the following question\.\{original\_prompt\}
Alignment Probe: Relation\-Swapped Prompt for VSRRequired Input:•Caption: \{caption\}•Relation\-Swapped Caption: \{relation\_swapped\_caption\}Prompt Template:Look at the image carefully\. Is the following statement true or false?Statement: “\{relation\_swapped\_caption\}”Answer with only ‘True’ or ‘False’\.
##### Relation\-swap construction\.
For VSR, we construct relation\-swapped captions using dataset relations such asleft,right,above,below,in front of, andbehind\.
### 9\.3Probe\-Level Entanglement Stress Tests
These probes test whether targeted perturbations produce diagonal or off\-diagonal responses in the measured Perturbation\-Response Signature\.
Language\-Ruined ProbeRequired Input:•Decoy Question: \{decoy\_question\}•Answer Constraint: \{constraint\}Example Decoy Question:What is the capital city of France?Prompt Template:\{decoy\_question\}\{constraint\}
Alignment\-Ruined Probe: Fallback Opposite\-Answer PromptUse Case:This fallback is used when deterministic relation inversion or polarity inversion is unavailable\.Required Input:•Original Prompt: \{original\_prompt\}Prompt Template:Answer the OPPOSITE of what you would normally answer to the following question\.\{original\_prompt\}
### 9\.4Intervention Routing Policies
We provide the full prompt and parameter catalog for intervention routing\. After computing the normalized dominant sourcesnorm∗\(x,M\)s^\{\*\}\_\{\\mathrm\{norm\}\}\(x,M\), we choose one of three matched correction paths: visual enhancement for visual\-dominant cases, anti\-prior prompting for L\-Dominant Cases, and grounding or relation checking for alignment\-dominant cases\. These templates and parameters define the intervention implementation used in the routing results\. Appendix[10\.11](https://arxiv.org/html/2608.29193#S10.SS11)reports only the empirical outcomes\.
#### 9\.4\.1Generic Intervention Baseline
The generic baseline applies the same correction prompt to every example\.
Generic Reasoning InterventionRequired Input:•Original Prompt: \{original\_prompt\}Prompt Template:Think step by step about the image and question before answering\.\{original\_prompt\}
#### 9\.4\.2Matched Intervention for Visual\-Dominant Cases
For visual\-dominant cases, we apply image enhancement and keep the original prompt fixed\.
Visual\-Dominant Intervention: Image EnhancementRequired Input:•Image: \{image\}•Original Prompt: \{original\_prompt\}Image Enhancement:•Upscale the image by1\.5×1\.5\\timesusing LANCZOS interpolation\.•Increase color saturation by1\.1×1\.1\\times\.•Increase sharpness by2\.5×2\.5\\times\.Prompt Template:\{original\_prompt\}
#### 9\.4\.3Matched Intervention for L\-Dominant Cases
For L\-Dominant Cases, we add an anti\-prior instruction before the original prompt\.
L\-Dominant Intervention: Anti\-Prior PromptRequired Input:•Original Prompt: \{original\_prompt\}Prompt Template:Look at the image very carefully before answering\. Do NOT guess based on common sense or what is usually true\. Base your answer ONLY on what you can actually see in this specific image\.\{original\_prompt\}
#### 9\.4\.4Matched Intervention for Alignment\-Dominant Cases
For alignment\-dominant cases, we use grounded answering for general inputs and relation checking for VSR\.
Alignment\-Dominant Intervention: Grounded AnsweringRequired Input:•Original Prompt: \{original\_prompt\}Prompt Template:Look at the image carefully\. First, describe what you see in the image in one sentence\. Then answer the following question based on your description\.\{original\_prompt\}
Alignment\-Dominant Intervention: VSR Relation CheckRequired Input:•Caption: \{caption\}Prompt Template:Follow these steps:1\. First, list the objects you can see in the image\.2\. Then, describe the spatial relationship between them\.3\. Finally, is this statement true or false: “\{caption\}”Answer with only ‘True’ or ‘False’ on the last line\.
### 9\.5Answer Normalization
We apply answer normalization before comparing clean, perturbed, and intervened outputs\. For binary yes/no datasets, surface variants such as ‘yes’, ‘Yes\.’, and ‘yes, it is’ are mapped to the canonical labelyes; analogous variants are mapped tono\. For VSR,trueandfalseare the canonical labels\. For VizWiz,unanswerablevariants are mapped to a single answerability label, and short open\-ended answers are lowercased and stripped of punctuation before exact matching\. These normalization rules are applied consistently to clean predictions, diagnostic probes, scalarization comparisons, and intervention\-routing evaluations\.
## 10Additional Results
We provide additional result that support the Perturbation\-Response view\. We first report probe\-design controls for image\-removal confidence\-retention saturation and alignment availability, and then specify dataset composition and source\-profile statistics\. We then examine controlled interventions and their off\-target effects\. After that, we analyze failure\-family diagnosis, dataset\-identity controls, human validation, abstention, and intervention routing\. The additional tables form a single empirical trace from source prevalence to diagnostic utility\. The main pattern is consistent across the appendix\. Multimodal source signals are entangled under intervention, yet their joint signature remains informative for failure diagnosis\. The same signature should not be treated as a direct scalar confidence correction\.
### 10\.1Image\-Removal Saturation Controls
The default image\-removal scoreL=LconfL=L\_\{\\mathrm\{conf\}\}in Eq\.[4](https://arxiv.org/html/2608.29193#S3.E4)measures confidence retention after removing image evidence and does not require the blank\-image answer to match the clean answer\. Because the clipped confidence ratio can saturate, we run implementation\-level controls to check whether the main conclusions depend on this formulation\. We also evaluate the stricter answer\-persistence scoreLpersistL\_\{\\mathrm\{persist\}\}from Eq\.[5](https://arxiv.org/html/2608.29193#S3.E5), which additionally requires the original answer to survive image removal\. These controls do not replace the default score used in the main paper\. They test whether image\-removal saturation alone explains the diagnostic results\.
We compare four variants\. First,LconfL\_\{\\mathrm\{conf\}\}is the default bounded confidence\-retention score from Eq\.[4](https://arxiv.org/html/2608.29193#S3.E4)\. Second,LpersistL\_\{\\mathrm\{persist\}\}additionally requires the blank\-image answer to match the clean answer\. Third,LmatchL\_\{\\mathrm\{match\}\}removes the confidence ratio and keeps only blank\-image answer survival:
Lmatch=𝟏\[match\(y^,y~ℓ\)\]\.\\begin\{split\}&L\_\{\\mathrm\{match\}\}=\\mathbf\{1\}\[\\mathrm\{match\}\(\\hat\{y\},\\tilde\{y\}^\{\\ell\}\)\]\.\\end\{split\}\(24\)Fourth,LstableL\_\{\\mathrm\{stable\}\}keeps the confidence\-ratio interpretation but suppresses low\-denominator amplification by replacing the clean\-confidence denominator with a fixed floorτc\\tau\_\{c\}:
Lstable\\displaystyle L\_\{\\mathrm\{stable\}\}=𝟏\[match\(y^,y~ℓ\)\]\\displaystyle=\\mathbf\{1\}\[\\mathrm\{match\}\(\\hat\{y\},\\tilde\{y\}^\{\\ell\}\)\]\(25\)⋅min\(c\(y~ℓ\)max\(c\(y^\),τc\)\+ϵ,1\)\.\\displaystyle\\cdot\\min\\left\(\\frac\{c\(\\tilde\{y\}^\{\\ell\}\)\}\{\\max\(c\(\\hat\{y\}\),\\tau\_\{c\}\)\+\\epsilon\},1\\right\)\.We setτc=0\.5\\tau\_\{c\}=0\.5in all experiments\.
Table 10:Image\-removal saturation controls\.LLDom\. denotes the fraction of examples for whichLLis the raw dominant coordinate\. AUROC and Macro\-F1 use the same pooled failure\-family diagnosis setting with a stratified 5\-fold logistic\-regression classifier, replacing only the image\-removal coordinate\.ForLpersistL\_\{\\mathrm\{persist\}\}, observed clean/blank answer agreement is51\.7%51\.7\\%, compared with28\.8%28\.8\\%agreement expected from the answer marginals, giving chance\-adjusted agreement0\.3210\.321\. XGBoost AUROC remains0\.9520\.952with\(V,Lpersist,A\)\(V,L\_\{\\mathrm\{persist\}\},A\)and0\.9670\.967when scalar confidence is added\.
We observe from Table[10](https://arxiv.org/html/2608.29193#S10.T10)that the raw dominance rate ofLLdrops sharply when answer matching is required, from99\.7%99\.7\\%forLconfL\_\{\\mathrm\{conf\}\}to about51\.651\.6–51\.7%51\.7\\%for the stricter variants\. However, the diagnostic result remains stable\. ReplacingLconfL\_\{\\mathrm\{conf\}\}withLpersistL\_\{\\mathrm\{persist\}\},LmatchL\_\{\\mathrm\{match\}\}, orLstableL\_\{\\mathrm\{stable\}\}changes AUROC only slightly, while Macro\-F1 remains comparable or improves\. Thus, the main diagnostic pattern does not depend on the broader confidence\-retention formulation\. The defaultLLshould be read as confidence retention under image removal, not as answer persistence or a causal source label\.
### 10\.2Alignment Availability Controls
The alignment score in Eq\.[6](https://arxiv.org/html/2608.29193#S3.E6)is availability\-aware\. The grounding re\-check component is available for all datasets, while relation\-swap probing is available only when the input contains an explicit perturbable relation\. In our evaluation suite, this applies to VSR\. Therefore, rawAAmagnitudes should not be interpreted as directly comparable source intensities across all datasets\. We instead useAAas a behavioral alignment\-stress coordinate and rely on dataset/model controls and within\-setting normalization when the score is used across heterogeneous benchmarks\.
We run an availability control to test whether the diagnostic value ofAAis driven by the VSR\-only relation\-swap branch or by the0\.4/0\.60\.4/0\.6grounding\-relation weighting\. This control removes VSR entirely from the pooled diagnosis setting\. Since VSR is the only relation\-eligible dataset, the no\-VSR setting contains no relation\-swap examples and no example for which the relation weight can affectAA\. The comparison therefore asks whether the Perturbation\-Response Signature remains diagnostic when only the non\-relation benchmark suite is used\.
Table 11:Alignment availability control\. We compare scalar confidenceccwith the Perturbation\-Response Signature\(V,L,A\)\(V,L,A\)under the same stratified 5\-fold logistic\-regression failure\-family diagnosis setting\. The no\-VSR setting removes the only relation\-swap\-eligible dataset, so any remaining gain cannot be attributed to the VSR\-only relation branch or to the grounding\-relation weight in Eq\.[6](https://arxiv.org/html/2608.29193#S3.E6)\.We observe from Table[11](https://arxiv.org/html/2608.29193#S10.T11)that removing VSR does not reduce the diagnostic value of the Perturbation\-Response Signature\. With all datasets,\(V,L,A\)\(V,L,A\)improves AUROC over scalar confidence from0\.7520\.752to0\.8280\.828and Macro\-F1 from0\.2600\.260to0\.3330\.333\. After excluding VSR, the gain remains nearly the same in AUROC and becomes larger in Macro\-F1: AUROC improves from0\.7610\.761to0\.8390\.839, while Macro\-F1 improves from0\.3290\.329to0\.4220\.422\. This indicates that the main diagnosis result is not an artifact of the relation\-swap branch or the particular0\.4/0\.60\.4/0\.6weighting used when relation\-swap probing is available\. The relation\-swap branch should be read as a stronger task\-valid alignment stress for relation\-bearing inputs, not as the source of the pooled diagnostic effect\.
### 10\.3Visual\-Perturbation Robustness
We test whether the diagnosis result depends on the choice or strength of the visual perturbations\. Removing blur, crop, brightness, or noise individually keeps diagnosis AUROC between0\.8220\.822and0\.8240\.824, while normalized source assignments agree with the complete four\-probe version on89\.8%89\.8\\%–97\.8%97\.8\\%of examples\.
We further evaluate weak, default, and strong perturbations on the same10,54010\{,\}540model\-output instances\.
Table 12:Visual\-perturbation severity analysis\. Stronger perturbations increase both mean visual sensitivity and answer\-flip rate\.Instance rankings remain stable relative to the default setting, with Spearman correlation0\.8920\.892for weak and0\.9240\.924for strong perturbations\. WithLpersistL\_\{\\mathrm\{persist\}\}, diagnosis AUROC is0\.7850\.785,0\.7860\.786, and0\.7830\.783under weak, default, and strong perturbations, respectively, compared with0\.6320\.632for scalar confidence\. Macro\-F1 remains0\.4050\.405–0\.4060\.406, compared with0\.1790\.179for scalar confidence\. All paired\-bootstrap intervals for the AUROC and Macro\-F1 gains exclude zero\.
Table 13:Full source\-profile with accuracy and raw dominant\-source \(Dom\) counts\. LP Match denotes exact\-answer agreement between the original image and blank\-image condition\.Table 14:Largest\-Coordinate Counts\. We compute the largest coordinate usingargmax\(V,L,A\)\\arg\\max\(V,L,A\)\. The largest coordinate is overwhelminglyLLbecauseLLis near saturation in most settings; this count should therefore be read as prevalence, not diagnostic value\.
### 10\.4Full Source Profiles
We use the full source\-profile tables to separate raw prevalence from diagnostic value\. We observe from Table[13](https://arxiv.org/html/2608.29193#S10.T13)thatLLis high across nearly all model\-dataset settings, with means from0\.9140\.914to1\.0001\.000and blank\-image match rates from37\.5%37\.5\\%to96\.5%96\.5\\%\. In contrast,VVandAAshow stronger task dependence: VizWiz gives the largest visual\-fragility values, while VSR gives the largest alignment values for Gemma 4 and Qwen3\-VL\. We observe from Table[14](https://arxiv.org/html/2608.29193#S10.T14)that rawargmax\(V,L,A\)\\arg\\max\(V,L,A\)selectsLLfor58,287/58,44058\{,\}287/58\{,\}440pooled instances\. This saturation makes raw dominance a poor proxy for diagnostic value, motivating our use of the full\(V,L,A\)\(V,L,A\)vector for failure\-family diagnosis and normalized dominance for routing\.
Table 15:Magnitude\-based diagonal\-dominance pass counts for the interventional entanglement test\. A check passes when the intended source coordinate has the largest absolute change under the corresponding ruined condition\.Table 16:Interventional EntanglementΔ\\Deltafor Gemma 4\. A check passes when the intended source coordinate has the largest absolute change under the corresponding ruined condition\. Resultsare computed using unroundedΔ\\Delta\.Table 17:Interventional EntanglementΔ\\Deltafor LLaVA\-Mistral\. A check passes when the intended source coordinate has the largest absolute change under the corresponding ruined condition\. Results are computed using unroundedΔ\\Delta\.Table 18:Interventional EntanglementΔ\\Deltafor LLaVA\-Vicuna\. A check passes when the intended source coordinate has the largest absolute change under the corresponding ruined condition\. Results are computed using unroundedΔ\\Delta\.Table 19:Interventional EntanglementΔ\\Deltafor Qwen3\-VL\. A check passes when the intended source coordinate has the largest absolute change under the corresponding ruined condition\. Results are computed using unroundedΔ\\Delta\.
### 10\.5Interventional Entanglement Across Models
We next test whether targeted interventions isolate the intended source score\. This test is stricter than source profiling\. If the three scores behaved as independent source components, visual\-ruined inputs would produce the largest absolute change inVV, language\-ruined inputs would produce the largest absolute change inLL, and alignment\-ruined inputs would produce the largest absolute change inAA\. We observe from Table[15](https://arxiv.org/html/2608.29193#S10.T15)that this magnitude\-based diagonal\-dominance pattern holds only partially\. Across all model\-dataset\-intervention checks,1818of4848pass\. Visual\-ruined inputs pass in1111of1616settings, while language\-ruined inputs pass in22of1616and alignment\-ruined inputs pass in55of1616\. This supports a more precise interpretation of\(V,L,A\)\(V,L,A\)\. Visual ruin is often source\-selective in magnitude, although the direction is not always positive\. Language and alignment interventions remain much less source\-selective\. Therefore, the vector should be read as a joint behavioral signature rather than as a clean causal decomposition\.
The model\-specific tables show how off\-target responses arise across backbones\. We observe from Table[16](https://arxiv.org/html/2608.29193#S10.T16)that Gemma 4 satisfies magnitude\-based dominance for visual\-ruined inputs in33of44settings, but the signed response is not always intuitive\. On VizWiz, visual ruin reducesVVby0\.1980\.198rather than increasing it\. On HallusionBench, visual ruin also increasesAAby0\.1260\.126, which shows that visual corruption can change alignment behavior\. These rows separate two effects: the intended coordinate may change most in magnitude, while the sign and off\-target movement still reveal non\-monotone source behavior\.
For LLaVA\-Mistral, we observe from Table[17](https://arxiv.org/html/2608.29193#S10.T17)that visual\-ruined inputs pass in33of44settings, while language\-ruined inputs pass in none\. VizWiz language ruin increasesAAby0\.2550\.255, and VSR visual ruin changes bothVVandAAin the negative direction\. These responses indicate that entanglement is not limited to one model family\. They also show why magnitude and signed direction should be interpreted separately\.
For LLaVA\-Vicuna, we observe from Table[18](https://arxiv.org/html/2608.29193#S10.T18)that only22of1212checks pass overall\. HallusionBench alignment ruin passes, while most other checks fail the magnitude\-based criterion\. POPE visual ruin increasesAAby0\.2390\.239, even though the intended target is visual\. VSR language ruin also increasesAAby0\.1970\.197\. These rows reinforce the joint\-signature interpretation because off\-target alignment responses can dominate or substantially accompany the intended response\.
For Qwen3\-VL, we observe from Table[19](https://arxiv.org/html/2608.29193#S10.T19)that77of1212checks pass, including all four visual\-ruined settings\. However, the signed deltas still show non\-monotone behavior\. On HallusionBench, visual ruin reducesVVby0\.0850\.085rather than increasing it, while language ruin reducesLLby0\.0860\.086and increasesAAby0\.1750\.175\. On POPE, language ruin reducesLLby0\.1490\.149and increasesAAby0\.2930\.293\. These cases show why independent source attribution is too strong\. The vector records how the answer changes under controlled pressure, and a ruined condition can shift the model toward a different answering route\.
### 10\.6Perturbation\-Response Signatures By Failure Family
We connect Perturbation\-Response Signatures to failure families before turning to predictive diagnosis\. Table[20](https://arxiv.org/html/2608.29193#S10.T20)reports pooled source means by raw failure\-family label\. Object hallucination is associated with very high image\-removal confidence retention, withL¯=0\.996\\bar\{L\}=0\.996, and a high scalar score of0\.9670\.967\. Answerability cases also have near\-saturated image\-removal confidence\-retention scores, withL¯=0\.997\\bar\{L\}=0\.997, although their accuracy is lower at77\.4%77\.4\\%\. The no\-annotated\-family row has the highest mean visual and alignment values, withV¯=0\.289\\bar\{V\}=0\.289andA¯=0\.587\\bar\{A\}=0\.587, because this row is heavily shaped by answerable VizWiz instances\. This row should not be read as the correctness\-conditioned no\-failure target used in the outcome\-family controls\. These values explain why single\-coordinate interpretations are insufficient\. Failure families do not map cleanly onto one isolated source\. Instead, they occupy different regions of the joint Perturbation\-Response space\.
Table 20:Pooled source\-signature means by failure family\.
### 10\.7Full Failure\-Family Diagnosis
We report model\-wise failure\-family AUROC in Table[21](https://arxiv.org/html/2608.29193#S10.T21)\. The table compares scalar confidence, the Perturbation\-Response Signature\(V,L,A\)\(V,L,A\), and the combined representation\(V,L,A,c\)\(V,L,A,c\)across classifier families\. We report tree\-based feature importance in Table[22](https://arxiv.org/html/2608.29193#S10.T22)\. These results provide the full supporting evidence for the main diagnosis claim summarized in Section[5](https://arxiv.org/html/2608.29193#S5)\.
Table 21:Failure\-family prediction AUROC by model and classifier\.ΔVLA−S\\Delta\_\{VLA\-S\}denotes the gain ofV\+L\+AV\+L\+Aover scalar confidence\.Table 22:Feature importance for tree\-based failure\-family diagnosis\.
### 10\.8Dataset\-Identity And Within\-Dataset Controls
The evaluation suite is deliberately heterogeneous: POPE primarily stresses object hallucination, VSR stresses spatial and relation errors, VizWiz stresses answerability, and HallusionBench contributes mixed hallucination cases\. This design supports the diagnosis\-first claim, although it also means that pooled failure\-family prediction can partly reflect benchmark identity\. We therefore use dataset\-identity, within\-dataset, leave\-dataset\-out, and matched source\-stress controls to test whether Perturbation\-Response Signatures add information beyond coarse dataset and model identifiers\.
We observe from Table[23](https://arxiv.org/html/2608.29193#S10.T23)that dataset and model identifiers already give a strong raw\-family baseline\. Dataset ID alone reaches AUROC0\.9760\.976and Macro\-F10\.5580\.558, while adding scalar confidence raises the result to AUROC0\.9860\.986and Macro\-F10\.7050\.705\. AddingV\+L\+AV\+L\+Ato dataset and model identifiers improves AUROC to0\.9910\.991and Macro\-F1 to0\.7830\.783\. The AUROC gain is small because the controlled baseline is already high, while the Macro\-F1 gain indicates better handling of less dominant failure labels\.
Table 23:Dataset\-ID controlled failure\-family diagnosis using a stratified held\-out split\. The target is the benchmark\-provided failure\-family label used in the pooled diagnosis experiment\.We observe from Table[24](https://arxiv.org/html/2608.29193#S10.T24)that the stricter outcome\-family control gives a smaller but positive gain\. Here correct predictions are mapped tono failure, so the classifier must separate both correctness and error family\. Dataset ID and model ID remain weak for Macro\-F1, both at0\.1450\.145\. Adding scalar confidence raises Macro\-F1 to0\.3080\.308, and the combined representation reaches the best result with AUROC0\.8530\.853and Macro\-F10\.3480\.348\.
Table 24:Dataset\-ID controlled outcome\-family diagnosis using a stratified held\-out split\. Correct predictions are mapped to no\-failure and incorrect predictions to their observed failure family\.Table 25:Within\-dataset diagnostic controls\. Each row fixes the dataset and includes model identity in all feature sets\. Improvements fromV\+L\+AV\+L\+Atherefore cannot be attributed to dataset identity alone\.We see in Table[25](https://arxiv.org/html/2608.29193#S10.T25)that the clearest fixed\-dataset gains appear on HallusionBench and VizWiz\. On HallusionBench, scalar confidence reaches AUROC0\.6340\.634and F10\.6210\.621, while\(V,L,A\)\(V,L,A\)reaches AUROC0\.7690\.769and F10\.7180\.718\. The combined representation reaches AUROC0\.8010\.801and F10\.7390\.739\. On VizWiz, scalar confidence reaches AUROC0\.7070\.707and F10\.6930\.693, while\(V,L,A\)\(V,L,A\)reaches AUROC0\.8170\.817and F10\.7820\.782\. The combined representation reaches AUROC0\.8250\.825and F10\.7840\.784\.
POPE and VSR give more conservative fixed\-dataset results\. On POPE, scalar confidence reaches AUROC0\.8230\.823, while\(V,L,A\)\(V,L,A\)reaches0\.8390\.839; the combined representation reaches AUROC0\.8600\.860\. On VSR, scalar confidence and\(V,L,A\)\(V,L,A\)are nearly matched, with AUROC0\.7220\.722and0\.7240\.724, respectively; the combined representation reaches AUROC0\.7350\.735\. These rows bound the diagnosis claim: Perturbation\-Response Signatures help most when the target contains multiple failure modes or answerability\-sensitive distinctions\.
Table 26:Leave\-dataset\-out binary failure detection\. This is a stress test only, because multiclass failure\-family support differs across datasets\.Table 27:Matched source\-stress slices from existing metadata\. These slices check whether source signatures align with dataset\-specific stress factors under fixed task conditions\.We include leave\-dataset\-out binary transfer as a stress test in Table[26](https://arxiv.org/html/2608.29193#S10.T26)\. The results are weak overall, which is expected because the datasets differ in task structure and failure\-label support\. Holding out POPE gives AUROC0\.2010\.201with model ID and scalar confidence,0\.3750\.375with model ID andV\+L\+AV\+L\+A, and0\.2290\.229with the combined representation\. Holding out HallusionBench gives0\.3390\.339,0\.4920\.492, and0\.3710\.371, respectively\. We therefore use this table as a transfer stress test, not as the main diagnosis evidence\. We find from Table[27](https://arxiv.org/html/2608.29193#S10.T27)that matched source\-stress slices align with the same interpretation\. In POPE, image\-removal confidence\-retention scores remain near saturation across adversarial, popular, and random splits, withL¯\\bar\{L\}between0\.9970\.997and0\.9980\.998, while accuracy drops from90\.3%90\.3\\%on random to86\.5%86\.5\\%on adversarial\. In VizWiz, answerable examples have higher visual and alignment scores than unanswerable examples, withV¯=0\.289\\bar\{V\}=0\.289andA¯=0\.587\\bar\{A\}=0\.587versusV¯=0\.124\\bar\{V\}=0\.124andA¯=0\.479\\bar\{A\}=0\.479\. VSR relation slices show stable high image\-removal confidence\-retention scores and moderate alignment scores across relations\. These slices provide interpretive support for reading the full\(V,L,A\)\(V,L,A\)vector rather than a single coordinate\.
### 10\.9Human Audit Results
We report the completed human\-audit protocol in Table[28](https://arxiv.org/html/2608.29193#S10.T28)\. The audit follows the protocol in Appendix[11](https://arxiv.org/html/2608.29193#S11)\. We use200200examples, with5050examples each from HallusionBench, POPE, VSR, and VizWiz\. Three annotators assign one of six labels: visual evidence insufficiency, language\-prior reliance, alignment error, mixed, unclear, or no failure\. Annotators see the image, question or caption, reference answer, model answer, and correctness flag\. They do not seeVV,LL,AA, scalar confidence, raw dominant source, normalized dominant source, or routing decision\. We manually adjudicate3636disagreement cases before computing the final audit labels\.
Table 28:Human\-audit reporting template\. Annotators label the dominant failure source as visual, language\-prior, alignment, mixed, unclear, or no\-failure\. We report agreement \(Fleiss’sκ\\kappa\) before adjudication and model\-human agreement after adjudication\.We report annotator agreement and adjudicated label counts in Table[29](https://arxiv.org/html/2608.29193#S10.T29)\. We read the audit as bounded semantic support for the Perturbation\-Response view\. Full\-label Fleiss’κ\\kappais0\.2360\.236, and source versus non\-source Fleiss’κ\\kappais0\.4800\.480\. Exact all\-annotator agreement is20\.5%20\.5\\%, while at least two annotators agree in82\.0%82\.0\\%of examples\. These values indicate fair full\-label agreement and stronger agreement when the task is reduced to source versus non\-source attribution\. This behavior is expected under the paper’s interpretation, since visual evidence, language priors, and cross\-modal binding can interact in the same example\.
QuantityValueExamples200Annotators3Adjudicated disagreement cases36Exact all\-annotator agreement20\.5%At least two annotators agree82\.0%Full\-label Fleiss’κ\\kappa0\.236Source vs non\-source Fleiss’κ\\kappa0\.480Alignment71No failure4Unclear87Language\-prior20Visual16Mixed2Table 29:Human\-audit agreement and adjudicated label distribution\. The audit uses 200 examples, with 50 examples per dataset\.After adjudication,107107examples receive a clear source label:7171alignment,2020language\-prior, and1616visual\. The remaining cases are8787unclear and44no\-failure, and22mixed cases after adjudication\. Among the140140originally incorrect examples, the adjudicated labels are7171alignment,3232unclear,1919language\-prior,1616visual, and22mixed\. This distribution differs sharply from the raw largest\-coordinate count, which is almost alwaysLL\-dominant\. It instead matches the main diagnostic pattern in the paper: image\-removal confidence retention is the most prevalent measured signal, while alignment is more useful for separating failure behavior\.
We compare the adjudicated human label with raw and normalized dominant sources only on clear\-source cases\. Mixed, unclear, and no\-failure cases are reported separately rather than forced into one of the three source coordinates\. We observe from Table[30](https://arxiv.org/html/2608.29193#S10.T30)that raw dominant source has low agreement with human clear\-source labels because it collapses to language\-prior\. Raw dominance reaches clear\-source accuracy0\.1650\.165, Macro\-F10\.0940\.094, and Cohen’sκ=0\.000\\kappa=0\.000\. Normalized dominance improves over raw dominance, with clear\-source accuracy0\.3630\.363, Macro\-F10\.3070\.307, and Cohen’sκ=0\.053\\kappa=0\.053, although the agreement remains limited\. This supports an important distinction\. Rawargmax\(V,L,A\)\\arg\\max\(V,L,A\)is useful for measuring prevalence, not for assigning a human\-like source label\.
Table 30:Human\-audit confusion\-matrix template for clear\-source examples\.We also evaluate a small audit\-side diagnostic classifier on the107107clear\-source examples\. We report this comparison in Table[31](https://arxiv.org/html/2608.29193#S10.T31)\. Scalar confidence reaches weighted AUROC0\.4300\.430for clear\-source label prediction, while raw\(V,L,A\)\(V,L,A\)reaches0\.5970\.597\. Normalized\(zV,zL,zA\)\(z\_\{V\},z\_\{L\},z\_\{A\}\)reaches0\.5560\.556, and raw\(V,L,A\)\(V,L,A\)with scalar confidence reaches0\.5870\.587\. The result is exploratory because the audit sample is small, although it is aligned with the main failure\-family diagnosis result\. The joint Perturbation\-Response Signature carries more source\-attribution signal than scalar confidence\. At the same time, the audit does not validate\(V,L,A\)\(V,L,A\)as independent causal sources\. It supports the restrained claim used throughout the paper: Perturbation\-Response Signatures are behavioral diagnostic objects, many examples remain genuinely ambiguous, and argmax collapse loses information that is present in the full vector\.
Table 31:Human audit\-side clear\-source prediction\. The classifier is evaluated on the 107 examples adjudicated as visual, language\-prior, or alignment\.
### 10\.10Abstention And Source\-Aware Scalarization
We examine whether Perturbation\-Response Signatures can be collapsed into scalar confidence scores\. We observe from Table[32](https://arxiv.org/html/2608.29193#S10.T32)that scalar confidence remains the strongest correctness\-ranking signal under the tested max\-penalty, weighted, and geometric scalarizations\. The pooled AUROC is0\.7830\.783for scalar confidence and0\.7670\.767for the best source\-aware scalarization\. The same pattern holds across model families, with the largest gap for Qwen3\-VL, where AUROC drops from0\.8090\.809to0\.7320\.732\. This supports the scalarization boundary: the Perturbation\-Response Signature can help diagnose failure family while still weakening abstention when compressed into a penalty\-adjusted confidence score\.
Table 32:Aggregate AUROC for all scalarization variants\.We observe from Table[33](https://arxiv.org/html/2608.29193#S10.T33)that this pattern is not only a pooled artifact\. Source\-aware scalarization is usually worse than scalar confidence across model\-dataset slices\. The largest drop appears for Qwen3\-VL on POPE, from AUROC0\.8260\.826to0\.7020\.702\. HallusionBench also shows clear drops for Gemma 4 and LLaVA\-Mistral, from0\.6900\.690to0\.6050\.605and from0\.6090\.609to0\.5520\.552, respectively\. The main exception is LLaVA\-Vicuna on VSR, where AUROC improves from0\.6850\.685to0\.7020\.702\. These exceptions are useful boundary cases, although the dominant pattern remains negative for naive scalarization\.
Table 33:Per\-dataset abstention comparison between scalar confidence and the best source\-aware scalarization\.We observe from Table[34](https://arxiv.org/html/2608.29193#S10.T34)that scalar confidence is especially stronger in the low\-coverage regime\. At5%5\\%coverage, scalar confidence has risk0\.0180\.018, compared with0\.0230\.023for weighted scalarization,0\.2900\.290for max\-penalty, and0\.2250\.225for geometric scalarization\. At10%10\\%coverage, scalar confidence has risk0\.0330\.033, while max\-penalty and geometric scalarization remain much higher at0\.1940\.194and0\.1710\.171\. These results locate the proper role of\(V,L,A\)\(V,L,A\): Perturbation\-Response Signatures are better suited for failure typing and routing than for naive scalar confidence replacement\.
Table 34:Pooled risk\-coverage values for scalar confidence and source\-aware scalarizations\.
### 10\.11Intervention Routing
We report the full intervention\-routing results in Table[35](https://arxiv.org/html/2608.29193#S10.T35)\. The comparison uses three conditions: \(i\) no intervention, \(ii\) a generic reasoning intervention applied to every example, and \(iii\) matched routing based on the normalized dominant Perturbation\-Response Signature\. The prompt templates and image\-enhancement parameters are given in Appendix[9\.4](https://arxiv.org/html/2608.29193#S9.SS4)\. Here we focus only on routing outcomes\. We observe from Table[35](https://arxiv.org/html/2608.29193#S10.T35)that matched routing is conditionally useful rather than a universal accuracy booster\. In the pooled result, no intervention reaches76\.6%76\.6\\%accuracy, generic intervention reaches75\.7%75\.7\\%, and matched routing reaches75\.9%75\.9\\%\. The aggregate accuracy difference is small\. The more relevant pattern is break\-rate control: generic intervention has a pooled break rate of4\.1%4\.1\\%, while matched routing lowers it to3\.9%3\.9\\%\. The clearest case appears for Qwen3\-VL on VSR\. Generic intervention reduces accuracy from87\.4%87\.4\\%to76\.2%76\.2\\%and breaks15\.5%15\.5\\%of originally correct predictions\. Matched routing reaches87\.6%87\.6\\%accuracy and lowers break rate to2\.7%2\.7\\%\. Gemma 4 on HallusionBench shows the same direction, with matched routing reducing break rate from15\.1%15\.1\\%to4\.9%4\.9\\%while recovering the no\-intervention accuracy level\. On LLaVA\-Mistral/VizWiz, both interventions reduce accuracy relative to no intervention, although matched routing still lowers break rate from21\.4%21\.4\\%to17\.4%17\.4\\%\. These cases show the intended use of routing: reducing harmful overcorrection when a generic prompt is disruptive\.
We observe from Table[36](https://arxiv.org/html/2608.29193#S10.T36)that matched routing beats generic intervention in10/1610/16model\-dataset settings and reduces break rate in10/1610/16settings\. It beats no intervention in8/168/16settings\. Thus,HalluPrismshould not be read as a general mitigation method\. Its decision\-facing value is conditional actionability: Perturbation\-Response Signatures help decide when one\-size\-fits\-all correction is risky and when a source\-matched correction is safer\.
Table 35:Full intervention\-routing results\. Generic applies one prompt to all samples\. Matched chooses an intervention from the dominant source signature\.Table 36:Model\-level routing win counts\.##### Additional Routing Control\.
We additionally compare the probe\-informed policy with blanket generic CoT on6,5616\{,\}561outputs from four model–dataset settings\. Relative to generic CoT, probe\-informed routing improves pooled accuracy by3\.543\.54percentage points \(95% CI\[2\.82,4\.25\]\[2\.82,\\,4\.25\]\) and reduces break rate by5\.125\.12points \(95% CI\[4\.12,6\.09\]\[4\.12,\\,6\.09\]\)\. The policy improves accuracy relative to generic CoT in all four settings, although it is not universally better than no intervention; VizWiz is a clear boundary case\. This control therefore supports only the narrow claim that probe\-informed correction can be safer than applying the same generic reasoning prompt everywhere\.
## 11Human Audit Protocol
We use a small human audit as a semantic plausibility check for the Perturbation\-Response interpretation\. The audit does not validate hidden causal mechanisms inside a Multimodal Large Language Model \(MLLM\)\. It only asks whether a human reader, given the image, question, reference answer, model answer, and correctness flag, can assign a plausible dominant failure source that agrees with the normalized Perturbation\-Response Signature\. The audit is therefore aligned with the main paper’s diagnosis\-first claim: Perturbation\-Response Signatures are behavioral diagnostic objects, not ground\-truth causes\.
### 11\.1Audit Objective
The audit checks whether the source labels used byHalluPrismare interpretable at the example level\. In particular, we ask whether model outputs with a visually dominant,LL\-dominant, or alignment\-dominant normalized signature correspond to human\-plausible explanations of the observed prediction behavior\. We do not use the audit to train the diagnostic classifier, tune source scores, select routing prompts, or choose scalarization weights\. All main quantitative results are computed without human audit feedback\.
### 11\.2Sampling Strategy
We sample examples from the same evaluated model\-dataset outputs used in the main experiments\. The audit sample contains200200examples, with5050examples from each benchmark: HallusionBench, POPE, VizWiz\-VQA, and Visual Spatial Reasoning \(VSR\)\. Within each dataset, we stratify by normalized dominant sourcesnorm∗\(x,M\)s^\{\*\}\_\{\\mathrm\{norm\}\}\(x,M\)whenever possible\. This choice is necessary because raw dominance is strongly affected by image\-removal confidence\-retention saturation, as shown in Appendix[10](https://arxiv.org/html/2608.29193#S10)\. When a source stratum has fewer available examples, we include all available examples from that stratum and fill the remaining slots with examples sampled from the other normalized\-source strata\. We also maintain a mixture of correct and incorrect predictions, since the audit checks source plausibility rather than only error explanation\.
Table 37:Human\-audit sampling plan\. The audit uses normalized dominant source for stratification because raw dominance is heavily affected by image\-removal confidence\-retention saturation\.
### 11\.3Annotator Recruitment And Payment
The human audit was conducted by student annotators recruited locally from undergraduate programs\. The annotators were in the 18\-22 age group and were studying Computer Science or Economics\. We selected annotators with sufficient technical background to understand image\-question answering tasks, model outputs, correctness flags, and the distinction between visual evidence, language\-prior reliance, and alignment\-related errors\. No annotator was asked to provide personal, demographic, medical, behavioral, or otherwise sensitive information beyond the minimal background needed to report the audit setting\. Annotators were not compensated in cash\. The compensation arrangement was communicated before participation\. The audit involved semantic labeling of benchmark examples and model outputs rather than user\-facing data collection or evaluation of the annotators themselves\. Annotators were not shown scalar confidence, source scores\(V,L,A\)\(V,L,A\), raw or normalized dominant sources, routing decisions, or model\-derived explanations\. This separation reduces score\-induced bias and keeps the audit focused on visible example\-level evidence\.
### 11\.4Annotator View
Annotators see only information that would be available in a semantic error inspection setting\. They are not shown scalar confidence, source scores\(V,L,A\)\(V,L,A\), raw dominant source, normalized dominant source, routing decision, or any model\-derived explanation\. This prevents the audit from simply reproducing the model\-derived signature\.
Table 38:Information shown to annotators and information hidden during the audit\.
### 11\.5Annotation Labels
Each annotator assigns one label from six categories\. The first three labels correspond to the Perturbation\-Response coordinates\. Two labels preserve ambiguity rather than forcing a clean source assignment, while a final label handles cases with no visible source\-sensitive risk\. The correctness flag is shown to annotators only as context, while the annotation target remains the plausible source of risk or source\-sensitive behavior in the example\.
Table 39:Human\-audit label rubric\. The labels are diagnostic descriptions of the observed example, not causal claims about model internals\.
### 11\.6Annotation Instructions
Annotators are instructed to label the most plausible source of risk in the specific example\. They should not infer a model’s hidden reasoning process\. They should also avoid using demographic, cultural, or identity\-based assumptions not required by the task\. For correct predictions, annotators assign a source label only when the prediction appears fragile or source\-dependent from the visible evidence\. If no source\-sensitive risk is visible, they useUnclear\. This convention keeps the audit aligned withHalluPrism: the Perturbation\-Response Signature is defined for both correct and incorrect predictions, while the human label remains an example\-level diagnostic judgment\. The instruction shown to annotators is:
> You are given an image, a question or caption, a reference answer, a model answer, and whether the model answer is correct\. Your task is to judge the most plausible source of risk or source\-sensitive behavior in this example\. UseVisualif the relevant visual evidence appears weak or insufficient\. UseLanguage\-priorif the answer seems plausible from the question alone or from common world knowledge rather than the image\. UseAlignmentif the image contains the relevant entities but the answer appears to fail due to relation, attribute, grounding, or image\-text binding\. UseMixedif more than one source is plausible\. UseUnclearif the source cannot be determined or if the gold reference answer is factually incorrect\. UseNo Failureif the model answer is completely correct\. Do not guess the model’s internal reasoning\. Label only the example\-level evidence visible to you\.
Table 40:Human\-audit reporting fields\. We report mixed and unclear cases separately rather than forcing them into one of the three source coordinates\.
### 11\.7Agreement and Adjudication
Three annotators independently label each example\. We compute raw agreement and Fleiss’sκ\\kappaover the six\-label set:Visual,Language\-prior,Alignment,Mixed,Unclear, andNo Failure\. In addition to the full 6\-label agreement, we compute a collapsed Fleiss’sκ\\kappathat groups the three source labels \(Visual, Language\-prior, Alignment\) against the non\-source labels \(Mixed, Unclear, No Failure\)\. This isolates whether humans can reliably distinguish source\-driven failures from generic ambiguity\. Disagreements are adjudicated after the independent pass\. During adjudication, annotators may revise a label only by referring to the visible evidence in the image, question, reference answer, model answer, and correctness flag\. They do not see source scores during adjudication\. After adjudication, we compute model\-human agreement between the adjudicated human label and the normalized dominant sourcesnorm∗\(x,M\)s^\{\*\}\_\{\\mathrm\{norm\}\}\(x,M\)\. This comparison is restricted to clear\-source cases labeledVisual,Language\-prior, orAlignment\. Mixed and unclear cases are reported separately\.
### 11\.8Reporting Metrics
We report eight audit quantities: \(i\)raw agreement, the fraction of examples on which annotators assign the same label before adjudication; \(ii\) Fleiss’sκ\\kappa, the chance\-corrected agreement over the six\-label set; \(iii\)collapsed Fleiss’sκ\\kappa, the chance\-corrected agreement for source vs\. non\-source classification; \(iv\)clear\-source coverage, the fraction of adjudicated examples assigned toVisual,Language\-prior, orAlignment; \(v\)model\-human agreement, the agreement between the normalized dominant source and the adjudicated human label on clear\-source examples; \(vi\)mixed rate, the fraction of examples adjudicated asMixed; \(vii\)unclear rate, the fraction adjudicated asUnclear; and \(viii\)no failure rate, the fraction adjudicated asNo Failure\. Lets^\(x\)=snorm∗\(x,M\)\\widehat\{s\}\(x\)=s^\{\*\}\_\{\\mathrm\{norm\}\}\(x,M\)\. Letvhv\_\{h\},lhl\_\{h\},aha\_\{h\},mhm\_\{h\},uhu\_\{h\}, andnhn\_\{h\}denote the adjudicated labelsVisual,Language\-prior,Alignment,Mixed,Unclear, andNo Failure, respectively\. We define the clear\-source label set, clear\-source examples, label\-coordinate map, and agreement indicator as follows:
𝒞src=\{vh,lh,ah\},𝒳c=\{x∈𝒳:h\(x\)∈𝒞src\},ψ\(vh\)=V,ψ\(lh\)=L,ψ\(ah\)=Aδx=\[s^\(x\)=ψ\(h\(x\)\)\]\.\\begin\{split\}&\\mathcal\{C\}\_\{\\mathrm\{src\}\}=\\\{v\_\{h\},l\_\{h\},a\_\{h\}\\\},\\ \\mathcal\{X\}\_\{c\}=\\\{x\\in\\mathcal\{X\}:h\(x\)\\in\\mathcal\{C\}\_\{\\mathrm\{src\}\}\\\},\\\\ &\\psi\(v\_\{h\}\)=\\mathrm\{V\},\\hskip 9\.24994pt\\psi\(l\_\{h\}\)=\\mathrm\{L\},\\hskip 9\.24994pt\\psi\(a\_\{h\}\)=\\mathrm\{A\}\\\\ &\\delta\_\{x\}=\\mathbf\{1\}\\\!\\left\[\\widehat\{s\}\(x\)=\\psi\(h\(x\)\)\\right\]\.\\end\{split\}\(26\)
The reported rates are then:
ClearCov=\|𝒳c\|\|𝒳\|,MHAgree=1\|𝒳c\|∑x∈𝒳cδx,MixedRate=1\|𝒳\|∑x∈𝒳\[h\(x\)=mh\],UnclearRate=1\|𝒳\|∑x∈𝒳\[h\(x\)=uh\],NoFailureRate=1\|𝒳\|∑x∈𝒳\[h\(x\)=nh\]\.\\begin\{split\}&\\mathrm\{ClearCov\}=\\frac\{\|\\mathcal\{X\}\_\{c\}\|\}\{\|\\mathcal\{X\}\|\},\\\\ &\\mathrm\{MHAgree\}=\\frac\{1\}\{\|\\mathcal\{X\}\_\{c\}\|\}\\sum\_\{x\\in\\mathcal\{X\}\_\{c\}\}\\delta\_\{x\},\\\\ &\\mathrm\{MixedRate\}=\\frac\{1\}\{\|\\mathcal\{X\}\|\}\\sum\_\{x\\in\\mathcal\{X\}\}\\mathbf\{1\}\\\!\\left\[h\(x\)=m\_\{h\}\\right\],\\\\ &\\mathrm\{UnclearRate\}=\\frac\{1\}\{\|\\mathcal\{X\}\|\}\\sum\_\{x\\in\\mathcal\{X\}\}\\mathbf\{1\}\\\!\\left\[h\(x\)=u\_\{h\}\\right\],\\\\ &\\mathrm\{NoFailureRate\}=\\frac\{1\}\{\|\\mathcal\{X\}\|\}\\sum\_\{x\\in\\mathcal\{X\}\}\\mathbf\{1\}\\\!\\left\[h\(x\)=n\_\{h\}\\right\]\.\\end\{split\}\(27\)
Hereh\(x\)h\(x\)denotes the adjudicated human label\. The mapψ\\psilinks each clear\-source human label to the corresponding Perturbation\-Response coordinate\. We report the confusion matrix over clear\-source examples and separately report the number of mixed and unclear cases\.
### 11\.9Interpretation
The audit is intentionally conservative\. High model\-human agreement would support the semantic plausibility of the Perturbation\-Response labels\. Low agreement would not invalidate the main quantitative results, because the main claims concern behavioral probe responses, failure\-family prediction, scalarization boundary checks, and routing breakage\. Instead, low agreement would indicate that the normalized dominant source is not always human\-interpretable as a single source\. This outcome would still be consistent with the paper’s entanglement claim\. Therefore, we treat the audit as an interpretability check on the source\-label interface, not as evidence for causal source recovery\.
## 12Computational Cost And Resource Use
We report the computational cost ofHalluPrismbecause the method is a diagnosis layer rather than a single\-pass confidence score\. Each evaluated prediction requires the clean answer, four visual perturbation answers, one blank\-image answer, and one or two alignment\-probe answers\. Thus, the core Perturbation\-Response computation uses77forward passes for datasets without relation\-swap probing and88forward passes for Visual Spatial Reasoning \(VSR\), where explicit relation perturbation is available\. This cost is the main practical tradeoff of the method:HalluPrismgives source\-aware diagnostic information, but it is not intended as a zero\-overhead scalar confidence replacement\.
### 12\.1Inference Budget
The evaluation contains58,44058\{,\}440model\-output instances across four datasets and four model families\. The Perturbation\-Response computation uses the clean prediction, four visual\-perturbation predictions, one blank\-image prediction, and one grounding re\-check prediction for every image\-paired example\. Therefore, HallusionBench, POPE, and VizWiz\-VQA use77forward passes per model\-example\. Visual Spatial Reasoning \(VSR\) additionally uses the relation\-swap alignment probe, yielding88forward passes per model\-example\.
LetNdN\_\{d\}denote the number of examples in datasetdd, and letpdp\_\{d\}denote the number of forward passes used for Perturbation\-Response measurement in that dataset\. We compute the core Perturbation\-Response budget as:
Ncore=∑m∈ℳ∑d∈𝒟Ndpd,pd=\{7,without relation\-swap probing,8,with relation\-swap probing\.\\begin\{split\}&N\_\{\\mathrm\{core\}\}=\\sum\_\{m\\in\\mathcal\{M\}\}\\sum\_\{d\\in\\mathcal\{D\}\}N\_\{d\}p\_\{d\},\\\\ &p\_\{d\}=\\begin\{cases\}7,&\\text\{without relation\-swap probing\},\\\\ 8,&\\text\{with relation\-swap probing\}\.\\end\{cases\}\\end\{split\}\(28\)
In our evaluation, relation\-swap probing is available only for VSR\. Hence, the core Perturbation\-Response budget is:
Ncore=4\(951⋅7\+9000⋅7\+4319⋅7\+340⋅8\)=410,440\.\\begin\{split\}N\_\{\\mathrm\{core\}\}&=4\\bigl\(951\\cdot 7\+9000\\cdot 7\+4319\\cdot 7\+340\\cdot 8\\bigr\)\\\\ &=410\{,\}440\.\\end\{split\}\(29\)
Intervention routing adds two further branches for each model\-output instance: one generic intervention and one matched intervention\. Since the pooled evaluation contains58,44058\{,\}440model\-output instances, routing adds:
Nroute=2⋅58,440=116,880\.N\_\{\\mathrm\{route\}\}=2\\cdot 58\{,\}440=116\{,\}880\.\(30\)
The full diagnosis\-plus\-routing experiment therefore uses:
Ntotal=Ncore\+Nroute=527,320N\_\{\\mathrm\{total\}\}=N\_\{\\mathrm\{core\}\}\+N\_\{\\mathrm\{route\}\}=527\{,\}320\(31\)forward passes, excluding optional self\-reported confidence baselines and implementation\-level retries caused by model\-serving failures\.
Dataset / BranchExamples Per ModelModelsPasses Per ExampleForward PassesHallusionBench9514726,628POPE9,00047252,000VizWiz\-VQA4,31947120,932VSR3404810,880Core Perturbation\-Response measurement14,6104–410,440Generic and matched routing branches14,61042116,880Full diagnosis\-plus\-routing experiment14,6104–527,320Table 41:Corrected inference budget for HalluPrism\. HallusionBench, POPE, and VizWiz\-VQA use77passes per model\-example\. VSR uses88passes because relation\-swap probing is available\. Routing adds two additional branches per model\-output instance\.
### 12\.2Training Cost
We do not fine\-tune any Multimodal Large Language Model \(MLLM\)\. All MLLMs are evaluated with deterministic decoding\. The only learned models in the paper are lightweight diagnostic classifiers trained on cached scalar confidence and Perturbation\-Response features\. These classifiers include logistic regression, XGBoost, and LightGBM\. Their inputs are four\-dimensional at most, namely\[V,L,A,c\]\[V,L,A,c\], and their cost is negligible relative to MLLM inference\. The diagnostic classifiers are used only for analysis\. They are not used to update the evaluated MLLMs\.
### 12\.3Decoding And Runtime Controls
We use greedy decoding with temperature00, disabled sampling, and a maximum response length of128128tokens\. These choices reduce stochastic variation in the perturbation\-response measurements\. They also make the compute budget easier to interpret, since repeated generations under the same input are not used as uncertainty samples\. This differs from sampling\-based uncertainty methods, where multiple generations are required for each clean input before any perturbation is applied\.
The Perturbation\-Response cost scales linearly in the number of probes\. If a deployment setting needs only a subset of the diagnostic sources, the cost can be reduced by disabling unused probes\. For example, a system concerned only with image\-removal confidence retention can run the clean and blank\-image branches\. A system concerned with spatial binding can retain the relation\-swap branch\. In the full paper, we keep all probes active because the central empirical question is whether the joint\(V,L,A\)\(V,L,A\)signature diagnoses failure families and guides correction\.
### 12\.4Hardware And Wall\-Clock Reporting
We report the hardware and wall\-clock runtime in Table[42](https://arxiv.org/html/2608.29193#S12.T42)\. Exact runtime depends on the serving backend, batch size, model implementation, image resolution, and GPU type\. The main reproducibility\-critical quantity is therefore the number of forward passes, decoding configuration, model checkpoint, and prompt regime\. These are fixed by the protocol and reported in Appendix[9](https://arxiv.org/html/2608.29193#S9)and Table[41](https://arxiv.org/html/2608.29193#S12.T41)\.
Table 42:Hardware and runtime used for the submitted experiments\. Runtime depends on serving backend, batching, model implementation, and image preprocessing, so forward\-pass counts in Table[41](https://arxiv.org/html/2608.29193#S12.T41)remain the primary reproducibility quantity\.
### 12\.5Resource Use And Environmental Reporting
The experiments are inference\-only for the evaluated MLLMs and do not require model fine\-tuning\. This substantially reduces the resource cost compared with training or adapting large vision\-language models\. However,HalluPrismstill increases inference cost because each prediction is evaluated under multiple perturbation branches\. We therefore report forward\-pass counts explicitly rather than only reporting the number of original examples\. If the final execution environment logs GPU\-hours or energy use, we will report those values with the hardware description in Table[42](https://arxiv.org/html/2608.29193#S12.T42)\. If such logs are unavailable, we report the fixed forward\-pass budget and deterministic decoding settings as the reproducible compute description\. This is less informative than a full energy audit, but it avoids inventing carbon or power estimates that were not directly measured\.
### 12\.6Practical Deployment Tradeoff
The compute cost should be interpreted together with the intended role of the method\.HalluPrismis not designed to run as a cheap scalar abstention score for every low\-risk query\. It is more suitable as an audit\-time diagnostic, a high\-risk inference\-time check, or a routing layer for cases where generic correction may damage correct answers\. In latency\-critical deployments, the full protocol can be reserved for uncertain, high\-impact, or audit\-selected examples\. This preserves the main contribution: source\-aware uncertainty should expose the kind of risk present before a system decides whether to answer or correct\.
Table 43:Dataset artifacts used inHalluPrism\. We use public benchmark datasets and follow the corresponding license and access terms\.Table 44:Model artifacts used inHalluPrism\. We use the listed models for deterministic inference only and do not redistribute model weights\.
## 13Artifact, License, and Data Statement
We report the artifacts used inHalluPrismto make dataset use, model access, and release boundaries explicit\. All experiments use public benchmark datasets and publicly accessible model checkpoints or model families\. We do not collect new user images, and we do not redistribute benchmark images or model weights\. When redistribution is restricted or unclear, we release only derived metadata, evaluation scripts, prompt templates, normalized predictions, source scores, and aggregate result tables, subject to the original artifact licenses\.
### 13\.1Dataset Artifacts
We use four public multimodal benchmarks under their stated access and license terms\. We report the dataset\-level license and access summary in Table[43](https://arxiv.org/html/2608.29193#S12.T43)\. For HallusionBench, we use only the image\-paired evaluation subset and do not redistribute raw images\. For POPE and VSR, we use the released benchmark prompts, annotations, and examples subject to the corresponding repository terms\. For VizWiz\-VQA, we treat the raw images with additional care because they may contain user\-provided visual contexts\. Across all datasets, any released artifact from this paper should contain only derived metadata, prompt templates, normalized outputs, source scores, and aggregate results unless the original dataset license explicitly permits redistribution of raw images\.
### 13\.2Model Artifacts
We evaluate open\-weight and publicly accessible Multimodal Large Language Models \(MLLMs\)\. We use these models only for inference\. We do not fine\-tune, redistribute, or modify their weights\. All generated outputs are used for research evaluation of Perturbation\-Response Signatures, abstention scalarization, and intervention routing\. A summary of the model artifacts is given in Table[44](https://arxiv.org/html/2608.29193#S12.T44)\.
### 13\.3Derived Artifacts
We create derived artifacts from model outputs and perturbation probes\. These include clean predictions, normalized answers, scalar confidence values, source scores\(V,L,A\)\(V,L,A\), dominant\-source labels, failure\-family labels, abstention scalarization scores, and intervention\-routing outcomes\. These artifacts are produced from public benchmark examples and model inference\. They do not contain new human\-subject data\. If released, they should be shared as derived metadata keyed by dataset identifiers rather than as redistributed benchmark images, unless the original dataset license explicitly permits redistribution\.
Table 45:Derived artifacts produced by theHalluPrismpipeline\. These artifacts support reproducibility without requiring redistribution of raw benchmark images or model weights\.
### 13\.4License Compliance
We use each artifact under its stated research or open\-weight access terms\. The paper cites the original dataset and model creators in the main bibliography\. The released code will include a license file for our own implementation and will document external dependencies separately from dataset and model licenses\. Any user of the released artifacts remains responsible for complying with the original licenses and terms for HallusionBench, POPE, VSR, VizWiz\-VQA, Gemma 4, Qwen3\-VL, LLaVA\-Mistral, and LLaVA\-Vicuna\.
### 13\.5Data Sensitivity
VizWiz\-VQA contains images and questions originating from blind or low\-vision users\. Although the dataset is public and licensed for research use, the images may include private environments, personal objects, or accessibility\-sensitive contexts\. We therefore avoid redistributing raw images and treat VizWiz\-derived outputs as evaluation metadata\. More generally, multimodal datasets can contain visual details that are more sensitive than the corresponding text prompts\. Downstream users should inspect dataset licenses, privacy notes, access terms, and intended\-use statements before reusing any raw images or derived artifacts\.
### 13\.6Statement Of Use
We use these artifacts only to evaluate the diagnosis\-first claim ofHalluPrism\. The artifacts are not used to train a new deployed MLLM, infer private attributes, identify people, or make decisions about real users\. The human audit protocol, if completed, uses benchmark examples and does not collect new sensitive personal information\. Any public release of audit labels should exclude annotator\-identifying information and should preserve only example identifiers, adjudicated labels, and aggregate agreement statistics\.
## 14Use Of AI Writing Assistance
AI writing assistance was used during manuscript preparation for language editing and polishing of LATEX tables\. The authors remained responsible for all scientific content, including the problem formulation, experimental design, implementation, data processing, result interpretation, citations, mathematical statements, and final manuscript text\. The authors manually verified the reported claims, numerical results, tables, citations, proofs, and responsible\-use statements before submission\. AI assistance was not used as a substitute for author judgment, was not treated as authorship, and did not determine the paper’s scientific conclusions\.
Table 46:Responsible\-use boundaries forHalluPrism\. The method is intended for diagnosis and matched correction analysis, not for standalone safety certification\.
## 15Ethical Considerations and Responsible Use
We studyHalluPrismas a behavioral diagnostic layer for Multimodal Large Language Models \(MLLMs\)\. The intended use is model analysis, audit support, and safer correction\-path selection\. It should not be used as a standalone safety certificate, a causal explanation of model internals, or an automatic deployment approval mechanism\. This distinction is central to the paper\.HalluPrismmeasures how an output changes under controlled visual, image\-removal, and grounding/relation pressure\. It does not prove why the model produced the answer, and it does not guarantee that an answer is safe for a new open\-world instance\.
### 15\.1Intended Use
HalluPrismis intended for model auditing, benchmark\-level error analysis, and guarded correction\-path selection\. Appropriate uses include: \(i\) comparing model behavior across multimodal hallucination benchmarks, \(ii\) identifying broad failure\-family patterns that are not visible from scalar confidence alone, and \(iii\) selecting a correction path only when validation shows that such routing reduces harmful breakage\. Any deployment use should specify the task, user population, validation distribution, acceptable risk level, fallback policy, and monitoring plan before Perturbation\-Response Signatures are used to guide correction\.
### 15\.2Non\-Intended Use
HalluPrismshould not be used as a standalone safety certificate, a deployment approval mechanism, or a replacement for human judgment in high\-stakes settings\. It should not certify answers in medical, legal, financial, educational, accessibility\-critical, or public\-sector contexts without domain\-specific validation, expert review, and post\-deployment monitoring\. The source scores should also not be interpreted as ground\-truth causes of model behavior\. They are bounded diagnostic signals produced under a fixed evaluation protocol\. Treating them as causal explanations or complete reliability guarantees would overstate what the method establishes\.
### 15\.3Risks From Diagnostic Misuse
The main misuse risk is over\-trust\. A system that treats the Perturbation\-Response Signature as a complete reliability certificate recreates the same failure mode that motivates the paper’s caution about scalar abstention\. The signature is bounded by the evaluated datasets, models, prompts, perturbations, and routing choices\. It can miss risks outside these conditions, including domain expertise gaps, cultural context, ambiguous user intent, missing background information, and harms that are not expressed through the measured source coordinates\. A second risk is correction harm\. A correction prompt can improve one model\-dataset condition and damage another\. Therefore, routing should be treated as conditional actionability rather than universal mitigation\. Systems usingHalluPrismshould log the original answer, routed action, corrected answer, fix indicator, break indicator, and any human\-review outcome, since accuracy alone can hide harmful overcorrection\.
### 15\.4Risks For Affected Users
Several datasets used in this paper involve user\-facing visual question answering\. VizWiz\-VQA is especially relevant because it contains images and questions from blind or low\-vision users\. Errors in such settings may create practical harm because users may not be able to independently verify a visual answer\. A model that confidently guesses from a language prior may give a fluent answer that is not supported by the image\. A generic correction prompt may also change an originally correct answer\. We therefore viewHalluPrismas an audit aid for identifying risk structure, not as a substitute for accessibility\-aware system design, user consent, domain\-specific evaluation, or human fallback mechanisms\.
### 15\.5Data And Privacy Considerations
We use public benchmark datasets and do not collect new user images for the main experiments\. The human audit protocol, if completed, uses examples sampled from the evaluated benchmark outputs and does not require collecting personal information from annotators beyond standard annotation logistics\. Annotators do not see source scores or routing decisions, which limits score\-induced bias during the audit\. Public image datasets can still contain sensitive visual content, private environments, or user\-specific contexts\. Any released artifact should therefore include dataset source information, license terms, access constraints, and a clear description of which fields are original, derived, or model\-generated\.
### 15\.6Responsible Deployment Guidance
Before usingHalluPrismin deployment, a system owner should document: \(i\) the target task and user population; \(ii\) the validation distribution and expected deployment shifts; \(iii\) the acceptable risk level and fallback policy; \(iv\) the conditions under which Perturbation\-Response Signatures trigger correction or human review; \(v\) the logging fields for the original answer, Perturbation\-Response Signature, routed action, corrected answer, fix indicator, and break indicator; and \(vi\) the monitoring plan for post\-deployment drift\. In high\-impact or latency\-critical settings, the full protocol should be reserved for audit\-selected, uncertain, or high\-risk examples\. Perturbation\-Response Signatures should guide review or guarded correction rather than automatically certifying an answer as safe\.Similar Articles
Unified Hallucination Fuzzing for Multimodal Large Language Models
This paper presents UniHall, a fine-grained hallucination benchmark with a unified taxonomy, and Self-Adaptive Multimodal Fuzzing (SAMF), a self-evolving stress-testing framework for multimodal LLMs. Experiments show SOTA models degrade significantly under fuzzing and reveal a helpfulness-hallucination trade-off.
ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning
ClinHallu is a benchmark for diagnosing and mitigating hallucinations in medical multimodal large language models by decomposing reasoning into visual recognition, knowledge recall, and reasoning integration stages, using trace-supervised fine-tuning to reduce errors.
OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
OmniHallu introduces a unified hallucination detection framework for multimodal large language models, covering comprehension and generation tasks across image, video, and audio modalities, with a benchmark and multi-agent architecture.
PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations
Researchers propose PRISM, a diagnostic benchmark that breaks down LLM hallucinations into four dimensions (knowledge missing/errors, reasoning errors, instruction-following errors) across three generation stages (memory, instruction, reasoning), evaluating 24 LLMs to reveal trade-offs in mitigation strategies.
Diversity-Oriented Fine-Tuning for Uncertainty-Based Hallucination Detection
This paper proposes diversity-oriented fine-tuning strategies to improve uncertainty-based hallucination detection in LLMs by encouraging varied generations, making hallucinations more detectable via semantic entropy.