Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflicts
Summary
This paper examines how the order of conflicting evidence in multimodal large language models affects judgments, revealing cross-modal evidence noncommutativity where placing perceptual evidence later increases model reliance on it.
View Cached Full Text
Cached at: 09/24/26, 09:12 AM
# EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH-TEXT CONFLICTS
Source: [https://arxiv.org/html/2609.26986](https://arxiv.org/html/2609.26986)
## SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS
Boxuan WangXiaowei HuangYi Dong⋆††thanks:⋆Corresponding author: yi\.dong@liverpool\.ac\.uk
###### Abstract
For multimodal large language models, when images or speech conflict with accompanying text, measured text reliance can entangle modality preference with evidence position\. Earlier studies of text bias often used a fixed evidence order or moved task instructions with the evidence, leaving the contribution of order unclear\. In this paper, we use a paired comparison that keeps the instructions and evidence content fixed and swaps only the positions of the two sources to quantify this potential influence\. Across vision and speech models, placing an image or recording after conflicting text consistently shifts answers toward its content\. We also revisit previous studies and analyze why their experimental settings can lead to misleading conclusions\. These findings reveal*cross\-modal evidence noncommutativity*: the same evidence can lead to different judgments when its order changes, and placing perceptual evidence later can increase the model’s reliance on its content\.
###### Index Terms:
Multimodal large language models, evidence conflict, order sensitivity, noncommutativity
††address:School of Computer Science and Informatics, University of Liverpool, United Kingdom## 1Introduction
Multimodal large language model \(MLLM\) assistants can process charts, document pages, and speech recordings alongside user\-provided descriptions and questions\[[3](https://arxiv.org/html/2609.26986#bib.bib15),[29](https://arxiv.org/html/2609.26986#bib.bib21)\]\. These words may reflect the user’s interpretation or recollection of the attachment\. Spelling mistakes or memory errors can make that description conflict with the attachment\. For example, a user may describe Method A as the best performer in a results plot that ranks Method B first\. A recording may state that tomorrow’s meeting starts at 9 a\.m\., while the user writes, “Schedule tomorrow’s meeting at 8 a\.m\. based on the manager’s recording\.” The model should understand the attachment and answer or act according to the source specified by the user\.
Studies of cross\-modal conflict find that vision\-language models often rely excessively on text when it contradicts an image\[[22](https://arxiv.org/html/2609.26986#bib.bib1),[7](https://arxiv.org/html/2609.26986#bib.bib2),[32](https://arxiv.org/html/2609.26986#bib.bib3),[33](https://arxiv.org/html/2609.26986#bib.bib4)\]\. Research on input order shows that rearranging long\-context information, answer options, multiple images, or image–text pairs can change model responses\[[24](https://arxiv.org/html/2609.26986#bib.bib7),[18](https://arxiv.org/html/2609.26986#bib.bib9),[23](https://arxiv.org/html/2609.26986#bib.bib10)\]\. Together, these findings raise a question:*how much of the measured text reliance really reflects modality preference, while how much actually reflects the position impact of the competing evidence?*
Prior work has tried to examine how evidence order affects modality preference\[[7](https://arxiv.org/html/2609.26986#bib.bib2)\]\. However, we find that this comparison swaps the image with a text block containing the description, task instructions, and cautionary warnings\. This changes both evidence order and instruction placement, while the warnings may also influence how the model treats the text\. This design leaves the order’s impact still entangled\. Other conflict benchmarks measure modality preference under a fixed input layout\[[22](https://arxiv.org/html/2609.26986#bib.bib1),[28](https://arxiv.org/html/2609.26986#bib.bib26)\], without enough exploration of evidence order for different modalities, which may lead to the over\- or underestimation of the modality bias\.
In this paper, we address the gaps by redesigning the prompt structure around everyday requests\. We place the instruction to answer from the attachment before both evidence sources and manually construct faithful and conflicting textual descriptions for existing multimodal benchmarks\. We then swap only the positions of the two sources, keeping all other components fixed, and compare the model’s relative reliance on the two modalities\. Controls involving unrelated text, within\-modality evidence swaps, and alternative instruction wording allow us to quantifycross\-modal evidence noncommutativity\.
Our contributions are as follows\.\(1\) Cross\-modal evidence noncommutativity\.We measure how judgments change when identical evidence is reordered\.\(2\) Evidence across vision and speech\.We demonstrate systematic noncommutativity across model families and extend the evaluation to diverse audio tasks, addressing a gap in fixed\-order audio conflict benchmarks\.\(3\) Protocol diagnosis\.We show that moving task instructions with the evidence can confound order comparisons; our simple protocol keeps the instructions fixed to isolate the effect of evidence order\.
Figure 1:Everyday requests in which user descriptions conflict with image or speech attachments\.
## 2Related Work
Cross\-modal conflict and text relianceStudies of cross\-modal conflict examine how models use competing sources\. Mixed Signals finds that modality reliance varies with the model and task difficulty\[[22](https://arxiv.org/html/2609.26986#bib.bib1)\]\. Words or Vision examines when models follow misleading text and how image–text order affects their answers\[[7](https://arxiv.org/html/2609.26986#bib.bib2)\]\. ConText\-VQA shows that persuasive false descriptions can override visual evidence\[[32](https://arxiv.org/html/2609.26986#bib.bib3)\]\. Other work studies conflicts among images, text, and a model’s prior knowledge\[[33](https://arxiv.org/html/2609.26986#bib.bib4),[35](https://arxiv.org/html/2609.26986#bib.bib5),[19](https://arxiv.org/html/2609.26986#bib.bib6)\]\. Together, these studies show how textual misinformation influences vision\-language models\. Under a fixed template, the measured text reliance can also reflect evidence position\. The concurrent preprint\[[14](https://arxiv.org/html/2609.26986#bib.bib8)\]studies modality and order in conflicts between parametric knowledge and external knowledge presented as text or rendered images\. Imitating Words or Vision, MCR\-BENCH tests the textual preference but still leaves the order effect blank\[[28](https://arxiv.org/html/2609.26986#bib.bib26)\]\. In this paper, we pair a natural image or real speech recording with an external description and measure what changes when the two sources exchange positions\.
Input order influenceLanguage\-model responses vary with information position and answer\-option order\[[18](https://arxiv.org/html/2609.26986#bib.bib9),[23](https://arxiv.org/html/2609.26986#bib.bib10)\]\. Multimodal models are also sensitive to image and image–text arrangements\[[24](https://arxiv.org/html/2609.26986#bib.bib7)\]\. We counterbalance option assignments while preserving each source’s content and semantic role\. Matched, conflicting, and unrelated descriptions then distinguish order effects across evidence relations\.
## 3Method
### 3\.1Evidence and paired swaps
We usePPfor perceptual evidence, an image or speech recording, andTTfor the accompanying text\. Each question has an attachment\-supported answeryPy\_\{P\}and a competing answeryT≠yPy\_\{T\}\\neq y\_\{P\}\. We construct three text conditions: conflictingT−T^\{\-\}supportsyTy\_\{T\}; matchedT\+T^\{\+\}agrees withPP; and unrelatedT0T^\{0\}supports neither answer\. The superscripts−\-,\+\+, and00identify these conditions throughout\. For each text condition, we keep task instructionIIbefore both sources and question/optionsqqafter them\. The paired inputs are
xP→T=\(I,P,T,q\),xT→P=\(I,T,P,q\)\.x\_\{P\\to T\}=\(I,P,T,q\),\\qquad x\_\{T\\to P\}=\(I,T,P,q\)\.\(1\)OnlyPPandTTexchange positions; the system prompt, evidence content, and generation settings stay fixed\. We evaluate both A/B label assignments and combine results by the underlying answer to counterbalance option\-label bias\.
### 3\.2Order\-sensitivity metrics
LetAAdenote the rate at which the model selects the attachment\-supported answeryPy\_\{P\}\. Our primary metric is the change in this rate under conflicting text:
ΔA−=A\(T−→P\)−A\(P→T−\)\.\\Delta A^\{\-\}=A\(T^\{\-\}\\\!\\to\\\!P\)\-A\(P\\\!\\to\\\!T^\{\-\}\)\.\(2\)We reportΔA−\\Delta A^\{\-\}in percentage points\. A positive value means the model followsPPmore often when it appears last\.
We also measure changes in answer probabilities within each conflict pair\. For a modelpθp\_\{\\theta\}, letℓP,ℓT∈\{A,B\}\\ell\_\{P\},\\ell\_\{T\}\\in\\\{\\mathrm\{A\},\\mathrm\{B\}\\\}denote the labels assigned toyP,yTy\_\{P\},y\_\{T\}in the current option ordering\. Using their probabilities at the first answer token, we define
sθ\(x\)\\displaystyle s\_\{\\theta\}\(x\)=logpθ\(ℓP∣x\)pθ\(ℓT∣x\),\\displaystyle=\\log\\frac\{p\_\{\\theta\}\(\\ell\_\{P\}\\mid x\)\}\{p\_\{\\theta\}\(\\ell\_\{T\}\\mid x\)\},\(3\)𝒞θ\\displaystyle\\mathcal\{C\}\_\{\\theta\}=sθ\(xT→P\)−sθ\(xP→T\)\.\\displaystyle=s\_\{\\theta\}\(x\_\{T\\to P\}\)\-s\_\{\\theta\}\(x\_\{P\\to T\}\)\.\(4\)A positive log\-odds swap score𝒞θ\\mathcal\{C\}\_\{\\theta\}indicates a shift towardyPy\_\{P\}whenPPis placed last\. We report the mean across paired evaluations\.
### 3\.3Controls and uncertainty
The matched and unrelated conditions test order sensitivity when the sources agree or the text provides no evidence for either answer\. Their selection\-rate changes are denoted byΔA\+\\Delta A^\{\+\}andΔA0\\Delta A^\{0\}, with the question and candidate answers held fixed across all three conditions\. To compare conflict with the unrelated control, we report
D=ΔA−−ΔA0\.D=\\Delta A^\{\-\}\-\\Delta A^\{0\}\.\(5\)A positiveDDindicates a larger shift towardPPunder conflicting than unrelated text\. We compute confidence intervals with a paired bootstrap over original images or recordings, keeping each example’s evidence orders, option assignments, and text conditions together\.
## 4Experiments
### 4\.1Experimental setup
Figure 2:Controlled evidence swaps in vision–text and speech–text\. Keeping instructions and content fixed lets us measure how order changes which source the model follows\.Visual data\.We manually inspect 240 natural images each from GQA/Visual Genome\[[10](https://arxiv.org/html/2609.26986#bib.bib11),[12](https://arxiv.org/html/2609.26986#bib.bib12)\]and VQAv2/COCO\[[9](https://arxiv.org/html/2609.26986#bib.bib13),[17](https://arxiv.org/html/2609.26986#bib.bib14)\]\. To constructT−T^\{\-\}, we change the queried fact \(includingcolor, count, object identity, and location\) and keep other scene details unchanged wherever possible\. For example, for a cup\-color question, we change the color while retaining the object and its location\. Unrelated textT0T^\{0\}comes from other images and supports neither candidate answer\.
Speech data\.We select 240 recordings each from Fluent Speech Commands \(FSC\)\[[20](https://arxiv.org/html/2609.26986#bib.bib18)\]and SLURP\[[4](https://arxiv.org/html/2609.26986#bib.bib19)\]\. We allocate 80 examples to each FSC field \(includingaction, object, and location\) and change only one field when constructingT−T^\{\-\}\. The SLURP subset covers 15 broad intent categories, with 16 examples per category\. Conflicting textT−T^\{\-\}comes from another intent within the same broad category, while unrelated textT0T^\{0\}comes from a different category\. The exploratory and confirmatory speech sets are disjoint\.
Models\.Vision models are Qwen2\.5\-VL\-3B\-Instruct\[[3](https://arxiv.org/html/2609.26986#bib.bib15)\], Idefics3\-8B\-Llama3\[[13](https://arxiv.org/html/2609.26986#bib.bib16)\], and Pixtral\-12B\[[1](https://arxiv.org/html/2609.26986#bib.bib17)\]\. Speech models are Qwen2\-Audio\-7B\-Instruct\[[6](https://arxiv.org/html/2609.26986#bib.bib20)\], Qwen2\.5\-Omni\-7B\[[29](https://arxiv.org/html/2609.26986#bib.bib21)\], and Phi\-4 Multimodal\[[21](https://arxiv.org/html/2609.26986#bib.bib22)\]\. These recent models support swapping the order of perceptual and text tokens\.
Input conditions\.Each example has nine input conditions: three single\-source baselines \(PPonly,T\+T^\{\+\}only, andT−T^\{\-\}only\) and six paired inputs \(both orders ofPPwith each ofT\+T^\{\+\},T−T^\{\-\}, andT0T^\{0\}\)\.We run each condition twice, reversing the A/B option positions to counterbalance option\-letter preference\[[23](https://arxiv.org/html/2609.26986#bib.bib10),[34](https://arxiv.org/html/2609.26986#bib.bib23)\]\.
Prompt construction\.We constructed the prompts to reflect everyday requests combining an attachment, a description, and a question\. It asks the model to answer from the attachment, introduces the user’s description naturally, and leaves its correctness unspecified\. The visual prompt is “Answer the question according to the image attachment, and here is also the image description\.” The speech prompt uses the corresponding audio\-attachment wording\.
### 4\.2Quantifying cross\-modal evidence noncommutativity
Table 1:Main image–text and speech–text results\.PPdenotes perceptual evidence;T−T^\{\-\}denotes conflicting text\. The number is the question selection accuracy \(pp\)\.ΔA−\\Delta A^\{\-\}is the change of rate\.𝒞\\mathcal\{C\}is the mean log\-odds swap score with its 95% confidence interval\.We first test whether models that understand an attachment change their judgments when its position relative to conflicting text is reversed\. As shown in Table[1](https://arxiv.org/html/2609.26986#S4.T1), with attachments alone, accuracy ranges from 88\.3–99\.4% across both vision and speech settings\. These baselines indicate that the models can generally recover the relevant information from the attachments\. Adding conflicting text often draws answers away from the attachment under both input orders, consistent with earlier findings on text reliance\.
Crucially,placing the attachment last systematically shifts judgments back toward its content\. Attachment\-answer selection increases by 9\.8–28\.1 percentage points across all twelve model–dataset combinations\. The mean log\-odds swap score is also positive throughout, with every 95% confidence interval excluding zero\. This consistent shift not only demonstrates noncommutativity in the positions of perceptual and textual evidence, but also suggests better judgment performance when evidence is presented later\.
### 4\.3Controlling for the effect of reordering
We next test whether the attachment\-last advantage reflects a general effect of reordering\. We repeat the swaps with matchedT\+T^\{\+\}and unrelatedT0T^\{0\}, keeping the attachment, question, and candidate answers fixed\. The unrelated condition provides a baseline for order effects when the text supports neither answer\. We subtract this baseline from the conflict condition to obtainD=ΔA−−ΔA0D=\\Delta A^\{\-\}\-\\Delta A^\{0\}\.
Table 2:Order effects under evidence control\. Values show changes in attachment\-answer selection whenPPappears last \(pp\)\.DD=ΔA−\\Delta A^\{\-\}\-ΔA0\\Delta A^\{0\}shows the net influence under evidence relation control\. The arrows are omitted\.PPandT\+T^\{\+\}mean only use that evidence\.In Table[2](https://arxiv.org/html/2609.26986#S4.T2),ΔA\+\\Delta A^\{\+\}shows that matched text usually produces little change\. An exception is Idefics3 on GQA, where placing the image last reduces accuracy by 6\.3 pp\. In this setting, matched\-text\-only accuracy is 98\.5%, compared with 89\.6% for the image alone\. One possible explanation is that placing the image last shifts reliance toward the source the model interprets less accurately\.
With unrelated text, order effectsΔA0\\Delta A^\{0\}are generally small and vary in sign\. After subtracting this baseline,DDremains positive across all twelve model–dataset combinations, ranging from 9\.4–28\.8 pp for vision and 10\.4–24\.0 pp for speech\. Thus,the attachment\-last advantage persists after accounting for the unrelated\-text order effect, strengthening the evidence for cross\-modal noncommutativity under conflict\.
### 4\.4Ruling out explanations
We next investigate what might explain the consistent attachment\-last advantage\. Both evidence sources precede the answer position and are available during prediction, yet exchanging their positions systematically changes the model’s judgments\. We probe several possible contributors to this order sensitivity and rule them out\.
Text\-only recency\.Removing the image and swapping correct and conflicting text yields effects from−11\.4\-11\.4to\+55\.6\+55\.6pp across six combinations, with inconsistent directions\. In the vision tests, all six combinations favor the image more when it appears last\. Text recency remains a possible contributor to the observed effects\.
Explicit source\-priority instructions\.In the vision tests, adding “If the description conflicts with the image, prioritize the image” remains 89–145% of the original order effect\. For FSC with Qwen2\-Audio, neutral, main, and stronger speech\-priority prompts yield effects of\+22\.5\+22\.5,\+27\.5\+27\.5, and\+34\.6\+34\.6pp, respectively\. MLLMs remain sensitive to order even when explicitly told which source to follow\.
Attachment wording\.On the same 60 exploratory SLURP examples, the original prompt yields\+21\.7\+21\.7pp\. Changing it to “Answer the question according to the audio attachment\.” yields\+23\.3\+23\.3pp, with a 95% confidence interval of\[14\.2,33\.3\]\[14\.2,33\.3\]\. The paired difference is\+1\.7\+1\.7pp,\[−1\.7,5\.0\]\[\-1\.7,5\.0\], and its interval includes zero\. This exploratory comparison applies to these examples\.
Evidence distance\.Inserting 64 or 256 non\-informative tokens between the visual and textual evidence increases the order effect in five combinations and decreases it for Qwen2\.5\-VL on GQA\. The effect of evidence distance therefore varies across models\.
Free\-form responses\.On FSC with Qwen2\-Audio, the order effect remains\+33\.3\+33\.3pp,\[27\.5,39\.2\]\[27\.5,39\.2\], showing that order also matters in free\-form responses\. Unrelated text can elicit a third answer, so comparisons with the conflict condition need a scoring rule that accounts for such responses\.
### 4\.5Revisiting existing protocols
The attachment\-last advantage persists across the prompt and response variants tested above\. We now revisit earlier studies to understand why their input designs can lead to different conclusions about modality preference\.
Reproducing and diagnostics\.*Words or Vision*\[[7](https://arxiv.org/html/2609.26986#bib.bib2)\]reports greater text preference in Phi\-3\.5 when the image follows the conflicting text\. We observe the same pattern with Qwen2\.5\-VL\-3B\-Instruct on 1,000 VQAv2 examples using its warning prompt and free\-form protocol: moving the image last reduces image\-answer accuracy from 42\.6% to 28\.9% \(−13\.7\-13\.7pp\)\. In this protocol, the image exchanges positions with a block containing the task framing, a warning, and the conflicting description, so the task instructions move along with the text\. The warning also describes the text as potentially irrelevant, incomplete, or inaccurate and advises caution\. Prior studies of negative\-token attention and negated prompts\[[31](https://arxiv.org/html/2609.26986#bib.bib24),[11](https://arxiv.org/html/2609.26986#bib.bib25)\]suggest that such wording may influence how the model processes the evidence\. Our exploratory attention comparison shows different attention distributions under the natural and warning prompts \(Fig\.[3](https://arxiv.org/html/2609.26986#S4.F3)a\), providing descriptive evidence of this sensitivity\.
We therefore apply our protocol to the same Qwen2\.5\-VL model: remove the warning, fix the instruction to answer from the image before both sources, and place the question and A/B options after them\. Only the image and description exchange positions\. On 807 examples answered correctly from each source in isolation, moving the image last increases image\-answer selection from 14\.9% to 32\.6% \(\+17\.7\+17\.7pp\), matching the direction found in our main experiments\. To test instruction placement directly, we retain the original framing and warning and fix both before the evidence \(Fig\.[3](https://arxiv.org/html/2609.26986#S4.F3)b\)\. On the same examples, moving the image last changes image\-answer selection by−28\.6\-28\.6pp under the original protocol and\+9\.5\+9\.5pp once the framing is fixed, a difference of\+38\.1\+38\.1pp\. This paired ablation shows that moving task instructions with the evidence can reverse the conclusion about which order favors image\-supported answers\.
Figure 3:Exploratory Qwen2\.5\-VL\-3B diagnostics on 20 examples: \(a\) attention moved by warning; \(b\) instruction placement ablation\.Extending the evaluation to MCR\-BENCH\.MCR\-BENCH\[[28](https://arxiv.org/html/2609.26986#bib.bib26)\]builds on*Words or Vision*and compares faithful, adversarial, and irrelevant text under a fixed audio–text order\. We evaluate both evidence orders on 1,000 examples each for audio question answering \(AQA\), speech emotion recognition \(SER\), and vocal sound classification \(VSC\)\. As shown in Table[3](https://arxiv.org/html/2609.26986#S5.T3), with the original warning retained, moving audio after conflicting text increases accuracy by\+17\.0\+17\.0,\+15\.1\+15\.1, and\+51\.3\+51\.3pp, respectively\. Under the audio\-attachment prompt without the warning, the corresponding gains are\+18\.2\+18\.2,\+11\.5\+11\.5, and\+44\.1\+44\.1pp\. The average gain across the three tasks is\+27\.8\+27\.8pp with the warning and\+24\.6\+24\.6pp without it, showing that placing audio last improves accuracy under both prompts\. The comparisons show why both evidence orders matter when assessing modality preference\. Instruction placement can reverse which order favors image\-supported answers, while a fixed audio–text layout leaves the gains from placing audio last unmeasured\.
## 5Discussion
Order contributes to measured text relianceText reliance under a fixed template reflects modality, position, distance from the question, and conflict between sources\. In our tests, moving the same image or speech recording later shifts answers toward it, while average modality preferences differ across models\. Paired swaps measure how order contributes to source reliance alongside possible textual shortcuts\. A source’s influence depends on where and how it appears in the input, including whether another source contradicts it\. Evaluations should test bothPP→\\toTTandTT→\\toPPand report average modality reliance alongside the change caused by swapping order\.
Implications for deployed systemsSpeech assistants often combine audio with an ASR transcript, a user’s description, device state, or retrieved information\. Our results suggest that the placement of these sources should be considered alongside their origin and consistency\. The same issue arises in multimodal retrieval\-augmented generation and agents, where retrieval ranking, attachment placement, and tool\-return order determine the input sequence\. Reordering these sources may change the answer even when their content stays the same\. A practical check is to reorder key evidence and query the model again\. If the answer changes, the system can review the sources, flag the disagreement, or seek clarification before a high\-risk action\.
Table 3:External benchmarks under original and aligned protocols in pp\.PPdenotes images for Words or Vision \(WoV\)\[[7](https://arxiv.org/html/2609.26986#bib.bib2)\]and audio for MCR\-BENCH \(MCR\)\[[28](https://arxiv.org/html/2609.26986#bib.bib26)\]\. Yellow highlights gains from placing audio last with the original warning retained\.
## 6Conclusion
In this paper, we demonstrate cross\-modal evidence noncommutativity: the same evidence can lead to different judgments when its order changes\. Across vision and speech models, placing an attachment after conflicting text consistently shifts judgments toward its content\. Revisiting earlier evaluations also reveals protocol confounds that can reverse the measured direction of these shifts and lead to misleading conclusions about modality preference\.
Our additional experiments probe several possible explanations and show that the shifts persist under alternative instructions and response formats\. Understanding their internal causes will require further analysis of perceptual representations, positional encoding, attention, and decoding\. We are also extending this study to multiple perceptual sourcesP1,…,PmP\_\{1\},\\ldots,P\_\{m\}and textual sourcesT1,…,TnT\_\{1\},\\ldots,T\_\{n\}to examine how different source combinations and orderings affect the interactions between evidence sources and the resulting judgments\.
ACKNOWLEDGEMENTS
This work is partially funded by the European Union \(under grant agreement ID 101212818\)\. Views and opinions expressed are however those of the author\(s\) only and do not necessarily reflect those of the European Union or European Health and Digital Executive Agency \(HADEA\)\. Neither the European Union nor the granting authority can be held responsible for them\. This work is partially supported by Innovate UK through AI\-PASSPORT under Grant 10126404\. This work was awarded a grant by the AI Security Institute \(AISI\) via the Alignment Project \(Rare\-Event Estimation in Large Language Models via Subset Simulation\) and funded by EPSRC\. Yi’s contribution is partially supported through the Royal Society international exchanges programme and in part by the Engineering and Physical Sciences Research Council, through funding from RAi UK \[EP/Y009800/1\]\.
The authors declare no relevant financial or non\-financial conflicts of interest\.
COMPLIANCE WITH ETHICAL STANDARDS
No new human participants were recruited and no new human\-subject data were collected for this study\. All experiments use existing publicly available benchmark datasets under their respective terms of use\.
## References
- \[1\]P\. Agrawalet al\.\(2024\)Pixtral 12b\.Note:arXiv:2410\.07073Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p3.1)\.
- \[2\]A\. Azulay, J\. Dubiński, Z\. Li, A\. Mittal, and Y\. Gandelsman\(2026\)Jailbreaking vision\-language models through the visual modality\.InProc\. ICML,External Links:[Link](https://openreview.net/forum?id=4VuMV9beC1)Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[3\]S\. Baiet al\.\(2025\)Qwen2\.5\-vl technical report\.Note:arXiv:2502\.13923Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p3.1)\.
- \[4\]E\. Bastianelli, A\. Vanzo, P\. Swietojanski, and V\. Rieser\(2020\)SLURP: a spoken language understanding resource package\.InProc\. EMNLP,pp\. 7252–7262\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.588)Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p2.1)\.
- \[5\]L\. Chen, Y\. Gao, J\. Wang, K\. Ji, and H\. Chen\(2026\)Do system prompts leave behavioral fingerprints? a large\-scale empirical study of clone detection via output similarity\.External Links:2608\.24461,[Link](https://arxiv.org/abs/2608.24461)Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[6\]Y\. Chuet al\.\(2024\)Qwen2\-audio technical report\.Note:arXiv:2407\.10759Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p3.1)\.
- \[7\]A\. Deng, T\. Cao, Z\. Chen, and B\. Hooi\(2025\)Words or vision: do vision\-language models have blind faith in text?\.InProc\. IEEE/CVF CVPR,pp\. 3867–3876\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52734.2025.00366)Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p2.1),[§1](https://arxiv.org/html/2609.26986#S1.p3.1),[§2](https://arxiv.org/html/2609.26986#S2.p1.1),[§4\.5](https://arxiv.org/html/2609.26986#S4.SS5.p2.1),[Table 3](https://arxiv.org/html/2609.26986#S5.T3)\.
- \[8\]Y\. Gaoet al\.\(2026\)PathMark: protecting intellectual property of mixture\-of\-expert llms via path watermarks\.Note:arXiv:2607\.03688Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[9\]Y\. Goyal, T\. Khot, A\. Agrawal, D\. Summers\-Stay, D\. Batra, and D\. Parikh\(2019\)Making the v in vqa matter: elevating the role of image understanding in visual question answering\.Int\. J\. Comput\. Vis\.127\(4\),pp\. 398–414\.External Links:[Document](https://dx.doi.org/10.1007/s11263-018-1116-0)Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p1.1)\.
- \[10\]D\. A\. Hudson and C\. D\. Manning\(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.InProc\. IEEE/CVF CVPR,pp\. 6693–6702\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2019.00686)Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p1.1)\.
- \[11\]J\. Jang, S\. Ye, and M\. Seo\(2023\)Can large language models truly understand prompts? a case study with negated prompts\.InProc\. TL4NLP Workshop,pp\. 52–62\.Cited by:[§4\.5](https://arxiv.org/html/2609.26986#S4.SS5.p2.1)\.
- \[12\]R\. Krishnaet al\.\(2017\)Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations\.Int\. J\. Comput\. Vis\.123\(1\),pp\. 32–73\.External Links:[Document](https://dx.doi.org/10.1007/s11263-016-0981-7)Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p1.1)\.
- \[13\]H\. Laurençon, A\. Marafioti, V\. Sanh, and L\. Tronchon\(2024\)Building and better understanding vision\-language models: insights and future directions\.Note:arXiv:2408\.12637Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p3.1)\.
- \[14\]J\. Lee, Y\. Yoon, and T\. Kim\(2026\)Same semantics, different outcome: on the modality robustness of multimodal LLMs under knowledge conflict\.Note:arXiv:2609\.00550Cited by:[§2](https://arxiv.org/html/2609.26986#S2.p1.1)\.
- \[15\]Z\. Li, B\. Wang, J\. Hu, X\. Huang, and Y\. Dong\(2026\)FragileFlow: spectral control of correct\-but\-fragile predictions for foundation model robustness\.Note:arXiv:2605\.08896Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[16\]Z\. Li, B\. Wang, C\. Wu, X\. Huang, and Y\. Dong\(2026\)SCOPE: sequential conformal probing for reliable ood rejection in llm services\.Note:arXiv:2606\.21255Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[17\]T\.\-Y\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan,et al\.\(2014\)Microsoft coco: common objects in context\.InProc\. ECCV,pp\. 740–755\.Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p1.1)\.
- \[18\]N\. F\. Liuet al\.\(2024\)Lost in the Middle: How Language Models Use Long Contexts\.Trans\. Assoc\. Comput\. Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p2.1),[§2](https://arxiv.org/html/2609.26986#S2.p2.1)\.
- \[19\]X\. Liu, W\. Wang, Y\. Yuan, J\.\-T\. Huang, Q\. Liu, P\. He,et al\.\(2025\)Insight over sight: exploring the vision\-knowledge conflicts in multimodal LLMs\.InProc\. ACL,pp\. 17825–17846\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.872)Cited by:[§2](https://arxiv.org/html/2609.26986#S2.p1.1)\.
- \[20\]L\. Lugosch, M\. Ravanelli, P\. Ignoto, V\. S\. Tomar, and Y\. Bengio\(2019\)Speech Model Pre\-Training for End\-to\-End Spoken Language Understanding\.InProc\. INTERSPEECH,pp\. 814–818\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2019-2396)Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p2.1)\.
- \[21\]Microsoftet al\.\(2025\)Phi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-loras\.Note:arXiv:2503\.01743Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p3.1)\.
- \[22\]P\. Pezeshkpour, M\. Aminnaseri, and E\. Hruschka\(2025\)Mixed signals: decoding VLMs’ reasoning and underlying bias in vision\-language conflict\.InFindings ACL: EMNLP,pp\. 24833–24848\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1351)Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p2.1),[§1](https://arxiv.org/html/2609.26986#S1.p3.1),[§2](https://arxiv.org/html/2609.26986#S2.p1.1)\.
- \[23\]P\. Pezeshkpour and E\. Hruschka\(2024\)Large language models sensitivity to the order of options in multiple\-choice questions\.InFindings ACL: NAACL,pp\. 2006–2017\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.130)Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p2.1),[§2](https://arxiv.org/html/2609.26986#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p4.1)\.
- \[24\]Z\. Tan, X\. Chu, W\. Li, and T\. Mo\(2024\)Order matters: exploring order sensitivity in multimodal large language models\.Note:arXiv:2410\.16983Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p2.1),[§2](https://arxiv.org/html/2609.26986#S2.p2.1)\.
- \[25\]B\. Wang, Z\. Li, X\. Huang, and Y\. Dong\(2025\)Rethinking multi\-agent intelligence through the lens of small\-world networks\.External Links:2512\.18094,[Link](https://arxiv.org/abs/2512.18094)Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[26\]B\. Wang, Z\. Li, X\. Huang, and Y\. Dong\(2026\)Dive into ambiguity: a\*\-inspired multi\-agents commonsense obfuscation attack on llm prompts\.External Links:2606\.01441,[Link](https://arxiv.org/abs/2606.01441)Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[27\]B\. Wanget al\.\(2026\)Chain\-of\-thought as a lens: evaluating structured reasoning alignment between human preferences and large language models\.InProc\. ACL \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 39514–39530\.External Links:[Link](https://aclanthology.org/2026.acl-long.1834/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1834),ISBN 979\-8\-89176\-390\-6Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[28\]C\. Wang, G\. Deng, X\. Yang, H\. Qiu, and T\. Zhang\(2025\)When audio and text disagree: revealing text bias in large audio\-language models\.InProc\. EMNLP,pp\. 4878–4888\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.246)Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p3.1),[§2](https://arxiv.org/html/2609.26986#S2.p1.1),[§4\.5](https://arxiv.org/html/2609.26986#S4.SS5.p4.1),[Table 3](https://arxiv.org/html/2609.26986#S5.T3)\.
- \[29\]J\. Xuet al\.\(2025\)Qwen2\.5\-omni technical report\.Note:arXiv:2503\.20215Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p3.1)\.
- \[30\]J\. Yuet al\.\(2025\)Black\-box adversarial defense based on image decomposition and reconstruction\.IEEE Transactions on Multimedia27\(\),pp\. 5909–5921\.External Links:[Document](https://dx.doi.org/10.1109/TMM.2025.3565987)Cited by:[SAME EVIDENCE, DIFFERENT JUDGMENTS: EVIDENCE NONCOMMUTATIVITY IN VISION/SPEECH\-TEXT CONFLICTS](https://arxiv.org/html/2609.26986#p1.1)\.
- \[31\]S\. Yu, J\. Song, B\. Hwang, H\. Kang, S\. Cho, J\. Choi,et al\.\(2025\)Correcting negative bias in large language models through negative attention score alignment\.InProc\. NAACL,pp\. 9979–10001\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.503)Cited by:[§4\.5](https://arxiv.org/html/2609.26986#S4.SS5.p2.1)\.
- \[32\]C\. Zhang, W\. Ding, J\. Liu, M\. Wu, Q\. Wu, and R\. Mooney\(2026\)Do images speak louder than words? investigating the effect of textual misinformation in VLMs\.InProc\. EACL,pp\. 6872–6895\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.323)Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p2.1),[§2](https://arxiv.org/html/2609.26986#S2.p1.1)\.
- \[33\]Z\. Zhang, W\. Zhou, J\. Zhao, and H\. Li\(2025\)Robust multimodal large language models against modality conflict\.InProc\. ICML,pp\. 77233–77253\.Cited by:[§1](https://arxiv.org/html/2609.26986#S1.p2.1),[§2](https://arxiv.org/html/2609.26986#S2.p1.1)\.
- \[34\]C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. Huang\(2024\)Large language models are not robust multiple choice selectors\.InProc\. ICLR,Cited by:[§4\.1](https://arxiv.org/html/2609.26986#S4.SS1.p4.1)\.
- \[35\]T\. Zhu, Q\. Liu, F\. Wang, Z\. Tu, and M\. Chen\(2024\)Unraveling cross\-modality knowledge conflicts in large vision\-language models\.Note:arXiv:2410\.03659Cited by:[§2](https://arxiv.org/html/2609.26986#S2.p1.1)\.
[15](https://arxiv.org/html/2609.26986#bib.bib27),[16](https://arxiv.org/html/2609.26986#bib.bib28),[2](https://arxiv.org/html/2609.26986#bib.bib32),[8](https://arxiv.org/html/2609.26986#bib.bib29),[30](https://arxiv.org/html/2609.26986#bib.bib31),[5](https://arxiv.org/html/2609.26986#bib.bib30),[27](https://arxiv.org/html/2609.26986#bib.bib33),[26](https://arxiv.org/html/2609.26986#bib.bib34),[25](https://arxiv.org/html/2609.26986#bib.bib35)Similar Articles
Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
This paper investigates why multimodal LLMs fail on vision-centric tasks when visual evidence conflicts with language priors, introducing the WhatIfVis benchmark and showing that models often encode visual evidence but cannot reliably control their reliance on it.
Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
该论文提出一个两参数贝叶斯线索组合模型,解释视觉-语言模型在文本与图像证据冲突时如何整合信息,并能准确预测模型在未见过的输入退化组合下的偏好,统一了此前关于文本主导和视觉主导的矛盾报告。
Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts
The paper introduces Tri-PvP, a benchmark that exposes visual bias and asymmetric evidence-form preferences in omni-modal large language models, revealing deep-seated modality biases that are linearly decodable from early layers and resistant to surface mitigation.
Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models
This paper investigates how vision-language models resolve conflicts between visual evidence and world knowledge, revealing that visual grounding is the default while prior knowledge depends on a small set of late-layer attention heads. The authors perform causal analysis across three VLM families, demonstrating an asymmetric structure where ablating these heads shifts predictions from knowledge-grounded to visually grounded answers.
Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
The paper examines how vision-language models handle conflicts between in-context and parametric knowledge, revealing asymmetric biases where models favor in-context information for text entities but parametric information for image entities due to slower visual processing.