When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

# When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models
Source: [https://arxiv.org/html/2608.19208](https://arxiv.org/html/2608.19208)
Yinfeng Wang Zhiyuan Yao Zheren Fu Lei Zhang Zhendong Mao University of Science and Technology of China Hefei, China wyf666@mail\.ustc\.edu\.cn

###### Abstract

Multimodal large language models \(MLLMs\) are frequently exposed to auxiliary textual context, the impact of which on visually grounded tasks remains underexplored\. In this paper, we investigate the influence of task\-irrelevant context by formulating it as a controlled intervention within a binary visual judgment framework\. By maintaining an invariant prompt structure while varying auxiliary inputs, we observe that irrelevant text consistently biases model predictions across diverse benchmarks\. To move beyond performance metrics, we characterize this sensitivity through a decision margin defined by the log\-probability difference between binary candidates\. Our analysis reveals a robust geometric regularity: context\-conditioned margins follow a consistent affine transformation of their context\-free counterparts\. This finding demonstrates that irrelevant context does not manifest as unstructured stochastic noise but as a estimable distortion of model preference\. We further interpret the fitted affine parameters as metrics for visual commitment preservation and directional answer bias\. These findings provide a margin\-level diagnostic view of irrelevant\-context effects in MLLMs and offer a basis for future studies on noisy\-context robustness\. We release all the code in[https://github\.com/Wangyf1998/Irrelevant\_Text\_Matters](https://github.com/Wangyf1998/Irrelevant_Text_Matters)\.

When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

Yinfeng Wang Zhiyuan Yao Zheren Fu Lei Zhang Zhendong Mao††thanks:Corresponding author\.University of Science and Technology of ChinaHefei, Chinawyf666@mail\.ustc\.edu\.cn

![Refer to caption](https://arxiv.org/html/2608.19208v1/x1.png)Figure 1:Overview of the controlled intervention and its behavioral effects\. \(a\) For each image\-question pair, we compare a context\-free input with a paired context\-conditioned input, where an irrelevant context is inserted while the prompt structure is kept unchanged\. \(b\) Across multiple benchmarks, irrelevant context not only reduces predictive accuracy, but also shifts model decisions toward negative responses\.## 1Introduction

Multimodal large language models \(MLLMs\) have demonstrated strong cross\-modal understanding capabilities, enabling complicated vision\-language tasks such as visual question answering, multi\-turn reasoning, and multimodal agents\. In these scenarios, the input image is often accompanied by auxiliary textual context, such as retrieved passages, user\-provided side information, or history dialogue\. Such context is not always aligned with the current task: retrieved passages may be off\-topicMortahebet al\.\([2025](https://arxiv.org/html/2608.19208#bib.bib35)\), while instructions and dialogue may also contain irrelevant informationWanget al\.\([2024b](https://arxiv.org/html/2608.19208#bib.bib55)\); Parket al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib56)\)\. This raises a question:Does irrelevant context affect MLLM predictions on vision\-language tasks?

Our question is motivated by a broader observation in NLP community: large language models \(LLMs\) can be sensitive to textual context that is not part of the task\-relevant evidence\. Prior studies have shown that irrelevant or distracting context can mislead LLMs in question answering and reasoning tasksJia and Liang \([2017](https://arxiv.org/html/2608.19208#bib.bib20)\); Shiet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib21)\); Amirazet al\.\([2025](https://arxiv.org/html/2608.19208#bib.bib18)\)\. Similar issues arise in retrieval\-augmented generation, where retrieved passages containing irrelevant information will affect the model’s performanceYoranet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib22)\); Yanet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib24)\); Cuconasuet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib19)\)\. In MLLMs, recent work shows that they may over\-rely on additional textual context and resulting in a strong biasDenget al\.\([2025](https://arxiv.org/html/2608.19208#bib.bib9)\); Chenet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib33)\); Zhanget al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib34)\)\. However, despite these observations, there remains a lack of systematic investigation into how irrelevant context transforms MLLM decision\-making in terms of behavioral effects and the underlying mechanisms\.

![Refer to caption](https://arxiv.org/html/2608.19208v1/x2.png)Figure 2:Illustration of the margin\-level analysis\. \(a\) The decision margin is defined as the log\-probability difference between the positive and negative answers\. \(b\) Adding irrelevant context transforms the context\-free marginm0m\_\{0\}into the context\-conditioned marginmcm\_\{c\}, which follows an approximate affine relationmc≈a​m0\+bm\_\{c\}\\approx am\_\{0\}\+b\. \(c\) The affine transformation shifts the effective decision boundary fromm0=0m\_\{0\}=0tom0=−b/am\_\{0\}=\-b/a\. This shift expands the Negative Region and compresses the Positive Region\. As a result, samples in the Flip Region, i\.e\.,0<m0<−b/a0<m\_\{0\}<\-b/a, are likely to change from positive to negative predictions, inducing a systematic shift in decision preference\.To bridge this gap, we conduct a systematic investigation into the influence of image\-irrelevant text, utilizing it as a controlled intervention within binary visually grounded prediction tasks\. Specifically, as illustrated in Figure[1](https://arxiv.org/html/2608.19208#S0.F1)\(a\), for each image\-question pair, we treat the original input containing only the image and the question as the*context\-free*condition\. We then construct a paired*context\-condition*input by inserting an irrelevant textual passage randomly sampled from an external corpus, while keeping the overall prompt structure unchanged\. As shown in Figure[1](https://arxiv.org/html/2608.19208#S0.F1)\(b\), evaluation across multiple benchmarks shows that irrelevant context not only degrades the model’s overall performance, but also induces a consistent shift in decision preference toward negative responses\.

To better characterize this behavior, we introduce the*decision margin*, defined as the log\-probability difference between the positive and negative answers, as illustrated in Figure[2](https://arxiv.org/html/2608.19208#S1.F2)\(a\)\. Surprisingly, as shown in Figure[2](https://arxiv.org/html/2608.19208#S1.F2)\(b\), the context\-condition margin can be well approximated by a simple affine transformation of the context\-free margin, i\.e\.,mc≈a​m0\+bm\_\{c\}\\approx am\_\{0\}\+b, wherem0m\_\{0\}andmcm\_\{c\}denote the margins under the context\-free and context\-conditioned inputs, respectively\. This affine structure further explains the observed prediction flips\. As illustrated in Figure[2](https://arxiv.org/html/2608.19208#S1.F2)\(c\), the effective decision boundary shifts fromm0=0m\_\{0\}=0tom0=−b/am\_\{0\}=\-b/a\. Consequently, samples with weak positive margins are likely to cross the shifted boundary and flip to negative predictions, accounting for the systematic tendency toward negative responses\.

Empirically, this affine relation exhibits two notable regularities\. First, it appears consistently across different MLLMs\. Second, for a given model, the fitted parameters remain close across different benchmarks\. Inspired by the implicit Bayesian interpretation of in\-context learningXieet al\.\([2021](https://arxiv.org/html/2608.19208#bib.bib32)\), we interpret the slopeaaas the preservation of the original visual decision structure, and the interceptbbas the directional answer offset introduced by irrelevant text\. Based on the above observations, we design a lightweight post\-hoc calibration, to partially mitigate the effect of irrelevant context at inference time\.

Our analysis extends the understanding of irrelevant\-context effects in MLLMs\. We show that such effects can follow a stable and estimable affine pattern at the margin level, and provide new insight on how textual noise biases multimodal decision\-making\. These findings suggest that MLLMs remain sensitive to irrelevant context, even when it provides no useful evidence\. We hope our work can serve as a building block for future research on noisy\-context robustness, and contribute to the development of more trustworthy and robust multimodal systems\.

## 2Irrelevant Context Affects Prediction

In this section, we first introduce our experiment setup, then empirically show that irrelevant context does change MLLM predictions\.

![Refer to caption](https://arxiv.org/html/2608.19208v1/x3.png)Figure 3:Visualization of three input conditions\. These three conditions represent different degrees to which the irrelevant context is framed as potentially image\-related\.### 2\.1Problem Formulation

Given an imageII, a questionQQ, and an image\-irrelevant textual contextCC, we compare model predictions under two input conditions:

xo=\(I,Q\),xc=\(I,C,Q\),x\_\{o\}=\(I,Q\),\\qquad x\_\{c\}=\(I,C,Q\),\(1\)Here,xox\_\{o\}denotes the context\-free input, andxcx\_\{c\}denotes the context\-conditioned input\.

We formulate each sample as a binary visual judgment\. The model is asked to decide whether the visual claim implied by the question is supported by the image\. We denote the binary label space as

𝒴=\{y\+,y−\},\\mathcal\{Y\}=\\\{y^\{\+\},y^\{\-\}\\\},\(2\)wherey\+y^\{\+\}denotes a positive judgment that the claim is supported, andy−y^\{\-\}denotes a negative judgment that the claim is not supported\. For a modelfθf\_\{\\theta\}, the corresponding predictions are

y^o=fθ​\(xo\),y^c=fθ​\(xc\),\\hat\{y\}\_\{o\}=f\_\{\\theta\}\(x\_\{o\}\),\\qquad\\hat\{y\}\_\{c\}=f\_\{\\theta\}\(x\_\{c\}\),\(3\)wherey^o,y^c∈𝒴\\hat\{y\}\_\{o\},\\hat\{y\}\_\{c\}\\in\\mathcal\{Y\}\.

This formulation adopts two key simplifications for controlled analysis: we cast open\-ended visually grounded predictions into binary positive/negative judgments, and we treat auxiliary textual context as a controlled intervention by varying only the irrelevant context\. Models typically answer with "Yes" and "No" under the default instruction\. See Appendix[C\.2](https://arxiv.org/html/2608.19208#A3.SS2)for analysis of different verbalizations, and Appendix[E\.1](https://arxiv.org/html/2608.19208#A5.SS1)for more discussions on problem formulation\.

Table 1:Behavioral effects of image\-irrelevant context across datasets and models\. Irrelevant context generally degrades performance, induces a negative\-response shift, and has stronger effects when presented as potentially image\-related\.Freeindicates the context\-free condition, whileΔ\\DeltaNeu\.andΔ\\DeltaCon\.report changes under the context\-neutral and context\-condition setting relative to context\-free condition\.Yes Ratereports the proportion of positive responses among all answers, indicating the model’s tendency toward affirmative predictions\.Flip Ratereports the percentage of samples whose prediction changes\. All values are reported in percentage points\.![Refer to caption](https://arxiv.org/html/2608.19208v1/x4.png)Figure 4:Affine margin patterns for LLaVA\-1\.5\-7B across four benchmarks\. Each panel plots the context\-free marginmom\_\{o\}against the context\-conditioned marginmcm\_\{c\}\. Points are colored by prediction transitions between the two conditions, and the solid line shows the fitted affine relationmc=a​mo\+bm\_\{c\}=am\_\{o\}\+b; the dashed line denotes the identity relationmc=mom\_\{c\}=m\_\{o\}\. The consistently highR2R^\{2\}and correlation values indicate that textual context induces a structured affine transformation of the original decision margin rather than unstructured perturbations\. Notably, changing the semantic framing from Neutral to Condition substantially decreases the fitted slopeaa, while leaving the negative offsetbblargely stable\. Results are shown for LLaVA\.
### 2\.2Experiment Setup

#### Irrelevant Context Sampling

Given an image\-question pair\(I,Q\)\(I,Q\), we first sample irrelevant contextsCCfrom WikiTextMerityet al\.\([2017](https://arxiv.org/html/2608.19208#bib.bib30)\)with a fixed random seed\. We control the length of each sampled sentence within a predefined range, and apply filtering process to ensure that the selected context is unrelated to the image\. See Appendix[A\.2](https://arxiv.org/html/2608.19208#A1.SS2)for implementation details\.

#### Input Construction

After irrelevant context sampling, we construct inputs under three conditions\. A case is shown in Figure[3](https://arxiv.org/html/2608.19208#S2.F3)\. For all settings, the question is followed by the same answer instruction:

> Answer the question using a single word or phrase\.

Thecontext\-freeinput contains only the image and the question:

> \[Image\] \[Question\]

Thecontext\-neutralinput appends the sampled irrelevant context:

> \[Image\] \[Question\] \[Context\]

Thecontext\-conditioninput further introduces a relevance template as:

> The following caption is possibly related to the image:

The template is placed before the context:

> \[Image\] \[Question\] \[Template\] \[Context\]

These three conditions represent different degrees to which the irrelevant text is framed as potentially image\-related\. All the samples are then fed into models, and use greedy decoding to get the final answer\.

#### Evaluation Benchmarks and Models

We evaluate on four widely used vision\-language benchmarks that cover diverse forms of visually grounded judgment\.POPELiet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib8)\)focuses on object\-existence verification and is commonly used to evaluate object hallucination in MLLMs\.AMBERWanget al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib26)\)extends this evaluation to multiple hallucination\-related dimensions, including object existence, attributes, and relations\.MMEFuet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib27)\)provides a broader diagnostic benchmark, covering perception\-oriented tasks such as color, count, OCR, and position recognition, as well as cognition\-oriented tasks such as commonsense and code reasoning\.GQAHudson and Manning \([2019](https://arxiv.org/html/2608.19208#bib.bib28)\)evaluates compositional visual reasoning over objects, attributes, relations, and spatial layouts\. See Appendix[A\.3](https://arxiv.org/html/2608.19208#A1.SS3)for data statistics and implementation detail\.

For models, we report results onLLaVA\-1\.5\-7bLiuet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib12)\),Qwen2\-VLWanget al\.\([2024a](https://arxiv.org/html/2608.19208#bib.bib37)\)family \(2b and 7b\) andIntern\-VL3\-8bZhuet al\.\([2025](https://arxiv.org/html/2608.19208#bib.bib38)\)\.

#### Discussion of Confounds

We further examine whether the observed behavioral shift can be explained by simple experimental confounds\. Firstly, the effect is not tied to a particular context corpus: replacing WikiText with alternative context sources, such as COCO captionsChenet al\.\([2015](https://arxiv.org/html/2608.19208#bib.bib31)\)or shuffled WikiText, preserves the main trend, although image\-like captions tend to induce stronger interference\. Secondly, it is also not explained by context length alone, since the affine distortion remains observable across different length buckets\. Finally, prompt formatting affects the magnitude but not the existence of the phenomenon\. Detailed settings and results are provided in Appendix[C\.1](https://arxiv.org/html/2608.19208#A3.SS1)\.

### 2\.3Experimental Results

We reportAccuracy,Recall, andYes\-rateunder the Context\-free condition, together with their changes under the context\-neutral and context\-condition settings\. We further reportFlip\-rateto measure sample\-level prediction changes\.

The results in Table[1](https://arxiv.org/html/2608.19208#S2.T1)reveal three main patterns:

#### Adding irrelevant context hurts performance\.

Across models and benchmarks, adding image\-irrelevant context generally reduces predictive performance\. This degradation appears under both context\-neutral and context\-conditioned settings, indicating that auxiliary text can interfere with visually grounded prediction\.

#### The behavioral shift is directional\.

The performance drop is accompanied by a systematic change in answer preference\. In most settings, the Yes rate decreases after irrelevant context is added, indicating that the induced errors are not merely random\.

#### Semantic framing amplifies the effect\.

The context\-conditioned setting usually produces larger changes than the context\-neutral setting, especially in Yes rate and flip rate\. This suggests that models are more affected when the same irrelevant text is framed as potentially image\-related\.

These behavioral patterns suggest that irrelevant context does not act as unstructured noise; instead, it systematically changes the model’s decision preferences\. In the next section, we investigate this effect in detail through a margin\-level analysis\. See Appendix[B\.1](https://arxiv.org/html/2608.19208#A2.SS1)for complement experiment result\.

## 3Affine Margin Shift

In this section, we analyze the effect of irrelevant context in logit space, and identify a simple regularity: its influence can be approximated by an affine transformation\.

Table 2:Fitted affine parameters across models and benchmarks\. The affine relationship exhibits consistently high goodness of fit, indicating that such affine is structural and stable\.### 3\.1Decision Margin Design

Given an inputxx, we define thedecision marginas the log\-probability difference between the two candidate answers:

m​\(x\)=log⁡P​\(y\+∣x\)−log⁡P​\(y−∣x\),m\(x\)=\\log P\(y^\{\+\}\\mid x\)\-\\log P\(y^\{\-\}\\mid x\),\(4\)
Our design is motivated by DPORafailovet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib39)\): its sign indicates the model preference, while its magnitude reflects the confidence\. See Appendix[A\.1](https://arxiv.org/html/2608.19208#A1.SS1)for implementation details\.

### 3\.2Affine Regularity of Decision Margins

Figure[4](https://arxiv.org/html/2608.19208#S2.F4)shows the affine margin patterns for LLaVA\-1\.5\-7B across four benchmarks\. We identify a regularity: the impact of irrelevant text can be characterized by an affine transformation:

mc≈a⋅m0\+b,m\_\{c\}\\approx a\\cdot m\_\{0\}\+b,\(5\)whereaaandbbrepresent model\-specific parameters estimated via least squares\.

To step further, we have following observations:

#### The affine form is consistent across different MLLMs\.

Table[2](https://arxiv.org/html/2608.19208#S3.T2)shows the parameters under context\-neutral settings across different benchmarks and models\. The consistently highR2R^\{2\}values confirm that the affine relationship is a fundamental score\-level response to auxiliary textual input\. This regularity persists independently of specific task distributions, suggesting a shared behavioral constraint in MLLMs\.

#### The affine parameters remain stable across benchmarks\.

For a model, the fitted slopeaaand offsetbbremain close across different benchmarks, despite substantial differences in task type and data distribution\. This suggests that the affine pattern is not a dataset\-specific artifact, but reflects a model\-specific response to auxiliary textual context\.

#### Different semantic framings affect the scaling component\.

Compared with the Neutral, the Condition setting consistently yields a smaller slopeaa, indicating stronger margin compression\. In contrast, the offset termbbremains negative and of comparable magnitude across framings, suggesting that the answer\-direction shift is relatively stable\. We will provide an explanation for this phenomenon in the following section\.

These observations suggest that the influence of irrelevant context is a systematic distortion\. See Appendix[B\.2](https://arxiv.org/html/2608.19208#A2.SS2)and Appendix[D](https://arxiv.org/html/2608.19208#A4)for more results and robust verification\.

### 3\.3Boundary Shift Explains Prediction Flips

Above results also explain the observed behavioral changes\. Without context, the decision boundary is located atm0=0m\_\{0\}=0\. After adding context, the margin is approximately transformed asmc≈a​m0\+bm\_\{c\}\\approx am\_\{0\}\+b, so the context\-condition decision boundary satisfiesa​m0\+b=0am\_\{0\}\+b=0\. Mapped back to the original margin axis, the effective boundary becomes

m0=−ba,m\_\{0\}=\-\\frac\{b\}\{a\},\(6\)Thus, an originally affirmative sample must have a sufficiently large positive margin to remain affirmative after irrelevant context is added; samples with weak margins can otherwise fall below the shifted boundary and flip\.

We verify this explanation by grouping samples according to the magnitude of their context\-free margin\|m0\|\|m\_\{0\}\|and computing the flip rate within each bin\. As shown in Figure[5](https://arxiv.org/html/2608.19208#S3.F5), flip rates are consistently highest for low\-margin samples and decrease as\|m0\|\|m\_\{0\}\|grows across models and benchmarks\. This confirms that context\-induced flips are not uniformly distributed over samples, but are concentrated near the original decision boundary\.

![Refer to caption](https://arxiv.org/html/2608.19208v1/x5.png)Figure 5:Flip rate across context\-free margin bins\. Samples with smaller\|m0\|\|m\_\{0\}\|are much more likely to change their predictions after adding irrelevant context\. This indicates that context\-induced flips are concentrated near the original decision boundary\.

## 4Interpreting the Affine Parameters

In this section, we use a lightweight Bayesian view to interpret the fitted parameters\.

### 4\.1Preliminary

We first introduce a lightweight Bayesian\-inspired view to describe how irrelevant context may enter the model’s prediction process\.

Let𝒳\\mathcal\{X\}denote the original input without context\. The model defines a context\-free predictive distributionp​\(Y∣𝒳\)p\(Y\\mid\\mathcal\{X\}\)\. When an additional contextCCis provided, the model may use it as an additional conditioning signal when interpreting the input\. Following the prior workXieet al\.\([2021](https://arxiv.org/html/2608.19208#bib.bib32)\), we introduce a latent interpretation variableZ∈𝒵Z\\in\\mathcal\{Z\}, which represents a task\-specific “concept”\. The context\-conditioned prediction can then be written as posterior\-weighted marginalization:

p​\(Y∣𝒳,C\)=∫𝒵p​\(Y∣𝒳,Z\)​p​\(Z∣𝒳,C\)​𝑑Z,p\(Y\\mid\\mathcal\{X\},C\)=\\int\_\{\\mathcal\{Z\}\}p\(Y\\mid\\mathcal\{X\},Z\)\\,p\(Z\\mid\\mathcal\{X\},C\)\\,dZ,\(7\)
Under this view, addingCCupdates the model’s belief over latent interpretations:

p​\(Z∣𝒳,C\)∝p​\(C∣Z,𝒳\)​p​\(Z∣𝒳\),p\(Z\\mid\\mathcal\{X\},C\)\\propto p\(C\\mid Z,\\mathcal\{X\}\)p\(Z\\mid\\mathcal\{X\}\),\(8\)In our experiments,CCprovides no useful evidence, i\.e\.,Y⟂C∣𝒳Y\\perp C\\mid\\mathcal\{X\}\. However, image\-text alignment priors may lead the model to treatCCas useful evidence rather than irrelevant noise\. Consequently,p​\(C∣Z,𝒳\)p\(C\\mid Z,\\mathcal\{X\}\)may be non\-uniform across latent interpretations, shiftingp​\(Z∣𝒳,C\)p\(Z\\mid\\mathcal\{X\},C\)away from the original beliefp​\(Z∣𝒳\)p\(Z\\mid\\mathcal\{X\}\)\.

### 4\.2Explanation of the Affine Parameters

The Bayesian formulation provides a plausible lens\. Under this interpretation, the fitted affine map summarizes two observable effects: how much of the original visual decision structure is preserved, and whether the added context introduces a directional answer offset\.

#### Slope as commitment preservation\.

We interpret the slopeaaas the degree to which the original visual decision structure is preserved after context\-conditioned inference\. A largeraaindicates that the original margin geometry is better preserved, while a smalleraaindicates stronger attenuation of the original visual commitment\.

#### Intercept as answer\-direction offset\.

The interceptbbcaptures the directional offset introduced by the added context\. Since the margin is defined such that positive values favorYes, a negativebbcorresponds to a shift towardNo\. For image\-irrelevant context, this term is especially informative because the added text does not provide label\-conditioned visual evidence and therefore tends to appear as a global directional bias in the answer space\.

This interpretation explains the effect of semantic framing observed in Sec\.[3\.2](https://arxiv.org/html/2608.19208#S3.SS2)\. The Neutral and Condition settings contain the same type of image\-irrelevant text, but the Condition template frames the text as potentially related to the image: this framing increases the likelihood that the model treats the irrelevant context as useful contextual evidence, thereby reweighting the latent interpretation more strongly and weakening the preservation of the original visual margin\. At the same time, because the context is unchanged, the induced answer\-direction prior is not fundamentally changed\.

![Refer to caption](https://arxiv.org/html/2608.19208v1/x6.png)Figure 6:Experiment results for Validating the Parameter Interpretation\. Points are colored by ground\-truth label\. Arrows show the mean label\-conditioned margin shift from the identity line to the observed context\-conditioned mean\. The monotonic trendasup\>airr\>aconfa\_\{\\mathrm\{sup\}\}\>a\_\{\\mathrm\{irr\}\}\>a\_\{\\mathrm\{conf\}\}is consistent with the interpretation ofaaas commitment preservation\.

### 4\.3Validating the Parameter Interpretation

To test above implications, we construct a controlled context\-type intervention\.

#### Intervention setup

Our key idea is that, Ifaaandbbrespectively capture visual\-decision preservation and directional answer offset, then varying the evidential role of the context should change the affine parameters\. Therefore, we construct three context types:

Supporting contextis consistent with the ground\-truth answer, which states that the queried object is present and absent for negative samples\.Irrelevant contextis the same as in the context\-conditioned setting\.Conflicting contextstates absence for positive samples and presence for negative samples\. All context types are inserted using the same template and position in the input\.

#### Results

Figure[6](https://arxiv.org/html/2608.19208#S4.F6)reports the fitted affine parameters under the three context types\. The slope changes monotonically with the evidential role of the context:

asup\>airr\>aconf,a\_\{\\mathrm\{sup\}\}\>a\_\{\\mathrm\{irr\}\}\>a\_\{\\mathrm\{conf\}\},\(9\)This ordering is consistent with the interpretation ofaaas commitment preservation\. Supporting context preserves the original margin structure the most, irrelevant context introduces stronger attenuation, and conflicting context weakens the original structure most substantially\.

However, although supporting context is related, its slope still remains below one\. We interpret this behavior as evidence that MLLMs integrate supporting context as imperfect evidence\. See Appendix[C\.3](https://arxiv.org/html/2608.19208#A3.SS3)for more discussion\.

## 5Post\-Hoc Calibration Test

In this section, we use post\-hoc calibration as a diagnostic test\.

### 5\.1Calibration Protocol

Given a held\-out calibration split𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}, we compute the context\-free marginmom\_\{o\}and the context\-conditioned marginmcm\_\{c\}for each calibration sample, and fit the affine relation\. On the test split, we apply the inverse transformation:

mcal=mc−b^a^\.m\_\{\\mathrm\{cal\}\}=\\frac\{m\_\{c\}\-\\hat\{b\}\}\{\\hat\{a\}\}\.\(10\)

### 5\.2Experiment Setup and results

#### Calibration settings

We consider two calibration settings\. In the in\-domain setting, affine parameters are fitted on a calibration split and applied to the held\-out split of the same benchmark\.

In the cross\-dataset transfer setting, affine parameters are fitted on one source dataset and directly applied to other target datasets\. This setting uses no target\-dataset calibration samples and tests whether the affine parameters capture a transferable model\-level response to irrelevant context\.

#### Experiment Results

Table[6](https://arxiv.org/html/2608.19208#A2.T6)reports results of in\-domain calibration, and Table[7](https://arxiv.org/html/2608.19208#A2.T7)reports the cross\-dataset transfer result\. These results show that the context\-induced margin distortion is structured and partially transferable: inverse\-affine correction recovers context\-free decisions on held\-out data and under cross\-dataset transfer\. See Appendix[B\.3](https://arxiv.org/html/2608.19208#A2.SS3)for impelentation details and more experiment results\.

Table 3:In\-domain post\-hoc affine calibration under WikiText\-after\-possibly context\. Affine parameters are fitted on calibration splits and evaluated on held\-out test splits\. We report the Accuracy and Flip rate with percentage\.Table 4:Cross\-dataset affine calibration transfer result\. Affine parameters are fitted on POPE and directly applied to other benchmarks without target calibration samples\. We report the Accuracy and Flip rate with percentage\.

## 6Related Work

### 6\.1Context Effects in LLMs

LLMs are known to be sensitive to contextual information that is not part of the task\-relevant evidence\. Early work on adversarial reading comprehension shows that irrelevant distractor sentences can mislead question\-answering modelsJia and Liang \([2017](https://arxiv.org/html/2608.19208#bib.bib20)\), and recent studies further demonstrate that large language models can be distracted by irrelevant context in reasoning and question\-answering tasksShiet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib21)\); Amirazet al\.\([2025](https://arxiv.org/html/2608.19208#bib.bib18)\)\. Related observations also appear in retrieval\-augmented generation, where retrieved passages may contain noisy, weakly related, or irrelevant information that affects the final answerYoranet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib22)\); Yanet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib24)\); Cuconasuet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib19)\)\.

### 6\.2Context Effects in MLLMs

Textual bias has long been observed in vision\-language tasks\. Early VQA studies show that models can exploit language priors and insufficient visual groundingGoyalet al\.\([2017](https://arxiv.org/html/2608.19208#bib.bib29)\); Agrawalet al\.\([2018](https://arxiv.org/html/2608.19208#bib.bib40)\)\. Subsequent work further studies how to reduce such unimodal biases and force models to rely more on visual evidenceCadeneet al\.\([2019](https://arxiv.org/html/2608.19208#bib.bib41)\); KV and Mittal \([2020](https://arxiv.org/html/2608.19208#bib.bib42)\); Denget al\.\([2025](https://arxiv.org/html/2608.19208#bib.bib9)\)\. In multimodal RAG, prior work shows that irrelevant retrieved passages or conflicting knowledge may reduce overall performanceChenet al\.\([2022](https://arxiv.org/html/2608.19208#bib.bib43)\); Kortukovet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib44)\); Caffagniet al\.\([2024](https://arxiv.org/html/2608.19208#bib.bib36)\)\.

### 6\.3In\-Context Learning and Bias Calibration

In\-context learning \(ICL\) shows that language models can adapt their predictions from contextual examples without parameter updatesBrownet al\.\([2020](https://arxiv.org/html/2608.19208#bib.bib49)\); Liuet al\.\([2022](https://arxiv.org/html/2608.19208#bib.bib50)\)\. A theoretical explanation further views ICL as implicit Bayesian inference, where the model predicts by inferring a latent concept from the observed contextXieet al\.\([2021](https://arxiv.org/html/2608.19208#bib.bib32)\)\.

Despite its ability, prior work indicates that prompts in ICL may introduce systematic biases, which can be mitigated by estimating and calibrating such biases at inference timeZhaoet al\.\([2021](https://arxiv.org/html/2608.19208#bib.bib45)\); Hanet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib46)\); Nieet al\.\([2022](https://arxiv.org/html/2608.19208#bib.bib47)\); Zhouet al\.\([2023](https://arxiv.org/html/2608.19208#bib.bib48)\)\. More broadly, this connects to post\-hoc calibration methods such as Platt scaling and temperature scaling, which adjust model scores without updating model parametersPlatt \([1999](https://arxiv.org/html/2608.19208#bib.bib53)\); Guoet al\.\([2017](https://arxiv.org/html/2608.19208#bib.bib54)\)\.

## 7Conclusion

In this paper, we study how image\-irrelevant textual context affects visually grounded decisions in MLLMs\. By treating irrelevant context as a controlled intervention, we show that it can degrade predictive performance, shift answer preferences, and induce prediction flips across benchmarks and models\. To characterize this effect beyond aggregate metrics, we introduce a margin\-level analysis based on the log\-probability difference between positive and negative answers\. Our results show that context\-conditioned margins can be well approximated by an affine transformation of context\-free margins, indicating that irrelevant context induces a structured distortion of model preference rather than unstructured noise\. We further analyze the affine parameters as compact descriptors of margin preservation and directional answer shift, and develop a post\-hoc calibration method that partially compensates for context\-induced margin shifts without updating model parameters\. Our findings provide a diagnostic perspective on irrelevant\-context effects in MLLMs and may serve as a useful basis for future work on robust and trustworthy multimodal systems\.

## 8Limitations

#### Closed\-form evaluation\.

Our analysis is mainly conducted in binary visual\-question settings, and therefore does not fully capture the complexity of open\-ended multimodal generation\. We provide a preliminary study in Appendix[E\.3](https://arxiv.org/html/2608.19208#A5.SS3), where we observe similar context\-induced effects in open\-ended generation\. When irrelevant context is introduced, we observe that the model tends to generate shorter responses and focus more on salient objects in the image\. One plausible explanation is that the context\-induced affine transformation suppresses marginal visual claims with weaker confidence\. In this view, objects or attributes that originally lie near the decision boundary are more likely to be filtered out after the margin shift, while highly salient objects with stronger margins remain in the output\. This may explain why irrelevant context reduces response coverage and makes the generation more conservative\. However, systematically characterizing whether the same affine margin pattern extends to open\-ended outputs remains an important direction for future work\.

#### Limited control experiment settings\.

Both our experimental setting and calibration method are intentionally simplified\. The irrelevant contexts considered in this work are controlled interventions, rather than the heterogeneous auxiliary contexts that may arise in real applications, such as long dialogue histories, partially relevant retrieved passages, tool\-use traces, or multi\-turn reasoning contexts\. As a result, our experiments do not fully capture more complex multimodal interaction scenarios\. Moreover, our affine calibration is only a lightweight post\-hoc correction for binary decision margins\. It can partially compensate for context\-induced margin shifts, but it does not remove the irrelevant context from the input or fully eliminate its influence\.

#### Limited theoretical explanation\.

Our current analysis does not provide a complete theoretical explanation for the observed affine law, nor does it identify the causal mechanism underlying this phenomenon\. Instead, we characterize the effect mainly at the behavioral and margin levels, and use the Bayesian\-inspired view only as an interpretive lens\. In addition, the fitted affine relation can be affected by question format and context presentation, including prompt wording, context position, and input structure\. This suggests that the affine law observed in our controlled setting may be a particular manifestation of a broader context\-interference phenomenon\. Developing a deeper theoretical and causal account of this phenomenon remains an important direction for future work\.

## References

- A\. Agrawal, D\. Batra, D\. Parikh, and A\. Kembhavi \(2018\)Don’t just assume; look and answer: overcoming priors for visual question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 4971–4980\.Cited by:[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- C\. Amiraz, F\. Cuconasu, S\. Filice, and Z\. Karnin \(2025\)The distracting effect: understanding irrelevant passages in rag\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 18228–18258\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1),[§6\.1](https://arxiv.org/html/2608.19208#S6.SS1.p1.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p1.1)\.
- R\. Cadene, C\. Dancette, H\. Ben\-younes, M\. Cord, and D\. Parikh \(2019\)RUBi: reducing unimodal biases in visual question answering\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- D\. Caffagni, F\. Cocchi, N\. Moratelli, S\. Sarto, M\. Cornia, L\. Baraldi, and R\. Cucchiara \(2024\)Wiki\-llava: hierarchical retrieval\-augmented generation for multimodal llms\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 1818–1826\.Cited by:[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- H\. Chen, M\. J\. Q\. Zhang, and E\. Choi \(2022\)Rich knowledge sources bring complex knowledge conflicts: recalibrating models to reflect conflicting evidence\.arXiv preprint arXiv:2210\.13701\.Cited by:[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- X\. Chen, H\. Fang, T\. Lin, R\. Vedantam, S\. Gupta, P\. Dollár, and C\. L\. Zitnick \(2015\)Microsoft COCO captions: data collection and evaluation server\.arXiv preprint arXiv:1504\.00325\.External Links:[Link](https://arxiv.org/abs/1504.00325)Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px4.p1.1)\.
- Z\. Chen, C\. Xu, Y\. Qi, and J\. Guo \(2024\)MLLM is a strong reranker: advancing multimodal retrieval\-augmented generation via knowledge\-enhanced reranking and noise\-injected training\.arXiv preprint arXiv:2407\.21439\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1)\.
- F\. Cuconasu, G\. Trappolini, F\. Siciliano, S\. Filice, C\. Campagnano, Y\. Maarek, N\. Tonellotto, and F\. Silvestri \(2024\)The power of noise: redefining retrieval for rag systems\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 719–729\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1),[§6\.1](https://arxiv.org/html/2608.19208#S6.SS1.p1.1)\.
- A\. Deng, T\. Cao, Z\. Chen, and B\. Hooi \(2025\)Words or vision: do vision\-language models have blind faith in text?\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1),[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun, Y\. Wu, and R\. Ji \(2023\)MME: a comprehensive evaluation benchmark for multimodal large language models\.arXiv preprint arXiv:2306\.13394\.External Links:[Link](https://arxiv.org/abs/2306.13394)Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px3.p1.1)\.
- Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. Parikh \(2017\)Making the V in VQA matter: elevating the role of image understanding in visual question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 6904–6913\.External Links:[Link](https://openaccess.thecvf.com/content_cvpr_2017/html/Goyal_Making_the_V_CVPR_2017_paper.html)Cited by:[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p2.1)\.
- Z\. Han, Y\. Hao, L\. Dong, Y\. Sun, and F\. Wei \(2023\)Prototypical calibration for few\-shot learning of language models\.InInternational Conference on Learning Representations,Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p2.1)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 6700–6709\.External Links:[Link](https://openaccess.thecvf.com/content_CVPR_2019/html/Hudson_GQA_A_New_Dataset_for_Real-World_Visual_Reasoning_and_Compositional_CVPR_2019_paper.html)Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px3.p1.1)\.
- R\. Jia and P\. Liang \(2017\)Adversarial examples for evaluating reading comprehension systems\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,Copenhagen, Denmark,pp\. 2021–2031\.External Links:[Document](https://dx.doi.org/10.18653/v1/D17-1215),[Link](https://aclanthology.org/D17-1215/)Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1),[§6\.1](https://arxiv.org/html/2608.19208#S6.SS1.p1.1)\.
- A\. T\. Kalai, O\. Nachum, S\. S\. Vempala, and E\. Zhang \(2025\)Why language models hallucinate\.arXiv preprint arXiv:2509\.04664\.Cited by:[§E\.2](https://arxiv.org/html/2608.19208#A5.SS2.p1.1)\.
- E\. Kortukov, A\. Rubinstein, E\. Nguyen, and S\. J\. Oh \(2024\)Studying large language model behaviors under context\-memory conflicts with real documents\.arXiv preprint arXiv:2404\.16032\.Cited by:[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- G\. KV and A\. Mittal \(2020\)Reducing language biases in visual question answering with visually\-grounded question encoder\.arXiv preprint arXiv:2007\.06198\.Cited by:[§6\.2](https://arxiv.org/html/2608.19208#S6.SS2.p1.1)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen \(2023\)Evaluating object hallucination in large vision\-language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px3.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px3.p2.1)\.
- J\. Liu, D\. Shen, Y\. Zhang, B\. Dolan, L\. Carin, and W\. Chen \(2022\)What makes good in\-context examples for GPT\-3?\.InProceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures,pp\. 100–114\.Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1609.07843)Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px1.p1.2)\.
- M\. Mortaheb, M\. A\. A\. Khojastepour, S\. T\. Chakradhar, and S\. Ulukus \(2025\)Re\-ranking the context for multimodal retrieval augmented generation\.arXiv preprint arXiv:2501\.04695\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p1.1)\.
- F\. Nie, M\. Chen, Z\. Zhang, and X\. Cheng \(2022\)Improving few\-shot performance of language models via nearest neighbor calibration\.arXiv preprint arXiv:2212\.02216\.Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p2.1)\.
- D\. Park, Z\. Qian, G\. Han, and S\. Lim \(2024\)Mitigating dialogue hallucination for large vision language models via adversarial instruction tuning\.arXiv preprint arXiv:2403\.10492\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p1.1)\.
- J\. C\. Platt \(1999\)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.InAdvances in Large Margin Classifiers,A\. J\. Smola, P\. Bartlett, B\. Schölkopf, and D\. Schuurmans \(Eds\.\),pp\. 61–74\.Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p2.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§3\.1](https://arxiv.org/html/2608.19208#S3.SS1.p2.1)\.
- F\. Shi, X\. Chen, K\. Misra, N\. Scales, D\. Dohan, E\. H\. Chi, N\. Schärli, and D\. Zhou \(2023\)Large language models can be easily distracted by irrelevant context\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 31210–31227\.External Links:[Link](https://proceedings.mlr.press/v202/shi23a.html)Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1),[§6\.1](https://arxiv.org/html/2608.19208#S6.SS1.p1.1)\.
- J\. Wang, Y\. Wang, G\. Xu, J\. Zhang, Y\. Gu, H\. Jia, J\. Wang, H\. Xu, M\. Yan, J\. Zhang, and J\. Sang \(2023\)AMBER: an LLM\-free multi\-dimensional benchmark for MLLMs hallucination evaluation\.arXiv preprint arXiv:2311\.07397\.External Links:[Link](https://arxiv.org/abs/2311.07397)Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px3.p1.1)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024a\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px3.p2.1)\.
- X\. Wang, J\. Pan, L\. Ding, and C\. Biemann \(2024b\)Mitigating hallucinations in large vision\-language models with instruction contrastive decoding\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 15840–15853\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p1.1)\.
- S\. M\. Xie, A\. Raghunathan, P\. Liang, and T\. Ma \(2021\)An explanation of in\-context learning as implicit bayesian inference\.arXiv preprint arXiv:2111\.02080\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p5.2),[§4\.1](https://arxiv.org/html/2608.19208#S4.SS1.p2.4),[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p1.1)\.
- S\. Yan, J\. Gu, Y\. Zhu, and Z\. Ling \(2024\)Corrective retrieval augmented generation\.arXiv preprint arXiv:2401\.15884\.External Links:[Link](https://arxiv.org/abs/2401.15884)Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1),[§6\.1](https://arxiv.org/html/2608.19208#S6.SS1.p1.1)\.
- O\. Yoran, T\. Wolfson, O\. Ram, and J\. Berant \(2024\)Making retrieval\-augmented language models robust to irrelevant context\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZS4m74kZpH)Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1),[§6\.1](https://arxiv.org/html/2608.19208#S6.SS1.p1.1)\.
- T\. Zhang, Z\. Zhang, Z\. Ma, Y\. Chen, Z\. Qi, C\. Yuan, B\. Li, J\. Pu, Y\. Zhao, Z\. Xie, J\. Ma, Y\. Shan, and W\. Hu \(2024\)MR2ag: multimodal retrieval\-reflection\-augmented generation for knowledge\-based vqa\.arXiv preprint arXiv:2411\.15041\.Cited by:[§1](https://arxiv.org/html/2608.19208#S1.p2.1)\.
- T\. Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh \(2021\)Calibrate before use: improving few\-shot performance of language models\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 12697–12706\.Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p2.1)\.
- H\. Zhou, X\. Wan, L\. Proleev, D\. Mincu, J\. Chen, K\. Heller, and S\. Roy \(2023\)Batch calibration: rethinking calibration for in\-context learning and prompt engineering\.arXiv preprint arXiv:2309\.17249\.Cited by:[§6\.3](https://arxiv.org/html/2608.19208#S6.SS3.p2.1)\.
- J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao,et al\.\(2025\)Internvl3: exploring advanced training and test\-time recipes for open\-source multimodal models\.arXiv preprint arXiv:2504\.10479\.Cited by:[§2\.2](https://arxiv.org/html/2608.19208#S2.SS2.SSS0.Px3.p2.1)\.

## Appendix AExperimental Setup and Implementation Details

This appendix provides implementation details and supplementary analyses that support the main text\. We organize the material into experimental protocols, robustness checks, full results, and formulation discussions\.

### A\.1Decision\-Margin Computation Details

All margin analyses in this paper use the same scoring convention\. Given a rendered multimodal inputx=\(I,T\)x=\(I,T\), whereIIis the image andTTis the complete text prompt, we define the binary decision margin as

m​\(x\)=log⁡Pθ​\(v\+∣I,T\)−log⁡Pθ​\(v−∣I,T\),m\(x\)=\\log P\_\{\\theta\}\(v\_\{\+\}\\mid I,T\)\-\\log P\_\{\\theta\}\(v\_\{\-\}\\mid I,T\),wherev\+v\_\{\+\}andv−v\_\{\-\}are the affirmative and negative verbalizers\. In the main experiments,v\+=Yesv\_\{\+\}=\\texttt\{Yes\}andv−=Nov\_\{\-\}=\\texttt\{No\}\.

We score both candidate continuations under teacher forcing\. For a candidate verbalizerv=\(v1,…,vK\)v=\(v\_\{1\},\\ldots,v\_\{K\}\)tokenized intoKKtokens, its sequence log\-probability is computed as

log⁡Pθ​\(v∣I,T\)=∑k=1Klog⁡Pθ​\(vk∣I,T,v<k\)\.\\log P\_\{\\theta\}\(v\\mid I,T\)=\\sum\_\{k=1\}^\{K\}\\log P\_\{\\theta\}\(v\_\{k\}\\mid I,T,v\_\{<k\}\)\.We then subtract the sequence log\-probabilities of the two verbalizers\. This sequence\-level scoring is used even when a verbalizer consists of more than one token, so the reported margin is not restricted to a single next\-token logit difference\.

The final binary prediction induced by the margin is determined by its sign:

y^​\(x\)=\{Yes,m​\(x\)\>0,No,m​\(x\)≤0\.\\hat\{y\}\(x\)=\\begin\{cases\}\\texttt\{Yes\},&m\(x\)\>0,\\\\ \\texttt\{No\},&m\(x\)\\leq 0\.\\end\{cases\}

### A\.2Irrelevant Text Sampling

We construct image\-irrelevant textual contexts from an offline WikiText sentence pool\. The contexts are not dynamically generated during inference; instead, each sample is assigned a fixed context before evaluation\. This makes the context assignment deterministic and reproducible under the same random seed\.

We first split corpus into sentence\-level candidates and remove duplicate sentences\. The candidate pool is then shuffled with a fixed seed, and contexts are assigned to samples by iterating through this shuffled pool\.

The filtering process consists of two steps\. First, each candidate context is processed by a rule\-based script to remove judgment\-related words, such as “yes” and “no”\. Second, we compute the CLIP similarity between the candidate context and the input image, and discard candidates with high similarity scores to avoid selecting visually related sentences\.

For CLIP filter, we calculate the similarity score between context and image:

sim​\(i,t\)=⟨CLIP​\(i\),CLIP​\(t\)⟩,\\mathrm\{sim\}\(i,t\)=\\langle\\mathrm\{CLIP\}\(i\),\\mathrm\{CLIP\}\(t\)\\rangle,\(11\)whereiiis the input image andttis a candidate context\.

![Refer to caption](https://arxiv.org/html/2608.19208v1/x7.png)Figure 7:The affine transformation results across all models and benchmarks\.
### A\.3Benchmark Statistics and Implementation Details

In the main experiments, we use four benchmarks\. POPE, MME, and AMBER\-discriminative are already binary judgment tasks, so we preserve their original question format and only add a unified answer instruction\. POPE contains 8,910 questions, including 4,500 Yes and 4,410 No instances \(50\.5%/49\.5%\); its three evaluation categories contain 3,000 adversarial, 3,000 popular, and 2,910 random questions\. The AMBER discriminative subset contains 14,216 questions, with 4,789 Yes and 9,427 No instances \(33\.7%/66\.3%\), and is therefore the most label\-imbalanced benchmark in our main evaluation\. The MME input contains 2,374 questions, evenly split into 1,187 Yes and 1,187 No instances across its perception and cognition subtasks\.

For GQA, we further divide the benchmark into two subsets: GQA\-Binary and GQA\-Judgment\. GQA\-Binary contains the native binary questions in GQA, for which we preserve the original question format and only append the unified answer instruction\. It contains 4,525 naturally yes/no questions, with 2,328 Yes and 2,197 No instances \(51\.4%/48\.6%\)\. GQA\-Judgment is constructed from the remaining open\-ended questions by converting each sample into an answer\-verification task: given the image, the original question, and a candidate answer, the model is asked to judge whether the candidate answer is supported by the image\. See Appendix[E\.2](https://arxiv.org/html/2608.19208#A5.SS2)for detailed conversion process and corresponding experiment results\.

Our experiments show that this conversion changes the task format from the model’s perspective\. Although GQA\-Judgment still exhibits the affine margin pattern observed in our main analysis, its fitted affine parameters differ from those of native binary tasks\. Therefore, we report GQA\-Binary in the main text for consistency with the primary binary\-judgment setting, and provide GQA\-Judgment results as a complementary evaluation\.

Table 5:Absolute behavioral metrics across tested models and benchmarks\. Free denotes context\-free input, Neutral denotes WikiText\-after\-neutral, and Condition denotes WikiText\-after\-possibly\. Part \(b\) reports the original\-style MME score\.

## Appendix BSupplement Experient Results

### B\.1Absolute Behavioral Metrics

Table[5](https://arxiv.org/html/2608.19208#A1.T5)reports the absolute behavioral metrics for all tested model families\. Neutral uses WikiText\-after\-neutral, while Condition uses WikiText\-after\-possibly\. All binary metrics are percentages\.

### B\.2Full Affine Transformation Results

Figure[7](https://arxiv.org/html/2608.19208#A1.F7)shows the affine transformation results across all models and benchmarks\.

### B\.3Calibration Experiments

#### In\-domain calibration\.

First, we test whether affine calibration can recover context\-free behavior when the affine parameters are estimated and evaluated on held\-out splits from the same benchmark\. Specifically, for each model and each dataset, we split the data into a calibration split and a test split\. The parametersa^\\hat\{a\}andb^\\hat\{b\}are fitted on the calibration split of the same dataset, and then applied to the corresponding test split\.

#### Cross\-dataset transfer calibration\.

Second, we evaluate a stronger transfer setting\. Instead of fitting affine parameters on the target dataset, we fita^\\hat\{a\}andb^\\hat\{b\}on one source dataset and directly apply them to other target datasets:

mcals→t=mct−b^sa^s,m\_\{\\mathrm\{cal\}\}^\{s\\rightarrow t\}=\\frac\{m\_\{c\}^\{t\}\-\\hat\{b\}\_\{s\}\}\{\\hat\{a\}\_\{s\}\},\(12\)wherea^s\\hat\{a\}\_\{s\}andb^s\\hat\{b\}\_\{s\}are estimated from the source datasetss, andmctm\_\{c\}^\{t\}denotes the context\-conditioned margin on the target datasettt\. This setting uses no target\-dataset calibration samples\.

Table[6](https://arxiv.org/html/2608.19208#A2.T6)reports the full in\-domain calibration on LLaVA\-1\.5\-7B\. Inverse\-affine calibration consistently recovers a substantial part of the context\-induced degradation: it raises accuracy relative to Context by 3\.2–7\.9 points and reduces flips by 10\.5–21\.5 points across all four benchmarks\. This shows that the fitted affine distortion remains predictive on held\-out samples\.

Table 6:In\-domain post\-hoc affine calibration under WikiText\-after\-possibly context\. Affine parameters are fitted on calibration splits and evaluated on held\-out test splits\. Values are percentages averaged over five splits\. Flip is relative to Origin predictions\.Table[7](https://arxiv.org/html/2608.19208#A2.T7)reports cross\-dataset transfer for LLaVA\-1\.5\-7B\. In every source–target pair, transferred parameters improve accuracy relative to the uncalibrated Context condition and sharply reduce flips\. In particular, POPE\-fitted parameters improve target accuracy by 2\.4 points on GQA\-Binary and by over 5 points on AMBER and MME, without using any target calibration examples\. These results support the interpretation that, for LLaVA, irrelevant context induces a substantial model\-level distortion rather than an unrelated benchmark\-specific artifact\.

Table 7:Cross\-dataset affine calibration transfer on LLaVA\-1\.5\-7B under WikiText\-after\-possibly context\. Source\-fitted parameters are applied directly to the target benchmark without target calibration samples\. Values are percentages\.Overall, these results show that for LLaVA, the context\-induced margin distortion is structured and partially transferable: inverse\-affine correction recovers context\-free decisions on held\-out data and under cross\-dataset transfer, while also consistently improving accuracy and F1\. As detailed in Appendix, other model families can exhibit a weaker relation between recovery of clean decisions and improvement of ground\-truth accuracy\.

Table 8:Prompt, template, length, and source ablations on POPE with LLaVA\-1\.5\-7B\. Acc, Recall, YesRate, and Flip are reported in percentages\.

## Appendix CAdditional Empirical Analysis

### C\.1Prompt, Template, and Context\-Source Ablations

We further test whether the affine effect depends on a particular prompt rendering choice\. All experiments in this subsection use POPE with LLaVA\-1\.5\-7B and reuse the same context\-free origin outputs\. We vary four factors: the position of the context relative to the question, the wording of the context template, the context length, and the context source\. We also include template\-only controls, where the template phrase is inserted without any external context text\. Table[8](https://arxiv.org/html/2608.19208#A2.T8)shows that the affine structure is robust across prompt variants, but the fitted parameters are sensitive to how the context is introduced\.

Putting WikiText before the question has a much weaker effect than putting it after the question\. For the possibly\-related template, the before\-question variant hasa=0\.726a=0\.726and only3\.7%3\.7\\%flips, whereas the after\-question variant hasa=0\.554a=0\.554and16\.5%16\.5\\%flips\. However, in many practical in\-context learning pipelines, especially settings where auxiliary textual information is appended as side information or retrieved evidence before answer generation, the added context is appended after the question\. Our after\-question setting is designed to reflect this practically relevant regime rather than to claim position\-invariant behavior\. This position ablation shows that the affine structure remains observable across position variants, therefore does not affect the conclusions in the main paper\.

The template\-only controls address the concern that the comparison between context\-free and context\-conditioned prompts may be confounded by the added template itself\. Even without external context text, the after\-question template changes the margin distribution\. However, the neutral template\-only setting is substantially weaker than the possibly\-related template\-only setting: its slope is closer to identity \(a=0\.819a=0\.819\) and its flip rate is lower \(6\.7%6\.7\\%vs\.13\.5%13\.5\\%\)\. Thus, template wording is not irrelevant, but it does not eliminate the need to model the full context\-conditioned transformation\.

Length and source controls provide additional checks\. Short, medium, and long WikiText contexts all preserve strong affine fits and similar behavioral metrics when placed before the question\. Shuffling WikiText also preserves a strong affine relation, suggesting that the effect does not require coherent evidence about the queried object\. In contrast, COCO captions placed after the question produce the strongest degradation among these controls, witha=0\.475a=0\.475and a flip rate of20\.7%20\.7\\%\. Overall, these ablations support the main interpretation: irrelevant text systematically transforms the decision margin, and the strength of the transformation depends on the semantic role and placement of the added text\.

### C\.2Label Verbalization

Our primary analyses define a binary decision margin using the verbalizersYesandNo\. This raises a potential confound: the observed affine relation might arise from the lexical properties of this particular answer pair, rather than from a structured transformation of the model’s visual decision\. We therefore evaluate alternative answer verbalizations with LLaVA\-1\.5\-7B\. The image, question, label, and fixed WikiText context assignment are kept unchanged; only the response instruction and the two verbalizer sequences used for margin computation are replaced\. For an affirmative/negative pair\(v\+,v−\)\(v\_\{\+\},v\_\{\-\}\), the semantic binary margin is

mv=log⁡P​\(v\+∣I,Q,C\)−log⁡P​\(v−∣I,Q,C\)\.m\_\{v\}=\\log P\(v\_\{\+\}\\mid I,Q,C\)\-\\log P\(v\_\{\-\}\\mid I,Q,C\)\.We evaluate the context\-free input \(Origin\), WikiText introduced through a neutral template \(Neutral\), and WikiText introduced through the possibly\-related template \(Possibly\)\.

Table[9](https://arxiv.org/html/2608.19208#A3.T9)first reports the POPE results\. In addition to the standardYes/Noformat, we useTrue/Falseas a natural semantic alternative\.

Table 9:Label\-verbalization controls on POPE with LLaVA\-1\.5\-7B\. For each verbalizer pair, margins score the semantically affirmative token against the negative token\. Behavioral metrics and Flip are percentages; affine quantities are fitted between Origin and each context condition\.To verify that the result is not restricted to POPE, we further evaluate the naturalTrue/Falsealternative on the AMBER\. As shown in Table[10](https://arxiv.org/html/2608.19208#A3.T10), affine compression remains visible on this additional benchmark under both prompt templates\.

Table 10:Natural verbalizer control on AMBER discriminative with LLaVA\-1\.5\-7B usingTrue/False\. Behavioral metrics and Flip are reported in percentages\.The natural\-verbalizer results support the main conclusion\. WithTrue/False, the estimated slope is below one in every condition:a=0\.600/0\.510a=0\.600/0\.510on POPE anda=0\.578/0\.473a=0\.578/0\.473on AMBER for the Neutral/Possibly settings\. Thus, the margin\-compression pattern is neither specific to the literalYes/Notokens nor confined to one benchmark\. However, the intercept is not verbalizer invariant:Yes/Noyields negative shifts on POPE, whileTrue/Falseyields positive shifts and substantially higher positive\-answer rates, particularly under Possibly\. This result suggests that the preference shift induced by irrelevant context is governed by a structured affine transformation of decision margins, which cannot be fully captured by output performance metrics alone\.

### C\.3Why Supporting Context Still Compresses Margins

The context\-role intervention in Sec\. 4 introduces a seemingly counter\-intuitive result\. Since supporting context is constructed to be consistent with the ground\-truth answer, one might expect it to strengthen the clean visual decision and therefore yield a slope larger than one\. This is not what we observe\. Table[11](https://arxiv.org/html/2608.19208#A3.T11)reports the controlled intervention on POPE with LLaVA\-1\.5\-7B, where the image, question, context position, and rendering template are fixed while only the semantic role of the inserted context is changed\.

Table 11:Context\-role intervention on POPE with LLaVA\-1\.5\-7B\.Δ​m\+\\Delta m\_\{\+\}andΔ​m−\\Delta m\_\{\-\}denote the mean shiftmc−m0m\_\{c\}\-m\_\{0\}for ground\-truthYesandNosamples, respectively\.Supporting context is substantially less destructive than the alternatives: it has the largest slope, the smallest negative intercept, and a much smaller negative shift for ground\-truthYessamples than irrelevant or conflicting context\. Nevertheless, its slope remains below one \(a=0\.626a=0\.626\)\. This does not contradict the usefulness of supporting evidence, because the fitted slope measures the global dynamic range of the context\-conditioned margins relative to clean margins; it does not directly measure whether a context is label\-consistent for an individual sample\.

In particular, the supporting intervention is label\-conditioned: positive questions receive a statement that the queried object is present, whereas negative questions receive a statement that it is absent\. Its effect is therefore more appropriately summarized as

mc=α​m0\+δ​\(y\)\+ϵ,m\_\{c\}=\\alpha m\_\{0\}\+\\delta\(y\)\+\\epsilon,whereδ​\(y\)\\delta\(y\)is a label\-dependent shift, rather than as a pure margin amplifiermc=γ​m0m\_\{c\}=\\gamma m\_\{0\}withγ\>1\\gamma\>1\. On POPE, the clean mean margins for ground\-truthYesandNosamples are2\.102\.10and−2\.76\-2\.76, giving a mean class separation of4\.864\.86\. Under supporting context, the corresponding means are1\.161\.16and−2\.15\-2\.15, reducing the separation to3\.313\.31\. Supporting text preserves the decision direction better than other context roles, while still narrowing the overall margin range; an ordinary least\-squares fit thus naturally yieldsa<1a<1\.

This compression is also consistent with how the context is presented to the model\. The supporting statement is introduced as caption\-like natural language under a possibly\-related template, rather than as an oracle answer\. Consequently, the model may treat it as uncertain evidence, reduce reliance on the original visual signal, or imperfectly process the negated form used for negative samples\. The latter possibility is reflected byΔ​m−=0\.61\\Delta m\_\{\-\}=0\.61: because positive margins favorYes, a perfectly effective supporting statement for aNoexample would shift the margin in the negative direction, whereas the observed average shift is positive\.

We therefore interpret supporting context as reducing harmful context\-induced distortion rather than amplifying clean visual margins beyond their original scale\. The ordering

asupporting\>airrelevant\>aconflictinga\_\{\\mathrm\{supporting\}\}\>a\_\{\\mathrm\{irrelevant\}\}\>a\_\{\\mathrm\{conflicting\}\}still shows that the semantic role of text systematically changes the affine operator\. At the same time,asupporting<1a\_\{\\mathrm\{supporting\}\}<1indicates that even label\-consistent context is integrated as imperfect prompt\-conditioned evidence, not as deterministic supervision\.

## Appendix DAffine Robustness Analysis

Table 12:Context\-only prior subtraction on POPE with LLaVA\-1\.5\-7B\. We report the affine fit betweenm0m\_\{0\}and the prior\-subtracted marginmsub=mc−mpriorm\_\{\\mathrm\{sub\}\}=m\_\{c\}\-m\_\{\\mathrm\{prior\}\}\. Acc, F1, YesRate, and Flip are reported in percentages\.### D\.1Context\-Only Prior Subtraction

Another alternative explanation is that irrelevant context only contributes an independent answer prior\. Under this view, the context\-conditioned margin could be decomposed as the clean margin plus a context\-only bias term\. To test this, we estimate a context prior marginmpriorm\_\{\\mathrm\{prior\}\}by removing the original image\-question evidence and then subtract it from the context\-conditioned margin:

msub=mc−mprior\.m\_\{\\mathrm\{sub\}\}=m\_\{c\}\-m\_\{\\mathrm\{prior\}\}\.We consider three prior estimators: context\-only with a blank image, question\-plus\-context with a blank image, and context\-only with the original image\.

Table[12](https://arxiv.org/html/2608.19208#A4.T12)reports the POPE results with LLaVA\-1\.5\-7B\. The context\-only prior itself is almost uncorrelated with the clean margin: across variants, its fitted slope is close to zero andR2R^\{2\}is nearly zero\. After subtracting this prior, the affine relation remains strong, and the slope remains substantially below one\. This is the key observation: subtracting an additive answer prior improves behavior, but it does not remove the margin compression effect\.

Table 13:Nested margin\-model comparison on POPE with LLaVA\-1\.5\-7B\. The context variant is WikiText\-after\-neutral\. Fit metrics evaluatemcm\_\{c\}prediction on held\-out samples\. Calibration metrics evaluate the corrected margin as a binary prediction\.Table 14:Local margin analysis under WikiText\-after\-possibly\. For each model and dataset, we report the global affine fit and the flip rate within low\-, mid\-, and high\-margin regions\. Flip rates are percentages\.For the possibly\-related context, the uncorrected context margin hasa=0\.554a=0\.554,b=−1\.046b=\-1\.046, andR2=0\.896R^\{2\}=0\.896\. Subtracting the context\-only prior changes the intercept and improves behavioral metrics, but the slope remains around0\.550\.55\. Similarly, under the neutral context, the prior\-subtracted slope remains around0\.700\.70\. Therefore, the irrelevant\-context effect cannot be fully explained as a standalone Yes/No prior added to the decision margin\. Instead, it also changes the effective scaling of the original visual evidence, consistent with the affine interpretation in the main text\.

### D\.2Nested Margin Models

One possible concern of the affine law is that irrelevant context only induces a simpler transformation, such as a global Yes/No answer\-prior shift or a uniform confidence scaling\. To verify this concern, we compare four nested margin models:

mc\\displaystyle m\_\{c\}=m0,mc=m0\+b,\\displaystyle=m\_\{0\},\\qquad m\_\{c\}=m\_\{0\}\+b,mc\\displaystyle m\_\{c\}=a​m0,mc=a​m0\+b\.\\displaystyle=am\_\{0\},\\qquad m\_\{c\}=am\_\{0\}\+b\.These correspond to identity, shift\-only, scale\-only, and full affine transformations, respectively\. We fit each model on a calibration split and evaluate it on held\-out samples\. In addition to held\-outR2R^\{2\}and MAE, we also evaluate whether each model predicts which samples will flip after adding context\.

Table[13](https://arxiv.org/html/2608.19208#A4.T13)reports the available result on POPE with LLaVA\-1\.5\-7B under the WikiText\-after\-neutral setting\. The full affine model provides the best held\-out fit, reducing MAE from1\.341\.34for identity and0\.930\.93for shift\-only to0\.520\.52, and increasing held\-outR2R^\{2\}to0\.920\.92\. It also gives the strongest flip prediction performance, with F1 increasing from0\.710\.71for shift\-only to0\.850\.85\. This supports the conclusion that the context effect is not merely a constant answer\-direction shift, nor merely a confidence scaling effect\.

For reference, the clean prediction accuracy on the same held\-out split is86\.4%86\.4\\%with F185\.9%85\.9\\%\. Adding context without correction reduces accuracy to80\.0%80\.0\\%and F1 to75\.6%75\.6\\%, with a flip rate of14\.5%14\.5\\%\. The full affine correction reduces the flip rate to5\.0%5\.0\\%and recovers most of the lost F1\. The fact that the full affine model substantially improves held\-out margin prediction over both shift\-only and scale\-only baselines is the key diagnostic: irrelevant context changes both the scale of the original margin and its answer\-direction offset\.

### D\.3Local Fits Across Margin Regions

The global affine relation could be inflated by the dynamic range of high\-confidence samples\. To check this, we fit the affine relation separately within low\-, mid\-, and high\-margin regions, where the regions are defined by quantiles of\|m0\|\|m\_\{0\}\|for each model–dataset pair\. We also repeat the global fit after trimming extreme margins\. This analysis serves two purposes\. First, it tests whether affine regularity remains visible after removing extreme margins\. Second, it quantifies where prediction flips occur\. Table[14](https://arxiv.org/html/2608.19208#A4.T14)shows two consistent patterns\. First, the affine fit remains strong after trimming the extreme55–10%10\\%of margins\. For example, LLaVA on POPE has globalR2=0\.896R^\{2\}=0\.896and trimmedR2=0\.899R^\{2\}=0\.899, while Qwen2\-2B on AMBER has globalR2=0\.863R^\{2\}=0\.863and trimmedR2=0\.848R^\{2\}=0\.848\. Thus, the affine law is not solely an artifact of a few very large margins\. Second, flips are concentrated near the decision boundary\. Across both model families and all four benchmarks, low\-margin samples have much higher flip rates than high\-margin samples\. For LLaVA, the low\-margin flip rate ranges from42\.2%42\.2\\%on POPE to56\.3%56\.3\\%on GQA\-Binary, while the corresponding high\-margin flip rates range only from0\.1%0\.1\\%to9\.6%9\.6\\%\. Qwen2\-2B shows the same pattern\. This supports the analysis in the main text: irrelevant context changes the margin distribution globally, but the final prediction changes primarily when the original decision is close to the boundary\. It also motivates gated calibration, where correction is focused on near\-boundary examples instead of being applied uniformly to all samples\.

### D\.4Random Seed Robustness

The main experiment assigns one fixed irrelevant WikiText context to each POPE example\. To test whether the affine relation is an artifact of a particular random assignment, we independently rebuild the WikiText assignment with seeds 13 and 21, while keeping the model, samples, context position, and possibly\-related rendering template fixed\. We reuse the context\-free margins and rerun only the context\-conditioned inference\.

Table 15:Random\-context\-seed robustness on POPE with LLaVA\-1\.5\-7B under the WikiText\-after\-possibly condition\. Behavioral metrics and Flip are reported in percentages\.The estimated parameters are nearly invariant across the two additional context assignments\. Moreover, they closely match the main seed\-42 fit \(a=0\.5538a=0\.5538,b=−1\.0465b=\-1\.0465,R2=0\.8964R^\{2\}=0\.8964, andr=0\.9468r=0\.9468\)\. Thus, the reported affine transformation is not attributable to an idiosyncratic WikiText sampling seed\.

![Refer to caption](https://arxiv.org/html/2608.19208v1/x8.png)Figure 8:Residual diagnostics for the affine fit under the WikiText\-after\-possibly context\. Columns denote benchmarks and rows denote models\. Each point is a sample residualmc−\(a​m0\+b\)m\_\{c\}\-\(am\_\{0\}\+b\); the red curve is the binned mean residual\. All panels use shared symmetric axes so that residual dispersion is directly comparable\.
### D\.5Residual Analysis

We further examine whether the fitted affine transformation leaves systematic structure in the prediction error\. For each model–benchmark pair under the WikiText\-after\-possibly condition, we compute the residual

ϵi=mc\(i\)−\(a​m0\(i\)\+b\),\\epsilon\_\{i\}=m\_\{c\}^\{\(i\)\}\-\\left\(am\_\{0\}^\{\(i\)\}\+b\\right\),\(13\)whereaaandbbare fitted on all samples of the corresponding setting\. Figure[8](https://arxiv.org/html/2608.19208#A4.F8)plots these residuals against the context\-free margin for four model families and four binary benchmarks\. The red curve shows the mean residual within bins ofm0m\_\{0\}and therefore exposes systematic departures from a single global affine map\.

For LLaVA\-1\.5\-7B, the residual cloud remains comparatively compact across all benchmarks, withR2R^\{2\}ranging from0\.740\.74on GQA\-Binary to0\.900\.90on POPE\. InternVL3\-8B also retains high explanatory power \(R2=0\.87R^\{2\}=0\.87–0\.960\.96\), although its larger margin scale produces a wider absolute residual spread\. Qwen2\-VL\-2B is broadly compatible with the affine approximation, with reduced fit quality on MME and GQA\-Binary\. In contrast, Qwen2\-VL\-7B exhibits visibly larger and margin\-dependent residual dispersion on AMBER, MME, and GQA\-Binary, whereR2R^\{2\}drops to0\.630\.63,0\.420\.42, and0\.510\.51, respectively\. Thus, the residual analysis supports the affine law as a strong first\-order description across models, while also identifying settings in which a single global affine operator is incomplete\. This qualification is consistent with the Qwen2\-VL calibration results above: calibration can reduce context\-induced decision changes even when the remaining residual structure prevents uniform accuracy recovery\.

## Appendix EAdditional Discussion and Experiments

### E\.1Why Binary Judgment?

We adopt a binary judgment formulation to obtain a controlled and comparable measure of model preference\. Open\-ended multimodal generation involves many confounding factors, including answer phrasing, verbosity, decoding strategy, and automatic evaluation noise\. These factors make it difficult to determine whether a change in the output reflects a genuine change in visual judgment or merely a change in surface generation\. By constraining the answer space to\{Yes,No\}\\\{\\texttt\{Yes\},\\texttt\{No\}\\\}, we isolate the model’s binary decision\-making from the confounding effects of open\-ended generation and uncontrolled answer formats\.

This formulation also enables margin\-level analysis\. For each input, the model assigns probabilities to the same two candidate answers, allowing us to define a decision margin as the log\-probability difference betweenYesandNo\. Because the candidate set is fixed across the context\-free and context\-conditioned inputs, the margin provides a direct score\-level probe of how the same image\-question pair is affected by the added context\. This is essential for identifying the affine relation between the original and context\-conditioned margins\.

The binary setting is not intended to replace open\-ended evaluation\. Rather, it serves as a controlled abstraction for studying whether image\-irrelevant context changes visually grounded decisions\. Several benchmarks, such as POPE and MME, are already naturally formulated as yes/no questions; for open\-ended benchmarks, we convert them into answer\-verification questions so that the same analysis can be applied consistently\. This design allows us to compare predictions, flip rates, and margin shifts under a unified evaluation protocol\.

### E\.2GQA\-judgment Conversion and Results

For non\-binary GQA questions, FollowingKalaiet al\.\([2025](https://arxiv.org/html/2608.19208#bib.bib25)\),, we convert them into answer\-verification questions using scene\-graph annotations\. Given an original question and its ground\-truth answer, we construct a positive verification sample by using the ground\-truth answer as the candidate answer\. To construct a negative sample, we sample an incorrect candidate from the GQA scene graph while preserving the semantic type of the answer whenever possible\. The final question is converted into following format:

> Question: \[Question\] Candidate answer: \[Answer\]\. Is the candidate answer correct according to the image?

Table 16:GQA\-Judgment behavior and affine fit for LLaVA under the WikiText\-after\-possibly context\. Origin accuracy is evaluated without irrelevant context, while context metrics are evaluated after inserting irrelevant context\.Table[16](https://arxiv.org/html/2608.19208#A5.T16)shows that GQA\-judgment behaves differently from the cleaner binary existence benchmarks\. We use LLaVA as the testing model\. Without irrelevant context, the converted task is solvable: LLaVA reaches81\.3%81\.3\\%accuracy\. adding irrelevant context can strongly suppress affirmative verification judgments\. Under WikiText\-after\-possibly, the fitted slope is onlya=0\.336a=0\.336withR2=0\.580R^\{2\}=0\.580, and the YesRate drops to14\.5%14\.5\\%\. Additionally, the affine parameter is different, indicating that the prompt template or task will change the affine parameter\.

![Refer to caption](https://arxiv.org/html/2608.19208v1/x9.png)Figure 9:A case of our open\-ended generation experiment\. We use the green mask to indicate the region covered by the model’s prediction under the context\-condition setting\. After adding irrelevant context, the model tends to describe only salient objects in the image, while peripheral objects are no longer mentioned\.
### E\.3Preliminary Study on Open\-Ended Generation

To further examine whether irrelevant context affects open\-ended generation, we conduct a preliminary study using AMBER generative set\. We follow the setting of context\-free and context\-condition, and ask the model to describe each given image\. To be specific, for all settings, the question is:

> Describe this image in detail\.

Thecontext\-freeinput contains only the image and the question:

> \[Image\] \[Question\]

Thecontext\-conditioninput appends the sampled irrelevant context and the template:

> \[Image\] \[Question\] \[Template\] \[Context\]

Table 17:AMBER generative evaluation with LLaVA\-1\.5\-7B\.FreeandConditiondenote context\-free and context\-conditioned generation, respectively\. Hal\., Cov\., Cog\., and Len\. denote hallucination rate, object coverage, cognitive error, and average output length\.The experiment result is in Table[17](https://arxiv.org/html/2608.19208#A5.T17)\. We find that adding irrelevant context substantially changes the model’s behavior: the generated responses become significantly shorter, accompanied by a large drop in coverage\.

The example in Figure[9](https://arxiv.org/html/2608.19208#A5.F9)provides an intuitive illustration of this behavioral shift: the model tends to describe only salient objects in the image, while peripheral objects are no longer mentioned\.

One possible interpretation is that irrelevant context changes the model’s effective decision margin rather than simply injecting random noise into generation\. Under the context\-induced affine transformation, many visual claims are shifted toward a more conservative decision region\. Consequently, objects or attributes that originally receive only weak or moderate support are more likely to fall below the implicit threshold for verbalization\. These marginal claims often correspond to small objects, peripheral regions, fine\-grained attributes, or visually ambiguous details\. By contrast, highly salient objects tend to have stronger visual evidence and larger margins, making them more robust to the margin shift and more likely to remain in the output\. This explains why the model still describes the dominant objects in the scene while omitting many peripheral details\.

From this perspective, the reduced coverage is not merely a side effect of shorter responses, but reflects a systematic change in the model’s generation policy: irrelevant context makes the model more selective about which visual claims to express, suppressing uncertain details and producing more conservative descriptions\.

However, further analysis is needed to verify the causal link between margin suppression and reduced coverage, and to better understand which visual claims are most affected\.