Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding

# Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding
Source: [https://arxiv.org/html/2606.09859](https://arxiv.org/html/2606.09859)
## Mitigating Manifold Departure: Uncertainty\-Aware Subspace Rectification for Trustworthy MLLM Decoding

Jingxiao YangMiao PanCheng TanYuxiang CaiSiwei TanChen ZhiXuhong ZhangJianwei YinJintao Chen

###### Abstract

MLLMs frequently hallucinate objects inconsistent with visual inputs\. This issue is typically attributed to the over\-reliance on language priors, which can override the visual context\. Recent training\-free decoding strategies address this by penalizing language priors\. However, these methods overlook the dual nature of language priors, where they can be both helpful and harmful depending on the alignment with visual evidence\. In particular, blindly suppressing language priors often disrupts the model’s semantic manifold, leading to performance degradation, a phenomenon we term Manifold Departure\. To address this, we propose Manifold\-Guided Adaptive Projection \(MGAP\), a geometry\-aware, training\-free decoding method that mitigates hallucinations while preserving representation structure\. MGAP first constructs a language\-prior subspace from blind hidden states via SVD\. During decoding, MGAP projects each multimodal hidden state onto this subspace and applies a consistency\-aware gate to adaptively attenuate only the projected prior component, yielding a subspace\-selective update that largely preserves the orthogonal semantic components\. Extensive experiments on POPE and CHAIR show that MGAP outperforms prior decoding baselines, achieving stronger hallucination suppression without sacrificing coherence\.

Multi\-model Large Language Model, Machine Learning, ICML, Hallucination, MLLM, Manifold Learning

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.09859v1/x1.png)Figure 1:The dual role of language priors in MLLM decoding\. When visual evidence and priors are aligned \(yellow banana\), priors sharpen and stabilize generation\. When they conflict \(blue banana\), priors can override the image and induce hallucination\.In recent years, Multimodal Large Language Models \(MLLMs\)\(Wanget al\.,[2024b](https://arxiv.org/html/2606.09859#bib.bib41); Chenet al\.,[2024b](https://arxiv.org/html/2606.09859#bib.bib42); Deitkeet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib43)\)have become strong general\-purpose interfaces for vision and language, enabling multimodal reasoning and generation across diverse benchmarks\(Yueet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib45); Liuet al\.,[2024b](https://arxiv.org/html/2606.09859#bib.bib13); Huet al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib46)\)\. Despite this progress, they remain vulnerable to object hallucination\(Liet al\.,[2023b](https://arxiv.org/html/2606.09859#bib.bib5); Baiet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib14); Rohrbachet al\.,[2018](https://arxiv.org/html/2606.09859#bib.bib37)\): the model produces fluent, plausible content that is simply not supported by the image\. This failure is often traced to the model’s language priors learned during pre\-training, which can overpower weak or ambiguous visual evidence\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21); Huanget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib23); Liet al\.,[2023a](https://arxiv.org/html/2606.09859#bib.bib38); Tonget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib39)\)\. Motivated by this, a growing line of training\-free decoding methods\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21); Faveroet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib6); Chenet al\.,[2024a](https://arxiv.org/html/2606.09859#bib.bib24); Wanget al\.,[2024c](https://arxiv.org/html/2606.09859#bib.bib40); Parket al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib27)\)attempts to mitigate hallucinations by subtracting a bias term \(e\.g\., an unconditional branch or a contrastive context\) from the decoding logits\. However, the core difficulty is that language priors are not uniformly harmful\. As illustrated in Figure[1](https://arxiv.org/html/2606.09859#S1.F1), priors play a dual role: they can trigger hallucinations under vision–language conflict \(blue banana\), yet serve as confidence anchors when aligned with the image \(yellow banana\), sharpening and stabilizing generation\. This makes indiscriminate prior suppression a blunt instrument: it may remove exactly the structure that helps normal, visually consistent cases\.

We further show that this unintended degradation has a geometric signature\. Viewed in representation space, valid decoding trajectories concentrate in a structured high\-density region\(Bengioet al\.,[2013a](https://arxiv.org/html/2606.09859#bib.bib47); Vermaet al\.,[2019](https://arxiv.org/html/2606.09859#bib.bib48); Stutzet al\.,[2019](https://arxiv.org/html/2606.09859#bib.bib49); Khoury and Hadfield\-Menell,[2018](https://arxiv.org/html/2606.09859#bib.bib50)\)\. Linear suppression methods apply a global, structure\-agnostic shift, which can push hidden states toward low\-density regions and destabilize decoding\. We term this failure mode*Manifold Departure*and analyze it in Section[3\.3](https://arxiv.org/html/2606.09859#S3.SS3)\(Figure[3](https://arxiv.org/html/2606.09859#S3.F3)\)\. To resolve this tension, we propose Manifold\-Guided Adaptive Projection \(MGAP\), a geometry\-aware, training\-free decoding framework\. MGAP explicitly models language priors as a low\-rank prior subspace estimated from blind hidden states via SVD, and performs a selective intervention during decoding: each hidden state is projected onto the prior subspace, and only this projected component is adaptively attenuated when vision–language conflict is detected\.

In summary, our main contributions are:

- •We identify Manifold Departure as a geometric failure mode of linear, training\-free prior suppression, explaining why such methods can degrade performance even when priors align with vision\.
- •We propose MGAP, which learns a language\-prior subspace from blind states and applies*adaptive, subspace\-selective projection*to suppress hallucinations without disturbing orthogonal semantics\.
- •Experiments on POPE\(Liet al\.,[2023b](https://arxiv.org/html/2606.09859#bib.bib5)\)and CHAIR\(Rohrbachet al\.,[2018](https://arxiv.org/html/2606.09859#bib.bib37)\)show that MGAP achieves stronger trade\-offs between hallucination mitigation and descriptive fidelity than prior training\-free decoding baselines\.

## 2Related Work

#### Hallucination in MLLMs

Object hallucination is a long\-standing issue in vision–language generation and is amplified in modern MLLMs\. Surveys summarize its phenomena and taxonomies\(Baiet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib14)\), while benchmarks/protocols quantify object\-level ungrounded content\(Liet al\.,[2023b](https://arxiv.org/html/2606.09859#bib.bib5),[c](https://arxiv.org/html/2606.09859#bib.bib15)\)\. Similar failures are also studied in other generation settings such as neural machine translation\(Guerreiroet al\.,[2023](https://arxiv.org/html/2606.09859#bib.bib16)\)\. Training\-time mitigation \(e\.g\., robust instruction tuning, factual alignment, preference optimization\) can reduce hallucinations but requires extra supervision or optimization\(Liuet al\.,[2024a](https://arxiv.org/html/2606.09859#bib.bib17); Sunet al\.,[2023](https://arxiv.org/html/2606.09859#bib.bib18); Zhaoet al\.,[2023](https://arxiv.org/html/2606.09859#bib.bib19); Jianget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib20)\)\. We instead study training\-free, decoding\-time control\.

#### Inference\-time intervention

Decoding\-time methods mitigate hallucination without parameter updates\. VCD suppresses language priors by contrasting multimodal decoding against a biased/unconditioned branch\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21)\)\. This idea is closely related to contrastive decoding in text generation\(Liet al\.,[2023a](https://arxiv.org/html/2606.09859#bib.bib38)\)and has been further explored in subsequent VCD\-style analyses\(Leeet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib22)\)\. OPERA regulates over\-trust via penalty and retrospection\-allocation\(Huanget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib23)\)\. Other approaches include instruction contrastive decoding\(Wanget al\.,[2024c](https://arxiv.org/html/2606.09859#bib.bib40)\), adaptive focal contrast decoding\(Chenet al\.,[2024a](https://arxiv.org/html/2606.09859#bib.bib24)\), and introspective/ensemble decoding\(Huoet al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib25); Choet al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib26); Parket al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib27)\)\. Many can be seen as global, structure\-agnostic linear manipulations in representation/logit space, which may hurt outputs even when priors align with vision\. Our method instead models a prior subspace and intervenes selectively in a geometry\-aware manner\. We additionally compare MGAP against recent decoding\-time methods including DeCo\(Wanget al\.,[2024a](https://arxiv.org/html/2606.09859#bib.bib1)\)and MoD\(Suet al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib2)\)\. MGAP consistently achieves the best overall trade\-off between hallucination mitigation and generation fidelity across both backbones\.

#### Manifold structure of latent representations

Representation geometry is widely used to explain and control model behavior via low\-dimensional structure, dominant subspaces, and concentration near a data manifold induced by the training distribution\. Following the common manifold assumption in representation learning, high\-dimensional observations are often assumed to lie on or near a smooth low\-dimensional manifold\(Roweis and Saul,[2000](https://arxiv.org/html/2606.09859#bib.bib51); Tenenbaumet al\.,[2000](https://arxiv.org/html/2606.09859#bib.bib52); Bengioet al\.,[2013b](https://arxiv.org/html/2606.09859#bib.bib53)\)\. In vision, transformer representations under large\-scale pretraining exhibit strong geometric regularities\(Dosovitskiyet al\.,[2020](https://arxiv.org/html/2606.09859#bib.bib28); Heet al\.,[2022](https://arxiv.org/html/2606.09859#bib.bib29); Beyeret al\.,[2023](https://arxiv.org/html/2606.09859#bib.bib30)\)\. Related work measures cross\-layer/model alignment using SVCCA/CKA\-style diagnostics\(Raghuet al\.,[2017](https://arxiv.org/html/2606.09859#bib.bib7); Kornblithet al\.,[2019](https://arxiv.org/html/2606.09859#bib.bib8); Morcoset al\.,[2018](https://arxiv.org/html/2606.09859#bib.bib9); Liet al\.,[2018](https://arxiv.org/html/2606.09859#bib.bib10)\)\. Decoding\-time behavior in language generation has also been connected to representation\-level objectives and contrastive directions\(Liet al\.,[2023a](https://arxiv.org/html/2606.09859#bib.bib38)\)\. Building on this view, we adopt the same geometric perspective for MLLM hidden states: we estimate a low\-rank language\-prior subspace from blind hidden states and apply bounded, selective projection to mitigate hallucination while staying near the data manifold\.

## 3Preliminary

In this section, we introduce notation for multimodal decoding and summarize training\-free linear suppression under a unified form\. We then clarify our manifold viewpoint that motivates geometry\-aware decoding interventions\.

### 3\.1Representation\-Level View of Training\-Free Decoding

Given an imagevvand a textual queryxx, a multimodal large language model generates tokens autoregressively\. At decoding steptt, the next\-token distribution is given by

P​\(yt∣y<t,x,v\)=Softmax​\(ℱ​\(ht\)\),P\(y\_\{t\}\\mid y\_\{<t\},x,v\)=\\mathrm\{Softmax\}\(\\mathcal\{F\}\(h\_\{t\}\)\),\(1\)whereht∈ℝdh\_\{t\}\\in\\mathbb\{R\}^\{d\}denotes the final\-layer hidden state\.

Recent training\-free hallucination mitigation methods intervene at decoding time by linearly suppressing language priors\. Despite different formulations, many such methods\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21); Wanget al\.,[2024c](https://arxiv.org/html/2606.09859#bib.bib40); Kimet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib12)\)can be unified as

Logitsfinal=Logitsmain−ρ⋅Logitsbias,\\mathrm\{Logits\}\_\{\\mathrm\{final\}\}=\\mathrm\{Logits\}\_\{\\mathrm\{main\}\}\-\\rho\\cdot\\mathrm\{Logits\}\_\{\\mathrm\{bias\}\},\(2\)whereLogitsbias\\mathrm\{Logits\}\_\{\\mathrm\{bias\}\}is obtained from a biased context \(e\.g\., blind or perturbed inputs\)\.

Importantly, Eq\. \([2](https://arxiv.org/html/2606.09859#S3.E2)\) implicitly assumes that hallucinations can be mitigated by a global linear translation in representation space\. In this work, we analyze and intervene directly on the hidden representationhth\_\{t\}, enabling a geometric understanding of why such linear suppression may fail\.

### 3\.2Hidden\-State Manifold and Manifold Departure

Letht∈ℝdh\_\{t\}\\in\\mathbb\{R\}^\{d\}denote the \(last\-layer\) decoder hidden state at decoding steptt\. Following the common manifold assumption in representation learning, high\-dimensional observations are often assumed to lie on or near a smooth low\-dimensional manifold\(Roweis and Saul,[2000](https://arxiv.org/html/2606.09859#bib.bib51); Tenenbaumet al\.,[2000](https://arxiv.org/html/2606.09859#bib.bib52); Bengioet al\.,[2013b](https://arxiv.org/html/2606.09859#bib.bib53)\)\. We adopt the same view for MLLM hidden states\.

###### Definition 3\.1\(Hidden\-state manifold\)\.

LetP0P\_\{0\}denote the distribution of hidden states produced by*normal decoding*\(without any decoding\-time intervention\)\. We say the hidden states concentrate near arr\-dimensional manifoldℳ⊂ℝd\\mathcal\{M\}\\subset\\mathbb\{R\}^\{d\}\(r≪dr\\ll d\) if there existε\>0\\varepsilon\>0and a smallδ∈\(0,1\)\\delta\\in\(0,1\)such that:

Prh∼P0⁡\[dist​\(h,ℳ\)≤ε\]≥1−δ\.\\Pr\_\{h\\sim P\_\{0\}\}\\\!\\big\[\\,\\mathrm\{dist\}\(h,\\mathcal\{M\}\)\\leq\\varepsilon\\,\\big\]\\geq 1\-\\delta\.\(3\)HerePrh∼P0⁡\[⋅\]\\Pr\_\{h\\sim P\_\{0\}\}\[\\cdot\]denotes probability whenhhis sampled fromP0P\_\{0\}\. Eq\. \([3](https://arxiv.org/html/2606.09859#S3.E3)\) means that at least a\(1−δ\)\(1\-\\delta\)fraction of normal\-decoding states lie within anε\\varepsilon\-neighborhood \(an “ε\\varepsilon\-tube”\) aroundℳ\\mathcal\{M\}\. We define the Euclidean distance to the manifold as:

dist​\(h,ℳ\)≜infm∈ℳ‖h−m‖2\.\\mathrm\{dist\}\(h,\\mathcal\{M\}\)\\triangleq\\inf\_\{m\\in\\mathcal\{M\}\}\\\|h\-m\\\|\_\{2\}\.\(4\)We refer toℳ\\mathcal\{M\}as the*semantic manifold*of normal decoding\.

#### A computable proxy

Sinceℳ\\mathcal\{M\}anddist​\(h,ℳ\)\\mathrm\{dist\}\(h,\\mathcal\{M\}\)are not directly computable, we approximate them using a reference bank of hidden states𝒮=\{h\(i\)\}i=1N\\mathcal\{S\}=\\\{h^\{\(i\)\}\\\}\_\{i=1\}^\{N\}collected from normal decoding \(gray points in Fig\.[3](https://arxiv.org/html/2606.09859#S3.F3)\)\. For any query stateh∈ℝdh\\in\\mathbb\{R\}^\{d\}, letNNk​\(h;𝒮\)⊂𝒮\\mathrm\{NN\}\_\{k\}\(h;\\mathcal\{S\}\)\\subset\\mathcal\{S\}denote the set of itskknearest neighbors in𝒮\\mathcal\{S\}under the chosen metric \(default: Euclidean distance\)\. We define an off\-manifoldness score by the averagekkNN distance:

dk​\(h;𝒮\)≜1k​∑s∈NNk​\(h;𝒮\)‖h−s‖2\.d\_\{k\}\(h;\\mathcal\{S\}\)\\triangleq\\frac\{1\}\{k\}\\sum\_\{s\\in\\mathrm\{NN\}\_\{k\}\(h;\\mathcal\{S\}\)\}\\\|h\-s\\\|\_\{2\}\.\(5\)\(Using cosine distance afterℓ2\\ell\_\{2\}\-normalization is an alternative in high dimensions\.\)

###### Definition 3\.2\(Manifold departure\)\.

Consider an interventionTTapplied at stepttthat mapshth\_\{t\}toh~t=T​\(ht\)\\tilde\{h\}\_\{t\}=T\(h\_\{t\}\)\. We sayTTinduces*manifold departure*at stepttif:

dk​\(h~t;𝒮\)\>τ,d\_\{k\}\(\\tilde\{h\}\_\{t\};\\mathcal\{S\}\)\>\\tau,\(6\)where the thresholdτ\\tauis set as the\(1−δ\)\(1\-\\delta\)\-quantile of the reference scores\{dk​\(s;𝒮\)\}s∈𝒮\\\{d\_\{k\}\(s;\\mathcal\{S\}\)\\\}\_\{s\\in\\mathcal\{S\}\}, i\.e\., only anδ\\deltafraction of normal\-decoding states would be labeled off\-manifold under this criterion\.

### 3\.3Manifold Departure from Prior Suppression

We identify a consistent*failure mechanism*underlying many training\-free linear decoding interventions:*indiscriminate*suppression of language\-prior components can push hidden states away from the normal\-decoding semantic manifold \(Definition[3\.1](https://arxiv.org/html/2606.09859#S3.Thmtheorem1)\), thereby degrading generation quality even when the language prior is semantically helpful\.

#### Empirical observation

![Refer to caption](https://arxiv.org/html/2606.09859v1/Figures/Method/vcd_bar_thin_reference_style_gray_blue.png)Figure 2:Uniform linear prior suppression can hurt\. VCD \(vs\. vanilla\) shows consistent performance drops on POPE across splits, including standard cases where priors align with visual evidence\.We compare vanilla decoding and Visual Contrastive Decoding \(VCD\)\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21)\)on the POPE benchmark using LLaVA v1\.5\-7B\. As shown in Figure[2](https://arxiv.org/html/2606.09859#S3.F2), VCD consistently degrades performance across all evaluation splits\. Notably, this degradation also occurs in standard cases where visual evidence and language priors are naturally aligned, suggesting a systematic limitation of applying the same linear suppression regardless of context\.

#### Geometric mechanism

To understand*why*such degradation happens, we analyze hidden representations through the lens of the semantic manifold defined in Section[3\.2](https://arxiv.org/html/2606.09859#S3.SS2)\. We focus on the final\-layer hidden stateht∈ℝDh\_\{t\}\\in\\mathbb\{R\}^\{D\}and its evolution under decoding\-time interventions\.

We extract hidden states from normal decoding trajectories on the COCO dataset and visualize their distribution using Principal Component Analysis \(PCA\)\. As shown in Figure[3](https://arxiv.org/html/2606.09859#S3.F3), hidden states corresponding to valid generations concentrate within a structured, low\-dimensional region of the representation space, which we interpret as the normal\-decoding semantic manifold \(Definition[3\.1](https://arxiv.org/html/2606.09859#S3.Thmtheorem1)\)\. Linear extrapolation along the VCD direction, however, progressively moves states away from this region, as quantified by increasing off\-manifoldness scores shown in the inset\.

![Refer to caption](https://arxiv.org/html/2606.09859v1/x2.png)Figure 3:Manifold departure under linear suppression\. Gray points are hidden states from normal decoding \(semantic manifold\); orange/blue/green mark blind prior, joint posterior, and ground\-truth states\. Red numeric labels denote different values of the VCD extrapolation coefficientα\\alpha\.*Inset*: kNN\-based off\-manifoldness scoredkd\_\{k\}of VCD states at differentα\\alpha, computed in the original hidden space\. The dashed line indicates the thresholdτ\\tauused to define manifold departure \(Eq\. \([6](https://arxiv.org/html/2606.09859#S3.E6)\)\); bars aboveτ\\tauare increasingly shaded to reflect larger deviations from the semantic manifold\.In contrast, linear prior suppression applies a fixed contrastive shift that extrapolates along the difference between the blind \(prior\) and joint \(visual\) contexts, e\.g\.,

h′=hjoint\+ρ​\(hjoint−hblind\)\.h^\{\\prime\}\\;=\\;h\_\{\\text\{joint\}\}\+\\rho\\bigl\(h\_\{\\text\{joint\}\}\-h\_\{\\text\{blind\}\}\\bigr\)\.\(7\)Herehblindh\_\{\\text\{blind\}\}follows VCD\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21)\)and is obtained by running the model on a visually distorted input, where the original image is perturbed with diffusion noise to form the blind view\. Such a global linear extrapolation is agnostic to the local geometry of the normal\-decoding manifold\. Even whenhjointh\_\{\\text\{joint\}\}lies in a typical \(high\-density\) region, stepping along the direction\(hjoint−hblind\)\(h\_\{\\text\{joint\}\}\-h\_\{\\text\{blind\}\}\)can move the state into a tail region that is rarely visited under normal decoding\. Formally, this corresponds to increased off\-manifoldness \(Eq\. \([5](https://arxiv.org/html/2606.09859#S3.E5)\)\) and can trigger the manifold\-departure conditiondk​\(h~t;𝒮\)\>τd\_\{k\}\(\\tilde\{h\}\_\{t\};\\mathcal\{S\}\)\>\\tau\(Definition[3\.2](https://arxiv.org/html/2606.09859#S3.Thmtheorem2)\), which is visualized along the VCD trajectory in Figure[3](https://arxiv.org/html/2606.09859#S3.F3)\(inset\)\. Once the hidden state departs from the high\-density region supported by normal decoding, the decoder operates in a poorly supported regime, often leading to less stable token distributions and degraded generation quality \(Fig[2](https://arxiv.org/html/2606.09859#S3.F2)\)\.

Importantly, this failure arises from the structure\-agnostic nature of uniform suppression, rather than from the intrinsic correctness or incorrectness of the language prior\. This diagnosis motivates a selective and geometry\-aware intervention, which we introduce next\.

![Refer to caption](https://arxiv.org/html/2606.09859v1/x3.png)Figure 4:Overview of the proposed Manifold\-Guided Adaptive Projection \(MGAP\)\. The language\-prior subspace is constructed offline using unlabeled blind inputs, while decoding\-time intervention is performed online via geometry\-aware adaptive projection\.Table 1:Main results on the POPE benchmark\. We report Accuracy \(%\), Precision \(%\), and F1\-score \(%\) on three splits\. Our method consistently outperforms existing training\-free decoding strategies across both backbones\.Table 2:Results on the CHAIR benchmark\. Lower CHAIR scores indicate fewer hallucinated objects\. Our method significantly reduces hallucination while maintaining competitive precision and F1\.

## 4Method

### 4\.1Label\-Free Language Prior Subspace Construction

Motivated by the diagnosis in Section[3\.3](https://arxiv.org/html/2606.09859#S3.SS3), we model language priors as a low\-dimensional subspace in the hidden\-state space\. We construct this subspace using only queries, requiring no labels, image content, or parameter updates\.

Given prompts\{x\(i\)\}i=1N\\\{x^\{\(i\)\}\\\}\_\{i=1\}^\{N\}, we collect final\-layer blind hidden states\{hblind\(i\)∈ℝd\}i=1N\\\{h\_\{\\mathrm\{blind\}\}^\{\(i\)\}\\in\\mathbb\{R\}^\{d\}\\\}\_\{i=1\}^\{N\}\. Leth¯blind≜1N​∑i=1Nhblind\(i\)\\bar\{h\}\_\{\\mathrm\{blind\}\}\\triangleq\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}h\_\{\\mathrm\{blind\}\}^\{\(i\)\}and define the centered matrix

H~blind≜\[\(hblind\(1\)−h¯blind\)⊤⋮\(hblind\(N\)−h¯blind\)⊤\]∈ℝN×d\.\\tilde\{H\}\_\{\\mathrm\{blind\}\}\\triangleq\\begin\{bmatrix\}\(h\_\{\\mathrm\{blind\}\}^\{\(1\)\}\-\\bar\{h\}\_\{\\mathrm\{blind\}\}\)^\{\\top\}\\\\ \\vdots\\\\ \(h\_\{\\mathrm\{blind\}\}^\{\(N\)\}\-\\bar\{h\}\_\{\\mathrm\{blind\}\}\)^\{\\top\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{N\\times d\}\.\(8\)We estimate the prior subspace as the top\-KKprincipal components, i\.e\.,

Vprior∈arg⁡maxV∈ℝd×K⁡Tr​\(V⊤​H~blind⊤​H~blind​V\)\.V\_\{\\mathrm\{prior\}\}\\in\\arg\\max\_\{V\\in\\mathbb\{R\}^\{d\\times K\}\}\\ \\mathrm\{Tr\}\\\!\\left\(V^\{\\top\}\\tilde\{H\}\_\{\\mathrm\{blind\}\}^\{\\top\}\\tilde\{H\}\_\{\\mathrm\{blind\}\}V\\right\)\.\(9\)HereTr​\(⋅\)\\mathrm\{Tr\}\(\\cdot\)denotes the trace \(sum of diagonal entries\);Tr​\(⋅\)\\mathrm\{Tr\}\\\!\\left\(\\cdot\\right\)is the total variance of blind hidden states captured by theKKorthonormal directionsVprior=\[v1,…,vK\]V\_\{\\mathrm\{prior\}\}=\[v\_\{1\},\\ldots,v\_\{K\}\], which has the closed\-form solution via SVD:

H~blind=U​Σ​V⊤,Vprior≜V\[:,1:K\]∈ℝd×K\.\\tilde\{H\}\_\{\\mathrm\{blind\}\}=U\\Sigma V^\{\\top\},\\qquad V\_\{\\mathrm\{prior\}\}\\triangleq V\_\{\[:,1:K\]\}\\in\\mathbb\{R\}^\{d\\times K\}\.\(10\)VpriorV\_\{\\mathrm\{prior\}\}captures dominant blind\-state variations induced by linguistic regularities; it is*not*assumed inherently harmful, motivating adaptive \(rather than uniform\) suppression in subsequent decoding\.

### 4\.2Consistency\-Aware Adaptive Projection

Given a hidden statehorigh\_\{\\mathrm\{orig\}\}produced during standard multimodal decoding, we first decompose it into components aligned with and orthogonal to the language prior subspace\. The projection onto the prior subspace is given by:

hproj=Vprior​Vprior⊤​horig\.h\_\{\\mathrm\{proj\}\}=V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}h\_\{\\mathrm\{orig\}\}\.\(11\)
Whilehprojh\_\{\\mathrm\{proj\}\}captures prior\-related components, indiscriminately removing it may cause Manifold Departure\. We therefore introduce an adaptive mechanism that selectively suppresses prior components only when they conflict with visual evidence\.

#### Prior\-Posterior alignment

We quantify the agreement between the original hidden state and its prior projection using cosine similarity:

δ=1−cos⁡\(horig,hproj\)\.\\delta=1\-\\cos\(h\_\{\\mathrm\{orig\}\},h\_\{\\mathrm\{proj\}\}\)\.\(12\)A smallδ\\deltaindicates that the current state is consistent with the prior subspace and does not require additional suppression, whereas a largeδ\\deltasignals a mismatch that warrants stronger attenuation of prior components\.

#### Uncertainty\-Aware gating

Hallucinations are often accompanied by increased predictive uncertainty\. We measure uncertainty using the Shannon entropy of the token distribution:

H=−∑yp​\(y\)​log⁡p​\(y\),H=\-\\sum\_\{y\}p\(y\)\\log p\(y\),\(13\)and derive a gating factor:

γ=tanh⁡\(λ​H\),\\gamma=\\tanh\(\\lambda H\),\(14\)which amplifies intervention when generation is unstable and suppresses it when the model is confident\.

#### Adaptive projection

The corrected hidden state is computed as:

hcut=horig−α⋅γ⋅δ⋅hproj\.h\_\{\\mathrm\{cut\}\}=h\_\{\\mathrm\{orig\}\}\-\\alpha\\cdot\\gamma\\cdot\\delta\\cdot h\_\{\\mathrm\{proj\}\}\.\(15\)Equivalently, lettingβ=α⋅γ⋅δ\\beta=\\alpha\\cdot\\gamma\\cdot\\delta, MGAP applieshcut=horig−β​hprojh\_\{\\mathrm\{cut\}\}=h\_\{\\mathrm\{orig\}\}\-\\beta h\_\{\\mathrm\{proj\}\}\. When visual evidence and language priors are consistent, bothγ\\gammaandδ\\deltabecome small, reducing the operation to an approximate identity mapping\. As a result, MGAP avoids the global linear translation that causes Manifold Departure in prior methods\.

### 4\.3Theoretical Insights

To clarify why MGAP avoids the indiscriminate behavior of uniform linear suppression, we summarize three theoretical properties below\. The proofs are provided in Appendix[B](https://arxiv.org/html/2606.09859#A2)\.

###### Theorem 4\.1\(Distance decrease under prior\-aligned residual\)\.

Lethorigh\_\{\\mathrm\{orig\}\}denote the original decoding hidden state,hprojh\_\{\\mathrm\{proj\}\}its projection onto the learned prior subspace, andhcut:=horig−β​hprojh\_\{\\mathrm\{cut\}\}:=h\_\{\\mathrm\{orig\}\}\-\\beta h\_\{\\mathrm\{proj\}\}the MGAP\-corrected state withβ\>0\\beta\>0\. Under a*prior\-aligned residual*condition \(formalized in Appendix[B](https://arxiv.org/html/2606.09859#A2)\), MGAP yields:

‖hcut−hgt‖2<‖horig−hgt‖2\.\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{gt\}\}\\\|^\{2\}<\\\|h\_\{\\mathrm\{orig\}\}\-h\_\{\\mathrm\{gt\}\}\\\|^\{2\}\.\(16\)

#### Interpretation

When the current error component lies along the prior projection directionhprojh\_\{\\mathrm\{proj\}\}, subtracting \(a suitable amount of\)hprojh\_\{\\mathrm\{proj\}\}moves the hidden state closer to the ground\-truth\-aligned statehgth\_\{\\mathrm\{gt\}\}\. This analysis is illustrative and does not assume access tohgth\_\{\\mathrm\{gt\}\}at inference

###### Theorem 4\.2\(Bounded coefficient and bounded step\)\.

Assumeβ:=α⋅γ⋅δ\\beta:=\\alpha\\cdot\\gamma\\cdot\\deltawithγ:=tanh⁡\(λ​H\)\\gamma:=\\tanh\(\\lambda H\)andδ:=1−cos⁡\(horig,hproj\)\\delta:=1\-\\cos\(h\_\{\\mathrm\{orig\}\},h\_\{\\mathrm\{proj\}\}\)\. Then0≤β<α0\\leq\\beta<\\alpha, and the update magnitude is bounded:

‖hcut−horig‖≤α​‖horig‖\.\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{orig\}\}\\\|\\leq\\alpha\\\|h\_\{\\mathrm\{orig\}\}\\\|\.\(17\)

#### Interpretation

Sinceγ\\gammaandδ\\deltascales the step size, the effective coefficientβ\\betais always bounded byα\\alpha, and the update magnitude is controlled\. This prevents overly large corrections and yields stable decoding\-time intervention\.

###### Theorem 4\.3\(Subspace\-selective update\)\.

LetVprior∈ℝd×KV\_\{\\mathrm\{prior\}\}\\in\\mathbb\{R\}^\{d\\times K\}be the prior\-subspace basis, and the prior projectionhproj:=Vprior​Vprior⊤​horigh\_\{\\mathrm\{proj\}\}:=V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}h\_\{\\mathrm\{orig\}\}\. MGAP only changes the prior component and preserves the orthogonal component:

hcut−Vprior​Vprior⊤​hcut=horig−Vprior​Vprior⊤​horig\.h\_\{\\mathrm\{cut\}\}\-V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}h\_\{\\mathrm\{cut\}\}=h\_\{\\mathrm\{orig\}\}\-V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}h\_\{\\mathrm\{orig\}\}\.\(18\)

#### Interpretation

MGAP only modifies the prior\-subspace component and leaves all non\-prior \(orthogonal\) semantics unchanged, avoiding a global shift\.

#### Takeaway

MGAP applies a bounded, subspace\-selective correction: its intervention strength is provably bounded \(Theorem[4\.2](https://arxiv.org/html/2606.09859#S4.Thmtheorem2)\), while preserving the orthogonal semantics \(Theorem[4\.3](https://arxiv.org/html/2606.09859#S4.Thmtheorem3)\) and avoiding a global shift\. Moreover, when the current error is positively aligned with the prior projection \(i\.e\.,⟨horig−hgt,hproj⟩\>0\\langle h\_\{\\mathrm\{orig\}\}\-h\_\{\\mathrm\{gt\}\},\\,h\_\{\\mathrm\{proj\}\}\\rangle\>0\), subtracting \(a suitable amount of\)hprojh\_\{\\mathrm\{proj\}\}provably moves the hidden state closer to a ground\-truth\-aligned reference \(Theorem[4\.1](https://arxiv.org/html/2606.09859#S4.Thmtheorem1)\)\.

## 5Experiments

### 5\.1Experimental Setup

Table 3:Ablation study on the POPE benchmark\. “w/o Prot” removes consistency protection, and “w/o Gate” removes uncertainty gating\. Removing either component leads to unstable trade\-offs\.Table 4:Ablation results on the CHAIR benchmark\. We analyze the contribution of consistency protection \(*w/o Prot*\) and uncertainty\-aware gating \(*w/o Gate*\) on two representative backbones\. Lower CHAIR scores indicate fewer hallucinated objects\.We evaluate MGAP on two representative multimodal large language models, LLaVA v1\.5\-7B and Qwen3\-VL\-8B\. Hallucination mitigation performance is assessed on POPE and CHAIR\. For each backbone, we construct the language prior subspace offline using blind inputs \(null image\)\. We sampleN=50N=50textual prompts and collect the corresponding final\-layer blind hidden states to formH~blind\\tilde\{H\}\_\{\\mathrm\{blind\}\}\(Eq\. \([8](https://arxiv.org/html/2606.09859#S4.E8)\)\)\. We then perform SVD and retain the top\-KKright singular vectorsVpriorV\_\{\\mathrm\{prior\}\}withK=5K=5\(Eq\. \([10](https://arxiv.org/html/2606.09859#S4.E10)\)\)\. All experiments are conducted on a single NVIDIA RTX A6000 GPU \(48GB\)\.

#### Baselines\.

We compare MGAP with several representative training\-free decoding methods: VCD\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21)\), which suppresses language priors via contrastive decoding with an unconditional branch; ICD\(Wanget al\.,[2024c](https://arxiv.org/html/2606.09859#bib.bib40)\), which applies instruction\-level contrastive decoding to penalize prior\-driven generations; CODE\(Kimet al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib12)\), which contrasts self\-generated descriptions to reduce hallucinations; and HalTrapper\(Zhenget al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib11)\), a recent method that attributes hallucinations to contextual accumulation in long responses and mitigates them by dynamically regulating context influence during decoding\. All baselines operate without additional training and are evaluated under their recommended settings\.

### 5\.2Main Results

We evaluate MGAP on the POPE benchmark using two representative backbones, LLaVA v1\.5\-7B and Qwen3\-VL\-8B\. Results are summarized in Table[1](https://arxiv.org/html/2606.09859#S3.T1)\.

#### Main Results on POPE

Across all splits and both backbones, MGAP consistently achieves the best performance\. Compared to vanilla decoding, our method improves F1 on all POPE splits, while avoiding the performance degradation observed in existing contrastive decoding approaches\. On theRandomandPopularsplits, MGAP achieves clear precision gains, indicating more faithful object grounding\. On theAdversarialsplit, which explicitly targets hallucination failure modes, MGAP yields the strongest overall performance on both backbones, demonstrating robustness under severe visual–linguistic conflict\. Compared with training\-free baselines such as VCD, ICD, and HalTrapper, MGAP consistently achieves superior trade\-offs across all splits\. In particular, VCD underperforms vanilla decoding even on non\-adversarial splits, confirming that indiscriminate linear suppression of language priors disrupts valid semantic configurations\. By contrast, MGAP selectively suppresses prior components only when they conflict with visual evidence, thereby avoiding unnecessary interference\. As a result, our method is the only approach that simultaneously improves precision and accuracy while maintaining or improving F1 across all evaluation settings\.

#### Main Results on CHAIR

We further evaluate MGAP on the CHAIR benchmark, which measures object\-level hallucinations in image captioning\. Results are reported in Table[2](https://arxiv.org/html/2606.09859#S3.T2)\. Across both backbones, MGAP achieves a substantial reduction in hallucination rates, significantly lowering both CHAIRs and CHAIRi compared to all baselines\. Notably, this reduction is accompanied by clear gains in precision, indicating that hallucination mitigation is achieved without overly conservative generation\. In contrast, contrastive decoding baselines \(e\.g\., VCD and ICD\) exhibit mixed behavior across metrics and backbones, often improving only one of CHAIRs / CHAIRi while worsening the other, reflecting an unstable trade\-off in object grounding\. These results demonstrate that MGAP generalizes beyond binary question answering to open\-ended captioning tasks\.

### 5\.3Generalization to Additional Benchmarks

Table 5:Results on the AMBER benchmark using LLaVA v1\.5 7B\.↓\\downarrowindicates lower is better, and↑\\uparrowindicates higher is better\. Best results are highlighted inbold\.Table 6:Inference efficiency comparison on 3000 samples\. We report total inference time and average latency per sample\. Our method is significantly more efficient than contrastive decoding baselines\.While POPE primarily evaluates object\-level hallucinations through binary probing questions, it remains unclear whether the learned hallucination priors generalize beyond this specific evaluation protocol\. To further assess the robustness of MGAP, we additionally evaluate on the AMBER benchmark, which contains more diverse hallucination scenarios and compositional visual reasoning challenges\.

Table 7:Number of off\-manifold samples \(out of 3000\) under different extrapolation coefficientsα\\alpha\. Lower values indicate that the representations are better constrained within the valid data manifold\. Best results are inbold, and second best areunderlined\.Table[5](https://arxiv.org/html/2606.09859#S5.T5)shows that MGAP consistently improves over existing decoding\-time baselines across both backbones\. In particular, MGAP achieves stronger hallucination mitigation while better preserving generation fidelity, suggesting that the learned prior subspace captures hallucination\-inducing directions that generalize across benchmarks\.

These results indicate that hallucination priors are not merely artifacts of the POPE benchmark, but instead exhibit transferable structural properties in the representation space\. This supports our hypothesis that selective prior correction can serve as a more general mechanism for mitigating hallucinations in large vision\-language models\.

### 5\.4Representation Analysis

To better understand the behavior of MGAP, we analyze the representation changes introduced during decoding\. Specifically, we measure the average projection magnitude onto the learned prior subspace for different decoding strategies\.

As shown in Table[7](https://arxiv.org/html/2606.09859#S5.T7), uniform suppression methods tend to produce larger global representation shifts, whereas MGAP introduces more localized corrections within the learned subspace\. This behavior is consistent across both backbones\.

These observations align with the design motivation of MGAP\. Instead of globally suppressing decoding trajectories, MGAP performs targeted representation correction using the learned prior subspace, which helps mitigate hallucinations while better preserving semantic fidelity\.

### 5\.5Ablation Study

We conduct ablation studies to analyze the contribution of each component in MGAP, namely the*consistency\-based protection*and the*uncertainty\-aware gating*\. Specifically, we consider two variants: \(1\)w/o Prot, which removes the consistency\-based protection term, and \(2\)w/o Gate, which removes the uncertainty\-aware gating mechanism\. Results are reported on both POPE and CHAIR benchmarks\.

![Refer to caption](https://arxiv.org/html/2606.09859v1/x4.png)Figure 5:Qualitative comparison on the CHAIR benchmark\. VCD introduces hallucinated object mentions \(highlighted in red\), while MGAP produces grounded descriptions without hallucinated objects\.#### Ablation Results on POPE

Table[3](https://arxiv.org/html/2606.09859#S5.T3)summarizes the ablation results on the POPE benchmark\. Removing either component leads to a noticeable degradation in overall performance, confirming that both are necessary for stable hallucination mitigation\. A notable phenomenon is that both w/o Prot and w/o Gate variants exhibit abnormally high precision, particularly on LLaVA v1\.5\-7B\. However, this precision gain comes at the cost of significantly reduced F1 scores and Accuracy\. This behavior indicates an imbalanced trade\-off: the model becomes overly conservative, suppressing a large portion of predictions to avoid hallucinations, but failing to correctly answer valid questions\. In contrast, the full MGAP achieves a more balanced trade\-off\. By jointly modeling visual–linguistic consistency and generation uncertainty, our method selectively suppresses prior components only when necessary\. As a result, MGAP maintains high precision while substantially improving F1 score, demonstrating stable and reliable hallucination mitigation\.

#### Ablation Results on CHAIR

Results on CHAIR in Table[4](https://arxiv.org/html/2606.09859#S5.T4)show a consistent trend\. While removing either protection or gating reduces hallucination rates compared to vanilla decoding, both variants exhibit inferior precision and F1 compared to the full model, indicating that both components are necessary for stable and accurate grounding\.

### 5\.6Efficiency Analysis

In addition to effectiveness, inference efficiency is a critical factor for training\-free decoding methods\. Many contrastive decoding approaches mitigate hallucinations by performing multiple forward passes with different contexts\(Lenget al\.,[2024](https://arxiv.org/html/2606.09859#bib.bib21); Wanget al\.,[2024c](https://arxiv.org/html/2606.09859#bib.bib40); Zhenget al\.,[2025](https://arxiv.org/html/2606.09859#bib.bib11)\), significantly increasing inference latency\. We compare the inference efficiency of MGAP with vanilla decoding and representative baselines, including VCD, ICD, and HalTrapper\. As shown in Table[6](https://arxiv.org/html/2606.09859#S5.T6), contrastive methods like VCD, ICD, and HalTrapper incur more than twice the inference time per sample compared to vanilla decoding due to multiple contrastive branches\. In contrast, MGAP only applies a lightweight intervention during a single forward pass, reducing inference latency by approximately 50 % compared to contrastive baselines, while remaining fully training\-free\. This efficiency–effectiveness trade\-off makes MGAP particularly suitable for practical deployment\.

### 5\.7Qualitative Case Study

We present qualitative case studies from CHAIR to illustrate how MGAP mitigates hallucinations\. Figure[5](https://arxiv.org/html/2606.09859#S5.F5)shows representative examples comparing Visual Contrastive Decoding \(VCD\) and MGAP\. In these cases, VCD hallucinates objects that are not supported by the visual content, such as chairs or tableware, likely due to over\-reliance on language priors associated with indoor scenes\. These hallucinations lead to reduced description faithfulness\. In contrast, MGAP avoids introducing unsupported object mentions\. By selectively regulating language priors based on visual–linguistic consistency and generation uncertainty, our method preserves informative descriptions while suppressing hallucinated content\. Importantly, the improvement does not stem from overly conservative generation: MGAP still provides detailed and coherent captions grounded in visual evidence\. These qualitative results align with the quantitative findings in Sections[5\.2](https://arxiv.org/html/2606.09859#S5.SS2.SSS0.Px2)and[5\.5](https://arxiv.org/html/2606.09859#S5.SS5), indivating that MGAP suppresses object hallucinations while maintaining descriptive quality\.

## 6Conclusion

We identify a common failure mode of training\-free decoding for hallucination mitigation: indiscriminate linear prior suppression disrupts hidden representations, which we term*Manifold Departure*\. We propose Manifold\-Guided Adaptive Projection \(MGAP\), which models a language\-prior subspace and applies uncertainty\-aware selective projection at decoding time\.111Our code and models are publicly available at:[https://github\.com/ZJU\-OmniAI/MGAP](https://github.com/ZJU-OmniAI/MGAP)Experiments prove improved trade\-offs between hallucination reduction and descriptive fidelity\.

## Impact Statement

This paper advances multimodal language models by improving hallucination mitigation\. We anticipate positive impacts in applications like visual question answering, with continued attention to fairness and robustness\.

## Acknowledgements

This work is supported by the Zhejiang Pioneer Project under Grant No\. 2025C06SA201986 and the Zhejiang Key Laboratory Project \(2024E10001\)\.

## References

- Z\. Bai, P\. Wang, T\. Xiao, T\. He, Z\. Han, Z\. Zhang, and M\. Z\. Shou \(2024\)Hallucination of multimodal large language models: a survey\.arXiv preprint arXiv:2404\.18930\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Bengio, A\. Courville, and P\. Vincent \(2013a\)Representation learning: a review and new perspectives\.IEEE Transactions on Pattern Analysis and Machine Intelligence35\(8\),pp\. 1798–1828\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2013.50)Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p2.1)\.
- Y\. Bengio, A\. Courville, and P\. Vincent \(2013b\)Representation learning: a review and new perspectives\.IEEE Transactions on Pattern Analysis and Machine Intelligence35\(8\),pp\. 1798–1828\.Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2606.09859#S3.SS2.p1.2)\.
- L\. Beyer, P\. Izmailov, A\. Kolesnikov, M\. Caron, S\. Kornblith, X\. Zhai, M\. Minderer, M\. Tschannen, I\. Abdulmohsin, and F\. Pavetic \(2023\)FlexiViT: one model for all patch sizes\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Chen, Z\. Zhao, H\. Luo, H\. Yao, B\. Li, and J\. Zhou \(2024a\)HALC: object hallucination reduction via adaptive focal\-contrast decoding\.arXiv preprint arXiv:2403\.00425\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Chen, W\. Wang, Y\. Cao, Y\. Liu, Z\. Gao, E\. Cui, J\. Zhu, S\. Ye, H\. Tian, Z\. Liu, L\. Gu, X\. Wang, Q\. Li, Y\. Ren, Z\. Chen, J\. Luo, J\. Wang, T\. Jiang, B\. Wang, C\. He, B\. Shi, X\. Zhang, H\. Lv, Y\. Wang, W\. Shao, P\. Chu, Z\. Tu, T\. He, Z\. Wu, H\. Deng, J\. Ge, K\. Chen, K\. Zhang, L\. Wang, M\. Dou, L\. Lu, X\. Zhu, T\. Lu, D\. Lin, Y\. Qiao, J\. Dai, and W\. Wang \(2024b\)Expanding performance boundaries of open\-source multimodal models with model, data, and test\-time scaling\.arXiv preprint arXiv:2412\.05271\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- Y\. Cho, K\. Kim, T\. Hwang, and S\. Cho \(2025\)Do you keep an eye on what i ask? mitigating multimodal hallucination via attention\-guided ensemble decoding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Deitke, C\. Clark, S\. Lee, R\. Tripathi, Y\. Yang, J\. S\. Park, M\. Salehi, N\. Muennighoff, K\. Lo, L\. Soldaini, J\. Lu, T\. Anderson, E\. Bransom, K\. Ehsani, H\. Ngo, Y\. Chen, A\. Patel, M\. Yatskar, C\. Callison\-Burch, A\. Head, R\. Hendrix, F\. Bastani, E\. VanderBilt, N\. Lambert, Y\. Chou, A\. Chheda, J\. Sparks, S\. Skjonsberg, M\. Schmitz, A\. Sarnat, B\. Bischoff, P\. Walsh, C\. Newell, P\. Wolters, T\. Gupta, K\. Zeng, J\. Borchardt, D\. Groeneveld, J\. Dumas, C\. Nam, S\. Lebrecht, C\. Wittlif, C\. Schoenick, O\. Michel, R\. Krishna, L\. Weihs, N\. A\. Smith, H\. Hajishirzi, R\. Girshick, A\. Farhadi, and A\. Kembhavi \(2024\)Molmo and pixmo: open weights and open data for state\-of\-the\-art multimodal models\.arXiv preprint arXiv:2409\.17146\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby \(2020\)An image is worth 16x16 words: transformers for image recognition at scale\.arXiv preprint arXiv:2010\.11929\.Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Favero, L\. Zancato, M\. Trager, S\. Choudhary, P\. Perera, A\. Achille, A\. Swaminathan, and S\. Soatto \(2024\)Multi\-modal hallucination control by visual information grounding\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- N\. M\. Guerreiro, E\. Voita, and A\. F\. T\. Martins \(2023\)Looking for a needle in a haystack: a comprehensive study of hallucinations in neural machine translation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,Note:arXiv:2208\.05309Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- K\. He, X\. Chen, S\. Xie, Y\. Li, P\. Dollár, and R\. Girshick \(2022\)Masked autoencoders are scalable vision learners\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Hu, P\. Wu, F\. Pu, W\. Xiao, Y\. Zhang, X\. Yue, B\. Li, and Z\. Liu \(2025\)Video\-mmmu: evaluating knowledge acquisition from multi\-discipline professional videos\.arXiv preprint arXiv:2501\.13826\.External Links:[Link](https://arxiv.org/abs/2501.13826)Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- Q\. Huang, X\. Dong, P\. Zhang, B\. Wang, C\. He, J\. Wang, D\. Lin, W\. Zhang, and N\. Yu \(2024\)OPERA: alleviating hallucination in multi\-modal large language models via over\-trust penalty and retrospection\-allocation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- F\. Huo, W\. Xu, Z\. Zhang, H\. Wang, Z\. Chen, and P\. Zhao \(2025\)Self\-introspective decoding: alleviating hallucinations for large vision\-language models\.arXiv preprint arXiv:2408\.02032\.Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Jiang, H\. Xu, M\. Dong, J\. Chen, W\. Ye, M\. Yan, Q\. Ye, J\. Zhang, F\. Huang, and S\. Zhang \(2024\)Hallucination augmented contrastive learning for multimodal large language model\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Khoury and D\. Hadfield\-Menell \(2018\)On the geometry of adversarial examples\.arXiv preprint arXiv:1811\.00525\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p2.1)\.
- J\. Kim, H\. Kim, Y\. Kim, and Y\. M\. Ro \(2024\)CODE: contrasting self\-generated description to combat hallucination in large multi\-modal models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2406\.01920External Links:[Link](https://arxiv.org/abs/2406.01920)Cited by:[§3\.1](https://arxiv.org/html/2606.09859#S3.SS1.p2.2),[§5\.1](https://arxiv.org/html/2606.09859#S5.SS1.SSS0.Px1.p1.1)\.
- S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton \(2019\)Similarity of neural network representations revisited\.InProceedings of the International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Lee, Y\. Tsai, and W\. Chiu \(2024\)Delve into visual contrastive decoding for hallucination mitigation of large vision\-language models\.arXiv preprint arXiv:2412\.06775\.Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Leng, H\. Zhang, G\. Chen, X\. Li, S\. Lu, C\. Miao, and L\. Bing \(2024\)Mitigating object hallucinations in large vision\-language models through visual contrastive decoding\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.09859#S3.SS1.p2.2),[§3\.3](https://arxiv.org/html/2606.09859#S3.SS3.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2606.09859#S3.SS3.SSS0.Px2.p3.4),[§5\.1](https://arxiv.org/html/2606.09859#S5.SS1.SSS0.Px1.p1.1),[§5\.6](https://arxiv.org/html/2606.09859#S5.SS6.p1.1)\.
- C\. Li, H\. Farkhoor, R\. Liu, and J\. Yosinski \(2018\)Measuring the intrinsic dimension of objective landscapes\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- X\. L\. Li, A\. Holtzman, D\. Fried, P\. Liang, J\. Eisner, T\. Hashimoto, L\. Zettlemoyer, and M\. Lewis \(2023a\)Contrastive decoding: open\-ended text generation as optimization\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Note:arXiv:2210\.15097Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen \(2023b\)Evaluating object hallucination in large vision\-language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://aclanthology.org/2023.emnlp-main.155/)Cited by:[3rd item](https://arxiv.org/html/2606.09859#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen \(2023c\)Evaluating object hallucination in large vision\-language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://aclanthology.org/2023.emnlp-main.155/)Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- F\. Liu, K\. Lin, L\. Li, J\. Wang, Y\. Yacoob, and L\. Wang \(2024a\)Mitigating hallucination in large multi\-modal models via robust instruction tuning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024b\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- A\. S\. Morcos, M\. Raghu, and S\. Bengio \(2018\)Insights on representational similarity in neural networks with canonical correlation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Park, D\. Lee, J\. Choe, and B\. Chang \(2025\)ConVis: contrastive decoding with hallucination visualization for mitigating hallucinations in multimodal large language models\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Raghu, J\. Gilmer, J\. Yosinski, and J\. Sohl\-Dickstein \(2017\)SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Rohrbach, L\. A\. Hendricks, K\. Burns, T\. Darrell, and K\. Saenko \(2018\)Object hallucination in image captioning\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Cited by:[3rd item](https://arxiv.org/html/2606.09859#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- S\. T\. Roweis and L\. K\. Saul \(2000\)Nonlinear dimensionality reduction by locally linear embedding\.Science290\(5500\),pp\. 2323–2326\.Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2606.09859#S3.SS2.p1.2)\.
- D\. Stutz, M\. Hein, and B\. Schiele \(2019\)Disentangling adversarial robustness and generalization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Note:arXiv:1812\.00740Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p2.1)\.
- J\. Su, J\. Chen, H\. Li, Y\. Chen, L\. Qing, and Z\. Zhang \(2025\)Activation steering decoding: mitigating hallucination in large vision\-language models through bidirectional hidden state intervention\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 12964–12974\.External Links:[Link](https://aclanthology.org/2025.acl-long.634/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.634),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Sun, S\. Shen, S\. Cao, H\. Liu, C\. Li, Y\. Shen, C\. Gan, L\. Gui, Y\. Wang, Y\. Yang,et al\.\(2023\)Aligning large multimodal models with factually augmented RLHF\.arXiv preprint arXiv:2309\.14525\.Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- J\. B\. Tenenbaum, V\. de Silva, and J\. C\. Langford \(2000\)A global geometric framework for nonlinear dimensionality reduction\.Science290\(5500\),pp\. 2319–2323\.Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2606.09859#S3.SS2.p1.2)\.
- S\. Tong, Z\. Liu, Y\. Zhai, Y\. Ma, Y\. LeCun, and S\. Xie \(2024\)Eyes wide shut? exploring the visual shortcomings of multimodal LLMs\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Note:arXiv:2401\.06209Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- V\. Verma, A\. Lamb, C\. Beckham, A\. Najafi, I\. Mitliagkas, D\. Lopez\-Paz, and Y\. Bengio \(2019\)Manifold mixup: better representations by interpolating hidden states\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.97,pp\. 6438–6447\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p2.1)\.
- C\. Wang, X\. Chen, N\. Zhang, B\. Tian, H\. Xu, S\. Deng, and H\. Chen \(2024a\)MLLM can see? dynamic correction decoding for hallucination mitigation\.External Links:2410\.11779,[Link](https://arxiv.org/abs/2410.11779)Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, Y\. Fan, K\. Dang, M\. Du, X\. Ren, R\. Men, D\. Liu, C\. Zhou, J\. Zhou, and J\. Lin \(2024b\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- X\. Wang, J\. Pan, L\. Ding, and C\. Biemann \(2024c\)Mitigating hallucinations in large vision\-language models with instruction contrastive decoding\.InFindings of the Association for Computational Linguistics \(Findings of ACL\),Note:arXiv:2403\.18715Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1),[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.09859#S3.SS1.p2.2),[§5\.1](https://arxiv.org/html/2606.09859#S5.SS1.SSS0.Px1.p1.1),[§5\.6](https://arxiv.org/html/2606.09859#S5.SS6.p1.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen \(2024\)MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of CVPR,Cited by:[§1](https://arxiv.org/html/2606.09859#S1.p1.1)\.
- Z\. Zhao, B\. Wang, L\. Ouyang, X\. Dong, J\. Wang, and C\. He \(2023\)Beyond hallucinations: enhancing lvlms through hallucination\-aware direct preference optimization\.External Links:2311\.16839Cited by:[§2](https://arxiv.org/html/2606.09859#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Zheng, J\. Qian, J\. Tang, and S\. Yang \(2025\)Why lvlms are more prone to hallucinations in longer responses: the role of context\.InInternational Conference on Computer Vision \(ICCV\),Cited by:[§5\.1](https://arxiv.org/html/2606.09859#S5.SS1.SSS0.Px1.p1.1),[§5\.6](https://arxiv.org/html/2606.09859#S5.SS6.p1.1)\.

## Appendix AHyper\-parameter Sensitivity Analysis

In this section, we investigate the sensitivity of our proposed MGAP to the hyper\-parametersKKandNN\. The experiments are conducted on the LLaVA v1\.5 7B backbone using the POPE benchmark\. Table[8](https://arxiv.org/html/2606.09859#A1.T8)reports the Accuracy, Precision, and F1 scores across the Random, Popular, and Adversarial evaluation splits under different settings ofK∈\{1,3,5,10\}K\\in\\\{1,3,5,10\\\}andN∈\{10,50,100,200\}N\\in\\\{10,50,100,200\\\}\.

Remarkably, as shown in Table[8](https://arxiv.org/html/2606.09859#A1.T8), MGAP demonstrates exceptional robustness to hyper\-parameter variations\. The performance metrics remain highly stable across all tested values ofKKandNN, indicating that our method does not rely on meticulous parameter tuning to achieve superior hallucination mitigation\. Specifically, increasingKKfrom 1 to 5 brings slight improvements, particularly on the more challenging Adversarial split, while further increasingKKto 10 yields no additional gains\. Similarly, regarding the parameterNN, the performance reaches its peak aroundN=50N=50\. ExpandingNNbeyond 50 does not significantly improve the generation quality but inevitably introduces additional computational overhead\. Therefore, to achieve an optimal balance between anti\-hallucination performance and computational efficiency, we adoptK=5K=5andN=50N=50as the default configuration for all main experiments\.

Table 8:Sensitivity analysis of hyper\-parametersKKandNNon the POPE benchmark using LLaVA v1\.5 7B\. Metrics are reported across Random \(R\), Popular \(P\), and Adversarial \(A\) splits in the format of R / P / A\. The best result for each individual split within theKKandNNblocks is marked inbold\.
## Appendix BTheoretical Properties of MGAP

This appendix provides several theoretical properties of MGAP that motivate its*geometry\-aware*and*stable*behavior during decoding\. We emphasize that MGAP performs a*low\-rank*intervention confined to a learned language\-prior subspace, and the overall correction magnitude is provably bounded\. These results help explain why MGAP avoids the “global translation” effect that may cause manifold departure in linear suppression methods\.

### B\.1Notation and Setup

LetV∈ℝd×KV\\in\\mathbb\{R\}^\{d\\times K\}denote the language prior basis with orthonormal columns, i\.e\.,V⊤​V=IV^\{\\top\}V=I\. Define the orthogonal projection onto the prior subspace as

P:=V​V⊤\.P:=VV^\{\\top\}\.\(19\)At decoding steptt, MGAP forms the prior projection

hproj:=P​horig,h\_\{\\mathrm\{proj\}\}:=Ph\_\{\\mathrm\{orig\}\},\(20\)and applies the adaptive correction

hcut:=horig−β​hproj,whereβ:=α⋅g⋅p\.h\_\{\\mathrm\{cut\}\}:=h\_\{\\mathrm\{orig\}\}\-\\beta\\,h\_\{\\mathrm\{proj\}\},\\quad\\text\{where\}\\quad\\beta:=\\alpha\\cdot g\\cdot p\.\(21\)Hereggis the uncertainty gate \(computed from entropy\) andppis the consistency\-based protection factor:

g:=tanh⁡\(λ​H\),p:=1−cos⁡\(horig,hproj\)\.g:=\\tanh\(\\lambda H\),\\qquad p:=1\-\\cos\(h\_\{\\mathrm\{orig\}\},h\_\{\\mathrm\{proj\}\}\)\.\(22\)For brevity, we analyze one decoding step and omit the time index\.

### B\.2Property I: MGAP Moves Toward a Ground\-truth\-Aligned State Under Prior\-Aligned Error

#### Intuition

The previous property ensures MGAP does not over\-correct, but it does not explain*when*the correction is beneficial\. This property states a sufficient condition under which subtracting the prior projection reduces the distance to a ground\-truth\-aligned reference state\. Concretely, if the current error is positively aligned with the prior component, then removing \(part of\) that component yields a strict improvement\.

###### Theorem B\.1\(Ground\-truth\-directed correction under prior\-aligned error\)\.

Lethgth\_\{\\mathrm\{gt\}\}be a reference hidden state aligned with the ground\-truth continuation \(e\.g\., obtained by teacher forcing\), and define the errore:=horig−hgte:=h\_\{\\mathrm\{orig\}\}\-h\_\{\\mathrm\{gt\}\}\. Lethproj=P​horigh\_\{\\mathrm\{proj\}\}=Ph\_\{\\mathrm\{orig\}\}andhcut=horig−β​hprojh\_\{\\mathrm\{cut\}\}=h\_\{\\mathrm\{orig\}\}\-\\beta h\_\{\\mathrm\{proj\}\}withβ≥0\\beta\\geq 0\. If⟨e,hproj⟩\>0\\langle e,\\,h\_\{\\mathrm\{proj\}\}\\rangle\>0, then for any

0<β<2​⟨e,hproj⟩‖hproj‖2,0<\\beta<\\frac\{2\\langle e,h\_\{\\mathrm\{proj\}\}\\rangle\}\{\\\|h\_\{\\mathrm\{proj\}\}\\\|^\{2\}\},the update strictly decreases the squared distance tohgth\_\{\\mathrm\{gt\}\}:

‖hcut−hgt‖2<‖horig−hgt‖2\.\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{gt\}\}\\\|^\{2\}<\\\|h\_\{\\mathrm\{orig\}\}\-h\_\{\\mathrm\{gt\}\}\\\|^\{2\}\.Moreover, the optimal step size \(minimizing‖hcut−hgt‖2\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{gt\}\}\\\|^\{2\}overβ≥0\\beta\\geq 0\) is

β⋆=⟨e,hproj⟩‖hproj‖2\.\\beta^\{\\star\}=\\frac\{\\langle e,h\_\{\\mathrm\{proj\}\}\\rangle\}\{\\\|h\_\{\\mathrm\{proj\}\}\\\|^\{2\}\}\.

###### Proof\.

We expand the squared distance:

‖hcut−hgt‖2=‖horig−β​hproj−hgt‖2=‖e−β​hproj‖2=‖e‖2−2​β​⟨e,hproj⟩\+β2​‖hproj‖2\.\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{gt\}\}\\\|^\{2\}=\\\|h\_\{\\mathrm\{orig\}\}\-\\beta h\_\{\\mathrm\{proj\}\}\-h\_\{\\mathrm\{gt\}\}\\\|^\{2\}=\\\|e\-\\beta h\_\{\\mathrm\{proj\}\}\\\|^\{2\}=\\\|e\\\|^\{2\}\-2\\beta\\langle e,h\_\{\\mathrm\{proj\}\}\\rangle\+\\beta^\{2\}\\\|h\_\{\\mathrm\{proj\}\}\\\|^\{2\}\.If⟨e,hproj⟩\>0\\langle e,h\_\{\\mathrm\{proj\}\}\\rangle\>0, the quadratic is strictly smaller than‖e‖2\\\|e\\\|^\{2\}whenever

−2​β​⟨e,hproj⟩\+β2​‖hproj‖2<0⇔0<β<2​⟨e,hproj⟩‖hproj‖2\.\-2\\beta\\langle e,h\_\{\\mathrm\{proj\}\}\\rangle\+\\beta^\{2\}\\\|h\_\{\\mathrm\{proj\}\}\\\|^\{2\}<0\\quad\\Leftrightarrow\\quad 0<\\beta<\\frac\{2\\langle e,h\_\{\\mathrm\{proj\}\}\\rangle\}\{\\\|h\_\{\\mathrm\{proj\}\}\\\|^\{2\}\}\.The minimizer overβ≥0\\beta\\geq 0follows by setting the derivative to zero, yieldingβ⋆=⟨e,hproj⟩/‖hproj‖2\\beta^\{\\star\}=\\langle e,h\_\{\\mathrm\{proj\}\}\\rangle/\\\|h\_\{\\mathrm\{proj\}\}\\\|^\{2\}\. ∎

### B\.3Property II: MGAP Has a Bounded Step Size

#### Intuition

Decoding\-time interventions can be brittle if the correction magnitude becomes too large\. MGAP prevents this by making the effective coefficientβ=α⋅Gate⋅Protection\\beta=\\alpha\\cdot\\mathrm\{Gate\}\\cdot\\mathrm\{Protection\}automatically bounded, which in turn bounds the update step‖hcut−horig‖\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{orig\}\}\\\|\. This yields a conservative, non\-explosive intervention in a single forward pass\.

###### Theorem B\.2\(Bounded coefficient and bounded step\)\.

AssumeV⊤​V=IV^\{\\top\}V=Iandhorig≠0h\_\{\\mathrm\{orig\}\}\\neq 0\. LetP=V​V⊤P=VV^\{\\top\},hproj=P​horigh\_\{\\mathrm\{proj\}\}=Ph\_\{\\mathrm\{orig\}\}, and define

Gate=tanh⁡\(λ​H\),Protection=1−cos⁡\(horig,hproj\),β=α⋅Gate⋅Protection\.\\mathrm\{Gate\}=\\tanh\(\\lambda H\),\\quad\\mathrm\{Protection\}=1\-\\cos\(h\_\{\\mathrm\{orig\}\},h\_\{\\mathrm\{proj\}\}\),\\quad\\beta=\\alpha\\cdot\\mathrm\{Gate\}\\cdot\\mathrm\{Protection\}\.Then

0≤Gate<1,0≤Protection≤1,0≤β<α\.0\\leq\\mathrm\{Gate\}<1,\\qquad 0\\leq\\mathrm\{Protection\}\\leq 1,\\qquad 0\\leq\\beta<\\alpha\.Moreover, the MGAP updatehcut=horig−β​hprojh\_\{\\mathrm\{cut\}\}=h\_\{\\mathrm\{orig\}\}\-\\beta h\_\{\\mathrm\{proj\}\}satisfies

‖hcut−horig‖=β​‖hproj‖≤α​‖horig‖\.\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{orig\}\}\\\|=\\beta\\\|h\_\{\\mathrm\{proj\}\}\\\|\\leq\\alpha\\\|h\_\{\\mathrm\{orig\}\}\\\|\.

###### Proof\.

SinceH≥0H\\geq 0andλ\>0\\lambda\>0, we havetanh⁡\(λ​H\)∈\[0,1\)\\tanh\(\\lambda H\)\\in\[0,1\), hence0≤Gate<10\\leq\\mathrm\{Gate\}<1\.

Becausehproj=P​horigh\_\{\\mathrm\{proj\}\}=Ph\_\{\\mathrm\{orig\}\}is an orthogonal projection, we can writehorig=hproj\+\(I−P\)​horigh\_\{\\mathrm\{orig\}\}=h\_\{\\mathrm\{proj\}\}\+\(I\-P\)h\_\{\\mathrm\{orig\}\}withhproj⟂\(I−P\)​horigh\_\{\\mathrm\{proj\}\}\\perp\(I\-P\)h\_\{\\mathrm\{orig\}\}\. Therefore,

cos⁡\(horig,hproj\)=⟨horig,hproj⟩‖horig‖​‖hproj‖=‖hproj‖2‖horig‖​‖hproj‖=‖hproj‖‖horig‖∈\[0,1\],\\cos\(h\_\{\\mathrm\{orig\}\},h\_\{\\mathrm\{proj\}\}\)=\\frac\{\\langle h\_\{\\mathrm\{orig\}\},h\_\{\\mathrm\{proj\}\}\\rangle\}\{\\\|h\_\{\\mathrm\{orig\}\}\\\|\\\|h\_\{\\mathrm\{proj\}\}\\\|\}=\\frac\{\\\|h\_\{\\mathrm\{proj\}\}\\\|^\{2\}\}\{\\\|h\_\{\\mathrm\{orig\}\}\\\|\\\|h\_\{\\mathrm\{proj\}\}\\\|\}=\\frac\{\\\|h\_\{\\mathrm\{proj\}\}\\\|\}\{\\\|h\_\{\\mathrm\{orig\}\}\\\|\}\\in\[0,1\],which impliesProtection∈\[0,1\]\\mathrm\{Protection\}\\in\[0,1\]and hence0≤β<α0\\leq\\beta<\\alpha\.

Finally,hcut−horig=−β​hprojh\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{orig\}\}=\-\\beta h\_\{\\mathrm\{proj\}\}so‖hcut−horig‖=β​‖hproj‖\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{orig\}\}\\\|=\\beta\\\|h\_\{\\mathrm\{proj\}\}\\\|\. SincePPis non\-expansive,‖hproj‖=‖P​horig‖≤‖horig‖\\\|h\_\{\\mathrm\{proj\}\}\\\|=\\\|Ph\_\{\\mathrm\{orig\}\}\\\|\\leq\\\|h\_\{\\mathrm\{orig\}\}\\\|, which yields‖hcut−horig‖≤α​‖horig‖\\\|h\_\{\\mathrm\{cut\}\}\-h\_\{\\mathrm\{orig\}\}\\\|\\leq\\alpha\\\|h\_\{\\mathrm\{orig\}\}\\\|\. ∎

### B\.4Property III: MGAP Modifies Only the Prior Subspace

#### Intuition

A core design goal of MGAP is to avoid perturbing semantic directions that are*not*attributable to language priors\. Since the correction is applied via the projectionhprojh\_\{\\mathrm\{proj\}\}that lies entirely in the prior subspace, the update should leave all components orthogonal to this subspace unchanged\. The following theorem formalizes this subspace\-selective behavior\.

###### Theorem B\.3\(Subspace\-selective update\)\.

LetVprior∈ℝd×KV\_\{\\mathrm\{prior\}\}\\in\\mathbb\{R\}^\{d\\times K\}have orthonormal columns \(Vprior⊤​Vprior=IKV\_\{\\mathrm\{prior\}\}^\{\\top\}V\_\{\\mathrm\{prior\}\}=I\_\{K\}\) and defineP:=Vprior​Vprior⊤∈ℝd×dP:=V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}\\in\\mathbb\{R\}^\{d\\times d\}\. Decomposehorigh\_\{\\mathrm\{orig\}\}into its subspace and orthogonal components:

horig=h∥\+h⟂,h∥:=P​horig,h⟂:=\(Id−P\)​horig,h\_\{\\mathrm\{orig\}\}=h\_\{\\parallel\}\+h\_\{\\perp\},\\qquad h\_\{\\parallel\}:=Ph\_\{\\mathrm\{orig\}\},\\;\\;h\_\{\\perp\}:=\(I\_\{d\}\-P\)h\_\{\\mathrm\{orig\}\},whereIdI\_\{d\}is thed×dd\\times didentity matrix\. Then MGAP yields

hcut=\(1−β\)​h∥\+h⟂\.h\_\{\\mathrm\{cut\}\}=\(1\-\\beta\)\\,h\_\{\\parallel\}\+h\_\{\\perp\}\.In particular, the orthogonal component is preserved:

\(Id−P\)​hcut=\(Id−P\)​horig\.\(I\_\{d\}\-P\)h\_\{\\mathrm\{cut\}\}=\(I\_\{d\}\-P\)h\_\{\\mathrm\{orig\}\}\.

Equivalent form\.SinceP=Vprior​Vprior⊤P=V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}, we have

\(Id−P\)​h=h−P​h=h−Vprior​Vprior⊤​h\.\(I\_\{d\}\-P\)h=h\-Ph=h\-V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}h\.Therefore,

\(Id−P\)​hcut=\(Id−P\)​horig\(I\_\{d\}\-P\)h\_\{\\mathrm\{cut\}\}=\(I\_\{d\}\-P\)h\_\{\\mathrm\{orig\}\}is equivalent to

hcut−Vprior​Vprior⊤​hcut=horig−Vprior​Vprior⊤​horig\.h\_\{\\mathrm\{cut\}\}\-V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}h\_\{\\mathrm\{cut\}\}=h\_\{\\mathrm\{orig\}\}\-V\_\{\\mathrm\{prior\}\}V\_\{\\mathrm\{prior\}\}^\{\\top\}h\_\{\\mathrm\{orig\}\}\.\(23\)
###### Proof\.

By definition of projection,hproj=P​horig=h∥h\_\{\\mathrm\{proj\}\}=Ph\_\{\\mathrm\{orig\}\}=h\_\{\\parallel\}\. MGAP applieshcut=horig−β​hprojh\_\{\\mathrm\{cut\}\}=h\_\{\\mathrm\{orig\}\}\-\\beta h\_\{\\mathrm\{proj\}\}\. Substitutinghorig=h∥\+h⟂h\_\{\\mathrm\{orig\}\}=h\_\{\\parallel\}\+h\_\{\\perp\}andhproj=h∥h\_\{\\mathrm\{proj\}\}=h\_\{\\parallel\}gives

hcut=\(h∥\+h⟂\)−β​h∥=\(1−β\)​h∥\+h⟂\.h\_\{\\mathrm\{cut\}\}=\(h\_\{\\parallel\}\+h\_\{\\perp\}\)\-\\beta h\_\{\\parallel\}=\(1\-\\beta\)h\_\{\\parallel\}\+h\_\{\\perp\}\.Applying\(Id−P\)\(I\_\{d\}\-P\)and using\(Id−P\)​h∥=0\(I\_\{d\}\-P\)h\_\{\\parallel\}=0and\(Id−P\)​h⟂=h⟂\(I\_\{d\}\-P\)h\_\{\\perp\}=h\_\{\\perp\}yields\(Id−P\)​hcut=h⟂=\(Id−P\)​horig\(I\_\{d\}\-P\)h\_\{\\mathrm\{cut\}\}=h\_\{\\perp\}=\(I\_\{d\}\-P\)h\_\{\\mathrm\{orig\}\}\. ∎

## Appendix CMGAP Decoding Algorithm

Algorithm[1](https://arxiv.org/html/2606.09859#alg1)summarizes the decoding\-time procedure of MGAP\. Given an imagevv, a queryxx, and the precomputed prior basisVpriorV\_\{\\mathrm\{prior\}\}\(Section[4\.1](https://arxiv.org/html/2606.09859#S4.SS1)\), MGAP computes the prior projectionhprojh\_\{\\mathrm\{proj\}\}, estimates mismatchδ\\deltaand uncertaintyγ\\gamma, and applies the adaptive correction to obtainhcuth\_\{\\mathrm\{cut\}\}for token generation\.

Algorithm 1MGAP Decoding0:Image

v\{v\}, query

xx, prior basis

Vprior\{V\}\_\{\\mathrm\{prior\}\}
0:Generated response

y=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\)
1:Initialize

y0←⟨BOS⟩y\_\{0\}\\leftarrow\\langle\\mathrm\{BOS\}\\rangle
2:for

t=1t=1to

TTdo

3:Compute hidden state

horigh\_\{\\mathrm\{orig\}\}
4:

hproj←𝐕prior​𝐕prior⊤​horigh\_\{\\mathrm\{proj\}\}\\leftarrow\\mathbf\{V\}\_\{\\mathrm\{prior\}\}\\mathbf\{V\}\_\{\\mathrm\{prior\}\}^\{\\top\}h\_\{\\mathrm\{orig\}\}
5:

δ←1−cos⁡\(horig,hproj\)\\delta\\leftarrow 1\-\\cos\(h\_\{\\mathrm\{orig\}\},h\_\{\\mathrm\{proj\}\}\)
6:

p​\(⋅\)←Softmax​\(ℱ​\(horig\)\)p\(\\cdot\)\\leftarrow\\mathrm\{Softmax\}\(\\mathcal\{F\}\(h\_\{\\mathrm\{orig\}\}\)\)
7:

H←−∑yp​\(y\)​log⁡p​\(y\)H\\leftarrow\-\\sum\_\{y\}p\(y\)\\log p\(y\)
8:

γ←tanh⁡\(λ​H\)\\gamma\\leftarrow\\tanh\(\\lambda H\)
9:

hcut←horig−α⋅γ⋅δ⋅hprojh\_\{\\mathrm\{cut\}\}\\leftarrow h\_\{\\mathrm\{orig\}\}\-\\alpha\\cdot\\gamma\\cdot\\delta\\cdot h\_\{\\mathrm\{proj\}\}
10:Generate

yt∼Softmax​\(ℱ​\(hcut\)\)y\_\{t\}\\sim\\mathrm\{Softmax\}\(\\mathcal\{F\}\(h\_\{\\mathrm\{cut\}\}\)\)
11:endfor

12:return

yy

## Appendix DHyper\-parameter Sensitivity

We study the sensitivity of MGAP to two key hyper\-parameters: the projection scaleα\\alphaand the uncertainty\-gating temperatureλ\\lambda\(LLaVA v1\.5\-7B on POPE\)\. Overall, MGAP is stable across a broad range of settings and exhibits a clear monotonic trend on the most challenging split\.

Effect ofα\\alphaIncreasingα\\alphaconsistently improves*Precision*on all splits, and yields monotonic gains on the Adversarial split \(F1: 82\.05→\\rightarrow84\.00 whenα\\alphaincreases from 0 to 1\.0\)\. On Random/Popular, moderateα\\alpha\(around 0\.5–0\.75\) gives slightly higher F1, while largerα\\alphafurther boosts Precision with a small F1 trade\-off, indicating a typical precision–recall shift\.

Table 9:Sensitivity to projection scaleα\\alphaon POPE \(LLaVA v1\.5\-7B\)\.Effect ofλ\\lambdaVaryingλ\\lambdashows that gating is not brittle: performance remains competitive forλ∈\[0\.25,1\.0\]\\lambda\\in\[0\.25,1\.0\]\. In particular, Adversarial improves substantially overλ=0\\lambda=0and peaks aroundλ=0\.5\\lambda=0\.5\(Acc: 80\.16→\\rightarrow84\.59; F1: 82\.05→\\rightarrow84\.46\), suggesting that moderate uncertainty amplification best targets conflict cases\. Very large gating can slightly over\-suppress on some splits \(e\.g\., Popular F1 drops atλ=0\.75\\lambda=0\.75\), but the overall trend remains stable\.

Table 10:Sensitivity to gating temperatureλ\\lambdaon POPE \(LLaVA v1\.5\-7B\)\.
## Appendix EComputational Complexity of MGAP

### E\.1Notation

Letdddenote the hidden dimension andKKthe prior subspace rank withK≪dK\\ll d\. LetNNdenote the number of blind hidden states used to construct the prior basis\. LetV∈ℝd×KV\\in\\mathbb\{R\}^\{d\\times K\}be the prior basis with orthonormal columns,

V⊤​V=IK\.V^\{\\top\}V=I\_\{K\}\.\(24\)Define the orthogonal projector

P:=V​V⊤∈ℝd×d\.P:=VV^\{\\top\}\\in\\mathbb\{R\}^\{d\\times d\}\.\(25\)At each decoding step, MGAP computes

hproj=P​h=V​V⊤​h,h\_\{\\mathrm\{proj\}\}=Ph=VV^\{\\top\}h,\(26\)and performs the update

hcut=h−β​hproj,h\_\{\\mathrm\{cut\}\}=h\-\\beta h\_\{\\mathrm\{proj\}\},\(27\)where

β=α⋅g⋅p\.\\beta=\\alpha\\cdot g\\cdot p\.\(28\)Hereg=tanh⁡\(λ​H\)g=\\tanh\(\\lambda H\)andp=1−cos⁡\(h,hproj\)p=1\-\\cos\(h,h\_\{\\mathrm\{proj\}\}\)are scalar factors\.

### E\.2Online \(Per\-token\) Complexity

###### Proposition E\.1\(MGAP per\-step time complexity\)\.

GivenV∈ℝd×KV\\in\\mathbb\{R\}^\{d\\times K\}, the additional time complexity incurred by MGAP at a decoding step is

TMGAP​\(d,K\)=𝒪​\(d​K\)\.T\_\{\\text\{MGAP\}\}\(d,K\)=\\mathcal\{O\}\(dK\)\.\(29\)

###### Proof\.

Computeu=V⊤​h∈ℝKu=V^\{\\top\}h\\in\\mathbb\{R\}^\{K\}:

This is a matrix–vector multiply of sizeK×dK\\times d, costing

𝒪​\(d​K\)\.\\mathcal\{O\}\(dK\)\.\(31\)Compute the projectionhproj=V​u∈ℝdh\_\{\\mathrm\{proj\}\}=Vu\\in\\mathbb\{R\}^\{d\}:

hproj=V​u\.h\_\{\\mathrm\{proj\}\}=Vu\.\(32\)This is a matrix–vector multiply of sized×Kd\\times K, costing

𝒪​\(d​K\)\.\\mathcal\{O\}\(dK\)\.\(33\)Compute cosine similarity requires an inner product and two norms:

⟨h,hproj⟩,‖h‖2,‖hproj‖2,\\langle h,h\_\{\\mathrm\{proj\}\}\\rangle,\\;\\;\\\|h\\\|\_\{2\},\\;\\;\\\|h\_\{\\mathrm\{proj\}\}\\\|\_\{2\},\(34\)which costs

𝒪​\(d\)\.\\mathcal\{O\}\(d\)\.\(35\)Compute the gated scalingβ=α​g​p\\beta=\\alpha gpis scalar\-only:

β=α​g​p,\\beta=\\alpha gp,\(36\)which costs

𝒪​\(1\)\.\\mathcal\{O\}\(1\)\.\(37\)Finally compute the residual update:

hcut=h−β​hproj,h\_\{\\mathrm\{cut\}\}=h\-\\beta h\_\{\\mathrm\{proj\}\},\(38\)which costs

𝒪​\(d\)\.\\mathcal\{O\}\(d\)\.\(39\)Thus the dominant cost is twod×Kd\\times K/K×dK\\times dmatrix–vector multiplies, yielding

TMGAP​\(d,K\)=𝒪​\(d​K\)\.T\_\{\\text\{MGAP\}\}\(d,K\)=\\mathcal\{O\}\(dK\)\.\(40\)∎

###### Proposition E\.2\(MGAP additional memory\)\.

The additional memory required by MGAP \(beyond the base model\) is

MMGAP​\(d,K\)=𝒪​\(d​K\),M\_\{\\text\{MGAP\}\}\(d,K\)=\\mathcal\{O\}\(dK\),\(41\)plus𝒪​\(d\)\\mathcal\{O\}\(d\)temporary vectors per decoding step\.

###### Proof\.

MGAP stores the prior basisV∈ℝd×KV\\in\\mathbb\{R\}^\{d\\times K\}, which takes

𝒪​\(d​K\)\\mathcal\{O\}\(dK\)\(42\)floating\-point numbers\. At runtime, MGAP may allocate temporary vectorsu∈ℝKu\\in\\mathbb\{R\}^\{K\}andhproj∈ℝdh\_\{\\mathrm\{proj\}\}\\in\\mathbb\{R\}^\{d\}, costing

𝒪​\(K\)\+𝒪​\(d\)=𝒪​\(d\)\.\\mathcal\{O\}\(K\)\+\\mathcal\{O\}\(d\)=\\mathcal\{O\}\(d\)\.\(43\)Therefore the asymptotic additional storage is dominated byVV:

MMGAP​\(d,K\)=𝒪​\(d​K\)\.M\_\{\\text\{MGAP\}\}\(d,K\)=\\mathcal\{O\}\(dK\)\.\(44\)∎

## Appendix FOffline Subspace Construction Complexity

LetHblind∈ℝN×dH\_\{\\mathrm\{blind\}\}\\in\\mathbb\{R\}^\{N\\times d\}be the centered blind\-state matrix:

Hblind=\[\(hblind\(1\)−h¯\)⊤⋮\(hblind\(N\)−h¯\)⊤\]\.H\_\{\\mathrm\{blind\}\}=\\begin\{bmatrix\}\(h^\{\(1\)\}\_\{\\mathrm\{blind\}\}\-\\bar\{h\}\)^\{\\top\}\\\\ \\vdots\\\\ \(h^\{\(N\)\}\_\{\\mathrm\{blind\}\}\-\\bar\{h\}\)^\{\\top\}\\end\{bmatrix\}\.\(45\)MGAP computes the top\-KKright singular vectors:

Hblind=U​Σ​V⊤,H\_\{\\mathrm\{blind\}\}=U\\Sigma V^\{\\top\},\(46\)and retains

Vprior:=V\[:,1:K\]∈ℝd×K\.V\_\{\\mathrm\{prior\}\}:=V\_\{\[:,1:K\]\}\\in\\mathbb\{R\}^\{d\\times K\}\.\(47\)
#### Full SVD \(exact\)\.

###### Proposition F\.1\(Offline exact SVD complexity\)\.

Computing an exact SVD ofHblind∈ℝN×dH\_\{\\mathrm\{blind\}\}\\in\\mathbb\{R\}^\{N\\times d\}has time complexity

TSVD\-full​\(N,d\)=𝒪​\(min⁡\{N​d2,N2​d\}\),T\_\{\\text\{SVD\-full\}\}\(N,d\)=\\mathcal\{O\}\\\!\\left\(\\min\\\{Nd^\{2\},\\,N^\{2\}d\\\}\\right\),\(48\)and memory complexity

MSVD\-full​\(N,d\)=𝒪​\(N​d\)\.M\_\{\\text\{SVD\-full\}\}\(N,d\)=\\mathcal\{O\}\(Nd\)\.\(49\)

###### Proof\.

Standard dense SVD for anN×dN\\times dmatrix has the classical bound

𝒪​\(min⁡\{N​d2,N2​d\}\)\.\\mathcal\{O\}\\\!\\left\(\\min\\\{Nd^\{2\},\\,N^\{2\}d\\\}\\right\)\.\(50\)Storing the data matrix requiresN​dNdentries:

𝒪​\(N​d\)\.\\mathcal\{O\}\(Nd\)\.\(51\)∎

#### Truncated SVD \(iterative, exact up to tolerance\)\.

In practice, MGAP only needs the top\-KKright singular vectors\. This can be obtained via Krylov subspace methods \(e\.g\., Lanczos\) using repeated matrix–vector products\.

###### Proposition F\.2\(Offline truncated SVD complexity\)\.

AssumeK≪min⁡\{N,d\}K\\ll\\min\\\{N,d\\\}and the top\-KKsingular vectors are computed by an iterative method requiringTTiterations\. Then the time complexity is

TSVD\-trunc​\(N,d,K\)=𝒪​\(T⋅N​d\),T\_\{\\text\{SVD\-trunc\}\}\(N,d,K\)=\\mathcal\{O\}\\\!\\left\(T\\cdot Nd\\right\),\(52\)and the additional memory beyond storingHblindH\_\{\\mathrm\{blind\}\}is

MSVD\-trunc​\(N,d,K\)=𝒪​\(\(N\+d\)​K\)\.M\_\{\\text\{SVD\-trunc\}\}\(N,d,K\)=\\mathcal\{O\}\\\!\\left\(\(N\+d\)K\\right\)\.\(53\)

###### Proof\.

Each iteration of Krylov/Lanczos methods uses one or a constant number of multiplications byHblindH\_\{\\mathrm\{blind\}\}andHblind⊤H\_\{\\mathrm\{blind\}\}^\{\\top\}\. A matrix–vector multiply costs

𝒪​\(N​d\)\.\\mathcal\{O\}\(Nd\)\.\(54\)WithTTiterations, the total is

𝒪​\(T​N​d\)\.\\mathcal\{O\}\(TNd\)\.\(55\)The method stores𝒪​\(K\)\\mathcal\{O\}\(K\)basis vectors inℝN\\mathbb\{R\}^\{N\}and/orℝd\\mathbb\{R\}^\{d\}, giving

𝒪​\(N​K\)\+𝒪​\(d​K\)=𝒪​\(\(N\+d\)​K\)\.\\mathcal\{O\}\(NK\)\+\\mathcal\{O\}\(dK\)=\\mathcal\{O\}\(\(N\+d\)K\)\.\(56\)∎

#### Randomized SVD \(approximate\)\.

A common alternative is randomized SVD, which forms a sketch of sizeK\+pK\+pwith oversamplingpp\. Letqqdenote the number of power iterations\.

###### Proposition F\.3\(Offline randomized SVD complexity\)\.

Randomized SVD for extractingKKcomponents fromHblind∈ℝN×dH\_\{\\mathrm\{blind\}\}\\in\\mathbb\{R\}^\{N\\times d\}has time complexity

TSVD\-rand​\(N,d,K\)=𝒪​\(N​d​\(K\+p\)\+q⋅N​d​\(K\+p\)\),T\_\{\\text\{SVD\-rand\}\}\(N,d,K\)=\\mathcal\{O\}\\\!\\left\(Nd\(K\+p\)\+q\\cdot Nd\(K\+p\)\\right\),\(57\)and memory complexity

MSVD\-rand​\(N,d,K\)=𝒪​\(\(N\+d\)​\(K\+p\)\)\.M\_\{\\text\{SVD\-rand\}\}\(N,d,K\)=\\mathcal\{O\}\\\!\\left\(\(N\+d\)\(K\+p\)\\right\)\.\(58\)

###### Proof\.

Randomized SVD forms a sketch using multiplications byHblindH\_\{\\mathrm\{blind\}\}\(and optionally power iterations\)\. Each pass costs𝒪​\(N​d​\(K\+p\)\)\\mathcal\{O\}\(Nd\(K\+p\)\)for multiplying by a\(d×\(K\+p\)\)\(d\\times\(K\+p\)\)test matrix\. Withqqpower iterations, the number of passes scales by\(1\+q\)\(1\+q\), yielding

𝒪​\(\(1\+q\)⋅N​d​\(K\+p\)\)\.\\mathcal\{O\}\\\!\\left\(\(1\+q\)\\cdot Nd\(K\+p\)\\right\)\.\(59\)The memory stores the sketch bases inℝN\\mathbb\{R\}^\{N\}andℝd\\mathbb\{R\}^\{d\}:

𝒪​\(\(N\+d\)​\(K\+p\)\)\.\\mathcal\{O\}\(\(N\+d\)\(K\+p\)\)\.\(60\)∎

#### Comparison to Multi\-branch Contrastive Decoding

MGAP performs a single\-pass representation update and adds only𝒪​\(d​K\)\\mathcal\{O\}\(dK\)arithmetic per token\. In contrast, contrastive decoding baselines that requireBBforward passes per token incur a multiplicative overhead in the base model cost\. LetCmodelC\_\{\\text\{model\}\}denote the per\-token compute cost of the backbone model\. Then their total per\-token cost scales as

Tcontrastive=Θ​\(B⋅Cmodel\),T\_\{\\text\{contrastive\}\}=\\Theta\(B\\cdot C\_\{\\text\{model\}\}\),\(61\)whereas MGAP scales as

TMGAP\-total=Θ​\(Cmodel\)\+𝒪​\(d​K\)\.T\_\{\\text\{MGAP\-total\}\}=\\Theta\(C\_\{\\text\{model\}\}\)\+\\mathcal\{O\}\(dK\)\.\(62\)WhenK≪dK\\ll dand the backbone dominates compute,𝒪​\(d​K\)\\mathcal\{O\}\(dK\)is negligible relative toCmodelC\_\{\\text\{model\}\}\.