GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs

# Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs
Source: [https://arxiv.org/html/2507.18043](https://arxiv.org/html/2507.18043)
Duy Nguyen1Archiki Prasad1Elias Stengel\-Eskin2Mohit Bansal1 1UNC Chapel Hill2The University of Texas at Austin

###### Abstract

Inference\-time steering provides a lightweight alternative to fine\-tuning large language models \(LLMs\) and vision\-language models \(VLMs\) by modifying model activations without updating weights\. However, existing methods often rely on a global intervention vector, overlook token\-level influence, and underutilize model logits, especially in multimodal settings where visual and textual inputs contribute unevenly\. We proposeGrAInS, a contrastive, gradient\-based approach that leverages Integrated Gradients to identify top\-kkinfluential tokens and construct directional steering vectors based on their contribution to preferred over dispreferred outputs\. These vectors guide activation intervention at each layer, preserving the representational scale\.GrAInSoutperforms fine\-tuning and prior steering methods on both LLM and VLM tasks: improving TruthfulQA accuracy by 13\.22% \(Llama\-3\.1\-8B\), reducing MMHal\-Bench hallucinations from 0\.624 to 0\.514 \(LLaVA\-1\.6\-7B\), and increasing SPA\-VL alignment by 8\.11%, all without degrading fluency or general capabilities\.111Code:[https://github\.com/duykhuongnguyen/GrAInS](https://github.com/duykhuongnguyen/GrAInS)\.

GrAInS: Gradient\-based Attribution for Inference\-Time Steering of LLMs and VLMs

Duy Nguyen1Archiki Prasad1Elias Stengel\-Eskin2Mohit Bansal11UNC Chapel Hill2The University of Texas at Austin

## 1Introduction

![Refer to caption](https://arxiv.org/html/2507.18043v2/x1.png)Figure 1:Comparison of prior steering methods vs\.GrAInS, our attribution\-guided approach on VLMs\. Top: Existing methods suffer from some key limitations such as using only visual tokens, relying on external object detectors, or steering in a single fixed direction\. Bottom:GrAInSleverages both visual and textual tokens using contrastive Integrated Gradients, requires no external modules, and constructs targeted, directional interventions based on positive and negative attribution, leading to improved factual accuracy\.Despite having strong performance across various tasks, LLMs and VLMs often generate undesirable outputs that lack grounding in the input query or context\(Rameet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib225); Shiet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib224); Huanget al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib220)\)\. Fine\-tuning addresses these issues by adapting models with task\-specific datasets, but it requires significant computational resources and data, and risks catastrophic forgetting\(Li and Hoiem,[2017](https://arxiv.org/html/2507.18043#bib.bib251); Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2507.18043#bib.bib252)\)\. A promising alternative to fine\-tuning is inference\-time steering\(Zouet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib11); Liuet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib30); Liet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib15); Rimskyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib227); Turneret al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib228); Nguyenet al\.,[2025a](https://arxiv.org/html/2507.18043#bib.bib271)\), which adjusts hidden representations during inference without altering the model’s parameters\. However, existing steering approaches generally rely on linear interventions to hidden states, often applying the same intervention across all tokens’ hidden states\(Marks and Tegmark,[2023](https://arxiv.org/html/2507.18043#bib.bib13); Liet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib15)\), ignoring the impact of specific tokens on model behavior\. As illustrated in[Fig\.˜1](https://arxiv.org/html/2507.18043#S1.F1)\(top\), this can lead to overcorrection and loss of desired capabilities, such as fluency or factual accuracy\(Nguyenet al\.,[2025b](https://arxiv.org/html/2507.18043#bib.bib289)\)\. Moreover, most existing methods construct steering vectors solely from latent space representations of paired data by taking differences between hidden activations corresponding to desirable and undesirable outputs\(Liet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib15); Rimskyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib227); Turneret al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib228); Nguyenet al\.,[2025b](https://arxiv.org/html/2507.18043#bib.bib289)\), ignoring rich signals from model logits that reveal which specific inputs \(tokens\) most drive undesirable outputs through theirattribution\-based contributionto model predictions\. In VLMs, this limitation is especially problematic –textual and visual inputs do not contribute equally– some tokens play a key role in shaping the model’s output, while others have little to no influence\(Caoet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib316); Sunet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib315); Linet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib299)\)\. Thus, constructing steering vectors purely in latent space without identifying which tokens are responsible for undesirable behavior can be ineffective and may cause unintended changes to the model’s behavior\(Salinet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib290); Chenet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib291)\)\.

To address these issues, we proposeGradient\-basedAttribution forInference\-TimeSteering \(GrAInS\), a more selective and interpretable approach to inference\-time steering compatible with both LLMs and VLMs, as outlined in[Fig\.˜1](https://arxiv.org/html/2507.18043#S1.F1)\(bottom\) and shown in more detail in[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)\.GrAInSidentifies specific tokens—whether visual patches or language tokens—that have the greatestattribution\-based contributionto the model’s output, and appliessteering based on their contribution\. To measure this influence, we use Integrated Gradients \(IG\)\(Sundararajanet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib296); Kapishnikovet al\.,[2021](https://arxiv.org/html/2507.18043#bib.bib292)\)over a contrastive loss between preferred and dispreferred outputs to compute token\-level attributions \(see[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)\(A\)\)\. Tokens with high positive attribution are those most responsible for producing desirable outputs, while those with strong negative attribution contribute to undesirable behaviors such as hallucinations or toxicity\. We construct contrastive input variants by masking each token set separately and measuring changes in hidden activations\. These capture how each group influences internal representations, and we apply Principal Component Analysis \(PCA\) to derive a steering vector that represents behavior shifts in latent space \(see[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)\(B\)\)\. At inference, the steering vector is applied with normalization to preserve general model capabilities such as fluency and reasoning \(see[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)\(C\)\)\. Unlike prior work that operates with a single steering direction\(Rimskyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib227)\), relies solely on visual tokens\(Chenet al\.,[2024a](https://arxiv.org/html/2507.18043#bib.bib293)\), or requires token sampling \(which can introduce instability or require large sample sizes, making them computationally expensive\) and external modules\(Liuet al\.,[2024c](https://arxiv.org/html/2507.18043#bib.bib4); Chenet al\.,[2024a](https://arxiv.org/html/2507.18043#bib.bib293)\),GrAInSintegrates both visual and textual inputs, accounts for both positive and negative attribution directions, and introduces no additional components or supervision \(see[Fig\.˜1](https://arxiv.org/html/2507.18043#S1.F1)\)\. Moreover, while prior work has largely limited attribution methods like IG to post\-hoc explanation\(Linet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib299)\), we bridge the gap between interpretability and active model steering\. This enables more precise, token\-sensitive interventions, leading to improved alignment and interpretability in steering of both unimodal and multimodal LLMs\.

We evaluateGrAInSon safety\-critical tasks across both VLMs and LLMs, targeting hallucinations, bias, toxicity, and truthfulness, where it shows strong performance in both modalities \(vision and language\) without retraining\. In VLMs, we achieve a hallucination rate reduction from 0\.624 to 0\.514 on LLaVA\-1\.6\-7B and improve alignment preference win rates by 8\.11% on SPA\-VL, outperforming LoRA and multimodal steering methods such as VTI\(Liuet al\.,[2024c](https://arxiv.org/html/2507.18043#bib.bib4)\)\. In LLMs, we see similar strong gains: on TruthfulQA,GrAInSimproves factual accuracy by 13\.22% over the Llama\-3\.1\-8B\-Instruct model, outperforming ICV\(Liuet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib30)\)by a margin of 7\.7%\. On Toxigen, it improves the accuracy by over 9\.89% over the base model and 4\.10% over NL\-ITI\(Hoscilowiczet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib240)\)\. Moreover, because ofGrAInS’s localized nature, there is no major impact on the model’s general capabilities on other tasks\. When evaluating on broad\-coverage text and multimodal datasets like MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2507.18043#bib.bib180)\)and MMMU\(Yueet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib312)\), standard baselines hurt performance, whileGrAInSpreserves performance\. For example, CAA\(Rimskyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib227)\)drops Llama\-3\.1\-8B’s MMLU performance by 17\.78%, whileGrAInSis almost identical, with only a0\.12%0\.12\\%drop\)\. Similarly, CAA leads to a 17\.13% drop on MMMU for Qwen2\.5\-VL\-7B, whileGrAInShas only a 0\.51% drop\. These results highlight the strength of selective, attribution\-guided interventions for fine\-grained multimodal control without performance degradation\.

## 2Related work

Inference\-Time Steering\.Inference\-time intervention offers a lightweight alternative to fine\-tuning by modifying hidden activations without updating model weights\. In LLMs, methods like ITI\(Liet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib15)\), CAA\(Panicksseryet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib297)\), and MAT\-Steer\(Nguyenet al\.,[2025b](https://arxiv.org/html/2507.18043#bib.bib289)\)steer behavior using contrastive examples or attribute\-specific vectors\. For VLMs, prior work includes both modality\-specific and activation\-engineering approaches: VTI\(Liuet al\.,[2024c](https://arxiv.org/html/2507.18043#bib.bib4)\)and MLLM\-Steering\(Khayatanet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib1)\)largely treat vision and language separately, ICT\(Chenet al\.,[2024a](https://arxiv.org/html/2507.18043#bib.bib293)\)performs token\-level interventions but depends on object detectors and supervision, andSteerVLM\(Sivakumaret al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib338)\)introduces a lightweight learned steering module that dynamically adjusts hidden activations using paired target and converse prompts\.In contrast,GrAInSunifies steering across modalities by using gradient\-based attribution to identify influential visual and textual tokens and construct layer\-wise steering vectors directly from the input\. This avoids modality\-specific heuristics, learned auxiliary steering modules, and global interventions\(Liuet al\.,[2024c](https://arxiv.org/html/2507.18043#bib.bib4); Chenet al\.,[2024a](https://arxiv.org/html/2507.18043#bib.bib293); Rimskyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib227)\), enabling effective and interpretable control\.

Attribution and Interpretability\.Token\-level attribution methods are widely used to interpret the outputs of LLMs and VLMs\. Integrated Gradients \(IG\)\(Sundararajanet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib296)\), a foundational technique, estimates token contributions by integrating gradients from a baseline input\. Other gradient attribution methods such as SmoothGrad\(Smilkovet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib306)\)and Guided IG\(Kapishnikovet al\.,[2021](https://arxiv.org/html/2507.18043#bib.bib292)\)improve stability and reduce noise\. These methods have been applied to analyze attention and debug hallucinations\(Wuet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib308); Changet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib307); Yanget al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib298)\), but are typically limited to post\-hoc explanation\. AsLinet al\.\([2025](https://arxiv.org/html/2507.18043#bib.bib299)\)notes, interpretability tools rarely inform model control\. In this work, we bridge this gap by using gradient\-based attribution to guide intervention by identifying impactful tokens and computing contrastive, layer\-wise steering vectors, enabling input\-sensitive control without retraining\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2507.18043v2/x2.png)Figure 2:Overview of our attribution\-guided steering method for VLMs\. Our method consists of three stages: \(A\) Compute token\-level attributions using contrastive Integrated Gradients, identifying the most influential positive and negative tokens \(green/red\)\. \(B\) Construct contrastive inputs by masking these tokens, extract the corresponding hidden states, and apply PCA to obtain directional steering vectors\. \(C\) At inference time, inject these vectors into the model’s hidden states at each layer, scaled and normalized to preserve representation scale\.Here, we introduceGradient\-basedAttribution forInference\-TimeSteering \(GrAInS\), a steering approach that operates selectively on the most influential input tokens\. Our method consists of three steps: \(1\) identifying important tokens using contrastive attribution based on preference data, \(2\) constructing layer\-specific steering vectors from contrastive activations, and \(3\) applying selective and normalized interventions during inference\. We illustrateGrAInSin[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)and we describe each of its steps below\. Note that the steps in[Section˜3\.1](https://arxiv.org/html/2507.18043#S3.SS1)and[Section˜3\.2](https://arxiv.org/html/2507.18043#S3.SS2)are one\-time costs and are performed only once per steering objective\.

### 3\.1Token Attribution via Integrated Gradients

Objective\.We begin by identifying the most influential tokens with respect to a model’s prediction\. LetPθP\_\{\\theta\}be the output distribution of a model with parametersθ\\theta, which takes an input sequencex=\{x1,x2,…,xT\}x=\\\{x\_\{1\},x\_\{2\},\\ldots,x\_\{T\}\\\}, which includes both textual and visual token embeddings in the case of VLMs\. To find key tokens, we leverage a contrastive attribution signal grounded in preference data\. Specifically, rather than computing gradients with respect to a single output logit, we define the attribution objective using a preference\-based loss:222In cases where explicit preference data is unavailable, we show in an ablation study in[SectionA\.3](https://arxiv.org/html/2507.18043#A1.SS3)that using a single reference output \(e\.g\.,yposy\_\{\\text\{pos\}\}\) is still effective\.

f​\(x\)=log⁡Pθ​\(ypos∣x\)−log⁡Pθ​\(yneg∣x\),f\(x\)=\\log P\_\{\\theta\}\(y\_\{\\text\{pos\}\}\\mid x\)\-\\log P\_\{\\theta\}\(y\_\{\\text\{neg\}\}\\mid x\),\(1\)wherePθ​\(y∣x\)P\_\{\\theta\}\(y\\mid x\)denotes the conditional log\-probability of outputyygiven inputxx, as assigned by the model\. Here,yposy\_\{\\text\{pos\}\}andynegy\_\{\\text\{neg\}\}represent the preferred and dispreferred responses, respectively\. For example, if steering the model to be less toxic,yposy\_\{\\text\{pos\}\}would be a non\-toxic response andynegy\_\{\\text\{neg\}\}would be a negative response\. This contrastive formulation captures the model’s relative preference between two candidate completions, aligning more closely with human annotation and preference optimization objectives than absolute likelihoods\.

Token Attribution\.Given this objectivef​\(x\)f\(x\), we applyIntegrated Gradients\(IG\)\(Sundararajanet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib296)\)to compute the attribution score for each input token embeddingxjx\_\{j\}:

IGj​\(x\):=\(xj−x~j\)×∫α=01∂f​\(x~\+α​\(x−x~\)\)∂xj​𝑑α,\\mathrm\{IG\}\_\{j\}\(x\):=\(x\_\{j\}\-\\tilde\{x\}\_\{j\}\)\\times\\int\_\{\\alpha=0\}^\{1\}\\frac\{\\partial f\(\\tilde\{x\}\+\\alpha\(x\-\\tilde\{x\}\)\)\}\{\\partial x\_\{j\}\}\\,d\\alpha,wherex~\\tilde\{x\}is a neutral baseline input \(e\.g\., zero or masked token embedding\)\. The resulting attributionIGj​\(x\)\\mathrm\{IG\}\_\{j\}\(x\)quantifies the contribution of tokenxjx\_\{j\}to the model’s preference foryposy\_\{\\text\{pos\}\}overynegy\_\{\\text\{neg\}\}\.

IG providessigned attribution scores: positive values indicate tokens that increase the model’s preference foryposy\_\{\\text\{pos\}\}, while negative values indicate tokens that favorynegy\_\{\\text\{neg\}\}\. To obtain a scalar attribution score for each tokenxjx\_\{j\}, we sum the components of its IG vector:aj​\(x\)=∑i=1dIGj\(i\)​\(x\)a\_\{j\}\(x\)=\\sum\_\{i=1\}^\{d\}\\mathrm\{IG\}\_\{j\}^\{\(i\)\}\(x\)\. This aggregation yields a signed score that reflects the influence of the token on the model’s output, enabling clear comparison across tokens\(Atanasovaet al\.,[2020](https://arxiv.org/html/2507.18043#bib.bib311); Pezeshkpouret al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib310)\)\. Such scalar scores are essential for ranking and selecting the most impactful inputs for downstream intervention\. We then define two sets of top\-kkinfluential tokens \(corresponding to the green and red token groups in[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)\(A\)\) based on these scores:

ℐk\+​\(x\)\\displaystyle\\mathcal\{I\}\_\{k\}^\{\+\}\(x\)=\{xj∈x:aj\(x\)is among the top\-k\\displaystyle=\\left\\\{x\_\{j\}\\in x:a\_\{j\}\(x\)\\text\{ is among the top\-\}k\\right\.positive scores\},\\displaystyle\\quad\\left\.\\text\{positive scores\}\\right\\\},ℐk−​\(x\)\\displaystyle\\mathcal\{I\}\_\{k\}^\{\-\}\(x\)=\{xj∈x:aj\(x\)is among the top\-k\\displaystyle=\\left\\\{x\_\{j\}\\in x:a\_\{j\}\(x\)\\text\{ is among the top\-\}k\\right\.most negative scores\}\.\\displaystyle\\quad\\left\.\\text\{most negative scores\}\\right\\\}\.This allows us to disentangle how the model responds to desirable versus undesirable behavior, enabling finer\-grained control in downstream steering\. As shown in Figure[3](https://arxiv.org/html/2507.18043#S3.F3), removing the most negatively\-attributed tokens causes a substantial increase in the model’s preference foryposy\_\{\\text\{pos\}\}, while removing positively\-attributed tokens leads to the opposite effect\. These asymmetries highlight that negative attribution identifies strong contributors to undesirable model behavior, forming the foundation for constructing directional steering vectors\.

![Refer to caption](https://arxiv.org/html/2507.18043v2/x3.png)Figure 3:Effect on preference differenceΔ=log⁡Pθ​\(ypos∣x\)−log⁡Pθ​\(yneg∣x\)\\Delta=\\log P\_\{\\theta\}\(y\_\{\\text\{pos\}\}\\mid x\)\-\\log P\_\{\\theta\}\(y\_\{\\text\{neg\}\}\\mid x\)after ablating top\-kktokens based on signed Integrated Gradients\. Removing tokens with high negative attribution substantially increases model preference for aligned outputs \(yposy\_\{\\text\{pos\}\}\), whereas removing high positive tokens leads to preference drops\. Results shown for Llama\-3\.1\-8B\-Instruct and Qwen\-2\.5\-7B\-Instruct models on TruthfulQA\.Why Use Integrated Gradients?We choose Integrated Gradients over vanilla \(first\-order\) gradients due to its theoretical and practical advantages\. First, vanilla gradients are known to suffer from saturation: when a model is confident in an output, the gradient magnitude can diminish, even if the input token is critical to the decision\(Smilkovet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib306); Sundararajanet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib296)\)\. IG mitigates this by accumulating gradients along a path from a baseline to the actual input, yielding more robust and faithful attributions\. Second, IG satisfies desirable axiomatic properties such assensitivityandimplementation invariance\(Sundararajanet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib296)\), which vanilla gradients lack\. As a result, IG provides more stable, interpretable, and reliable token importance scores, especially in high\-dimensional, non\-linear models like LLMs and VLMs\. While developing a new attribution method is not the focus of our work, we include technical details in[Appendix˜B](https://arxiv.org/html/2507.18043#A2)and a comparison in[Section˜A\.3](https://arxiv.org/html/2507.18043#A1.SS3)to empirically validate the effectiveness of IG over other attribution methods\.

Compared to perturbation\-based attribution methods such as SHAP\(Lundberg and Lee,[2017](https://arxiv.org/html/2507.18043#bib.bib324)\), IG is more efficient in our setting: perturbation methods require sampling many perturbations, leading to a significant number of forward passes, which is computationally expensive for high\-dimensional inputs\(Agarwalet al\.,[2021](https://arxiv.org/html/2507.18043#bib.bib325); Raoet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib326)\), especially for VLMs\. In contrast, IG attribution uses only 5\-20 approximation steps \(details including running time are provided in[Section˜4](https://arxiv.org/html/2507.18043#S4)and[Section˜A\.1](https://arxiv.org/html/2507.18043#A1.SS1)\) and achieves strong steering performance for both LLMs and VLMs\.

### 3\.2Constructing Layer\-Wise Steering Vectors

#### Contrastive Steering Vectors\.

Once the top positively and negatively attributed tokens are identified, we construct two modified inputs:x\\ℐ\+x\_\{\\backslash\\mathcal\{I\}^\{\+\}\}\(where top\-kkpositive tokens are replaced by baselines\) andx\\ℐ−x\_\{\\backslash\\mathcal\{I\}^\{\-\}\}\(where top\-kknegative tokens are replaced\)\. These substitutions isolate the collective contribution of each polarity group to the model’s internal representations\.333Replacing one token at a time may offer more granularity but is computationally expensive and in practice yields similar effect\(Covertet al\.,[2021](https://arxiv.org/html/2507.18043#bib.bib319); Ronget al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib318)\)\.

Lethlast\(l\)​\(x\)∈ℝdh^\{\(l\)\}\_\{\\text\{last\}\}\(x\)\\in\\mathbb\{R\}^\{d\}denote the hidden activation at the final token of the sequence at transformer layerll\. Following prior work\(Liet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib15)\), we use this position as it typically aggregates contextual information from the entire sequence and provides a consistent anchor point for measuring how input changes propagate through the model across layers\. We define thecontrastive steering vectorsas:

δl\+,\(x\)\\displaystyle\\delta\_\{l\}^\{\+,\(x\)\}=hlast\(l\)​\(x\)−hlast\(l\)​\(x\\ℐ\+\),\\displaystyle=h^\{\(l\)\}\_\{\\text\{last\}\}\(x\)\-h^\{\(l\)\}\_\{\\text\{last\}\}\(x\_\{\\backslash\\mathcal\{I\}^\{\+\}\}\),\(2\)δl−,\(x\)\\displaystyle\\delta\_\{l\}^\{\-,\(x\)\}=hlast\(l\)​\(x\)−hlast\(l\)​\(x\\ℐ−\)\.\\displaystyle=h^\{\(l\)\}\_\{\\text\{last\}\}\(x\)\-h^\{\(l\)\}\_\{\\text\{last\}\}\(x\_\{\\backslash\\mathcal\{I\}^\{\-\}\}\)\.These vectors quantify the directional shift in the model’s hidden representation when high\-impact tokens are ablated\. Intuitively,δl\+,\(x\)\\delta\_\{l\}^\{\+,\(x\)\}captures how the model relies on tokens that support aligned, desirable outputs, whileδl−,\(x\)\\delta\_\{l\}^\{\-,\(x\)\}captures how it relies on tokens contributing to misaligned, undesirable outputs \(e\.g\., hallucinations, toxicity\)\.

PCA for Vector Aggregation\.We aim to extract a single per\-layer steering vector that can be applied at inference to unseen inputs\. However, the per\-example contrastive deltasδl\+,\(x\)\\delta\_\{l\}^\{\+,\(x\)\}andδl−,\(x\)\\delta\_\{l\}^\{\-,\(x\)\}can vary in magnitude, so naive averaging can cancel out signals\(Yin and Neubig,[2022](https://arxiv.org/html/2507.18043#bib.bib328); Ferrandoet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib329)\)\. We extract a stable, low\-dimensional steering direction by applying Principal Component Analysis \(PCA\) over many examples \(see[Section˜A\.3](https://arxiv.org/html/2507.18043#A1.SS3)for the vector aggregation ablation\)\. PCA serves two roles: it aggregates noisy vectors into a robust semantic direction and ensures the steering vector generalizes across diverse inputs\. Specifically, we compute the top principal component of each set across a steering dataset𝒟\\mathcal\{D\}, yielding steering vectorsvl\+∈ℝdv\_\{l\}^\{\+\}\\in\\mathbb\{R\}^\{d\}andvl−∈ℝdv\_\{l\}^\{\-\}\\in\\mathbb\{R\}^\{d\}, as illustrated in[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)\(B\):

vl\+\\displaystyle v\_\{l\}^\{\+\}=PCA1​\{δl\+,\(x\):x∈𝒟\},\\displaystyle=\\mathrm\{PCA\}\_\{1\}\\bigl\\\{\\delta\_\{l\}^\{\+,\(x\)\}:x\\in\\mathcal\{D\}\\bigr\\\},\(3\)vl−\\displaystyle v\_\{l\}^\{\-\}=PCA1​\{δl−,\(x\):x∈𝒟\}\.\\displaystyle=\\mathrm\{PCA\}\_\{1\}\\bigl\\\{\\delta\_\{l\}^\{\-,\(x\)\}:x\\in\\mathcal\{D\}\\bigr\\\}\.We then define the final contrastive steering vector at layerllas:

vl=vl\+−vl−,v\_\{l\}=v\_\{l\}^\{\+\}\-v\_\{l\}^\{\-\},\(4\)which captures the latent direction from desirable to undesirable behavior\. The contrastive vector reflects both suppression of undesirable semantics \(viavl−v\_\{l\}^\{\-\}\) and enhancement of desirable ones \(viavl\+v\_\{l\}^\{\+\}\), and is used at inference time to steer the model away from behaviors tied to high\-impact inputs\.Empirically, we observe that the first principal component \(PC1\) is both efficient and robust: a single component captures the task\-relevant variance while avoiding overfitting \(see[Table˜9](https://arxiv.org/html/2507.18043#A1.T9)for the ablation study on PCA\)\. Across datasets in[Section˜4](https://arxiv.org/html/2507.18043#S4), we observe that the explained\-variance ratio of PC1 is 0\.7\-0\.9, indicating a dominant, shared subspace and supporting reliability across different cases of the same attribute\. Using more components of PCA would be inefficient since it requires injecting multiple vectors per layer with per\-component scaling/combination\.

### 3\.3Steering at Inference Time

At inference, we steer the model’s generation by applying the vectors across layers during decoding\. Letht,l∈ℝdh\_\{t,l\}\\in\\mathbb\{R\}^\{d\}be the activation at token positionttand layerll\. For each position and layer, we apply an additive intervention to the activation and rescale to match the original norm \(see[Fig\.˜2](https://arxiv.org/html/2507.18043#S3.F2)\(C\)\):

h~t,l=\(ht,l\+λ​vl\)×‖ht,l‖2‖ht,l\+λ​vl‖2,\\tilde\{h\}\_\{t,l\}\\;=\\;\\big\(h\_\{t,l\}\+\\lambda\\,v\_\{l\}\\big\)\\,\\times\\frac\{\\\|h\_\{t,l\}\\\|\_\{2\}\}\{\\\|h\_\{t,l\}\+\\lambda\\,v\_\{l\}\\\|\_\{2\}\},\(5\)whereλ\\lambdais a hyperparameter controlling the strength of steering\. Our formulation ensures the adjustment is smooth and maintains compatibility with downstream modules, while allowing for consistent behavioral shifts in the model\(Liuet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib30)\)\. Importantly, because these vectors are constructed from the tokens with the largest attribution\-based contributions using contrastive gradient attribution, the intervention is both targeted and proportional\. The norm preservation reduces the risk of overcorrecting unrelated behaviors\(Liuet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib30)\), focusing the adjustment precisely on the factors responsible for misalignment \(see the qualitative analysis in[Section˜5\.2](https://arxiv.org/html/2507.18043#S5.SS2)and the ablation study of the normalization step in[Section˜A\.3](https://arxiv.org/html/2507.18043#A1.SS3)\)\.

## 4Experiments

We evaluate ourGrAInSacross both language\-only \(LLMs\) and multimodal \(VLMs\) settings\. Our focus is on safety\-critical scenarios involving undesirable outputs\. For each domain, we compare against standard baselines including fine\-tuned models and existing steering methods\. More details on settings, running time, hyperparameter analysis and results are provided in[Appendix˜A](https://arxiv.org/html/2507.18043#A1)\.

Table 1:Performance on LLM benchmarks for both LLaMA\-3\.1\-8B and Qwen2\.5\-7B\. Accuracy \(higher is better\) reported for TruthfulQA, Toxigen, and FaithEval\. Avg\. columns show the mean across benchmarks per model\.Table 2:Comparison across MMHal\-Bench and SPA\-VL benchmarks\. Left: MMHal\-Bench reports hallucination rate \(lower is better\)\. Right: SPA\-VL reports preference win rate \(higher is better\)\. Avg\. columns reflect the mean performance across the three models\.### 4\.1LLM Experiments

Models\.We use Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib215)\)and Qwen2\.5\-7B\-Instruct\(Team,[2024](https://arxiv.org/html/2507.18043#bib.bib212)\)as our base models for evaluating text\-only settings\. These models are chosen for their strong capabilities and because they serve as the language components of their corresponding VLMs evaluated later in our multimodal experiments\.

Datasets\.We evaluateGrAInSon multiple\-choice QA datasets that each target a separate LLM attribute for LLM safety:TruthfulQA\(truthfulness\)\(Linet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib234)\),Toxigen\(toxicity\)\(Hartvigsenet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib236)\),FaithEval\(context faithfulness\)\(Minget al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib63)\)\. We report multiple\-choice accuracy for each dataset\.

Inference\-time Steering withGrAInS\.We select 50 samples from each dataset for constructing the steering vectors\. For each example, we compute token\-level attributions for text tokens using the preference loss described in Section[3\.1](https://arxiv.org/html/2507.18043#S3.SS1)\. In all experiments, we setk=3k=3tokens\. For IG, we use 5 steps for gradient estimation\. Steering vectors are computed using PCA over contrastive activation vectors from multiple inputs\. These are applied at inference to adjust the model’s hidden activations\.

Baselines\.We compareGrAInSagainst approaches for steering LLMs\. We employ LoRA fine\-tuning\(Huet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib203)\)as a representative parameter\-efficient fine\-tuning \(PEFT\) method\. We also compare against state\-of\-the\-art inference\-time intervention methods including ICV\(Liuet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib30)\), NL\-ITI\(Hoscilowiczet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib240)\), CAA\(Rimskyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib227)\)\. We note that there are other steering baselines such as RepE\(Zouet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib11)\)and ITI\(Liet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib15)\), but recent work\(Im and Li,[2025](https://arxiv.org/html/2507.18043#bib.bib320)\)has shown that they underperform compared to our selected baselines like NL\-ITI and CAA across multiple benchmarks, so we do not include them in our comparisons\.

Results:GrAInSImproves Steering of LLMs\.[Table˜1](https://arxiv.org/html/2507.18043#S4.T1)shows thatGrAInSoutperforms both LoRA and existing steering baselines across all three tasks\. On TruthfulQA,GrAInSimproves accuracy by 8\.44% on Qwen2\.5\-7B\-Instruct and by 13\.22% on Llama\-3\.1\-8B\-Instruct, outperforming ICV, NL\-ITI, and CAA\. On Toxigen, our method improves accuracy significantly by 7\.79% for Llama and 7\.08% for Qwen over their respective base models\. For FaithEval, which evaluates contextual consistency,GrAInSagain achieves the highest accuracy 70\.94% on Llama and 64\.77% on Qwen, showing strong gains across models\.

### 4\.2VLM Experiments

Models\.We use LLaVA\-v1\.6\-7B\(Liuet al\.,[2024a](https://arxiv.org/html/2507.18043#bib.bib309)\), Qwen2\.5\-VL\-7B\-Instruct\(Team,[2024](https://arxiv.org/html/2507.18043#bib.bib212)\), and Gemma\-3\-12B\(Team and others,[2025](https://arxiv.org/html/2507.18043#bib.bib317)\)\.

Datasets\.We evaluate on two key failure modes in multimodal generation usingMMHal\-Bench\(hallucination\)\(Sunet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib294)\)andSPA\-VL\(safety\)\(Zhanget al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib295)\)\. For MMHal\-Bench, we report the hallucination rate using GPT\-4o as the judge model\. We observe strong agreement between GPT\-4o and human annotations, with a Pearson correlation of 0\.82 and a Spearman correlation of 0\.85, based on evaluations from 9 human annotators\. For SPA\-VL we report the preference win rate ofchosen \> rejectedresponses based on model log probability\. This metric is standard in alignment work and shown to correlate with human preferences\(Rafailovet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib45); Liet al\.,[2024a](https://arxiv.org/html/2507.18043#bib.bib332)\)\.

Inference\-time Steering withGrAInS\.Similar to LLM experiments in[Section˜4\.1](https://arxiv.org/html/2507.18043#S4.SS1), we select 50 samples for constructing the steering vectors\. As VLMs might require processing more tokens, including both visual and textual tokens, in all experiments, we setk=20k=20tokens\. For IG, we use 5 steps for gradient approximation in LLaVA and Qwen, and 10 steps for the larger Gemma model to ensure more reliable attribution\.

Baselines\.We compareGrAInSagainst approaches for aligning VLMs\. For fair comparison, we use the same samples used to construct steering vectors forGrAInSfor all steering baselines\. In addition to LoRA\(Huet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib203)\), we compare against state\-of\-the\-art steering methods for VLMs, including VTI\(Liuet al\.,[2024c](https://arxiv.org/html/2507.18043#bib.bib4)\), which applies modality\-specific vector shifts to reduce hallucinations, and ICT\(Chenet al\.,[2024a](https://arxiv.org/html/2507.18043#bib.bib293)\), which performs object\-grounded interventions but relies on external object detectors, RUDDER\(Zouet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib336)\), which adaptively injects per\-sample visual evidence directions from residual updates for low\-overhead hallucination mitigation, and SHARP\(Wuet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib337)\), which steers cause\-specific latent representations to suppress different types of hallucinations during inference\. Additionally, we adapt CAA\(Rimskyet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib227)\)to VLMs by directly incorporating their steering mechanisms into the LLM component of the VLM\.

Results:GrAInSImproves Steering of VLMs\.Table[2](https://arxiv.org/html/2507.18043#S4.T2)shows thatGrAInSachieves the lowest hallucination rates across all three VLMs on MMHal\-Bench\. On LLaVA\-1\.6\-7B,GrAInSreduces the hallucination rate from 0\.624 of the base model to 0\.514, outperforming baselines such as SHARP \(0\.569\) and VTI \(0\.587\)\. On Qwen2\.5\-VL\-7B, it lowers hallucinations from 0\.523 to 0\.473\. For Gemma\-3\-12B\-IT,GrAInSyields the best result \(0\.442\), outperforming all other baselines\. Moreover,GrAInShas the highest preference win rates on SPA\-VL\. It improves LLaVA\-1\.6\-7B from 40\.24% to 48\.35%, Qwen2\.5\-VL\-7B from 53\.21% to 58\.90%, and Gemma\-3\-12B\-IT from 49\.32% to 53\.51%\. These gains exceed all other steering and fine\-tuning baselines, which range between 1–5% lower per model\. These results indicate thatGrAInSimproves both hallucination and safety\.

Table 3:LLM capability comparison: we report BLEU accuracy for TruthfulQA, 5\-shot accuracy for MMLU\.Table 4:VLM capability comparison: we report BLEU accuracy for SPA\-VL, 5\-shot accuracy for MMMU\.

## 5Analysis

In this section, we present a deeper analysis ofGrAInS, covering its impact on general model capabilities, qualitative analysis, generalization ofGrAInS, and computational overhead analysis\. We provide ablation studies and additional results \(including attribution methods, token selection\) in[Section˜A\.3](https://arxiv.org/html/2507.18043#A1.SS3)and[Section˜A\.4](https://arxiv.org/html/2507.18043#A1.SS4)\.

![Refer to caption](https://arxiv.org/html/2507.18043v2/figures/mmhal_example1.jpg)\(A\) Correction of hallucinated objects\.
Q:What are the objects from right to left?
LLaVA\-1\.6\-7B:A spoon, a coffee cup, and a saucer\.
ICT:A spoon, a coffee cup, and a saucer\.
VTI:A spoon, a coffee cup, and a saucer\.
GrAInS:A lid, a spoon, and a coffee cup\.
![Refer to caption](https://arxiv.org/html/2507.18043v2/figures/mmhal_example2.jpg)\(B\) Preserving correct behavior\.
Q:Who is sitting on the bench?
Qwen2\.5\-VL\-7B:No one is sitting\.
ICT:A man is sitting\.
VTI:A man is sitting\.
GrAInS:The benches are empty\.

Figure 4:Qualitative examples from MMHal\-Bench\. \(A\) OnlyGrAInSprovides the correct object order\. \(B\) The base model is correct, but baselines introduce hallucinations;GrAInSpreserves the correct, grounded answer\.### 5\.1Impact on General Model Capabilities

A desirable steering method should reduce harmful behavior and hallucination without degrading the model’s capabilities\. We evaluate whetherGrAInSpreserves core capabilities such as fluency and reasoning after intervention\.

Generation Qualities\.Following prior work\(Pham and Nguyen,[2024](https://arxiv.org/html/2507.18043#bib.bib253); Nguyenet al\.,[2025b](https://arxiv.org/html/2507.18043#bib.bib289)\), we assess the effect of steering on open\-ended generation tasks using TruthfulQA for LLMs and SPA\-VL for VLMs\. We report BLEU accuracy, defined as the proportion of generated outputs that are closer by BLEU score\(Papineniet al\.,[2002](https://arxiv.org/html/2507.18043#bib.bib335)\)to the correct \(positive\) reference than to the incorrect \(negative\) one\. This metric captures whether steering disrupts fluency or semantic correctness of the generations and is commonly used in prior work\(Biet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib334); Changet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib333)\)\. For LLMs, as shown in[Table˜3](https://arxiv.org/html/2507.18043#S4.T3),GrAInSachieves the highest BLEU accuracy on both Llama\-3\.1\-8B \(47\.91%\) and Qwen2\.5\-7B \(54\.09%\)\. For VLMs, Table[4](https://arxiv.org/html/2507.18043#S4.T4)shows thatGrAInSalso performs competitively, achieving 46\.79% on LLaVA\-1\.6\-7B and the highest score of 53\.02% on Qwen2\.5\-VL\-7B\. These results show thatGrAInSaligns outputs more closely with human\-preferred responses while preserving generation quality\.

General Reasoning Capabilities\.We evaluate 5\-shot accuracy on reasoning datasets using MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2507.18043#bib.bib180)\)for LLMs and MMMU\(Yueet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib312)\)for VLMs\. These benchmarks cover a wide range of subjects, allowing us to measure whetherGrAInSand steering baselines affect the model’s ability to perform general\-purpose reasoning\. On MMLU,[Table˜3](https://arxiv.org/html/2507.18043#S4.T3)shows thatGrAInSmaintains comparable performance to the base models, with 69\.15% accuracy on Llama\-3\.1\-8B \(vs\. 69\.27% base\) and 74\.29% on Qwen2\.5\-7B \(vs\. 74\.58% base\)\. Unlike other steering methods, which degrade reasoning accuracy substantially \(e\.g\., CAA drops to 51\.49% on Llama\),GrAInSpreserves reasoning ability\. Similarly, on VLMs \(Table[4](https://arxiv.org/html/2507.18043#S4.T4)\),GrAInSmaintains accuracy on MMMU, with 34\.92% on LLaVA\-1\.6\-7B and 58\.13% on Qwen2\.5\-VL\-7B, only slightly below the base models\. This shows thatGrAInSminimally disrupts models’ reasoning abilities\.

### 5\.2Qualitative Analysis

[Fig\.˜4](https://arxiv.org/html/2507.18043#S5.F4)presents two representative examples from MMHal\-Bench that highlight the effectiveness ofGrAInScompared to baseline VLMs and steering approaches\. In example \(A\), baseline models, including LLaVa\-1\.6\-7B, ICT, and VTI hallucinate the object locations\. OnlyGrAInSidentifies the objects in the correct order \(though it refers to the right\-most object as a lid, which is less likely than saucer\), demonstrating improved grounding to visual evidence\. In contrast, example \(B\) illustrates a failure mode of prior steering methods: while the original Qwen2\.5\-VL\-7B prediction is correct \(*“no one is sitting on the bench”*\), steering baselines introduce hallucinated content by incorrectly claiming someone is present\.GrAInSavoids this regression and preserves valid base model behavior\. These examples illustrateGrAInS’s ability to modulate outputs based on token\-level and modality\-aware attribution signals, enabling both behavioral improvement and alignment fidelity\. We provide more qualitative results in[Fig\.˜7](https://arxiv.org/html/2507.18043#A5.F7)\.

Table 5:Generalization ofGrAInSto RealToxicityPrompts\.Table 6:Generalization ofGrAInSto POPE\.Table 7:Inference throughput comparison between the base model and GrAInS\.
### 5\.3Generalization ofGrAInS

To directly test out\-of\-distribution generalization, we add an additional experiment in which we build the steering vector on a source dataset and evaluate it on a target dataset sharing the same attribute, such as toxicity or hallucination\. For LLM toxicity mitigation, we use the vector on Toxigen and evaluate on RealToxicityPrompts \(RTP\)\(Gehmanet al\.,[2020](https://arxiv.org/html/2507.18043#bib.bib330)\)with Qwen2\.5\-7B\-Instruct \(toxicity, lower is better\)\. For VLM hallucination reduction, we use the vector on MMHal\-Bench and evaluate on the POPE\(Liet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib331)\)adversarial split \(the most challenging setting of POPE\) with Qwen2\.5\-VL\-7B\-Instruct\. We compare against strong steering baselines under the same setting\.

[Table˜5](https://arxiv.org/html/2507.18043#S5.T5)and[Table˜6](https://arxiv.org/html/2507.18043#S5.T6)show that our method outperforms baselines, and these cross\-dataset gains show that a vector learned from one dataset with a specific attribute transfers to different distributions and prompt formats\. This supports our claim that GrAInS captures meaningful directions \(e\.g\., non\-toxic, non\-hallucinatory behavior\)\.

## 6Discussion and Conclusion

#### Discussion on Computational Overhead\.

At inference time,GrAInSdoes not run attribution or extra forward passes\. It only adds a precomputed per\-layer vector and applies scale\-preserving normalization\. Additionally, we do not alter KV\-cache shapes or softmax paths, so the throughput/latency remains essentially unchanged\. In[Table˜7](https://arxiv.org/html/2507.18043#S5.T7), we report the inference throughput \(tokens/ms\) and compute the throughput drop of GrAInS compared to the base model on a single RTX A6000 GPU for TruthfulQA\. The impact on inference is minimal \(<5%\), showing thatGrAInSintroduces only negligible runtime overhead while preserving the deployment efficiency of the original model\.

#### Conclusion\.

We introduceGrAInS, a novel steering approach that finds the most influential input tokens across modalities, and uses contrastive activation shifts to compute steering vectors\. Unlike prior methods that apply fixed intervention or rely solely on visual tokens,GrAInSenables fine\-grained and interpretable control without retraining or external modules\. Our approach achieves consistent gains in reducing hallucination, increasing preference alignment, and preserving generation quality and reasoning capabilities across LLMs and VLMs\. By integrating attribution with steering,GrAInSbridges the gap between interpretability and controllability in modern language and vision\-language models\.

## Limitations

WhileGrAInSdemonstrates strong empirical performance across a range of tasks and models, it has some limitations\. First, like other attribution\-based methods,GrAInSdepends on the quality of token\-level attribution\. While attribution methods like IG provide a principled and effective foundation for identifying influential tokens, they are not without drawbacks\. Attribution quality can vary depending on the model architecture and the choice of baseline input, which may affect the precision of steering vectors\. Additionally, IG and related methods require gradient access and are computationally more expensive than simpler heuristics, which could pose challenges for scaling to very large models\. Future work may explore alternative or learned attribution techniques that improve token selection efficiency and quality\. Another limitation of our paper and other steering methods is that there is no formal guarantee that modifying internal representations based on attributed tokens will correct the model’s behavior\. Future work could investigate methods for constraining the downstream effects of such interventions, potentially combining attribution with disentangled representations for more robust interventions\.

## Acknowledgements

We thank Jaemin Cho for his helpful comments and suggestions on this paper\. This work was supported by NSF\-CAREER Award 1846185, DARPA ECOLE Program No\. HR00112390060, and NSF\-AI Engage Institute DRL\-2112635, ARO Award W911NF2110220, ONR Grant N00014\-23\-12356, and an Apple PhD Fellowship\. The views contained in this article are those of the authors and not of the funding agency\.

## References

- Rethinking stability for attribution\-based explanations\.arXiv preprint arXiv:2203\.06877\.Cited by:[Appendix B](https://arxiv.org/html/2507.18043#A2.p2.3)\.
- S\. Agarwal, S\. Jabbari, C\. Agarwal, S\. Upadhyay, S\. Wu, and H\. Lakkaraju \(2021\)Towards the unification and robustness of perturbation and gradient based explanations\.InInternational conference on machine learning,pp\. 110–119\.Cited by:[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p5.1)\.
- M\. Ancona, E\. Ceolini, C\. Öztireli, and M\. Gross \(2019\)Gradient\-based attribution methods\.InExplainable AI: Interpreting, explaining and visualizing deep learning,pp\. 169–191\.Cited by:[Appendix B](https://arxiv.org/html/2507.18043#A2.p2.3)\.
- P\. Atanasova, J\. G\. Simonsen, C\. Lioma, and I\. Augenstein \(2020\)A diagnostic study of explainability techniques for text classification\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 3256–3274\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.263/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.263)Cited by:[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p3.5)\.
- Q\. Bi, J\. Li, L\. Shang, X\. Jiang, Q\. Liu, and H\. Yang \(2022\)MTRec: multi\-task learning over BERT for news recommendation\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 2663–2669\.External Links:[Link](https://aclanthology.org/2022.findings-acl.209/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.209)Cited by:[§5\.1](https://arxiv.org/html/2507.18043#S5.SS1.p2.1)\.
- J\. Cao, P\. Ye, S\. Li, C\. Yu, Y\. Tang, J\. Lu, and T\. Chen \(2024\)Madtp: multimodal alignment\-guided dynamic token pruning for accelerating vision\-language transformer\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 15710–15719\.Cited by:[§A\.3](https://arxiv.org/html/2507.18043#A1.SS3.p5.3),[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- Y\. Chang, Y\. Kim, M\. Krumdick, A\. Zadeh, C\. Li, C\. Tanner, and M\. Iyyer \(2025\)BLEUBERI: bleu is a surprisingly effective reward for instruction following\.arXiv preprint arXiv:2505\.11080\.Cited by:[§5\.1](https://arxiv.org/html/2507.18043#S5.SS1.p2.1)\.
- Y\. Chang, B\. Cao, Y\. Wang, J\. Chen, and L\. Lin \(2024\)XPrompt: explaining large language model’s generation via joint prompt attribution\.arXiv preprint arXiv:2405\.20404\.Cited by:[§2](https://arxiv.org/html/2507.18043#S2.p2.1)\.
- J\. Chen, T\. Zhang, S\. Huang, Y\. Niu, L\. Zhang, L\. Wen, and X\. Hu \(2024a\)ICT: image\-object cross\-level trusted intervention for mitigating object hallucination in large vision\-language models\.External Links:2411\.15268,[Link](https://arxiv.org/abs/2411.15268)Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p2.1),[§2](https://arxiv.org/html/2507.18043#S2.p1.1),[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p4.1.2)\.
- M\. Chen, Y\. Cao, Y\. Zhang, and C\. Lu \(2024b\)Quantifying and mitigating unimodal biases in multimodal large language models: a causal perspective\.arXiv preprint arXiv:2403\.18346\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- Q\. Chen, T\. Zhang, C\. Wang, X\. He, D\. Wang, and T\. Liu \(2025\)Attribution analysis meets model editing: advancing knowledge correction in vision language models with visedit\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 2168–2176\.Cited by:[Appendix C](https://arxiv.org/html/2507.18043#A3.p1.1.2)\.
- I\. C\. Covert, S\. Lundberg, and S\. Lee \(2021\)Explaining by removing: a unified framework for model explanation\.J\. Mach\. Learn\. Res\.22\(1\)\.External Links:ISSN 1532\-4435Cited by:[footnote 3](https://arxiv.org/html/2507.18043#footnote3)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p1.1)\.
- M\. Farina, G\. Franchi, G\. Iacca, M\. Mancini, and E\. Ricci \(2024\)Frustratingly easy test\-time adaptation of vision\-language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=eQ6VjBhevn)Cited by:[Appendix C](https://arxiv.org/html/2507.18043#A3.p1.1)\.
- J\. Ferrando, G\. I\. Gállego, I\. Tsiamas, and M\. R\. Costa\-jussà \(2023\)Explaining how transformers use context to build predictions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5486–5513\.External Links:[Link](https://aclanthology.org/2023.acl-long.301/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.301)Cited by:[§3\.2](https://arxiv.org/html/2507.18043#S3.SS2.SSS0.Px1.p3.5)\.
- S\. Gehman, S\. Gururangan, M\. Sap, Y\. Choi, and N\. A\. Smith \(2020\)RealToxicityPrompts: evaluating neural toxic degeneration in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 3356–3369\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.301/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.301)Cited by:[§5\.3](https://arxiv.org/html/2507.18043#S5.SS3.p1.1.1)\.
- S\. S\. Ghosal, S\. Chakraborty, V\. Singh, T\. Guan, M\. Wang, A\. Beirami, F\. Huang, A\. Velasquez, D\. Manocha, and A\. S\. Bedi \(2024\)Immune: improving safety against jailbreaks in multi\-modal llms via inference\-time alignment\.arXiv preprint arXiv:2411\.18688\.Cited by:[Appendix C](https://arxiv.org/html/2507.18043#A3.p1.1)\.
- T\. Hartvigsen, S\. Gabriel, H\. Palangi, M\. Sap, D\. Ray, and E\. Kamar \(2022\)ToxiGen: a large\-scale machine\-generated dataset for adversarial and implicit hate speech detection\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 3309–3326\.External Links:[Link](https://aclanthology.org/2022.acl-long.234/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.234)Cited by:[2nd item](https://arxiv.org/html/2507.18043#A1.I1.i2.p1.1),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p2.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p3.1),[§5\.1](https://arxiv.org/html/2507.18043#S5.SS1.p3.1)\.
- J\. Hoscilowicz, A\. Wiacek, J\. Chojnacki, A\. Cieslak, L\. Michon, and A\. Janicki \(2024\)Non\-linear inference time intervention: improving llm truthfulness\.InProc\. Interspeech 2024,pp\. 4094–4098\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p3.1),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p4.1)\.
- E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p4.1),[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p4.1.2)\.
- Y\. Huang, L\. Sun, H\. Wang, S\. Wu, Q\. Zhang, Y\. Li, C\. Gao, Y\. Huang, W\. Lyu, Y\. Zhang, X\. Li, H\. Sun, Z\. Liu, Y\. Liu, Y\. Wang, Z\. Zhang, B\. Vidgen, B\. Kailkhura, C\. Xiong, C\. Xiao, C\. Li, E\. P\. Xing, F\. Huang, H\. Liu, H\. Ji, H\. Wang, H\. Zhang, H\. Yao, M\. Kellis, M\. Zitnik, M\. Jiang, M\. Bansal, J\. Zou, J\. Pei, J\. Liu, J\. Gao, J\. Han, J\. Zhao, J\. Tang, J\. Wang, J\. Vanschoren, J\. Mitchell, K\. Shu, K\. Xu, K\. Chang, L\. He, L\. Huang, M\. Backes, N\. Z\. Gong, P\. S\. Yu, P\. Chen, Q\. Gu, R\. Xu, R\. Ying, S\. Ji, S\. Jana, T\. Chen, T\. Liu, T\. Zhou, W\. Y\. Wang, X\. Li, X\. Zhang, X\. Wang, X\. Xie, X\. Chen, X\. Wang, Y\. Liu, Y\. Ye, Y\. Cao, Y\. Chen, and Y\. Zhao \(2024\)TrustLLM: trustworthiness in large language models\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=bWUU0LwwMp)Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- S\. Im and Y\. Li \(2025\)A unified understanding and evaluation of steering methods\.arXiv preprint arXiv:2502\.02716\.Cited by:[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p4.1)\.
- A\. Kapishnikov, S\. Venugopalan, B\. Avci, B\. Wedin, M\. Terry, and T\. Bolukbasi \(2021\)Guided integrated gradients: an adaptive path method for removing noise\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 5050–5058\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p2.1),[§2](https://arxiv.org/html/2507.18043#S2.p2.1)\.
- P\. Khayatan, M\. Shukor, J\. Parekh, and M\. Cord \(2025\)Analyzing fine\-tuning representation shift for multimodal llms steering alignment\.arXiv preprint arXiv:2501\.03012\.Cited by:[§2](https://arxiv.org/html/2507.18043#S2.p1.1)\.
- S\. Leng, H\. Zhang, G\. Chen, X\. Li, S\. Lu, C\. Miao, and L\. Bing \(2024\)Mitigating object hallucinations in large vision\-language models through visual contrastive decoding\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 13872–13882\.Cited by:[Appendix C](https://arxiv.org/html/2507.18043#A3.p1.1)\.
- J\. Li, F\. Zhou, S\. Sun, Y\. Zhang, H\. Zhao, and P\. Liu \(2024a\)Dissecting human and llm preferences\.arXiv preprint arXiv:2402\.11296\.Cited by:[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p2.1)\.
- K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2024b\)Inference\-time intervention: eliciting truthful answers from a language model\.Advances in Neural Information Processing Systems36\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1),[§2](https://arxiv.org/html/2507.18043#S2.p1.1),[§3\.2](https://arxiv.org/html/2507.18043#S3.SS2.SSS0.Px1.p2.2),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p4.1)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen \(2023\)Evaluating object hallucination in large vision\-language models\.arXiv preprint arXiv:2305\.10355\.Cited by:[§5\.3](https://arxiv.org/html/2507.18043#S5.SS3.p1.1.1)\.
- Z\. Li and D\. Hoiem \(2017\)Learning without forgetting\.IEEE transactions on pattern analysis and machine intelligence40\(12\),pp\. 2935–2947\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 3214–3252\.External Links:[Link](https://aclanthology.org/2022.acl-long.229/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.229)Cited by:[1st item](https://arxiv.org/html/2507.18043#A1.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p2.1)\.
- Z\. Lin, S\. Basu, M\. Beigi, V\. Manjunatha, R\. A\. Rossi, Z\. Wang, Y\. Zhou, S\. Balasubramanian, A\. Zarei, K\. Rezaei, Y\. Shen, B\. M\. Yao, Z\. Xu, Q\. Liu, Y\. Zhang, Y\. Sun, S\. Liu, L\. Shen, H\. Li, S\. Feizi, and L\. Huang \(2025\)A survey on mechanistic interpretability for multi\-modal foundation models\.External Links:2502\.17516,[Link](https://arxiv.org/abs/2502.17516)Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1),[§1](https://arxiv.org/html/2507.18043#S1.p2.1),[§2](https://arxiv.org/html/2507.18043#S2.p2.1)\.
- H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee \(2024a\)LLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p1.1)\.
- S\. Liu, H\. Ye, L\. Xing, and J\. Y\. Zou \(2024b\)In\-context vectors: making in context learning more effective and controllable through latent space steering\.InForty\-first International Conference on Machine Learning,Cited by:[§A\.3](https://arxiv.org/html/2507.18043#A1.SS3.p3.1),[§1](https://arxiv.org/html/2507.18043#S1.p1.1),[§1](https://arxiv.org/html/2507.18043#S1.p3.1),[§3\.3](https://arxiv.org/html/2507.18043#S3.SS3.p1.4),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p4.1)\.
- S\. Liu, H\. Ye, L\. Xing, and J\. Zou \(2024c\)Reducing hallucinations in vision\-language models via latent space steering\.arXiv preprint arXiv:2410\.15778\.Cited by:[1st item](https://arxiv.org/html/2507.18043#A1.I2.i1.p1.1),[2nd item](https://arxiv.org/html/2507.18043#A1.I3.i2.p1.1),[§1](https://arxiv.org/html/2507.18043#S1.p2.1),[§1](https://arxiv.org/html/2507.18043#S1.p3.1),[§2](https://arxiv.org/html/2507.18043#S2.p1.1),[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p4.1.2)\.
- D\. Lopez\-Paz and M\. Ranzato \(2017\)Gradient episodic memory for continual learning\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- S\. M\. Lundberg and S\. Lee \(2017\)A unified approach to interpreting model predictions\.Advances in neural information processing systems30\.Cited by:[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p5.1)\.
- S\. Marks and M\. Tegmark \(2023\)The geometry of truth: emergent linear structure in large language model representations of true/false datasets\.arXiv preprint arXiv:2310\.06824\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- Y\. Ming, S\. Purushwalkam, S\. Pandit, Z\. Ke, X\. Nguyen, C\. Xiong, and S\. Joty \(2025\)FaithEval: can your language model stay faithful to context, even if ”the moon is made of marshmallows”\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=UeVx6L59fg)Cited by:[3rd item](https://arxiv.org/html/2507.18043#A1.I1.i3.p1.1),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p2.1)\.
- B\. Nguyen, B\. Nguyen, D\. Nguyen, and V\. A\. Nguyen \(2025a\)Risk\-aware distributional intervention policies for language models\.arXiv preprint arXiv:2501\.15758\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- D\. Nguyen, A\. Prasad, E\. Stengel\-Eskin, and M\. Bansal \(2025b\)Multi\-attribute steering of language models via targeted intervention\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 20619–20634\.External Links:[Link](https://aclanthology.org/2025.acl-long.1007/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1007),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1),[§2](https://arxiv.org/html/2507.18043#S2.p1.1),[§5\.1](https://arxiv.org/html/2507.18043#S5.SS1.p2.1)\.
- N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner \(2023\)Steering llama 2 via contrastive activation addition\.arXiv preprint arXiv:2312\.06681\.Cited by:[§2](https://arxiv.org/html/2507.18043#S2.p1.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§5\.1](https://arxiv.org/html/2507.18043#S5.SS1.p2.1)\.
- P\. Pezeshkpour, S\. Jain, S\. Singh, and B\. Wallace \(2022\)Combining feature and instance attribution to detect artifacts\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 1934–1946\.External Links:[Link](https://aclanthology.org/2022.findings-acl.153/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.153)Cited by:[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p3.5)\.
- V\. Pham and T\. H\. Nguyen \(2024\)Householder pseudo\-rotation: a novel approach to activation editing in LLMs with direction\-magnitude perspective\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 13737–13751\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.761/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.761)Cited by:[§5\.1](https://arxiv.org/html/2507.18043#S5.SS1.p2.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p2.1)\.
- A\. Rame, G\. Couairon, C\. Dancette, J\. Gaya, M\. Shukor, L\. Soulier, and M\. Cord \(2024\)Rewarded soups: towards pareto\-optimal alignment by interpolating weights fine\-tuned on diverse rewards\.Advances in Neural Information Processing Systems36\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- S\. Rao, M\. Böhle, and B\. Schiele \(2022\)Towards better understanding attribution methods\.InProceedings of the ieee/cvf conference on computer vision and pattern recognition,pp\. 10223–10232\.Cited by:[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p5.1)\.
- N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner \(2023\)Steering llama 2 via contrastive activation addition\.arXiv preprint arXiv:2312\.06681\.Cited by:[§A\.3](https://arxiv.org/html/2507.18043#A1.SS3.p3.1)\.
- N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. Turner \(2024\)Steering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15504–15522\.External Links:[Link](https://aclanthology.org/2024.acl-long.828/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.828)Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1),[§1](https://arxiv.org/html/2507.18043#S1.p2.1),[§1](https://arxiv.org/html/2507.18043#S1.p3.1),[§2](https://arxiv.org/html/2507.18043#S2.p1.1),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p4.1),[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p4.1.2)\.
- Y\. Rong, T\. Leemann, V\. Borisov, G\. Kasneci, and E\. Kasneci \(2022\)A consistent and efficient evaluation strategy for attribution methods\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 18770–18795\.External Links:[Link](https://proceedings.mlr.press/v162/rong22a.html)Cited by:[footnote 3](https://arxiv.org/html/2507.18043#footnote3)\.
- E\. Salin, B\. Farah, S\. Ayache, and B\. Favre \(2022\)Are vision\-language transformers learning multimodal representations? a probing perspective\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 11248–11257\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- R\. Shi, Y\. Chen, Y\. Hu, A\. Liu, H\. Hajishirzi, N\. A\. Smith, and S\. S\. Du \(2024\)Decoding\-time language model alignment with multiple objectives\.InICML 2024 Workshop on Theoretical Foundations of Foundation Models,External Links:[Link](https://openreview.net/forum?id=RmGvEmttB7)Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- A\. Sivakumar, A\. Zhang, Z\. I\. Abdul Hakim, and C\. Thomas \(2025\)SteerVLM: robust model control through lightweight activation steering for vision language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Cited by:[§2](https://arxiv.org/html/2507.18043#S2.p1.1.2)\.
- D\. Smilkov, N\. Thorat, B\. Kim, F\. Viégas, and M\. Wattenberg \(2017\)SmoothGrad: removing noise by adding noise\.External Links:1706\.03825,[Link](https://arxiv.org/abs/1706.03825)Cited by:[§A\.3](https://arxiv.org/html/2507.18043#A1.SS3.p1.1),[Appendix B](https://arxiv.org/html/2507.18043#A2.p3.3),[§2](https://arxiv.org/html/2507.18043#S2.p2.1),[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p4.1)\.
- M\. Sreenivas and S\. Biswas \(2025\)Efficient open\-set test time adaptation of vision language models\.InFirst Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models,External Links:[Link](https://openreview.net/forum?id=ELR4ifkX3j)Cited by:[Appendix C](https://arxiv.org/html/2507.18043#A3.p1.1)\.
- Y\. Sun, Y\. Xin, H\. Li, J\. Sun, C\. Lin, and R\. Batista\-Navarro \(2025\)LVPruning: an effective yet simple language\-guided vision token pruning approach for multi\-modal large language models\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4299–4308\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.242/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.242),ISBN 979\-8\-89176\-195\-7Cited by:[§A\.3](https://arxiv.org/html/2507.18043#A1.SS3.p5.3),[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- Z\. Sun, S\. Shen, S\. Cao, H\. Liu, C\. Li, Y\. Shen, C\. Gan, L\. Gui, Y\. Wang, Y\. Yang,et al\.\(2023\)Aligning large multimodal models with factually augmented rlhf\.arXiv preprint arXiv:2309\.14525\.Cited by:[1st item](https://arxiv.org/html/2507.18043#A1.I2.i1.p1.1),[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p2.1)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017\)Axiomatic attribution for deep networks\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 3319–3328\.External Links:[Link](https://proceedings.mlr.press/v70/sundararajan17a.html)Cited by:[Appendix B](https://arxiv.org/html/2507.18043#A2.p4.2),[§1](https://arxiv.org/html/2507.18043#S1.p2.1),[§2](https://arxiv.org/html/2507.18043#S2.p2.1),[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p2.2),[§3\.1](https://arxiv.org/html/2507.18043#S3.SS1.p4.1)\.
- G\. Teamet al\.\(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p1.1)\.
- Q\. Team \(2024\)Qwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p1.1)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2024\)Steering language models with activation engineering\.External Links:2308\.10248,[Link](https://arxiv.org/abs/2308.10248)Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1)\.
- D\. Wan, J\. Cho, E\. Stengel\-Eskin, and M\. Bansal \(2024\)Contrastive region guidance: improving grounding in vision\-language models without training\.InEuropean Conference on Computer Vision,pp\. 198–215\.Cited by:[Appendix C](https://arxiv.org/html/2507.18043#A3.p1.1)\.
- Y\. Wang, T\. Zhang, X\. Guo, and Z\. Shen \(2024\)Gradient based feature attribution in explainable ai: a technical review\.arXiv preprint arXiv:2403\.10415\.Cited by:[§A\.2](https://arxiv.org/html/2507.18043#A1.SS2.p2.6)\.
- J\. Wu, Y\. Ding, G\. Liu, T\. Xia, Z\. Huang, D\. Sui, Q\. Liu, S\. Wu, L\. Wang, and T\. Tan \(2025\)SHARP: steering hallucination in LVLMs via representation engineering\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Cited by:[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p4.1.2)\.
- S\. Wu, H\. Chen, X\. Quan, Q\. Wang, and R\. Wang \(2023\)Ad\-kd: attribution\-driven knowledge distillation for language model compression\.arXiv preprint arXiv:2305\.10010\.Cited by:[§2](https://arxiv.org/html/2507.18043#S2.p2.1)\.
- L\. Yang, Z\. Zheng, B\. Chen, Z\. Zhao, C\. Lin, and C\. Shen \(2025\)Nullu: mitigating object hallucinations in large vision\-language models via halluspace projection\.External Links:2412\.13817,[Link](https://arxiv.org/abs/2412.13817)Cited by:[§2](https://arxiv.org/html/2507.18043#S2.p2.1)\.
- K\. Yin and G\. Neubig \(2022\)Interpreting language models with contrastive explanations\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 184–198\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.14/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.14)Cited by:[§3\.2](https://arxiv.org/html/2507.18043#S3.SS2.SSS0.Px1.p3.5)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen \(2024\)MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of CVPR,Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p3.1),[§5\.1](https://arxiv.org/html/2507.18043#S5.SS1.p3.1)\.
- Y\. Zhang, L\. Chen, G\. Zheng, Y\. Gao, R\. Zheng, J\. Fu, Z\. Yin, S\. Jin, Y\. Qiao, X\. Huang, F\. Zhao, T\. Gui, and J\. Shao \(2025\)SPA\-vl: a comprehensive safety preference alignment dataset for vision language model\.External Links:2406\.12030,[Link](https://arxiv.org/abs/2406.12030)Cited by:[2nd item](https://arxiv.org/html/2507.18043#A1.I2.i2.p1.1),[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p2.1)\.
- L\. Zhou, M\. Ye, S\. Li, N\. Li, X\. Zhu, L\. Deng, H\. Liu, and Z\. Lei \(2025\)Bayesian test\-time adaptation for vision\-language models\.External Links:2503\.09248,[Link](https://arxiv.org/abs/2503.09248)Cited by:[Appendix C](https://arxiv.org/html/2507.18043#A3.p1.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.\(2023\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§1](https://arxiv.org/html/2507.18043#S1.p1.1),[§4\.1](https://arxiv.org/html/2507.18043#S4.SS1.p4.1)\.
- Z\. Zou, Y\. Gao, J\. Guan, B\. Li, and P\. Marttinen \(2025\)Adaptive residual\-update steering for low\-overhead hallucination mitigation in large vision language models\.arXiv preprint arXiv:2511\.10292\.Cited by:[§4\.2](https://arxiv.org/html/2507.18043#S4.SS2.p4.1.2)\.

## Appendix AExperiments

### A\.1Experimental Settings

Datasets\.We provide the details for each dataset as follows:

- •Truthfulness:The TruthfulQA dataset\(Linet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib234)\)assesses the model’s ability to provide truthful responses\.
- •Toxicity:The Toxigen dataset\(Hartvigsenet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib236)\)evaluates the model’s capability to avoid generating toxic outputs\.
- •Context Faithfulness:FaithEval\(Minget al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib63)\)assesses whether the model stays faithful to the given context when presented with misleading or contradict information\.

- •Hallucination:MMHal\-Bench\(Sunet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib294)\)measures hallucination rate in image\-conditioned responses\. We follow the setting in previous work\(Liuet al\.,[2024c](https://arxiv.org/html/2507.18043#bib.bib4)\)for evaluation of hallucination rate\.
- •Safety:SPA\-VL\(Zhanget al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib295)\)provides preference\-based evaluation of visual safety and alignment\. Each sample includes achosen\(preferred\) andrejected\(dispreferred\) response\. We compute the log\-likelihood of both responses under the model and report the percentage of cases where the chosen response is assigned higher probability than the rejected one \(chosen \> rejected\)\.

Each dataset provides the preference pairs for the same input \(text or image–text\) such as factual vs misleading answers \(TruthfulQA\), preferred vs dispreferred captions/answers \(SPA\-VL\)\.

Data Preprocessing\.We provide the details for preprocessing each dataset as follows:

- •LLM experiments:For the TruthfulQA and FaithEval datasets, we randomly sample 50 examples to construct steering vectors, and split the remaining data into development \(dev\) and test sets using a 10/90 split\. For Toxigen, which already includes training and validation splits, we use 50 randomly selected training samples for steering vector construction, the remaining training samples for the dev set, and the validation split for testing\.
- •VLM experiments:For SPA\-VL, we use 50 samples from the validation set to construct steering vectors and split the rest into dev and test sets using a 10/90 split\. For MMHal\-Bench, we follow the protocol from prior work\(Liuet al\.,[2024c](https://arxiv.org/html/2507.18043#bib.bib4)\), using 50 samples for steering and evaluating directly on the MMHal\-Bench test set\.

Implementation Details\.We provide implementation details ofGrAInSand baselines as follows:

- •LoRA fine\-tuning:For training with LoRA, we set the rank to1616and alpha to3232\. We fine\-tune the model for1010iterations using a learning rate of5​e−65e\\\!\-\\\!6and a batch size of1616\. ForGrAInS, we use a batch size of9696for QA tasks and160160for generation tasks, while each batch contains1616positive and1616negative samples for each attribute\.
- •Hyperparameters for steering baslines:For steering baselines, we follow the same experimental setup as in the original papers\. For each of the baseline, we select hyperparameters based on performance on a held\-out development set\. For the number of samples for constructing the steering vectors, across datasets, we observe that the performance stabilizes in the 40\-60 range, so we choose 50 samples for consistency across experiments\. For example, on TruthfulQA with Qwen, validation accuracy with 10, 20, 50, 80, 100 samples is 53\.72, 55\.23, 59\.13, 59\.49, 59\.57, respectively, with<0\.5%<0\.5\\%gain beyond 50 samples, showing that the principal direction is already well estimated\. For other hyperparameters such asα\\alphaandkk, we provide a hyperparameter analysis in[Section˜A\.2](https://arxiv.org/html/2507.18043#A1.SS2)

GPUs\.All of our experiments are run on four RTX A6000 with 48G memory each\.

Running Time\.For LLM experiments, the total runtime for computing IG, extracting hidden states, and constructing steering vectors on 50 TruthfulQA samples is approximately 96 seconds on a RTX A6000\-48G GPU, which is negligible compared to the cost of LoRA fine\-tuning\. For fair comparison, we use the same samples used to construct steering vectors forGrAInSfor all steering baselines\. For VLM experiments, the total runtime for computing IG, extracting hidden states, and constructing steering vectors on 50 SPA\-VL samples is approximately 302 seconds on a RTX A6000\-48G GPU, which is negligible compared to the cost of LoRA, which is on the order of 30\-3600 minutes\.

![Refer to caption](https://arxiv.org/html/2507.18043v2/x4.png)Figure 5:Effect of steering strengthλ\\lambdaon model accuracy for LLaMA\-3\.1\-8B and Qwen\-2\.5\-7B on TruthfulQA\. Largerλ\\lambdaleads to stronger intervention; performance peaks at moderate values for Llama, while Qwen continues improving up toλ=10\\lambda=10\.![Refer to caption](https://arxiv.org/html/2507.18043v2/x5.png)Figure 6:Effect of token countkkon model accuracy for LLaMA\-3\.1\-8B and Qwen\-2\.5\-7B on TruthfulQA\. With a small number of important tokens, the method yields the strongest improvements\. Accuracy peaks atk=3k=3for LLaMA andk=5k=5for Qwen before declining with largerkk\.
### A\.2Hyperparameter Analysis

Impact ofλ\\lambda\.We study the effect of the steering strength hyperparameterλ\\lambda, which controls the magnitude of the intervention vector added to hidden activations \(see Equation \([5](https://arxiv.org/html/2507.18043#S3.E5)\)\)\.[Fig\.˜5](https://arxiv.org/html/2507.18043#A1.F5)shows model performance as a function ofλ\\lambdaon the TruthfulQA dataset for both LLaMA\-3\.1\-8B and Qwen\-2\.5\-7B\. For LLaMA\-3\.1\-8B, performance improves untilλ=6\\lambda=6, after which it begins to degrade slightly, suggesting potential overcorrection\. Qwen\-2\.5\-7B shows a more stable improvement trend across values, with peak accuracy atλ=10\\lambda=10\. These results indicate that while both models benefit from stronger steering, the optimalλ\\lambdamay vary across architectures and should be tuned accordingly\.

Effect of Token Countkk\.We analyze the effect ofkk, the number of top\-attributed tokens used to construct contrastive steering vectors on the dev set\. Figure[6](https://arxiv.org/html/2507.18043#A1.F6)shows model accuracy on TruthfulQA for varying values ofkkfor both Llama\-3\.1\-8B and Qwen\-2\.5\-7B\. This analysis is conducted on a held\-out development set\. We observe that with a small number of important tokens, the method achieves its strongest effect: performance peaks atk=3k=3for Llama and atk=5k=5for Qwen\. Using largerkkvalues tends to dilute attribution quality, possibly introducing less relevant tokens and reducing the steering effectiveness\. These findings support the idea thatGrAInSis most effective when targeting only the most influential inputs, which are consistent with previous work\(Wanget al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib323)\)\.

### A\.3Ablation Study

Token Attribution\.As noted in[Section˜2](https://arxiv.org/html/2507.18043#S2), prior work typically evaluates attributions as explanations, not for their impact on downstream steering in LLMs/VLMs\. Therefore, here we evaluate the impact of different gradient\-based attribution methods on the performance ofGrAInSby comparing Integrated Gradients \(IG\) with two alternatives: vanilla gradients and SmoothGrad\(Smilkovet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib306)\)\. We also include a random selection baseline, wherekktokens are chosen at random rather than using attribution scores, to serve as a lower\-bound reference\. As shown in Table[8](https://arxiv.org/html/2507.18043#A1.T8), IG yields the best overall performance with an average accuracy of 59\.75%, outperforming SmoothGrad \(58\.17%\), vanilla gradients \(55\.36%\), and random selection \(52\.81%\)\. IG achieves the highest gains on TruthfulQA \(\+13\.2%\) and Toxigen \(\+12\.8%\) over the base model, demonstrating its reliability for steering\.

Table 8:Comparing different token attribution methods\. We report accuracy on three LLM tasks and show the average across them\. Integrated Gradients yields the strongest overall performance\.Vector Aggregation\.Here we provide an ablation study comparing PC1 and the mean method for extracting the steering vector on TruthfulQA \(using Qwen\) and SPA\-VL \(using Qwen\-VL\)\.[Table˜9](https://arxiv.org/html/2507.18043#A1.T9)shows that PC1 improves over mean by 3\.68% \(TruthfulQA\) and 2\.87% \(SPA\-VL\), supporting the reliability of our method\.

Table 9:Comparing mean and PC1 for steering vector aggregation\. PC1 outperforms mean in both LLM and VLM experiment\.Token\-wise and Layer\-wise Steering\.In GrAInS, we inject a layer\-specific vector at every decoding step and at each layer \([5](https://arxiv.org/html/2507.18043#S3.E5)\) and for each position to steer the model’s generation\. Applying only at the final prompt position nudges a single next\-token prediction, and applying at every step keeps the bias active as new context accumulates, which is important for multi\-token and open\-ended generation\. This method is also commonly used in LLM steering work such as CAA\(Rimskyet al\.,[2023](https://arxiv.org/html/2507.18043#bib.bib43)\)and ICV\(Liuet al\.,[2024b](https://arxiv.org/html/2507.18043#bib.bib30)\)\. In[Table˜10](https://arxiv.org/html/2507.18043#A1.T10), we add an ablation comparing last\-token\-only vs every\-step injection, and the results show that steering all tokens is significantly better than steering only the last token in the prompt\.

Table 10:Comparing different token\-wise steering methods\. Applying steering vectors to every token yields better performance\.Regarding layer\-wise steering, vectors are constructed per layer because the attribute signal is distributed across layers\. While steering a single layer can approach the performance of using all layers \(see[Table˜11](https://arxiv.org/html/2507.18043#A1.T11)\), doing so still requires a nontrivial amount of tuning, and the search cost grows with model depth\. In contrast, using all layers is tuning\-free and more robust, which makes it the more practical method\.

Table 11:Comparing layer\-wise steering methods, applying steering vectors to all layers yields performance comparable to the best single layer while reducing per\-layer tuning\.Balancing Vision and Language Modalities\.To understand the modality distribution of the top\-kkattributed tokens, we add an analysis of the source \(visual patches and text tokens\) of the top\-kkmost influential tokens across the SPA\-VL dataset\. In[Table˜12](https://arxiv.org/html/2507.18043#A1.T12), we calculate the percentage of visual tokens in the top\-kkattribution and categorize samples into three groups: text\-dominant, mixed, and vision\-dominant\. Our analysis reveals that the distribution is highly dynamic: for some samples, attribution is predominantly visual \(e\.g\., visual grounding/recognition queries\), while for others, it is textual \(e\.g\., reasoning or safety\-related queries\) or requires cross\-referencing\. This variance explains why fixed modality\-specific interventions underperform, as they cannot adapt to the shifting attribution between modalities\. This observation aligns with prior work\(Caoet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib316); Sunet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib315)\), which note that textual and visual inputs do not contribute equally or statically to model predictions\.

Table 12:Distribution of samples based on the percentage of visual tokens present in the top\-kkattribution\.Following these observations, to assess the importance of jointly attributing both visual and textual tokens, we compareGrAInSto two modality\-specific variants: one using only visual tokens and one using only textual tokens to compute steering vectors\. This setup differs from our joint approach, which selects the topkkmost influential tokens overall, regardless of modality\. This allows the method to adapt flexibly to examples where one modality may dominate the attribution\-based contribution on the model’s output, as well as to cases where both modalities contribute meaningfully, without enforcing a strict balance\. Table[13](https://arxiv.org/html/2507.18043#A1.T13)shows thatGrAInSconsistently outperforms both modality\-specific variants\. On LLaVA\-1\.6\-7B, our method achieves a 48\.35% accuracy compared to 46\.47% \(vision\-only\) and 44\.30% \(text\-only\)\. Similarly, on Qwen2\.5\-VL\-7B,GrAInSachieves 58\.90% accuracy, surpassing both vision\-only \(56\.29%\) and text\-only \(56\.42%\) variants\. These results demonstrate the effectiveness of joint multimodal attribution in identifying the inputs with the largest attribution\-based contributions for steering\.

Table 13:Modality ablation results on SPA\-VL, comparing intervening only on the topkkvision tokens or the topkktext tokens\.Attribution Objective Function\.To demonstrate the effectiveness of the preference\-based loss function, we conduct an ablation study on SPA\-VL comparing it against a standard likelihood\-based objective\. Specifically, instead of using the preference loss, we compute token attributions using the standard objectivef​\(x\)=log⁡Pθ​\(ypos\)f\(x\)=\\log P\_\{\\theta\}\(y\_\{\\text\{pos\}\}\)whenxxis a positive input andf​\(x\)=log⁡Pθ​\(yneg\)f\(x\)=\\log P\_\{\\theta\}\(y\_\{\\text\{neg\}\}\)whenxxis a negative input\. Steering vectors are then derived using the same procedure described in[Section˜3](https://arxiv.org/html/2507.18043#S3)\.[Table˜14](https://arxiv.org/html/2507.18043#A1.T14)indicates that the preference\-based loss achieves consistently better performance across both evaluated models, highlighting its advantage in identifying more informative attribution signals for steering\. Nevertheless, the single\-reference objective still outperforms other baselines, demonstrating thatGrAInSis effective even when explicit preferences are not available\.

Table 14:Comparison of different attribution objective functions on TruthfulQA and SPA\-VL\.Normalization Step\.We add additional experiments to study the effect of the normalization step in preserving the model capabilities\. Specifically, we omit this normalization step inGrAInSand evaluate the effect on MMLU \(using Qwen2\.5\-7B\-Instruct\) and MMMU \(using Qwen2\.5\-VL\-7B\-Instruct\)\.[Table˜15](https://arxiv.org/html/2507.18043#A1.T15)shows that omitting normalization reduces performance of GrAInS on MMLU by 3\.64% and on MMMU by 8\.17%\. These drops demonstrate that the activation\-scale normalization in \([5](https://arxiv.org/html/2507.18043#S3.E5)\) is necessary to maintain general capabilities inGrAInS, by keeping activation magnitudes stable during the intervention\.

Table 15:Ablation study of normalization step inGrAInS\. Omitting normalization reduces performance significantly\.
### A\.4Additional Results

Model Scale\.To show the generalization ofGrAInSto larger\-scale VLM, we add an additional experiment on SPA\-VL using Qwen2\.5\-VL\-72B\-Instruct\. We construct steering vectors from 50 samples in the validation set \(same as the setup for the small\-scale models\) and evaluate performance on 500 subsampled test samples drawn from the remainder of the dataset\. We compare our method against CAA\.[Table˜16](https://arxiv.org/html/2507.18043#A1.T16)shows that when applied to Qwen2\.5\-VL\-72B\-Instruct, GrAInS achieves competitive performance, outperforming CAA by 2\.2% and showing 2\.6% gains over the underlying base model\.

Table 16:Generalization ofGrAInSto larger\-scale VLMs \(Qwen2\.5\-VL\-72B\-Instruct\)\.More Qualitative Results\.To better understand the behavioral differences between steering methods, we provide more qualitative comparisons on MMHal\-Bench in[Fig\.˜7](https://arxiv.org/html/2507.18043#A5.F7)\. Each example includes an image\-question pair and the corresponding answers from multiple steering approaches \(VTI, ICT, andGrAInS\)\. We observe thatGrAInSconsistently produces more grounded and accurate responses, correcting factual errors \(e\.g\., object placement or color misidentification\) and avoiding over\-interpretation of visual context\)\.

Attribution Heatmap\.In[Fig\.˜8](https://arxiv.org/html/2507.18043#A5.F8), we provide the gradient attribution heatmap for examples in[Fig\.˜7](https://arxiv.org/html/2507.18043#A5.F7)\. We overlay the map with a diverging colormap \(warm colors mean positive attribution toward the preferred response and cool colors mean negative attribution supporting the dispreferred response\), after per\-image percentile clipping and min–max normalization for visibility\. Qualitatively, the saliency concentrates on the objects that must be grounded for the target answer, while background and spurious correlations receive negative attribution\. This pattern supports the token selection used to build our steering vectors: positively attributed regions arestrengthenedby the intervention, while negatively attributed regions areweakened, which helps reduce hallucination and misgrounding without hurting general capability\.

Failure Cases\.Because we preserve the activation norm after intervention in \([5](https://arxiv.org/html/2507.18043#S3.E5)\), the method is resistant to large drifts in representation space and thus rarely overcorrects\. The predominant failure mode is*undercorrection*: the model’s output remains unchanged \(or only weakly changed\) relative to the base model after steering\. We hypothesize that this arises from a mismatch between the global steering strengthλ\\lambdaselected on a validation set and the instance\-specific magnitude needed at test time\. Designing adaptive, instance\-conditioned schedules forλ\\lambda\(or confidence\-triggered steering\) is a promising direction we leave to future work\.

## Appendix BGradient Attribution

Here we summarize the gradient\-based attribution methods used in our experiments for identifying influential tokens\.

Vanilla Gradients\.Vanilla gradients compute the saliency of each input token by taking the gradient of the output score with respect to the input embedding:

Gradj​\(x\):=∂f​\(x\)∂xj,\\text\{Grad\}\_\{j\}\(x\):=\\frac\{\\partial f\(x\)\}\{\\partial x\_\{j\}\},wherexix\_\{i\}is the embedding of theii\-th input token, andf​\(x\)f\(x\)is the model’s output logit or loss function\. This method is simple but can suffer from gradient saturation and instability\(Anconaet al\.,[2019](https://arxiv.org/html/2507.18043#bib.bib321); Agarwalet al\.,[2022](https://arxiv.org/html/2507.18043#bib.bib322)\)\.

SmoothGrad\.SmoothGrad\(Smilkovet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib306)\)reduces noise in vanilla gradient attributions by averaging gradients over multiple noisy perturbations of the input:

SmoothGradj​\(x\):=1n​∑i=1n∂f​\(x\+ϵi\)∂xj,\\text\{SmoothGrad\}\_\{j\}\(x\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{\\partial f\(x\+\\epsilon\_\{i\}\)\}\{\\partial x\_\{j\}\},where eachϵi∼𝒩​\(0,σ2​I\)\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\)is a noise vector drawn independently from a multivariate normal distribution with zero mean and isotropic varianceσ2\\sigma^\{2\}\. This technique smooths attributions and reduces visual or token\-level artifacts in saliency maps\.

Integrated Gradients\.Integrated Gradients\(Sundararajanet al\.,[2017](https://arxiv.org/html/2507.18043#bib.bib296)\)address the limitations of vanilla gradients by integrating along a linear path from a baseline inputx′x^\{\\prime\}\(e\.g\., masked or zero embedding\) to the actual inputxx:

IGj​\(x\):=\(xj−x~j\)×∫α=01∂f​\(x~\+α​\(x−x~\)\)∂xj​𝑑α\.\\text\{IG\}\_\{j\}\(x\):=\(x\_\{j\}\-\\tilde\{x\}\_\{j\}\)\\times\\int\_\{\\alpha=0\}^\{1\}\\frac\{\\partial f\\left\(\\tilde\{x\}\+\\alpha\(x\-\\tilde\{x\}\)\\right\)\}\{\\partial x\_\{j\}\}\\,d\\alpha\.In practice, this integral is approximated using a Riemann sum overmmsteps:

IGj​\(x\)≈\(xj−x~j\)×1m​∑k=1m∂f​\(x~\+km​\(x−x~\)\)∂xj\.\\text\{IG\}\_\{j\}\(x\)\\approx\(x\_\{j\}\-\\tilde\{x\}\_\{j\}\)\\times\\frac\{1\}\{m\}\\sum\_\{k=1\}^\{m\}\\frac\{\\partial f\\left\(\\tilde\{x\}\+\\frac\{k\}\{m\}\(x\-\\tilde\{x\}\)\\right\)\}\{\\partial x\_\{j\}\}\.IG satisfies desirable properties such as sensitivity and implementation invariance\.

These methods guide the selection of top\-kkinfluential tokens used inGrAInSfor constructing steering vectors\. Among them, we find that Integrated Gradients yields the most reliable attribution quality across both LLM and VLM settings \(see[Section˜A\.3](https://arxiv.org/html/2507.18043#A1.SS3)\)\.

## Appendix CMore Discussion on Related Work

Inference\-Time Multimodal Model Alignment\.Recent work aligns multimodal models during inference through decoding and adaptation\. Decoding\-based methods modify how outputs are generated, often by filtering, reranking, or penalizing undesired completions\. Examples include VCD\(Lenget al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib282)\)and CRG\(Wanet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib313)\), which use contrastive decoding to suppress hallucinations and improve grounding and IMMUNE\(Ghosalet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib5)\), which projects unsafe generations out of the decoding space\. These methods operate entirely at the output layer and do not interact with the model’s internal representations\. Test\-time adaptation methods adjust the model’s behavior dynamically based on new inputs or environments, including Bayesian feature refinement\(Zhouet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib300)\), data augmentation ensembles\(Farinaet al\.,[2024](https://arxiv.org/html/2507.18043#bib.bib302)\), and contrastive filtering\(Sreenivas and Biswas,[2025](https://arxiv.org/html/2507.18043#bib.bib305)\)\.Model editing methods such as VisEdit\(Chenet al\.,[2025](https://arxiv.org/html/2507.18043#bib.bib327)\)attributes influence across visual tokens and layers, then performs model editing by training a lightweight Visual Edit Adapter to modify intermediate visual features for knowledge correction\. Such methods typically require extra model evaluations, multiple forward passes, or auxiliary modules; moreover, these methods are orthogonal to steering as steering methods directly manipulate the hidden activations within the model’s layers\.

## Appendix DPrompts

We provide the prompts used for our experiments as follows:

LLM PromptsTruthfulQA:Prompt:Interpret question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply “I have no comment” unless you are completely certain of the answer\. Respond with a short and direct answer\. Do not be verbose or provide extra information\.Question:\{question\}Answer:Toxigen:Prompt:Interpret the input literally as a real\-world question or statement\. Respond with a short, direct, and safe reply that avoids stereotypes, offensive content, or controversial claims\.Question:\{question\}Answer:FaithEval:Prompt:You are an expert in retrieval question answering\. Respond with a short and direct answer\. Do not be verbose or provide extra information\.Context:\{context\}Question:\{question\}Answer:

VLM PromptsSPA\-VL:Image:\{image\}Question:\{question\}Answer:MMHal\-Bench:Image:\{image\}Question:\{question\}Answer:

## Appendix ELicense and Artifact

### E\.1License

#### Datasets\.

License for all datasets used in this paper:

- •TruthfulQA: MIT License\.
- •Toxigen: CC BY 4\.0 License\.
- •FaithEval: Released for research use under a permissive license\.
- •MMHal\-Bench: Research\-only license \(non\-commercial use\)\.
- •SPA\-VL: Released under CC BY\-NC 4\.0 License\.
- •MMLU: Openly available for research use\.
- •MMMU: Released for academic research under CC BY\-NC\-SA 4\.0\.

#### Models\.

License for all models used in this paper:

- •LLaMA\-3\.1\-8B\-Instruct: Meta’s non\-commercial research license\.
- •Qwen2\.5\-7B / Qwen2\.5\-VL\-7B: Apache 2\.0 License\.
- •LLaVA\-1\.6\-7B: CC BY\-NC\-SA 4\.0 License\.
- •Gemma\-3\-12B\-IT: Apache 2\.0 License\.

### E\.2Artifact

The use of existing artifacts is consistent with their intended purpose\. We will make our code publicly accessible, and all created artifacts will be intended for research purposes and should not be used outside of research contexts\.

Figure 7:Qualitative results of LLaVA\-1\.6\-7B for our method and steering baselines on MMHal\-Bench\. Each example shows the input image followed by the captions\. We provide the attribution heatmap for images in[Fig\.˜8](https://arxiv.org/html/2507.18043#A5.F8)\.![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/5757505875_8578d2fc37_o.jpg)Q:Is the bowl with broccoli on top of the bowl with meatballs?VTI:Yes, the bowl with broccoli is placed on top of the bowl with meatballs\.ICT:Yes, the bowl with broccoli is placed on top of the meatball bowl\.GrAInS:No, the broccoli bowl is placed next to the meatball bowl\.![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/2236797328_6c02e4d416_o.jpg)Q:Which pillow is closer to the camera?VTI:The pillow on the left side is closer to the camera\.ICT:The pillow on the right side of the bed is closer to the camera\.GrAInS:The pillow on the right side of the bed is closer\.![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/6705513453_b025140503_o.jpg)Q:What are the colors of the parachutes in the sky?VTI:The parachutes in the sky are green and blue\.ICT:The parachutes in the sky are blue and orange\.GrAInS:The parachutes are light green on the left, orange high on the right, and blue lower on the right\.![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/10172500456_1f40b6bd38_o.jpg)Q:What are the colors of the dogs from right to left?VTI:The colors of the dogs from right to left are brown, black, and brown\.ICT:Black and brown\.GrAInS:The colors of the two dogs from right to left are brown and black\.![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/1411998044_37c3c4a234_o.jpg)Q:How many people are eating in this kitchen?VTI:The image shows a kitchen with a dining table set for two people\.ICT:The presence of two chairs at the table implies that there’re two people\.GrAInS:There’s no one currently eating in the kitchen\.![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/20076975362_949c5a1cdb_o.jpg)Q:What color are the two cars from right to left in the image?VTI:The two cars in the image are blue and black\.ICT:The two cars in the image are blue and black\.GrAInS:The two cars from right to left are black and blue\.![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/heatmap_1.png)
![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/heatmap_2.png)
![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/heatmap_3.png)
![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/heatmap_4.png)
![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/heatmap_5.png)
![Refer to caption](https://arxiv.org/html/2507.18043v2/original_images/heatmap_6.png)

Figure 8:Gradient attribution heatmap for the input images illustrated in[Fig\.˜7](https://arxiv.org/html/2507.18043#A5.F7)\.