Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

# A Geometrical Analysis of Self-Attention Transformers
Source: [https://arxiv.org/html/2606.07604](https://arxiv.org/html/2606.07604)
## Contribution Weights: A Geometrical Analysis of Self\-Attention Transformers

###### Abstract

Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models \(LLMs\)\. However, this approach has significant limitations as it neglects the geometric properties of the value vectors being aggregated\. To address this gap, we introduce*Contribution Weights*, a projection\-based metric that quantifies a token’s influence by accounting for it’s attention weight, value magnitude, and directional alignment with the layer output\. We demonstrate that contribution weights provide a more faithful measure of token importance, consistently outperforming attention\-based metrics in identifying semantically critical tokens across different decoder\-only models, tasks, and datasets\. Further, our metric enables novel mechanistic analysis of*attention sinks*\. While previous work characterized sinks as passive repositories for excess attention, we reveal they serve an active functional role, suppressing information through a convex relationship between sink rate and output norm, stabilizing representations by opposing the semantic drift of low\-confidence tokens\.

Machine Learning, ICML

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.07604v1/x1.png)Figure 1:Contribution weights\.\(Top\)Attention matrices and\(Bottom\)corresponding contribution weight matrices, at two randomly selected heads and layers, computed usingLLaMA\-3\.1\-8B\.Whilst attention weights place a large amount of attention mass on the first token, examining the contribution weights reveals that these tokens contribute very little to the output\.Self\-attention governs information mixing in Large Language Models \(LLMs\)\([Devlin et al\.,](https://arxiv.org/html/2606.07604#bib.bib11);[Brown et al\.,](https://arxiv.org/html/2606.07604#bib.bib5)\)by computing an input\-dependent weighted sum of tokens via attention weightsαi​j\\alpha\_\{ij\}\([Vaswani et al\.,](https://arxiv.org/html/2606.07604#bib.bib32)\)\. A common interpretability approach analyzes these weights directly, often implicitly treating large attention mass as evidence of functional importance\. However, many prior works have questioned this*attention\-as\-explanation*view\([Jain & Wallace,](https://arxiv.org/html/2606.07604#bib.bib21);[Bastings & Filippova,](https://arxiv.org/html/2606.07604#bib.bib2);[Lopardo et al\.,](https://arxiv.org/html/2606.07604#bib.bib25)\), arguing that attention weights alone provide an incomplete account of information aggregation, as they do not account for the latent geometry of the vectors being aggregated\(Kobayashi et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib23); Ferrando et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib15),[b](https://arxiv.org/html/2606.07604#bib.bib16)\)\.

For interpretability analysis, multi\-head attention \(MHA\) can be equivalently computed as a weighted sum of head\-wise independent value vectors, aggregating information over the sequence dimension, where each value vector is projected by a learned linear map that shapes value geometry\(Kobayashi et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib23);[Elhage et al\.,](https://arxiv.org/html/2606.07604#bib.bib14)\)\. From this perspective, the influence of a token on the layer output depends not only on it’s attention weight, but also on the magnitude of it’s value vector and its alignment with the all other attended to tokens\. Neglecting these geometric factors can lead to misleading conclusions by ignoring interference among vectors and by overestimating the importance of high\-attention tokens\. A prominent example is the study of*attention sinks*, where early tokens receive substantial attention despite contributing little to the resulting representation once value norms and directional alignment are accounted for\.

In this work, we introduce*contribution weights*, a projection\-based metric that accounts for the full geometry of the attention vector sum and that faithfully captures token importance\. We show that contribution weights decompose into the product of attention weight, relative magnitude, and directional alignment between value vectors and the output\. Using this decomposition, we study the functional role of value geometry in self\-attention, with particular focus on sink tokens\.

Our key findings and contributions are:

- •Contribution weights faithfully measure token importance\.We demonstrate through causal interventions that contribution weights consistently outperform attention\-based metrics in identifying critical tokens across multiple models, tasks, and datasets\.
- •Directional alignment is critical to token contribution\.Incorporating the directional alignment of value vectors substantially improves the accuracy of contribution estimates across all sequence positions, revealing that geometric orientation plays a more important role than magnitude in determining token influence\.
- •*Value geometry drives the attention sink phenomenon\.*Initial token value vectors are anti\-aligned with other tokens in the sequence, and their contribution to the output is independent of their attention scores, explaining why high attention on these positions does not translate to semantic influence\.
- •Sink tokens actively suppress information\.Contrary to existing characterizations of sinks as passive attention absorbers, we show that sink tokens function as active information suppressors that oppose the semantic direction of non\-sink tokens, thereby nullifying semantic drift introduced by weak attention accumulation\.

## 2Related Work

#### Geometry\-Aware Attention Analysis\.

To address the limitations of weight\-based attention analysis, several value\-aware measures incorporate geometric properties such as value norms\(Kobayashi et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib23),[b](https://arxiv.org/html/2606.07604#bib.bib24)\)and value\-output distances\(Ferrando et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib15)\)\. These methods more faithfully identify important tokens\(Kobayashi et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib23),[b](https://arxiv.org/html/2606.07604#bib.bib24)\)and enable effective token pruning\(Ferrando et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib15)\)and KV cache compression\([Devoto et al\.,](https://arxiv.org/html/2606.07604#bib.bib12); Guo et al\.,[b](https://arxiv.org/html/2606.07604#bib.bib19);[Shin et al\.,](https://arxiv.org/html/2606.07604#bib.bib29)\)\. Nevertheless, existing approaches capture only partial value geometry, neglecting effects such as inter\-token interference, which we show to be crucial for understanding attention’s functional role\.

#### Attention Sinks and Value Geometry\.

Understanding how value geometry operates in tandem with attention weights is crucial for explaining phenomena like attention sink, where tokens receive disproportionately high attention despite carrying minimal semantic content\([Xiao et al\.,](https://arxiv.org/html/2606.07604#bib.bib33)\)\. Recent work has characterized specific geometric properties of sink tokens, including high\-norm residual stream hidden states\([Sun et al\.,](https://arxiv.org/html/2606.07604#bib.bib30)\), small value norms\(Kobayashi et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib23);[Devoto et al\.,](https://arxiv.org/html/2606.07604#bib.bib12); Guo et al\.,[b](https://arxiv.org/html/2606.07604#bib.bib19)\), and negative cosine similarity with non\-sink hidden states\([Shin et al\.,](https://arxiv.org/html/2606.07604#bib.bib29);[Cancedda,](https://arxiv.org/html/2606.07604#bib.bib6)\)\. These properties suggest sinks function as implicit bias terms\([Sun et al\.,](https://arxiv.org/html/2606.07604#bib.bib30);[Darcet et al\.,](https://arxiv.org/html/2606.07604#bib.bib9)\), absorbing excess attention and deactivating heads by driving output norms toward zero\([Bondarenko et al\.,](https://arxiv.org/html/2606.07604#bib.bib4); Guo et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib18)\), slowing the rate of sequence mixing\([Barbero et al\.,](https://arxiv.org/html/2606.07604#bib.bib1)\)\. However, despite these insights, the precise geometric mechanisms by which sinks affect sequence mixing, particularly through inter\-token interference, remain under explored\.

## 3Background

In this section, we provide background on the attention mechanism, explicitly stating it’s form as a weighted sum of projected value vectors\.

### 3\.1Multi\-Head Attention as Weighted Sum

Given an input sequence𝐗∈ℝL×d\{\\mathbf\{X\}\}\\in\\mathbb\{R\}^\{L\\times d\}of lengthLLand dimensiondd, causal self\-attention\([Vaswani et al\.,](https://arxiv.org/html/2606.07604#bib.bib32)\)computes the output𝐘∈ℝL×d′\{\\mathbf\{Y\}\}\\in\\mathbb\{R\}^\{L\\times d^\{\\prime\}\}as a weighted summation over previous tokensj≤ij\\leq i\. At token positioni∈Li\\in L,

𝒚i=∑j=1iexp⁡\(𝒒i​𝒌jT/dh\)∑n=1iexp⁡\(𝒒i​𝒌nT/dh\)​𝒗j=∑j=1iαi​j​𝒗j\{\\bm\{y\}\}\_\{i\}=\\sum^\{i\}\_\{j=1\}\\frac\{\\exp\(\{\\bm\{q\}\}\_\{i\}\{\\bm\{k\}\}\_\{j\}^\{T\}/\\sqrt\{d\_\{h\}\}\)\}\{\\sum^\{i\}\_\{n=1\}\\exp\(\{\\bm\{q\}\}\_\{i\}\{\\bm\{k\}\}\_\{n\}^\{T\}/\\sqrt\{d\_\{h\}\}\)\}\{\\bm\{v\}\}\_\{j\}=\\sum^\{i\}\_\{j=1\}\\alpha\_\{ij\}\{\\bm\{v\}\}\_\{j\}\(1\)whereαi​j∈\[0,1\]\\alpha\_\{ij\}\\in\[0,1\]are the scalarattention weightssumming to11alongj≤ij\\leq i,𝐐,𝐊,𝐕=𝐗​𝑾Q,𝐗​𝑾K,𝐗​𝑾V∈ℝL×d′\{\\mathbf\{Q\}\},\{\\mathbf\{K\}\},\{\\mathbf\{V\}\}=\{\\mathbf\{X\}\}\\bm\{W\}\_\{Q\},\{\\mathbf\{X\}\}\\bm\{W\}\_\{K\},\{\\mathbf\{X\}\}\\bm\{W\}\_\{V\}\\in\\mathbb\{R\}^\{L\\times d^\{\\prime\}\}are the query, key and value projections, with𝑾Q,𝑾K,𝑾V∈ℝd×d′\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\}\\in\\mathbb\{R\}^\{d\\times d^\{\\prime\}\}learnable parameter matrices\.

The output of MHA𝒐i∈ℝd\{\\bm\{o\}\}\_\{i\}\\in\\mathbb\{R\}^\{d\}is conventionally expressed as the concatenation ofHH*independent*attention heads𝒚i\(h\)∈ℝd′\{\\bm\{y\}\}\_\{i\}^\{\(h\)\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\}, withd=H⋅d′d=H\\cdot d^\{\\prime\}, followed by a single output projection𝑾O∈ℝd×d\\bm\{W\}\_\{O\}\\in\\mathbb\{R\}^\{d\\times d\},

𝒐i=Concat​\(\{𝒚i\(h\)\}h=1H\)​𝑾O\\displaystyle\{\\bm\{o\}\}\_\{i\}=\\text\{Concat\}\(\\\{\{\\bm\{y\}\}\_\{i\}^\{\(h\)\}\\\}^\{H\}\_\{h=1\}\)\\bm\{W\}\_\{O\}\(2\)Partitioning the output projection matrix intoHHblocks,𝑾O=Concat​\(\{𝑾O\(h\)\}h=1H\)∈ℝH​d′×d\\bm\{W\}\_\{O\}=\\text\{Concat\}\(\\\{\\bm\{W\}\_\{O\}^\{\(h\)\}\\\}\_\{h=1\}^\{H\}\)\\in\\mathbb\{R\}^\{Hd^\{\\prime\}\\times d\}, where𝑾O\(h\)∈ℝd′×d\\bm\{W\}\_\{O\}^\{\(h\)\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\\times d\}, we can express the MHA output as a summation over attention heads\(Kobayashi et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib23);[Elhage et al\.,](https://arxiv.org/html/2606.07604#bib.bib14)\),

𝒐i=\[𝒚i\(1\),𝒚i\(2\),…\]​\[𝑾O\(1\)𝑾O\(2\)⋮\]=∑h=1H𝒚i\(h\)​𝑾O\(h\)\\begin\{split\}\{\\bm\{o\}\}\_\{i\}=\\begin\{bmatrix\}\{\\bm\{y\}\}\_\{i\}^\{\(1\)\},\{\\bm\{y\}\}\_\{i\}^\{\(2\)\},\\ldots\\end\{bmatrix\}\\begin\{bmatrix\}\\bm\{W\}^\{\(1\)\}\_\{O\}\\\\ \\bm\{W\}^\{\(2\)\}\_\{O\}\\\\ \\vdots\\end\{bmatrix\}=\\sum^\{H\}\_\{h=1\}\{\\bm\{y\}\}\_\{i\}^\{\(h\)\}\\bm\{W\}^\{\(h\)\}\_\{O\}\\end\{split\}\(3\)
From this perspective, MHA can be equivalently viewed as the sum of independently projected head outputs𝒐i\(h\):=𝒚i\(h\)​𝑾O\(h\)∈ℝd\{\\bm\{o\}\}\_\{i\}^\{\(h\)\}:=\{\\bm\{y\}\}\_\{i\}^\{\(h\)\}\\bm\{W\}\_\{O\}^\{\(h\)\}\\in\\mathbb\{R\}^\{d\}, each additively contributing to the final representation\.

### 3\.2Attention as Independent Linear Maps

Substituting the definition of attention \([Equation1](https://arxiv.org/html/2606.07604#S3.E1)\) into the additive representation of MHA \([Equation3](https://arxiv.org/html/2606.07604#S3.E3)\), one equivalently obtains𝒐i\{\\bm\{o\}\}\_\{i\}as

𝒐i=∑h=1H∑j=1iαi​j\(h\)​𝒙j​𝑾V\(h\)​𝑾O\(h\)\{\\bm\{o\}\}\_\{i\}=\\sum^\{H\}\_\{h=1\}\\sum^\{i\}\_\{j=1\}\\alpha\_\{ij\}^\{\(h\)\}\{\\bm\{x\}\}\_\{j\}\\bm\{W\}\_\{V\}^\{\(h\)\}\\bm\{W\}\_\{O\}^\{\(h\)\}\(4\)This expression makes explicit that MHA factors into two independent head\-level linear operations: \(i\) a weighted sum over thesequence dimensionwith scalar coefficientsαi​j\(h\)\\alpha\_\{ij\}^\{\(h\)\}, which adaptively determines how much tokenjjcontributes to positionii, and \(ii\) a*low\-rank*linear map over thefeature dimension𝑾V\(h\)​𝑾O\(h\)∈ℝd×d\\bm\{W\}\_\{V\}^\{\(h\)\}\\bm\{W\}\_\{O\}^\{\(h\)\}\\in\\mathbb\{R\}^\{d\\times d\}, which shapes the geometry of the𝑾O\(h\)\\bm\{W\}\_\{O\}^\{\(h\)\}\-projected value vectors\.

For ease of notation we refer to the inputs transformed by the product𝑾V\(h\)​𝑾O\(h\)\\bm\{W\}\_\{V\}^\{\(h\)\}\\bm\{W\}\_\{O\}^\{\(h\)\}asprojected value vectors,

𝒗~j\(h\):=𝒙j​𝑾V\(h\)​𝑾O\(h\)=𝒗j\(h\)​𝑾O\(h\)∈ℝd\.\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(h\)\}:=\{\\bm\{x\}\}\_\{j\}\\bm\{W\}^\{\(h\)\}\_\{V\}\\bm\{W\}\_\{O\}^\{\(h\)\}=\{\\bm\{v\}\}^\{\(h\)\}\_\{j\}\\bm\{W\}\_\{O\}^\{\(h\)\}\\in\\mathbb\{R\}^\{d\}\.\(5\)Combining these observations, the output of an attention layerllin a transformer can be expressed as a weighted sum over projected value vectors:

𝒐i\(l\)=∑h=1H∑j=1iαi​j\(l,h\)​𝒗~j\(l,h\)\{\\bm\{o\}\}^\{\(l\)\}\_\{i\}=\\sum^\{H\}\_\{h=1\}\\sum^\{i\}\_\{j=1\}\\alpha^\{\(l,h\)\}\_\{ij\}\{\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\}\(6\)where𝒗j~\(l,h\)=𝒉^j\(l−1\)​𝑾V\(l,h\)​𝑾O\(l,h\)\{\\tilde\{\{\\bm\{v\}\}\_\{j\}\}^\{\(l,h\)\}=\\hat\{\{\\bm\{h\}\}\}\_\{j\}^\{\(l\-1\)\}\\bm\{W\}\_\{V\}^\{\(l,h\)\}\\bm\{W\}\_\{O\}^\{\(l,h\)\}\}are the projected value vectors, and𝒉^j\(l\)=Norm\(l\)​\(𝒉j\(l\)\)\\hat\{\{\\bm\{h\}\}\}\_\{j\}^\{\(l\)\}=\\text\{Norm\}^\{\(l\)\}\(\{\\bm\{h\}\}\_\{j\}^\{\(l\)\}\)are the normalised hidden state from the preceding layer\.

This formulation of MHA highlights two key properties: \(i\) attention constructs hierarchical representations through successive stages of aggregation— value vectors are weighted and summed to form head outputs, which are in turn summed to produce layer outputs, \(ii\) attention explicitly shapes the geometry of value vectors, an often overlooked property that is central to our analysis of token contributions in the following section\.

## 4Contribution of Attention

We now introduce*contribution weights*, a projection based metric to analyse the behaviour of the attention mechanism, that accounts for both the magnitude of the attention weights and the geometry of value vectors\.

### 4\.1Determinants of Vector Summation

Consider a weighted summation ofNNvectors𝒗i∈ℝd\{\\bm\{v\}\}\_\{i\}\\in\\mathbb\{R\}^\{d\}, each scaled by a scalarαi∈ℝ\\alpha\_\{i\}\\in\\mathbb\{R\}:

𝒚=∑i=1Nαi​𝒗i\.\{\\bm\{y\}\}=\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\{\\bm\{v\}\}\_\{i\}\.\(7\)The resulting𝒚∈ℝd\{\\bm\{y\}\}\\in\\mathbb\{R\}^\{d\}is determined by three main factors:

1. 1\.Magnitude ofαi\\alpha\_\{i\}\.Each coefficientαi\\alpha\_\{i\}scales its corresponding vector𝒗i\{\\bm\{v\}\}\_\{i\}, modulating its relative influence\. Under SoftMax attention,αi∈\[0,1\]\\alpha\_\{i\}\\in\[0,1\]:αi=0\\alpha\_\{i\}=0implies no contribution,0<αi<10<\\alpha\_\{i\}<1attenuates the effect, andαi=1\\alpha\_\{i\}=1preserves the full vector\.
2. 2\.Norm ofvi\{\\bm\{v\}\}\_\{i\}\.Even whenαi\\alpha\_\{i\}values are equal, vectors with larger norms contribute more strongly, since‖αi​𝒗i‖=\|αi\|​‖𝒗i‖\\\|\\alpha\_\{i\}\{\\bm\{v\}\}\_\{i\}\\\|=\|\\alpha\_\{i\}\|\\\|\{\\bm\{v\}\}\_\{i\}\\\|\. Hence, the combined influence ofαi\\alpha\_\{i\}and‖𝒗i‖\\\|\{\\bm\{v\}\}\_\{i\}\\\|determines each vector’s effective strength in the summation\.
3. 3\.Alignment ofvi\{\\bm\{v\}\}\_\{i\}\.The relative orientation of vectors determines the degree of interference in the summation\. When pairwise cosine similaritiescos⁡\(𝒗i,𝒗j\)\>0\\cos\(\{\\bm\{v\}\}\_\{i\},\{\\bm\{v\}\}\_\{j\}\)\>0, the vectors reinforce one another, producing constructive interference and a larger resultant norm\. Whencos⁡\(𝒗i,𝒗j\)<0\\cos\(\{\\bm\{v\}\}\_\{i\},\{\\bm\{v\}\}\_\{j\}\)<0, opposing directions lead to destructive interference, attenuating or cancelling the combined output\.

### 4\.2Contribution weights

To better understand the aggregation behaviour of MHA, we introduce*contribution weights*, a projection\-based measure that accounts for a token’s attention weight, it’s norm and directional alignment with the output\. We show that our metric is naturally suited to evaluating the influence of representations across different levels of aggregation\.

#### Intra\-Head Contribution Weights\.

Consider the output at positioniifrom a*single*attention head,

𝒐i=∑j=1iαi​j​𝒗~j,\{\\bm\{o\}\}\_\{i\}=\\sum\_\{j=1\}^\{i\}\\alpha\_\{ij\}\\,\\tilde\{\{\\bm\{v\}\}\}\_\{j\},\(8\)whereαi​j\\alpha\_\{ij\}are the attention weights and𝒗~j\\tilde\{\{\\bm\{v\}\}\}\_\{j\}the projected value vectors\. The contribution of tokenjjdepends on how strongly the weighted value vectorαi​j​𝒗~j\\alpha\_\{ij\}\\tilde\{\{\\bm\{v\}\}\}\_\{j\}projects onto the final output direction𝒐i\{\\bm\{o\}\}\_\{i\}\. We define the*intra\-head contribution weights*as the normalized inner product:

cˇi​j:=⟨𝒐i,αi​j​𝒗~j⟩‖𝒐i‖2∈ℝ\.\\check\{c\}\_\{ij\}:=\\frac\{\\langle\{\\bm\{o\}\}\_\{i\},\\alpha\_\{ij\}\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\rangle\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|^\{2\}\}\\in\\mathbb\{R\}\.\(9\)Contribution weights represent the signed fraction of the output’s norm‖𝒐i‖\\\|\{\\bm\{o\}\}\_\{i\}\\\|attributable to each component in the output’s direction as they naturally sum to one across the sequence dimension,

∑j=1icˇi​j=⟨𝒐i,∑j=1iαi​j​𝒗~j⟩‖𝒐i‖2=⟨𝒐i,𝒐i⟩‖𝒐i‖2=1\.\\sum\_\{j=1\}^\{i\}\\check\{c\}\_\{ij\}=\\frac\{\\langle\{\\bm\{o\}\}\_\{i\},\\sum^\{i\}\_\{j=1\}\\alpha\_\{ij\}\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\rangle\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|^\{2\}\}=\\frac\{\\langle\{\\bm\{o\}\}\_\{i\},\{\\bm\{o\}\}\_\{i\}\\rangle\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|^\{2\}\}=1\.\(10\)
Whencˇi​j\>0\\check\{c\}\_\{ij\}\>0, tokenjjreinforces the output direction, whencˇi​j<0\\check\{c\}\_\{ij\}<0it opposes it, and whencˇi​j≈0\\check\{c\}\_\{ij\}\\approx 0it contributes orthogonally\. This contrasts with attention weights \(αi​j∈\[0,1\]\\alpha\_\{ij\}\\in\[0,1\]\), which only indicate selection strength without regard for directional alignment\.

#### Inter\-Head Contribution Weights\.

In multi\-head attention, the output at positioniiis the sum of projected value vectors from all heads,

𝒐i=∑h=1H∑j=1iαi​j\(h\)​𝒗~j\(h\),\{\\bm\{o\}\}\_\{i\}=\\sum\_\{h=1\}^\{H\}\\sum\_\{j=1\}^\{i\}\\alpha^\{\(h\)\}\_\{ij\}\\,\\tilde\{\{\\bm\{v\}\}\}^\{\(h\)\}\_\{j\},\(11\)We extend contribution weights to the multi\-head setting by projecting each head’s weighted value vectors onto the combined multi\-head output direction𝒐i\{\\bm\{o\}\}\_\{i\},

c^i​j\(h\)=⟨𝒐i,αi​j\(h\)​𝒗~j\(h\)⟩‖𝒐i‖2\.\\hat\{c\}^\{\(h\)\}\_\{ij\}=\\frac\{\\langle\{\\bm\{o\}\}\_\{i\},\\alpha^\{\(h\)\}\_\{ij\}\\tilde\{\{\\bm\{v\}\}\}^\{\(h\)\}\_\{j\}\\rangle\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|^\{2\}\}\.\(12\)Using the full multi\-head output𝒐i\{\\bm\{o\}\}\_\{i\}, rather than the head\-specific output𝒐i\(h\)\{\\bm\{o\}\}\_\{i\}^\{\(h\)\}, captures both intra\-head token influence and the relative contribution of each head to the combined output\. This contrasts with attention weights, which are independently normalised within each head\.

As a result,*inter\-head contribution weights*are normalised jointly over heads and positions:

∑h=1H∑j=1ic^i​j\(h\)=1\.\\sum\_\{h=1\}^\{H\}\\sum\_\{j=1\}^\{i\}\\hat\{c\}^\{\(h\)\}\_\{ij\}=1\.\(13\)
![Refer to caption](https://arxiv.org/html/2606.07604v1/x2.png)Figure 2:Contribution weight decomposition\.We visualise the multiplicative components of the inter\-head contribution weightsc^i​j\\hat\{c\}\_\{ij\}: \(i\) the attention weightsαi​j\\alpha\_\{ij\}, \(ii\) the relative norm‖𝒗~j\(l,h\)‖/‖𝒐i\(l\)‖\\\|\\tilde\{\{\\bm\{v\}\}\}^\{\(l,h\)\}\_\{j\}\\\|/\\\|\{\\bm\{o\}\}^\{\(l\)\}\_\{i\}\\\|, and \(iii\) the cosine similaritycos⁡\(𝒐i\(l\),𝒗~j\(l,h\)\)\\cos\(\{\\bm\{o\}\}^\{\(l\)\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}^\{\(l,h\)\}\_\{j\}\)\. All quantities are computed on the same input for a randomly selected layer and head of LLaMA\-3\.1\-8B\.

### 4\.3Decomposition of Contribution Weights

Using the cosine identity, contribution weights can be decomposed into three interpretable geometric factors111We omit the*iter*/*intra*notation difference here as the insight holds for both measures\.,

ci​j=⟨𝒐i,αi​j​𝒗~j⟩‖𝒐i‖2=αi​j​‖𝒗~j‖‖𝒐i‖​cos⁡\(𝒐i,𝒗~j\),c\_\{ij\}=\\frac\{\\langle\{\\bm\{o\}\}\_\{i\},\\alpha\_\{ij\}\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\rangle\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|^\{2\}\}=\\alpha\_\{ij\}\\frac\{\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|\}\\cos\(\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\),\(14\)wherecos⁡\(𝒐i,𝒗~j\)=⟨𝒐i,𝒗~j⟩‖𝒐i‖​‖𝒗~j‖\\cos\(\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\)=\\frac\{\\langle\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\rangle\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\}is the cosine similarity between the projected value and the output\.

This decomposition makes apparent that a token’s contribution is determined jointly by three independent factors:

1. 1\.Attention weightsαi​j\\alpha\_\{ij\}: determines the selection strength; how much the token is attended to relative to other tokens in the sequence\.
2. 2\.Relative norm‖𝒗~j‖‖𝒐i‖\\frac\{\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|\}: scales the contribution by the magnitude of the projected value vector relative to the output norm\.
3. 3\.Cosine similaritycos⁡\(𝒐i,𝒗~j\)\\cos\(\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\): identifies whether the token reinforces \(\>0\>0\) or opposes \(<0<0\) the output direction\.

Together, these components provide a complete account of how tokens influence the attention output, capturing the full geometry of vector summation\. As we’ll explore later, this decomposition enables fine\-grained analysis of how attention weights, value norms, and alignment interact both within and across attention heads to produce the final output\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x3.png)Figure 3:Contribution weights most reliably predict token importance\.Change in perplexity \(Δ\\DeltaPPL\) onGSM8Kunder increasing token\-removal rates, comparing four contribution measures: \(1\) contribution weights, \(2\) value\-weighted output\-normalised attention \(VW\-ON\), \(3\) value\-weighted attention, and \(4\) raw attention weights\.Top:Removal of top percentiles\.Bottom:Removal of bottom percentiles\. Contribution weights outperform all other metrics for every removal rate, model and dataset, highlighting how they better reflect a token’s importance\.

## 5Contribution Weights Reflect Token Importance

We first evaluate whether contribution weights produce more faithful explanations of a token’s importance than existing attention\-based metrics, paying particular attention to how attention weights, relative norms and cosine similarity contribute to increased faithfulness\.

### 5\.1Token Importance

Given an input𝐗∈ℝL×d\{\\mathbf\{X\}\}\\in\\mathbb\{R\}^\{L\\times d\}, we compute the contribution of tokenjjto the output at positioniifor layerlland headhhaccording to a chosen measureγ\\gamma\(e\.g\., attention weightsα\\alphaor contribution weightscc\)\. To quantify the importance of tokenjjaccording toγ\\gamma, we define itstoken\-scoreas its average contribution for all subsequent positionsi≥ji\\geq j:

token\-score​\(γ\)j\(l,h\)=1T−j\+1​∑i=jTγi​j\(l,h\)\\text\{token\-score\}\(\\gamma\)^\{\(l,h\)\}\_\{j\}=\\frac\{1\}\{T\-j\+1\}\\sum^\{T\}\_\{i=j\}\\gamma\_\{ij\}^\{\(l,h\)\}\(15\)
To evaluate a metric’s faithfulness, we compute both its Comprehensiveness and Sufficiency\(DeYoung et al\.,[2020](https://arxiv.org/html/2606.07604#bib.bib13)\)\.

Comprehensiveness \(↑\\uparrow\)measures the average change in model output after removing important tokens:

Comp​\(γ\)p=1N​∑i=1Nf​\(𝒗~\)−f​\(𝒗~:1−pγ\)\\text\{Comp\}\(\\gamma\)\_\{p\}=\\frac\{1\}\{N\}\\sum^\{N\}\_\{i=1\}f\(\\tilde\{\{\\bm\{v\}\}\}\)\-f\(\\tilde\{\{\\bm\{v\}\}\}^\{\\gamma\}\_\{:1\-p\}\)\(16\)whereffstands for model perplexity,𝒗~\\tilde\{\{\\bm\{v\}\}\}are the original projected values at all layers, and𝒗~:qγ\\tilde\{\{\\bm\{v\}\}\}^\{\\gamma\}\_\{:q\}denotes the bottom\-qqfraction projected values, at each layer, ranked bytoken\-score​\(γ\)j\(l,h\)\\text\{token\-score\}\(\\gamma\)\_\{j\}^\{\(l,h\)\}\. Larger drops indicate higher faithfulness\.

Sufficiency \(↓\\downarrow\)measures the average change in model output after removing unimportant tokens:

Suff​\(γ\)p=1N​∑i=1Nf​\(𝒗~\)−f​\(𝒗~∖𝒗~:pγ\)\\text\{Suff\}\(\\gamma\)\_\{p\}=\\frac\{1\}\{N\}\\sum^\{N\}\_\{i=1\}f\(\\tilde\{\{\\bm\{v\}\}\}\)\-f\(\\tilde\{\{\\bm\{v\}\}\}\\setminus\\tilde\{\{\\bm\{v\}\}\}^\{\\gamma\}\_\{:p\}\)\(17\)Lower scores indicate higher faithfulness, as predictions change minimally with only important tokens retained\.

We implement removal by masking projected values𝒗~j\(l,h\)\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}to𝟎\\mathbf\{0\}at each layerll\. For Comprehensiveness, we mask the top\-ppfraction of tokens; for Sufficiency, we mask the bottompp\. Masking is applied sequentially so that modifications at layerllpropagate through all subsequent layers\.

### 5\.2Experimental Setup

#### Measures\.

We compare four measures of token importanceγ\\gamma, each incorporating progressively more information about value geometry: \(i\) Attentionαi​j\(l,h\)\\alpha\_\{ij\}^\{\(l,h\)\}, \(ii\) Value\-weighted \(VW\) attentionαi​j\(l,h\)​‖𝒗~j\(l,h\)‖\\alpha\_\{ij\}^\{\(l,h\)\}\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\\\|, which scales attention by value magnitude, \(iii\) VW output\-normalized \(VW\-ON\) attentionαi​j\(l,h\)​‖𝒗~j\(l,h\)‖/‖𝒐i\(l\)‖\\alpha\_\{ij\}^\{\(l,h\)\}\{\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\\\|\}/\{\\\|\{\\bm\{o\}\}\_\{i\}^\{\(l\)\}\\\|\}, which additionally normalizes by the output norm, and \(iv\) Absolute inter\-head contribution \(IHC\)\|c^i​j\|\|\\hat\{c\}\_\{ij\}\|, our proposed measure\.

#### Non\-Semantic Tokens\.

To focus onsemanticinfluence, we excludesink tokens, identified by near\-perfect collinearity, a cosine similarity score\>0\.95\>0\.95, with the\[BOS\]token, from removal\. Deleting these semantically void tokens disproportionately degrades attention\-based and value\-weighted metrics\([Xiao et al\.,](https://arxiv.org/html/2606.07604#bib.bib33)\), confounding our assessment of semantic importance\.[AppendixC](https://arxiv.org/html/2606.07604#A3)presents results including sink token removal\.

#### Language Modeling\.

We evaluate 4 models: \(i\) Mistral\-7B\(Jiang et al\.,[2023](https://arxiv.org/html/2606.07604#bib.bib22)\), \(ii\) LLaMA\-3\.1\-8B\([Touvron et al\.,](https://arxiv.org/html/2606.07604#bib.bib31)\), \(iii\) Qwen3\-8B\(Yang et al\.,[2025](https://arxiv.org/html/2606.07604#bib.bib34)\), and \(iv\) Deepseek\-7B\(DeepSeek\-AI et al\.,[2024](https://arxiv.org/html/2606.07604#bib.bib10)\), on GSM8K\([Cobbe et al\.,](https://arxiv.org/html/2606.07604#bib.bib8)\)and FineWeb\-Edu\([Penedo et al\.,](https://arxiv.org/html/2606.07604#bib.bib27)\)with maximum sequence length 512\. For each model we compute comprehensiveness and sufficiency as the change in perplexity \(Δ​PPL\\Delta\\text\{PPL\}\) for different removal rateskk\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x4.png)Figure 4:Contribution weights maintain downstream performance under high token\-removal rates\.Change in accuracy on MMLU, BoolQ, Lambada and Winogrande under increasing token\-removal rates, comparing four contribution measures: \(1\) contribution weights, \(2\) value\-weighted output\-normalised attention \(VW\-ON\), \(3\) value\-weighted attention, and \(4\) raw attention weights\. Contribution weights maintain baseline performance at a higher removal rate than other metrics\.
#### Downstream Tasks\.

Using LLaMA\-3\.1\-8B, we also assess the faithfulness of each measure on downstream tasks that require isolating relevant contextual information\([Bick et al\.,](https://arxiv.org/html/2606.07604#bib.bib3);[Merullo et al\.,](https://arxiv.org/html/2606.07604#bib.bib26)\): \(i\) MMLU\([Hendrycks et al\.,](https://arxiv.org/html/2606.07604#bib.bib20)\), \(ii\) Winogrande\([Sakaguchi et al\.,](https://arxiv.org/html/2606.07604#bib.bib28)\), and \(iii\) BoolQ\([Clark et al\.,](https://arxiv.org/html/2606.07604#bib.bib7)\)\. We measure task accuracy before and after intervention and report sufficiency in[Figure4](https://arxiv.org/html/2606.07604#S5.F4)\. A faithful metric should minimize accuracy degradation\.

### 5\.3Results

#### Contribution weights are more faithful than other attention\-based metrics\.

Across all models and removal rates, contribution weights consistently outperform attention\-based metrics in both comprehensiveness and sufficiency \([Figure3](https://arxiv.org/html/2606.07604#S4.F3)\), providing the clearest signal of token functional importance\. On downstream tasks \([Figure4](https://arxiv.org/html/2606.07604#S5.F4)\), contribution weights better preserve model performance under extreme sparsity, maintaining baseline accuracy at higher removal rates than attention\-based alternatives\. These results provide clear empirical evidence for the importance of value geometry in the attention mechanism\.

#### All geometric information contributes to faithfulness improvements\.

Comparing importance metrics reveals a clear progression:*adding geometric information consistently improves faithfulness*\. Across all tasks, raw attention weights perform worst, confirming that attention scores alone don’t reliably indicate functional contribution\. Performance improves when we multiply attention weights by value norms, and the marginal gain from VW to VW\-ON in language modeling suggests that importance depends not only on intra\-head geometry but also on each head’s relative influence on the total layer output\. Contribution weights, which incorporate angular information, prove most faithful\. Adding relative alignment produces the largest performance increase among all metrics, suggesting that token influence is strongly affected by interference from other tokens\.

## 6Functional Role of Value Geometry

Having established that contribution weights are a faithful measure of a token’s functional importance, we now use them to examine*what determines a token’s contribution to the output*\. We first decompose token contributions into the product of attentionαi​j\\alpha\_\{ij\}, value magnitude‖𝒗~j‖/‖𝒐i‖\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|/\\\|\{\\bm\{o\}\}\_\{i\}\\\|, and directional alignmentcos⁡\(𝒐i,𝒗~j\)\\cos\(\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\), studying the their significance in shaping a token’s contribution in Llama\-3\.1\-8B\. Through causal interventions, we then examine the functional role of each component, revealing how the geometry of the sink token actively suppresses semantic tokens while concentrating contribution among them\.

Table 1:Regression analysis of contribution weight components\.We report the coefficient of determination \(R2R^\{2\}\) for linear regressions of contribution weights onto individual and combined multiplicative components\. Each row corresponds to a different regression model: \(1\) the scalar attention weightαi​j\\alpha\_\{ij\}, \(2\) the relative norm of the projected value vectornormi​j=‖𝒗~j‖2/‖𝒐i‖2\\mathrm\{norm\}\_\{ij\}=\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\_\{2\}/\\\|\{\\bm\{o\}\}\_\{i\}\\\|\_\{2\}, \(3\) the cosine similaritycos⁡\(𝒗~j,𝒐i\)\\cos\(\\tilde\{\{\\bm\{v\}\}\}\_\{j\},\{\\bm\{o\}\}\_\{i\}\), and \(4–6\) products of these components\. We evaluate regressions under three token groupings: \(i\)j∈\{1,T\}j\\in\\\{1,T\\\}\(all tokens\), \(ii\)j=1j=1\(initial tokens\), and \(iii\)j\>1j\>1\(non\-initial tokens\)\. HigherR2R^\{2\}values indicate that the corresponding component or combination explains more of the contribution weight variance\.![Refer to caption](https://arxiv.org/html/2606.07604v1/x5.png)Figure 5:Value vector geometry\.\(a\)Average alignment \(cosine similarity\) between the initial \(\[BOS\]/sink\) token’s value vector and the tail tokens’ value vectors as a function of relative depth\.\(b\)Sink rateαj,1\\alpha\_\{j,1\}\(x\-axis\) against output norm‖𝐨j‖\\\|\\mathbf\{o\}\_\{j\}\\\|\(y\-axis\) for all tokens in 32 batches, for a fixed\(l,h\)\(l,h\)choice inLlama\-3\.1\-8B\. We plot in red the prediction obtained via our model outlined in[Equation20](https://arxiv.org/html/2606.07604#S6.E20), and colour each token according to its relative position in the sequence\.\(c\)Change in perplexity \(Δ\\DeltaPPL\) as the sink token’s value\-vector norm is scaled by a factor in\[0,1\]\[0,1\], showing the mean and 25th–75th percentile band\.### 6\.1Component Importance

In[Figure2](https://arxiv.org/html/2606.07604#S4.F2)we plot the contribution weights and each multiplicative component for a sample head\. Noting the visual differences in distribution between components, we first examine how much of the variance in contribution can be explained by each subset of components and their interactions\. Leveraging knowledge of its true functional form \([Equation14](https://arxiv.org/html/2606.07604#S4.E14)\), we analyse the direct multiplicative relationships between components\. Specifically, for each subset of components,

S⊆\{αi​j\(l,h\),‖𝒗~j\(l,h\)‖/‖𝒐i\(l\)‖,cos⁡\(𝒐i\(l\),𝒗~j\(l,h\)\)\},S\\subseteq\\\{\\alpha^\{\(l,h\)\}\_\{ij\},\\;\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\\\|/\\\|\{\\bm\{o\}\}^\{\(l\)\}\_\{i\}\\\|,\\;\\cos\(\{\\bm\{o\}\}\_\{i\}^\{\(l\)\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\)\\\},\(18\)we construct the multiplicative featureηS=∏η∈Sη\\eta\_\{S\}=\\prod\_\{\\eta\\in S\}\\etaand then fit a simple linear modelci​j\(l,h\)=β0​ηS,i​j\(l,h\)c\_\{ij\}^\{\(l,h\)\}=\\beta\_\{0\}\\eta^\{\(l,h\)\}\_\{S,ij\}to the contribution weights\. We then compute the coefficient of determination \(R2R^\{2\}\), which measures the variance inci​j\(l,h\)c^\{\(l,h\)\}\_\{ij\}explained by the multiplicative interaction of components inSS\. See Appendix[D\.2](https://arxiv.org/html/2606.07604#A4.SS2)for experimental details\. In[Table1](https://arxiv.org/html/2606.07604#S6.T1)we report results on FineWeb\-Edu, observing a striking difference in results for initial or sink tokens \(j=1j=1\) versus tail tokens \(j\>1j\>1\)\.

#### Sink tokens: Value geometry dominates\.

For initial tokens, contribution is primarily determined by value geometry\. The product of relative norm and cosine similarity serves as the strongest predictor \(R2=0\.725R^\{2\}=0\.725\), while attention weights alone have negligible explanatory power \(R2=0\.079R^\{2\}=0\.079\)\. This reveals that first\-token contributions depend on the magnitude and directional alignment of value vectors rather than attention allocation\.

#### Tail tokens: attention and alignment jointly matter\.

For tail tokens, contribution depends on both attention and value geometry\. Attention alone is the dominant factor but still captures less than half the variance \(R2=0\.449R^\{2\}=0\.449\), while the joint effect of attention and value vector alignment provides the strongest prediction \(R2=0\.834R^\{2\}=0\.834\)\. This demonstrates that tail token contributions arise when a token receives attention and its value vector aligns with the output direction\.

### 6\.2Sink as Information Suppressor\.

Having established that sink token contributions are governed by value geometry, we now examine how this geometry functions mechanically\. Analyzing the relationship between sink token value norms and their alignment with semantic tokens reveals that, contrary to existing belief, sink tokens operate as active information suppressors\.

#### Sink tokens are anti\-aligned with semantic tokens\.

Prior explanations on the role of sink tokens have largely focused on value magnitudes: since initial value vectors𝒗~1\(l,h\)\\tilde\{\{\\bm\{v\}\}\}^\{\(l,h\)\}\_\{1\}have significantly smaller norms than semantic tokens\(Guo et al\.,[b](https://arxiv.org/html/2606.07604#bib.bib19)\), attending to them produces negligible head output,

‖∑j=1iαi​j​𝒗~j\(l,h\)\)‖≈‖αi​1​𝒗~1\(l,h\)‖≈‖𝒗~1\(l,h\)‖≈0\\\|\\sum^\{i\}\_\{j=1\}\\alpha\_\{ij\}\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\)\}\\\|\\approx\\\|\\alpha\_\{i1\}\\tilde\{\{\\bm\{v\}\}\}\_\{1\}^\{\(l,h\)\}\\\|\\approx\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{1\}^\{\(l,h\)\}\\\|\\approx 0\(19\)
However, beyond low magnitudes we observe that sink token value vectors exhibit systematic directional opposition to semantic tokens \([Figure5](https://arxiv.org/html/2606.07604#S6.F5), left\)\. Across layers, initial token values maintain negative cosine similarity with tail values,cos⁡\(𝒗~1\(l,h\),𝒗~j\(l,h\)\)≈−0\.2\\cos\(\\tilde\{\{\\bm\{v\}\}\}\_\{1\}^\{\(l,h\)\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\)\\approx\-0\.2for allj≠1j\\neq 1, while tail tokens exhibit positive mutual alignmentcos⁡\(𝒗~i\(l,h\),𝒗~j\(l,h\)\)≈0\.2\\cos\(\\tilde\{\{\\bm\{v\}\}\}\_\{i\}^\{\(l,h\)\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\)\\approx 0\.2fori,j≠1i,j\\neq 1\. This anti\-alignment pattern is unexplained by existing theories and, as we show in[Table1](https://arxiv.org/html/2606.07604#S6.T1), plays an important role in shaping sink token contributions\.

#### Sink tokens actively oppose semantic tokens\.

This raises a key question,*does this directional opposition serve a functional role?*To test this, we perform an intervention, scaling the norm of the initial token’s value vector to 0 in layers exhibiting attention sink \(2–30 for Llama\-8B\):𝒗^1\(l,h\)=γ​𝒗1\(l,h\)\\hat\{\{\\bm\{v\}\}\}\_\{1\}^\{\(l,h\)\}=\\gamma\{\\bm\{v\}\}\_\{1\}^\{\(l,h\)\}for0<γ<10<\\gamma<1\. Results are plotted in Figure[6](https://arxiv.org/html/2606.07604#S6.F6)c for Llama\-8B, averaging over 128 FineWeb\-Edu sequences of 1024 tokens\. If sink tokens were simply passive placeholders designed to absorb attention without contributing meaningfully, reducing their norm should minimally affect model performance\. Instead, perplexity increases monotonically asγ\\gammadecreases, indicating that initial tokens play an active role: by opposing the semantic direction of non\-sink tokens, they exert meaningful influence on model predictions despite their small magnitudes\.

#### Sink suppression exhibits convex geometry, not monotonic decay\.

To understand the purpose of the sink’s semantic opposition, we examine it’s effect on the head output norm\. Prior work assumes that sink value vectors𝒗~1\(l,h\)\\tilde\{\{\\bm\{v\}\}\}^\{\(l,h\)\}\_\{1\}exhibiting significantly lower norms than semantic tokens\(Guo et al\.,[b](https://arxiv.org/html/2606.07604#bib.bib19)\)renders the head inactive\(Guo et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib18)\)\. Under this perspective, one would assume a monotonic suppression mechanism where output norm scales with sink attentionαi​1\\alpha\_\{i1\}, and the maximal suppression \(minimum output norm\) would occur strictly when the sink captures all attention \(αi​1→1\\alpha\_\{i1\}\\to 1\)\.

We challenge this intuition, demonstrating that the relationship between sink rate and output norm is fundamentally*convex*, not monotonic\. As shown in Figure[5](https://arxiv.org/html/2606.07604#S6.F5)b, the minimum output norm is empirically achieved at sink rates significantly below11\. We formalize this in Appendix[E](https://arxiv.org/html/2606.07604#A5)using a geometric model that approximates the output norm as the root of a quadratic:

‖𝐨i‖≃α2​‖𝒗~1‖2\+2​α​\(1−α\)​γ\+\(1−α\)2​κ2\|\|\\mathbf\{o\}\_\{i\}\|\|\\simeq\\sqrt\{\\alpha^\{2\}\|\|\\tilde\{\{\\bm\{v\}\}\}\_\{1\}\|\|^\{2\}\+2\\alpha\(1\-\\alpha\)\\gamma\+\(1\-\\alpha\)^\{2\}\\kappa^\{2\}\}\(20\)whereα\\alphadenotes the sink rate,κ\\kappacaptures the average norm𝔼j\>1​\[‖𝒗~j‖\]\\mathbb\{E\}\_\{j\>1\}\[\\\|\{\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\}\\\|\]of non\-sink values, andγ\\gammatheir aggregate alignment𝔼j\>1​\[⟨𝒗~1,𝒗~j⟩\]\\mathbb\{E\}\_\{j\>1\}\[\\langle\\tilde\{\{\\bm\{v\}\}\}\_\{1\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\rangle\]with the sink token\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x6.png)Figure 6:Sinks oppose small magnitude attention weights\.Head outputs𝒐i\(l,h\)\{\\bm\{o\}\}\_\{i\}^\{\(l,h\)\}projected onto the semantic direction𝝁^\\hat\{\\bm\{\\mu\}\}and its orthogonal complement𝐮^\\hat\{\\mathbf\{u\}\}, coloured by output norm\. The dashed cone contains 90% of non\-sink values; the orange arrow shows the sink vector\. \(a\) With the sink token present, low\-norm heads \(dark blue\) lie outside the semantic cone, either orthogonal or negatively aligned with𝝁^\\hat\{\\bm\{\\mu\}\}\. \(b\) When the sink value norm is set to zero, the same low\-norm heads collapse into the semantic cone\. This demonstrates that sink tokens actively prevent weak attention from introducing noisy semantic drift\.Crucially, the location of the minimum depends onγ\\gamma\. If sink tokens were strongly aligned with semantic content \(γ≫0\\gamma\\gg 0\), suppression would indeed be monotonic in attention offloading\. However, we find that sink vectors are consistently*anti\-aligned*with other tokens \(γ<0\\gamma<0, see Figure[5](https://arxiv.org/html/2606.07604#S6.F5)a\)\. This directional opposition creates destructive interference, causing the output norm to collapse to its minimum at an intermediate attention thresholdαmin<1\\alpha\_\{\\min\}<1\. These findings reveal that attention sinks function as information suppressors not just through low magnitude, but through a delicate balance of value alignment and attention head characteristics\.

#### Sink tokens counteract Semantic Drift from weak attention\.

We hypothesize that directional opposition between sink and non\-sink tokens serves to nullify semantic drift introduced by many small but non\-zero attention weights\. For a given query, numerous attention weightsαi​j\\alpha\_\{ij\}are small yet non\-negligible, and because non\-sink tokens align along a shared semantic direction, their weak positive contributions accumulate and bias the attention output\.

To show how the sink tokens achieve we this we can use a simple geometric model\. Let us define𝝁^\\hat\{\\bm\{\\mu\}\}as the unit semantic direction defined as the mean of non\-sink token value vectors𝝁^=1T−1​∑j=2T𝒗~j\\hat\{\\bm\{\\mu\}\}=\\frac\{1\}\{T\-1\}\\sum^\{T\}\_\{j=2\}\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\. We write the sink value vector as𝒔=a​𝝁^\+b​𝒖^\{\\bm\{s\}\}=a\\hat\{\\bm\{\\mu\}\}\+b\\hat\{\{\\bm\{u\}\}\}, where𝒖^\\hat\{\{\\bm\{u\}\}\}is orthogonal to𝝁^\\hat\{\\bm\{\\mu\}\}anda,b∈ℝa,b\\in\\mathbb\{R\}\. This decomposition reveals two complementary roles\. The*cancellation*role is governed bya=𝒔⋅μ^<0a=\{\\bm\{s\}\}\\cdot\\hat\{\\mu\}<0: this negative component subtracts from the positive drift alongμ^\\hat\{\\mu\}induced by many small attention weights\. The*preservation*role is governed by the orthogonal componentbb, which prevents collapse into the semantic cone\.

Figure[6](https://arxiv.org/html/2606.07604#S6.F6)illustrates this geometry empirically, plotting the sink vector𝒗~1\(l,h\)\\tilde\{\{\\bm\{v\}\}\}\_\{1\}^\{\(l,h\)\}and the projections of the output𝒐i\(l,h\)\{\\bm\{o\}\}\_\{i\}^\{\(l,h\)\}of a given head onto the plane spanned by the semantic direction𝝁^\\hat\{\\bm\{\\mu\}\}and by𝐮^\\hat\{\\mathbf\{u\}\}\. The shaded region in each plot corresponds to the semantic cone, which we define as the region around𝝁^\\hat\{\\bm\{\\mu\}\}in which 90% of the value vectors lie within\.[Figure6](https://arxiv.org/html/2606.07604#S6.F6)a shows that with the sink token present, heads with small output norm lie outside of the semantic cone and are either slightly negatively aligned or orthogonal to the semantic direction\.[Figure6](https://arxiv.org/html/2606.07604#S6.F6)b however shows that if we intervene and set the norm of the sink value vector to 0, negating its cancellation effect, then all of the small norm head outputs lie within the semantic cone\. This provides evidence that without the sink tokens uninformative value vectors can contribute noisily along the semantic axis\.

## 7Conclusion

We introduced Contribution Weights, a geometry\-aware metric that accounts for value vector norms and alignment\. Through intervention analysis, we showed that contribution weights provide a more faithful measure of token functional importance, consistently outperforming existing attention and norm\-based approaches across the decoder\-only models we evaluate\. Decomposing contribution weights into attention, relative norm, and directional alignment revealed the critical role of value geometry in shaping token influence\. Our analysis uncovered a key mechanistic insight: sink tokens actively suppress information through directional opposition to semantic tokens, counteracting drift from weak attention accumulation via convex geometric relationships rather than simple monotonic suppression\. These findings demonstrate that analyzing value geometry is essential for faithful mechanistic understanding of decoder\-only transformers\. While our experiments focus on this setting, contribution weights depend only on the structure of the attention mechanism and are in principle applicable to any attention\-based architecture such as encoder\-only models and ViTs\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- \(1\)Barbero, F\., Arroyo, A\., Gu, X\., Perivolaropoulos, C\., Bronstein, M\., Veličković, P\., and Pascanu, R\.Why do LLMs attend to the first token?URL[http://arxiv\.org/abs/2504\.02732](http://arxiv.org/abs/2504.02732)\.
- \(2\)Bastings, J\. and Filippova, K\.The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?URL[http://arxiv\.org/abs/2010\.05607](http://arxiv.org/abs/2010.05607)\.
- \(3\)Bick, A\., Xing, E\., and Gu, A\.Understanding the skill gap in recurrent language models: The role of the gather\-and\-aggregate mechanism\.URL[http://arxiv\.org/abs/2504\.18574](http://arxiv.org/abs/2504.18574)\.
- \(4\)Bondarenko, Y\., Nagel, M\., and Blankevoort, T\.Quantizable transformers: Removing outliers by helping attention heads do nothing\.URL[http://arxiv\.org/abs/2306\.12929](http://arxiv.org/abs/2306.12929)\.
- \(5\)Brown, T\. B\., Mann, B\., Ryder, N\., Subbiah, M\., Kaplan, J\., Dhariwal, P\., Neelakantan, A\., Shyam, P\., Sastry, G\., Askell, A\., Agarwal, S\., Herbert\-Voss, A\., Krueger, G\., Henighan, T\., Child, R\., Ramesh, A\., Ziegler, D\. M\., Wu, J\., Winter, C\., Hesse, C\., Chen, M\., Sigler, E\., Litwin, M\., Gray, S\., Chess, B\., Clark, J\., Berner, C\., McCandlish, S\., Radford, A\., Sutskever, I\., and Amodei, D\.Language models are few\-shot learners\.URL[http://arxiv\.org/abs/2005\.14165](http://arxiv.org/abs/2005.14165)\.
- \(6\)Cancedda, N\.Spectral filters, dark signals, and attention sinks\.URL[http://arxiv\.org/abs/2402\.09221](http://arxiv.org/abs/2402.09221)\.version: 1\.
- \(7\)Clark, C\., Lee, K\., Chang, M\.\-W\., Kwiatkowski, T\., Collins, M\., and Toutanova, K\.BoolQ: Exploring the surprising difficulty of natural yes/no questions\.URL[http://arxiv\.org/abs/1905\.10044](http://arxiv.org/abs/1905.10044)\.
- \(8\)Cobbe, K\., Kosaraju, V\., Bavarian, M\., Chen, M\., Jun, H\., Kaiser, L\., Plappert, M\., Tworek, J\., Hilton, J\., Nakano, R\., Hesse, C\., and Schulman, J\.Training verifiers to solve math word problems\.URL[http://arxiv\.org/abs/2110\.14168](http://arxiv.org/abs/2110.14168)\.
- \(9\)Darcet, T\., Oquab, M\., Mairal, J\., and Bojanowski, P\.Vision transformers need registers\.URL[http://arxiv\.org/abs/2309\.16588](http://arxiv.org/abs/2309.16588)\.
- DeepSeek\-AI et al\. \(2024\)DeepSeek\-AI, :, Bi, X\., Chen, D\., Chen, G\., Chen, S\., Dai, D\., Deng, C\., Ding, H\., Dong, K\., Du, Q\., Fu, Z\., Gao, H\., Gao, K\., Gao, W\., Ge, R\., Guan, K\., Guo, D\., Guo, J\., Hao, G\., Hao, Z\., He, Y\., Hu, W\., Huang, P\., Li, E\., Li, G\., Li, J\., Li, Y\., Li, Y\. K\., Liang, W\., Lin, F\., Liu, A\. X\., Liu, B\., Liu, W\., Liu, X\., Liu, X\., Liu, Y\., Lu, H\., Lu, S\., Luo, F\., Ma, S\., Nie, X\., Pei, T\., Piao, Y\., Qiu, J\., Qu, H\., Ren, T\., Ren, Z\., Ruan, C\., Sha, Z\., Shao, Z\., Song, J\., Su, X\., Sun, J\., Sun, Y\., Tang, M\., Wang, B\., Wang, P\., Wang, S\., Wang, Y\., Wang, Y\., Wu, T\., Wu, Y\., Xie, X\., Xie, Z\., Xie, Z\., Xiong, Y\., Xu, H\., Xu, R\. X\., Xu, Y\., Yang, D\., You, Y\., Yu, S\., Yu, X\., Zhang, B\., Zhang, H\., Zhang, L\., Zhang, L\., Zhang, M\., Zhang, M\., Zhang, W\., Zhang, Y\., Zhao, C\., Zhao, Y\., Zhou, S\., Zhou, S\., Zhu, Q\., and Zou, Y\.Deepseek llm: Scaling open\-source language models with longtermism, 2024\.URL[https://arxiv\.org/abs/2401\.02954](https://arxiv.org/abs/2401.02954)\.
- \(11\)Devlin, J\., Chang, M\.\-W\., Lee, K\., and Toutanova, K\.BERT: Pre\-training of deep bidirectional transformers for language understanding\.URL[http://arxiv\.org/abs/1810\.04805](http://arxiv.org/abs/1810.04805)\.
- \(12\)Devoto, A\., Zhao, Y\., Scardapane, S\., and Minervini, P\.A simple and effective $l\_2$ norm\-based strategy for KV cache compression\.URL[http://arxiv\.org/abs/2406\.11430](http://arxiv.org/abs/2406.11430)\.
- DeYoung et al\. \(2020\)DeYoung, J\., Jain, S\., Rajani, N\. F\., Lehman, E\., Xiong, C\., Socher, R\., and Wallace, B\. C\.ERASER: A Benchmark to Evaluate Rationalized NLP Models, April 2020\.URL[http://arxiv\.org/abs/1911\.03429](http://arxiv.org/abs/1911.03429)\.arXiv:1911\.03429 \[cs\]\.
- \(14\)Elhage, N\., Nanda, N\., and Olsson, C\.A mathematical framework for transformer circuits\.
- Ferrando et al\. \(a\)Ferrando, J\., Gállego, G\. I\., and Costa\-jussà, M\. R\.Measuring the mixing of contextual information in the transformer, a\.URL[http://arxiv\.org/abs/2203\.04212](http://arxiv.org/abs/2203.04212)\.
- Ferrando et al\. \(b\)Ferrando, J\., Sarti, G\., Bisazza, A\., and Costa\-jussà, M\. R\.A primer on the inner workings of transformer\-based language models, b\.URL[http://arxiv\.org/abs/2405\.00208](http://arxiv.org/abs/2405.00208)\.
- Gromov et al\. \(2025\)Gromov, A\., Tirumala, K\., Shapourian, H\., Glorioso, P\., and Roberts, D\.The unreasonable ineffectiveness of the deeper layers\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=ngmEcEer8a](https://openreview.net/forum?id=ngmEcEer8a)\.
- Guo et al\. \(a\)Guo, T\., Pai, D\., Bai, Y\., Jiao, J\., Jordan, M\. I\., and Mei, S\.Active\-dormant attention heads: Mechanistically demystifying extreme\-token phenomena in LLMs, a\.URL[http://arxiv\.org/abs/2410\.13835](http://arxiv.org/abs/2410.13835)\.version: 1\.
- Guo et al\. \(b\)Guo, Z\., Kamigaito, H\., and Watanabe, T\.Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters, b\.URL[http://arxiv\.org/abs/2406\.12335](http://arxiv.org/abs/2406.12335)\.
- \(20\)Hendrycks, D\., Burns, C\., Basart, S\., Zou, A\., Mazeika, M\., Song, D\., and Steinhardt, J\.Measuring massive multitask language understanding\.URL[http://arxiv\.org/abs/2009\.03300](http://arxiv.org/abs/2009.03300)\.
- \(21\)Jain, S\. and Wallace, B\. C\.Attention is not explanation\.URL[http://arxiv\.org/abs/1902\.10186](http://arxiv.org/abs/1902.10186)\.
- Jiang et al\. \(2023\)Jiang, A\. Q\., Sablayrolles, A\., Mensch, A\., Bamford, C\., Chaplot, D\. S\., de las Casas, D\., Bressand, F\., Lengyel, G\., Lample, G\., Saulnier, L\., Lavaud, L\. R\., Lachaux, M\.\-A\., Stock, P\., Scao, T\. L\., Lavril, T\., Wang, T\., Lacroix, T\., and Sayed, W\. E\.Mistral 7b, 2023\.URL[https://arxiv\.org/abs/2310\.06825](https://arxiv.org/abs/2310.06825)\.
- Kobayashi et al\. \(a\)Kobayashi, G\., Kuribayashi, T\., Yokoi, S\., and Inui, K\.Attention is not only a weight: Analyzing transformers with vector norms, a\.URL[http://arxiv\.org/abs/2004\.10102](http://arxiv.org/abs/2004.10102)\.
- Kobayashi et al\. \(b\)Kobayashi, G\., Kuribayashi, T\., Yokoi, S\., and Inui, K\.Incorporating residual and normalization layers into analysis of masked language models, b\.URL[http://arxiv\.org/abs/2109\.07152](http://arxiv.org/abs/2109.07152)\.
- \(25\)Lopardo, G\., Precioso, F\., and Garreau, D\.Attention meets post\-hoc interpretability: A mathematical perspective\.
- \(26\)Merullo, J\., Vatsavaya, S\., Bushnaq, L\., and Lewis, O\.From memorization to reasoning in the spectrum of loss curvature\.URL[http://arxiv\.org/abs/2510\.24256](http://arxiv.org/abs/2510.24256)\.
- \(27\)Penedo, G\., Kydlíček, H\., allal, L\. B\., Lozhkov, A\., Mitchell, M\., Raffel, C\., Werra, L\. V\., and Wolf, T\.The FineWeb datasets: Decanting the web for the finest text data at scale\.URL[http://arxiv\.org/abs/2406\.17557](http://arxiv.org/abs/2406.17557)\.
- \(28\)Sakaguchi, K\., Bras, R\. L\., Bhagavatula, C\., and Choi, Y\.WinoGrande: An adversarial winograd schema challenge at scale\.URL[http://arxiv\.org/abs/1907\.10641](http://arxiv.org/abs/1907.10641)\.
- \(29\)Shin, S\., Oh, J\., and Oh, D\.OrthoRank: Token selection via sink token orthogonality for efficient LLM inference\.URL[http://arxiv\.org/abs/2507\.03865](http://arxiv.org/abs/2507.03865)\.
- \(30\)Sun, M\., Chen, X\., Kolter, J\. Z\., and Liu, Z\.Massive activations in large language models\.URL[http://arxiv\.org/abs/2402\.17762](http://arxiv.org/abs/2402.17762)\.
- \(31\)Touvron, H\., Lavril, T\., Izacard, G\., Martinet, X\., Lachaux, M\.\-A\., Lacroix, T\., Rozière, B\., Goyal, N\., Hambro, E\., Azhar, F\., Rodriguez, A\., Joulin, A\., Grave, E\., and Lample, G\.LLaMA: Open and efficient foundation language models\.URL[http://arxiv\.org/abs/2302\.13971](http://arxiv.org/abs/2302.13971)\.
- \(32\)Vaswani, A\., Shazeer, N\., Parmar, N\., Uszkoreit, J\., Jones, L\., Gomez, A\. N\., Kaiser, L\., and Polosukhin, I\.Attention is all you need\.URL[http://arxiv\.org/abs/1706\.03762](http://arxiv.org/abs/1706.03762)\.
- \(33\)Xiao, G\., Tian, Y\., Chen, B\., Han, S\., and Lewis, M\.Efficient streaming language models with attention sinks\.URL[http://arxiv\.org/abs/2309\.17453](http://arxiv.org/abs/2309.17453)\.
- Yang et al\. \(2025\)Yang, A\., Li, A\., Yang, B\., Zhang, B\., Hui, B\., Zheng, B\., Yu, B\., Gao, C\., Huang, C\., Lv, C\., Zheng, C\., Liu, D\., Zhou, F\., Huang, F\., Hu, F\., Ge, H\., Wei, H\., Lin, H\., Tang, J\., Yang, J\., Tu, J\., Zhang, J\., Yang, J\., Yang, J\., Zhou, J\., Zhou, J\., Lin, J\., Dang, K\., Bao, K\., Yang, K\., Yu, L\., Deng, L\., Li, M\., Xue, M\., Li, M\., Zhang, P\., Wang, P\., Zhu, Q\., Men, R\., Gao, R\., Liu, S\., Luo, S\., Li, T\., Tang, T\., Yin, W\., Ren, X\., Wang, X\., Zhang, X\., Ren, X\., Fan, Y\., Su, Y\., Zhang, Y\., Zhang, Y\., Wan, Y\., Liu, Y\., Wang, Z\., Cui, Z\., Zhang, Z\., Zhou, Z\., and Qiu, Z\.Qwen3 technical report, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.

## Appendix ANotation

Throughout this work we denote matrices with bold upper\-case letters \(e\.g\.𝐀\{\\mathbf\{A\}\},𝐁\{\\mathbf\{B\}\}\) and vectors with bold lower\-case letters \(e\.g\.𝒂i\{\\bm\{a\}\}\_\{i\},𝒃i\{\\bm\{b\}\}\_\{i\}\), using the same letter of the alphabet to show the rows of a matrix \(e\.g\.𝒂i\{\\bm\{a\}\}\_\{i\}is theii\-th row of𝐀\{\\mathbf\{A\}\}\) and italic upper\-case letters for learnable parameter matrices \(e\.g\.𝑾O\)\\bm\{W\}\_\{O\}\)\.

## Appendix BLimitations of Existing Contribution Measures

We now examine how existing approaches to measuring token contributions often overlook or simplify the three interacting effects identified above\. We analyze these limitations using empirical data from LLaMA\-3\.1\-8B, evaluated on Fineweb\-Edu\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/figures/llama_8B/fig_Llama-3.1-8B_v_distributions.png)Figure 7:Variation of attention weights, value norms and value alignment\.Pairwise relationships between attention weightsαi​j\(l,h\)\\alpha^\{\(l,h\)\}\_\{ij\}, projected value norms‖𝒗~j\(l,h\)‖2\\\|\\tilde\{\{\\bm\{v\}\}\}^\{\(l,h\)\}\_\{j\}\\\|\_\{2\}and alignment with the output directioncos⁡\(𝒐i\(l\),𝒗~j\(l,h\)\)\\cos\(\{\\bm\{o\}\}^\{\(l\)\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}^\{\(l,h\)\}\_\{j\}\)across all layerslland headshhof Llama\-8B\. The large dispersion across all three plots shows how these factors vary independently\. Indeed, high attention weights do not guarantee high large norms or positive alignment, violating the assumptions of weight and norm\-based contribution analysis\.#### Weight\-Based Analysis\.

Analyzing token contributions based solely on attention weightsαi​j\\alpha\_\{ij\}implicitly assumes that: \(1\) projected value norms are approximately constant‖𝒗~i‖2≈const\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{i\}\\\|\_\{2\}\\approx\\text\{const\}, and \(2\) all projected value vectors are similarly aligned with the output,cos⁡\(𝒐i,𝒗~j\)≈1\\cos\(\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\)\\approx 1\. Under these assumptions, all variation in contribution is explained entirely by differences in attention weights\. In reality, however, these assumptions are strongly violated\.[Figure7](https://arxiv.org/html/2606.07604#A2.F7)shows the joint distributions of attention weightsαi​j\(l,h\)\\alpha\_\{ij\}^\{\(l,h\)\}, projected value norms‖𝒗~j\(l,h\)‖2\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\\\|\_\{2\}, and cosine similaritiescos⁡\(𝒐i\(l,h\),𝒗~j\(l,h\)\)\\cos\(\{\\bm\{o\}\}^\{\(l,h\)\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\)for randomly sampled token pairs\. Even whenαi​j\\alpha\_\{ij\}is fixed, both‖𝒗~j‖2\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\_\{2\}andcos⁡\(𝒐i,𝒗~j\)\\cos\(\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\)exhibit substantial variation, particularly for lower attention weights\(αi​j<0\.2\)\(\\alpha\_\{ij\}<0\.2\)\.

#### Norm\-Based Analysis\.

Value\-weighted attention\(Kobayashi et al\.,[a](https://arxiv.org/html/2606.07604#bib.bib23),[b](https://arxiv.org/html/2606.07604#bib.bib24)\)extends weight\-based analysis by scaling attention weights by the norm of the projected value vector,αi​j​‖𝒗~j‖\\alpha\_\{ij\}\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\. While this accounts for magnitude differences between tokens, it still ignores directional alignment\. As shown in[Figure7](https://arxiv.org/html/2606.07604#A2.F7)b and[7](https://arxiv.org/html/2606.07604#A2.F7)c, cosine similarity between projected values and output vectors has a distribution approximately centered at zero \(mean≈0\.012\\approx 0\.012\)\. This indicates that value vectors are roughly isotropically oriented relative to the output direction, meaning their contributions can constructively or destructively interfere depending on their relative orientations\. Such interference effects cannot be captured by magnitude alone\.

#### Distance\-Based Analysis\.

To account for differing directionsFerrando et al\. \([a](https://arxiv.org/html/2606.07604#bib.bib15)\)propose a distance based metric that measures the contribution by theℓ1\\ell\_\{1\}\-distance between the projected value and the outputdi​j=‖𝒐i−𝒗j​𝑾O‖1d\_\{ij\}=\\\|\{\\bm\{o\}\}\_\{i\}\-\{\\bm\{v\}\}\_\{j\}\\bm\{W\}\_\{O\}\\\|\_\{1\}\. However, distance metrics are insensitive to the nature of interference and cannot distinguish constructive from destructive contributions\. Further, distance metrics do not decompose the output linearly, offering no principled way to normalize distancesdi​jd\_\{ij\}into contribution weights that sum to a meaningful total, such as the output norm or energy\.

## Appendix CExtended Token Importance Results

Here we ablate our analysis from[Section5](https://arxiv.org/html/2606.07604#S5)by exploring three alternative intervention settings:

1. 1\.Dataset\.We change the dataset to intervene on FineWeb\-Edu\.
2. 2\.Computation Graph\.We compute and apply*global percentiles*rather than layer\-specific thresholds\.
3. 3\.Sink\.We permit the removal of semantically void sink tokens\.

### C\.1Dataset

We first examine whether our findings generalize across domains by evaluating on FineWeb\-Edu \([Figure8](https://arxiv.org/html/2606.07604#A3.F8)\)\. The hierarchy of improvement observed on GSM8K remains consistent: contribution weights \(blue\) provide the most reliable signal for identifying both critical and dispensable tokens\. This confirms that our metric’s efficacy is robust to domain shifts and not an artifact of the specific reasoning patterns found in mathematical datasets\. The only notable deviation occurs in DeepSeek\-7B when removing high\-importance tokens, where attention exhibits a more pronounced concave profile than in GSM8K, though it rapidly converges to the relative ranking observed in the original experiment\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x7.png)Figure 8:Intervention dataset ablation\.Language modeling loss when removing tokens based on different importance metrics on FineWeb\-Edu\. Top row shows removal of high\-importance tokens; bottom row shows removal of low\-importance tokens\. Contribution weights \(blue\) consistently outperform attention\-based baselines \(red, orange\) across all models, demonstrating robustness to domain shifts from mathematical reasoning \(GSM8K\) to general web text\.
### C\.2Computation Graph

We next test whether layer\-specific normalization is necessary by computing importance scores using*global percentiles*\([Figure9](https://arxiv.org/html/2606.07604#A3.F9)\)\. This setting implicitly assumes layer contributions are directly comparable across network depths\. However, prior work has established that this assumption often does not hold, as later layers can be significantly less influential than preceding ones\(Gromov et al\.,[2025](https://arxiv.org/html/2606.07604#bib.bib17)\)\. Selecting tokens based off of their global importance disrupts pruning stability, particularly for models exhibiting multiple sinks such as Qwen3\-8B and DeepSeek\-7B, where results appear more volatile\. Despite these structural instabilities, contribution weights consistently maintain their advantage over attention\-based baselines across the majority of conditions\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x8.png)Figure 9:Computation graph ablation using global percentiles\.Language modeling loss when importance scores are computed using global percentiles across all layers rather than layer\-specific thresholds\. While this introduces instability in models with multiple sinks \(Qwen3\-8B, DeepSeek\-7B\), contribution weights \(blue\) maintain their advantage over attention\-based methods, though the gap narrows compared to layer\-specific normalization\.
### C\.3Sink Tokens

Finally, we examine the pathological behavior induced when attention sinks are not protected from removal \([Figure10](https://arxiv.org/html/2606.07604#A3.F10)\)\. When removing*high\-importance*tokens \(top row\), attention\-based metrics \(red\) induce immediate model collapse\. This confirms that attention heavily overfits to structural sinks, erroneously identifying them as the most critical tokens; their removal destroys the model’s functional integrity\. In contrast, contribution weights geometrically demote these sink tokens, prioritizing semantic content and yielding a significantly smoother degradation curve\. In the*low\-importance*removal regime \(bottom row\), Qwen3\-8B emerges as an outlier\. This deviation is attributable to the model’s reliance on multiple attention sinks, which are classified as marginal by most metrics, leading to their early removal and a consequent drop in performance not seen in single\-sink architectures\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x9.png)Figure 10:Sink token ablation\.Language modeling loss when attention sinks are permitted to be removed\. Top row \(high\-importance removal\): attention\-based metrics \(red\) cause immediate collapse by removing structural sinks, while contribution weights \(blue\) prioritize semantic content\. Bottom row \(low\-importance removal\): Qwen3\-8B shows early degradation due to multiple sinks being classified as low\-importance and removed, while single\-sink architectures remain stable\.

## Appendix DExtended Results on the Functional Role of Value Geometry

Here we provide extended results relative to[Section6\.1](https://arxiv.org/html/2606.07604#S6.SS1)\.

### D\.1Contribution components are weakly correlated\.

We quantify the independence of each component of the contribution weight decomposition by computing pairwise Pearson correlations between the log\-transformed components,

log⁡\|ci​j\|=log⁡αi​j\+log⁡\(‖𝒗~j‖‖𝒐i‖\)\+log⁡\|cos⁡\(𝒐i,𝒗~j\)\|,\\log\|c\_\{ij\}\|=\\log\\alpha\_\{ij\}\+\\log\\left\(\\frac\{\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\}\{\\\|\{\\bm\{o\}\}\_\{i\}\\\|\}\\right\)\+\\log\\bigl\|\\cos\(\{\\bm\{o\}\}\_\{i\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\)\\bigr\|,\(21\)The log transformation converts the multiplicative form into an additive one, enabling meaningful assessment of linear dependence\. Because contribution weights can be signed, all analyses are performed on their magnitudes\. Table[2](https://arxiv.org/html/2606.07604#A4.T2)reports pairwise correlations for LLaMA\-3\.1\-8B for 32 different sequences of length 128 across all token positions, heads and layers\. Attention is negatively correlated with relative norm \(r=−0\.287r=\-0\.287\), affirming our previous observations that when attention is large value norm is small and vice versa, and weakly correlated with cosine similarity \(r=0\.147r=0\.147\)\. Cosine similarity and relative norm are uncorrleated \(r=−0\.07r=\-0\.07\)\. Overall, these low correlations suggest that each component captures a distinct factor of variation, and that high attention alone does not imply high contribution—its effect depends on whether value geometry \(norms and alignment\) amplifies or attenuates the attention signal\.

Table 2:Correlation of contribution components\.Pairwise Pearson correlations between log\-transformed contribution components for LLaMA\-3\.1\-8B\. Correlations are computed across 32 sequences of length 128, aggregating over all token positions, attention heads, and layers\. Low correlations indicate that attention weight, relative norm, and cosine similarity capture distinct aspects of token contribution\.
### D\.2Variance Decomposition Experimental Setup\.

We evaluate three token groups: \(i\) all tokensj∈\{1,…,T\}j\\in\\\{1,\\ldots,T\\\}, \(ii\) the initial tokenj=1j=1, and \(iii\) non\-initial tokensj\>1j\>1\. We compute the contribution weights and their components for three models: LLaMA\-3\.1\-8B, using two datasets: FineWeb\-Edu and GSM8K, with 32 samples from each and a maximum sequence length of 512\. For each regression we uniformly sample 5 million data points performing 5 fold cross validation, evaluating on a held\-out test set\. Model\-wiseR2R^\{2\}results for LLaMA\-3\.1\-8B are reported in Table[3](https://arxiv.org/html/2606.07604#A4.T3)\.

### D\.3Extended Results on GSM8K

Below we provide further results, following the same experimental setup as described above but using GSM8K instead of FineWeb\-Edu\. TheR2R^\{2\}values are very similar across datasets, suggesting that the importance of each component is largely indpendent of the data source\.

Table 3:Regression analysis of contribution weight components\.We report the coefficient of determination \(R2R^\{2\}\) for linear regressions of contribution weights onto individual and combined components\. Each row corresponds to a different regression model: \(1\) the scalar attention weightlog⁡\(αi​j\)\\log\(\\alpha\_\{ij\}\), \(2\) the relative norm of the projected value vectornormi​j=log⁡\(‖𝒗~j‖2/‖𝒐i‖2\)\\mathrm\{norm\}\_\{ij\}=\\log\(\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}\\\|\_\{2\}/\\\|\{\\bm\{o\}\}\_\{i\}\\\|\_\{2\}\), \(3\) the cosine similaritylog⁡\|cos⁡\(𝒗~j,𝒐i\)\|\\log\\bigl\|\\cos\(\\tilde\{\{\\bm\{v\}\}\}\_\{j\},\{\\bm\{o\}\}\_\{i\}\)\\bigr\|, and \(4–6\) combinations of these components\. We evaluate regressions under three token groupings: \(i\)j∈\{1,T\}j\\in\\\{1,T\\\}\(all tokens\), \(ii\)j=1j=1\(initial tokens\), and \(iii\)j\>1j\>1\(non\-initial tokens\)\. HigherR2R^\{2\}values indicate that the corresponding component or combination explains more of the contribution weight variance\.
### D\.4Value geometry suppresses high attention\.

Initial tokens receive disproportionate attention mass but contribute far less than their attention weights suggest\.

We quantify this via the average first\-token inter\-head contribution𝔼​c1\\mathbb\{E\}\{c\}\_\{1\}, defined, for any given input sequence, as the mean first token contribution across all layers, heads, and query positions:

𝔼​\[c^1\(l\)\]=1T​∑h=1H∑i=1Tc^i​1\(l,h\),𝔼​\[c^1\]=1L​∑l=1L𝔼​\[c^1\(l\)\]\\mathbb\{E\}\[\\hat\{c\}^\{\(l\)\}\_\{1\}\]=\\frac\{1\}\{T\}\\sum^\{H\}\_\{h=1\}\\sum^\{T\}\_\{i=1\}\\hat\{c\}^\{\(l,h\)\}\_\{i1\},\\quad\\quad\\mathbb\{E\}\[\\hat\{c\}\_\{1\}\]=\\frac\{1\}\{L\}\\sum^\{L\}\_\{l=1\}\\mathbb\{E\}\[\\hat\{c\}\_\{1\}^\{\(l\)\}\]\(22\)where the layer\-wise average𝔼​c^\(l\)\\mathbb\{E\}\\hat\{c\}^\{\(l\)\}sums over heads such that contribution rows are normalized222For attention and value\-weighted attention, which are not normalized over heads, we compute layer\-wise averages as𝔼​α1\(l\)=1H​T​∑h=1H∑i=1Tαi​1\(l,h\)\\mathbb\{E\}\{\\alpha\}\_\{1\}^\{\(l\)\}=\\frac\{1\}\{HT\}\\sum^\{H\}\_\{h=1\}\\sum^\{T\}\_\{i=1\}\\alpha\_\{i1\}^\{\(l,h\)\}to maintain comparable scales\.\.[Table4](https://arxiv.org/html/2606.07604#A4.T4)compares initial\-token weights for LLaMA models on FineWeb\-Edu across three metrics: attention, value\-weighted attention, and inter\-head contribution\.

Table 4:Average weight on the first token\.Average weight on the first token \([Equation22](https://arxiv.org/html/2606.07604#A4.E22)\) for LLaMA models on FineWeb\-Edu across three metrics: 1\) attention, 2\) value\-weighted attention, and 3\) contribution\. Value\-weighted \(VW\) and contribution weights assign substantially less mass to the first token than attention, indicating that value magnitude and direction mitigate the overemphasis introduced by softmax normalisation\.Across all models, attention consistently overestimates first\-token importance\. Value\-weighted attention \(which incorporates value magnitude\) reduces this overemphasis, while contribution weights \(incorporating both value magnitude and directional alignment\) reduce it further still\. In LLaMA\-3\.1\-8B, for instance, attention assignsα^1\\hat\{\\alpha\}\_\{1\}=0\.708 to the first token, yet it contributes onlyc^1\\hat\{c\}\_\{1\}=0\.142, corresponding to a5×5\\timessuppression\.[Section6\.1](https://arxiv.org/html/2606.07604#S6.SS1)established that value geometry largely explains first\-token contributions, with the product of relative value norm and cosine similarity accounting forR2=0\.725R^\{2\}=0\.725of variance\. We now examine how these geometric factors jointly suppress first\-token influence despite high attention, exploring their functional role in the attention mechanism\.

### D\.5Value geometry concentrates contributions\.

For tail tokens \(j\>1j\>1\), value geometry concentrates rather than suppresses contributions\. We quantify concentration via entropyS\(l,h\)S^\{\(l,h\)\}for each head:

c¯i​j\(l,h\)=\|ci​j\(l,h\)\|∑k=1i\|ci​k\(l,h\)\|,S\(l,h\)=1T​∑i=1T∑j=1i−c¯i​j\(l,h\)​log⁡c¯i​j\(l,h\)\\bar\{c\}\_\{ij\}^\{\(l,h\)\}=\\frac\{\|c\_\{ij\}^\{\(l,h\)\}\|\}\{\\sum^\{i\}\_\{k=1\}\|c\_\{ik\}^\{\(l,h\)\}\|\},\\quad\\quad S^\{\(l,h\)\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}\\sum^\{i\}\_\{j=1\}\-\\bar\{c\}\_\{ij\}^\{\(l,h\)\}\\log\\bar\{c\}\_\{ij\}^\{\(l,h\)\}\(23\)where normalization accounts for signed contributions and lower entropy indicates greater concentration\.[Table5](https://arxiv.org/html/2606.07604#A4.T5)reports average entropy for LLaMA models, including and excluding the first token\. The attention sink strongly affects apparent attention concentration: for LLaMA\-8B, attention entropy isS=1\.21S=1\.21including the first token \(appearing highly concentrated due to the sink\), but rises toS=2\.61S=2\.61when excluding it \(revealing the true spread across tail tokens\)\. Comparing tail tokens only, value\-weighted attention hasS=2\.58S=2\.58and contribution weights haveS=2\.27S=2\.27, both lower than attention’sS=2\.61S=2\.61\. The entropy reduction from both attention metrics to contribution demonstrates that the cosine similarity between tokens and the output helps to concentrate attention: tokens with high attention but poor value\-output alignment contribute less than their attention weights indicate, while geometrically aligned tokens contribute more, resulting in a more concentrated effective distribution\.

Table 5:Average entropy of weight matrices\.Average entropySSof: 1\) attention, 2\) value\-weighted attention, and 3\) contribution weights \([Equation23](https://arxiv.org/html/2606.07604#A4.E23)\) across layers and heads of LLaMA models on FineWeb\-Edu\. Entropy is reported for all tokens and with the first token excluded\. Including the first token lowers entropy substantially due to its disproportionate attention mass\. When excluded, value\-weighted attention and contribution weights—which account for value magnitude and direction—show lower entropy, indicating that value information sharpens token selectivity\.
### D\.6Interference refines contributions\.

We hypothesize that this sharpening is a result of geometric filtering, where the alignment of the attended value vector with the layer’s output direction modulates the final contribution\. To quantify this effect we consider the relationship between a token’s contribution\-to\-attention ratio,

ci​j\(l,h\)αi​j\(l,h\)=‖𝒗~j\(l,h\)‖‖𝒐i\(l,h\)‖​cos⁡\(𝒐i\(l,h\),𝒗~j\(l,h\)\)\\frac\{c\_\{ij\}^\{\(l,h\)\}\}\{\\alpha\_\{ij\}^\{\(l,h\)\}\}=\\frac\{\\\|\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\\\|\}\{\\\|\{\\bm\{o\}\}\_\{i\}^\{\(l,h\)\}\\\|\}\\cos\(\{\\bm\{o\}\}\_\{i\}^\{\(l,h\)\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\)\(24\)and it’s cosine similarity with the outputcos⁡\(𝒐i\(l,h\),𝒗~j\(l,h\)\)\\cos\(\{\\bm\{o\}\}\_\{i\}^\{\(l,h\)\},\\tilde\{\{\\bm\{v\}\}\}\_\{j\}^\{\(l,h\)\}\), making sure to compute contribution with respect to the head\-level output𝒐i\(l,h\)\{\\bm\{o\}\}\_\{i\}^\{\(l,h\)\}to avoid interference between heads\. The contribution\-to\-attention ratio isolates the effect of value geometry and how it can boost \(positive ratio\) or suppress \(negative ratio\) a token’s contribution relative to it’s attention weight\. Fitting the contribution\-to\-attention ratio against cosine similarity we observe a very strong near\-linear relationship \(r=0\.949r=0\.949,p=0\.000p=0\.000\) where: the value vectors are positively aligned contributions are amplified, near\-orthogonal vectors are filtered regardless of their attention mass and negatively aligned vectors actively induce destructive interference in the output\.

## Appendix EA simple model for the relationship between Sink\-Rate and Output Norms

Recall the definition of output as sum of projected value vectors \(where we omit layer and head in the notation\):

𝐨i=αi,0​v~0\+∑1≤j≤iαi,j​v~j\\mathbf\{o\}\_\{i\}=\\alpha\_\{i,0\}\\tilde\{v\}\_\{0\}\+\\sum\_\{1\\leq j\\leq i\}\\alpha\_\{i,j\}\\tilde\{v\}\_\{j\}and define the vector𝐰i:=∑1≤j≤iαi,j​v~j=𝐨i−αi,0​v~0\\mathbf\{w\}\_\{i\}:=\\sum\_\{1\\leq j\\leq i\}\\alpha\_\{i,j\}\\tilde\{v\}\_\{j\}=\\mathbf\{o\}\_\{i\}\-\\alpha\_\{i,0\}\\tilde\{v\}\_\{0\}so that, omitting indices forα\\alpha,

‖𝐨i‖2=‖α​v~0\+𝐰i‖2=α2​‖v~0‖2\+2​α​⟨v~0,𝐰i⟩\+‖𝐰i‖2\\begin\{split\}\|\|\\mathbf\{o\}\_\{i\}\|\|^\{2\}&=\|\|\\alpha\\tilde\{v\}\_\{0\}\+\\mathbf\{w\}\_\{i\}\|\|^\{2\}\\\\ &=\\alpha^\{2\}\|\|\\tilde\{v\}\_\{0\}\|\|^\{2\}\+2\\alpha\\langle\\tilde\{v\}\_\{0\},\\mathbf\{w\}\_\{i\}\\rangle\+\|\|\\mathbf\{w\}\_\{i\}\|\|^\{2\}\\end\{split\}\(25\)
[Figure11](https://arxiv.org/html/2606.07604#A5.F11)\[bottom\] shows empirically for a specific choice of\(l,h\)\(l,h\)head how, noticing how atα=1\\alpha=1the variance is null and the resulting value is always 0, both

‖𝐰i‖≃κ​\(1−α\),κ\>0\|\|\\mathbf\{w\}\_\{i\}\|\|\\simeq\\kappa\(1\-\\alpha\),\\quad\\kappa\>0\(26\)and

⟨v~0,𝐰i⟩≃γ​\(1−α\),γ<0\\langle\\tilde\{v\}\_\{0\},\\mathbf\{w\}\_\{i\}\\rangle\\simeq\\gamma\(1\-\\alpha\),\\quad\\gamma<0\(27\)
so that

‖𝐨i‖2≃α2​‖v~0‖2\+2​α​\(α−1\)​γ\+\(1−α\)2​κ2\\begin\{split\}\|\|\\mathbf\{o\}\_\{i\}\|\|^\{2\}&\\simeq\\alpha^\{2\}\|\|\\tilde\{v\}\_\{0\}\|\|^\{2\}\+2\\alpha\(\\alpha\-1\)\\gamma\+\(1\-\\alpha\)^\{2\}\\kappa^\{2\}\\end\{split\}\(28\)and thus

‖𝐨i‖≃α2​‖v~0‖2\+2​α​\(1−α\)​γ\+\(1−α\)2​κ2\\begin\{split\}\|\|\\mathbf\{o\}\_\{i\}\|\|&\\simeq\\sqrt\{\\alpha^\{2\}\|\|\\tilde\{v\}\_\{0\}\|\|^\{2\}\+2\\alpha\(1\-\\alpha\)\\gamma\+\(1\-\\alpha\)^\{2\}\\kappa^\{2\}\}\\end\{split\}\(29\)
In[Figure11](https://arxiv.org/html/2606.07604#A5.F11)\[top\], we present the distribution ofR2R^\{2\}scores across heads for each layer\. The consistently highR2R^\{2\}values \(median\>0\.8\>0\.8and\>0\.7\>0\.7across most layers\) indicate that the linear model captures the relationship with high fidelity\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x10.png)

![Refer to caption](https://arxiv.org/html/2606.07604v1/x11.png)

Figure 11:Left: Linear fit of‖𝐰i\(l,h\)‖\|\|\\mathbf\{w\}^\{\(l,h\)\}\_\{i\}\|\|againstαi,1\(l,h\)\\alpha\_\{i,1\}^\{\(l,h\)\}\. Right: Linear fit of⟨v~1\(l,h\),𝐰i\(l,h\)⟩\\langle\\tilde\{v\}\_\{1\}^\{\(l,h\)\},\\mathbf\{w\}^\{\(l,h\)\}\_\{i\}\\rangleagainstαi,1\(l,h\)\\alpha\_\{i,1\}^\{\(l,h\)\}\. Top: AggregateR2R^\{2\}statistics\. Bottom: Fit for a fixed index\(l,h\)\(l,h\), each point corresponds to a token coloured according to its relative sequence position\.In an attempt to explain this phenomenon, recall how

𝐰i:=∑1≤j≤iαi,j​v~j\\mathbf\{w\}\_\{i\}:=\\sum\_\{1\\leq j\\leq i\}\\alpha\_\{i,j\}\\tilde\{v\}\_\{j\}and since the constraint on attention weight is1=αi,0\+∑1≤j≤iαi,j1=\\alpha\_\{i,0\}\+\\sum\_\{1\\leq j\\leq i\}\\alpha\_\{i,j\}it must hold that

∑1≤j≤iαi,j=1−αi,0\\sum\_\{1\\leq j\\leq i\}\\alpha\_\{i,j\}=1\-\\alpha\_\{i,0\}
From these observations it follows that if we definepi,j:=αi,j1−αi,0p\_\{i,j\}:=\\frac\{\\alpha\_\{i,j\}\}\{1\-\\alpha\_\{i,0\}\}for1≤j≤i1\\leq j\\leq ithen these scalars are positive and sum to11, indeed they are the renormalized attention distribution once the \[BOS\] token is removed\. Note that with this renormalisation in mind one has

𝐰i:=\(1−αi,0\)​∑1≤j≤ipi,j​v~j=\(1−αi,0\)​𝐰^i\\mathbf\{w\}\_\{i\}:=\(1\-\\alpha\_\{i,0\}\)\\sum\_\{1\\leq j\\leq i\}p\_\{i,j\}\\tilde\{v\}\_\{j\}=\(1\-\\alpha\_\{i,0\}\)\\hat\{\\mathbf\{w\}\}\_\{i\}where𝐰^i:=∑1≤j≤ipi,j​v~j=𝔼pi,⋅​\[v~\]\\hat\{\\mathbf\{w\}\}\_\{i\}:=\\sum\_\{1\\leq j\\leq i\}p\_\{i,j\}\\tilde\{v\}\_\{j\}=\\mathbb\{E\}\_\{p\_\{i,\\cdot\}\}\[\\tilde\{v\}\]\.

The previous empirical results are telling us that both‖𝐰^i‖\\\|\\hat\{\\mathbf\{w\}\}\_\{i\}\\\|and⟨𝐰^i,v~0⟩\\langle\\hat\{\\mathbf\{w\}\}\_\{i\},\\tilde\{v\}\_\{0\}\\rangleare approximately constant\. Since𝐰^i\\hat\{\\mathbf\{w\}\}\_\{i\}is the expected value underpi,⋅p\_\{i,\\cdot\}this roughly follows from other observations on thev~j\\tilde\{v\}\_\{j\}, namely that their norms are of the same magnitude and that their dot product withv~0\\tilde\{v\}\_\{0\}is constant \(and negative\)\.

Note that the approximate equality

‖𝐨i‖≃α2​‖v~0‖2\+2​α​\(1−α\)​γ\+\(1−α\)2​κ2\\begin\{split\}\|\|\\mathbf\{o\}\_\{i\}\|\|&\\simeq\\sqrt\{\\alpha^\{2\}\|\|\\tilde\{v\}\_\{0\}\|\|^\{2\}\+2\\alpha\(1\-\\alpha\)\\gamma\+\(1\-\\alpha\)^\{2\}\\kappa^\{2\}\}\\end\{split\}\(30\)implies that the minimum norm is not attained atα=1\\alpha=1as a simplistic understanding of sink token as suppressor would lead to think, but is expected at the point

α=κ2−γκ2−2​γ\+‖v~0‖2=‖𝐰^‖2−⟨v~0,𝐰^⟩‖𝐰^−v~0‖2\\alpha=\\frac\{\\kappa^\{2\}\-\\gamma\}\{\\kappa^\{2\}\-2\\gamma\+\\\|\\tilde\{v\}\_\{0\}\\\|^\{2\}\}=\\frac\{\\\|\\hat\{\\mathbf\{w\}\}\\\|^\{2\}\-\\langle\\tilde\{v\}\_\{0\},\\hat\{\\mathbf\{w\}\}\\rangle\}\{\\\|\\hat\{\\mathbf\{w\}\}\-\\tilde\{v\}\_\{0\}\\\|^\{2\}\}\(31\)
In fact the minimum in the interval\[0,1\]\[0,1\]occurs atα=1\\alpha=1only if

‖𝐰^‖2−⟨v~0,𝐰^⟩≥‖𝐰^‖2−2​⟨v~0,𝐰^⟩\+‖v~0‖2⇔‖v~0‖2≤⟨v~0,𝐰^⟩⇔‖v~0‖‖𝐰^‖≤cos⁡\(v~0,𝐰^\),\\\|\\hat\{\\mathbf\{w\}\}\\\|^\{2\}\-\\langle\\tilde\{v\}\_\{0\},\\hat\{\\mathbf\{w\}\}\\rangle\\geq\\\|\\hat\{\\mathbf\{w\}\}\\\|^\{2\}\-2\\langle\\tilde\{v\}\_\{0\},\\hat\{\\mathbf\{w\}\}\\rangle\+\\\|\\tilde\{v\}\_\{0\}\\\|^\{2\}\\iff\\\|\\tilde\{v\}\_\{0\}\\\|^\{2\}\\leq\\langle\\tilde\{v\}\_\{0\},\\hat\{\\mathbf\{w\}\}\\rangle\\iff\\frac\{\\\|\\tilde\{v\}\_\{0\}\\\|\}\{\\\|\\hat\{\\mathbf\{w\}\}\\\|\}\\leq\\cos\(\\tilde\{v\}\_\{0\},\\hat\{\\mathbf\{w\}\}\),which is empirically never the case as the anti\-alignment of the sink value with non\-sink values leads tocos⁡\(v~0,𝐰^\)<0\\cos\(\\tilde\{v\}\_\{0\},\\hat\{\\mathbf\{w\}\}\)<0\.

![Refer to caption](https://arxiv.org/html/2606.07604v1/x12.png)Figure 12:Contribution weight matrices\. Attention and contribution weights computed from LLaMA\-3\.1\-8B evaluated on FineWeb\-Edu\. Rows alternate between attention weights and contribution weights\. All attention weights and contribution weights are plotted with the same colourbar axis from\[0,1\]\[0,1\]and\[−0\.01,0\.15\]\[\-0\.01,0\.15\]respectively\.![Refer to caption](https://arxiv.org/html/2606.07604v1/x13.png)Figure 13:Contribution weight matrices\. Attention and contribution weights computed from Qwen2\.5\-7B evaluated on FineWeb\-Edu\. Rows alternate between attention weights and contribution weights\. All attention weights and contribution weights are plotted with the same colourbar axis from\[0,1\]\[0,1\]and\[−0\.01,0\.15\]\[\-0\.01,0\.15\]respectively\.![Refer to caption](https://arxiv.org/html/2606.07604v1/x14.png)Figure 14:Contribution weight matrices\. Attention and contribution weights computed from Qwen3\-8B evaluated on FineWeb\-Edu\. Rows alternate between attention weights and contribution weights\. All attention weights and contribution weights are plotted with the same colourbar axis from\[0,1\]\[0,1\]and\[−0\.01,0\.15\]\[\-0\.01,0\.15\]respectively\.