Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

# Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Source: [https://arxiv.org/html/2607.01002](https://arxiv.org/html/2607.01002)
Aryo Pradipta GemaQBeatrice AlexKPasquale MinerviniQ,V QUniversity of EdinburghKHeriot\-Watt UniversityVMiniml\.AI \{aryo\.gema, p\.minervini\}@ed\.ac\.ukb\.alex@hw\.ac\.uk

###### Abstract

In long\-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy\-pasting them\. Identifying which attention heads perform this synthesis matters for interpreting long\-context model behavior\. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal\-copy criterion that captures*where*a head reads but not*what*it writes through its output\-value \(OV\) circuit, the very mechanism that carries non\-literal retrieval\. We introduce Logit\-Contribution Scoring \(LOCOS\), a write\-aware detector that scores each head by the projection of its OV\-circuit output onto the answer\-token unembedding direction, contrasting needle and off\-needle source positions in a single forward pass\. Across three model families \(Qwen3, Gemma\-3, OLMo\-3\.1\), mean\-ablating the topLOCOSheads on the NoLiMa non\-literal retrieval benchmark collapses ROUGE\-L at lower head counts than prior attention\-based detections; on Qwen3\-8B, ablating 50 heads drives ROUGE\-L from0\.4010\.401to0\.0000\.000while the strongest baseline still retains0\.2920\.292\. The selected heads are retrieval\-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation\. On Qwen3\-8B, the same ablation also drops MuSiQue from0\.550\.55to0\.080\.08and BABILong from0\.620\.62to0\.200\.20, while a random\-heads control stays within0\.050\.05of baseline\.

[![[Uncaptioned image]](https://arxiv.org/html/2607.01002v1/GitHub-Mark.png)locos](https://github.com/aryopg/locos)[![[Uncaptioned image]](https://arxiv.org/html/2607.01002v1/hf-logo.png)locos\-results](https://huggingface.co/datasets/aryopg/locos-results)

## 1Introduction

The ability of large language models \(LLMs\) to retrieve information from their input context, rather than relying on memorized parametric knowledge, depends on a sparse set of attention heads known as*retrieval heads*\(Wuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib17)\), which build on earlier mechanistic work on induction heads\(Elhageet al\.,[2021](https://arxiv.org/html/2607.01002#bib.bib24); Olssonet al\.,[2022](https://arxiv.org/html/2607.01002#bib.bib25)\)\. However, in practice, context retrieval is rarely literal copy\-paste: a user’s question may share no lexical overlap with the relevant passage, and the model must identify the relevant snippet, parse its meaning, and synthesize an answer from it \(as illustrated in[Fig\.˜1](https://arxiv.org/html/2607.01002#S1.F1)\)\. Yet all existing identification methods, whether via token\-matching heuristics\(Wuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib17)\)or weighted attention accumulation\(Fuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib19); Linet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib18)\), evaluate heads on literal copying tasks and share a common observable: each head’s attention pattern over source positions\. This observable captures*where*a head allocates attention, but does not capture*what information*the head propagates through its output\-value \(OV\) circuit, the very mechanism by which non\-literal retrieval is accomplished\. Two heads with identical attention patterns but different OV circuits can propagate entirely different information from the same positions\. For literal retrieval, where the attended token is the answer token, attention and OV output are trivially aligned, and attention\-based scoring works\. In non\-literal retrieval, a head may attend to “Eiffel Tower” while writing the “Yuki” direction to the residual stream\. Attention\-based scoring sees the read site \(“Eiffel Tower”\); the OV circuit determines the write content \(“Yuki”\), and the two need not agree\. This distinction matters in practice, and no prior retrieval\-head detector makes this distinction\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x1.png)Figure 1:Non\-literal retrieval requires synthesis\.The same context answers two questions differently: a literal question requires reading “Eiffel Tower” directly from the needle, while a non\-literal question must produce “Yuki” after synthesizing the context\.Our method, Logit\-Contribution Scoring \(LOCOS\), measures how each attention head contributes to the correct answer token in the unembedding space \(See[Fig\.˜2](https://arxiv.org/html/2607.01002#S3.F2)\)\. For each head at each source position, the method computes the scalar projection of the head’s weighted OV circuit output onto the correct answer’s unembedding vector\. Aggregation uses*spatial contrast*: logit contributions from needle positions are compared against length\-normalized off\-needle contributions within a single decoding step\. The method requires only a single forward pass per probing trial\.

Ablation experiments on six configurations spanning three model families \(i\.e\.,Qwen3 \(8B, 14B, 32B\)\(Team,[2025b](https://arxiv.org/html/2607.01002#bib.bib33)\), Gemma\-3 \(12B, 27B\)\(Team,[2025a](https://arxiv.org/html/2607.01002#bib.bib34)\), and OLMo\-3\.1 \(32B\)\(Olmoet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib35)\)\) on the NoLiMa non\-literal retrieval benchmark heldout set\(Modarressiet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib20)\)validate the method: mean\-ablating the top\-rankedLOCOSheads produces a steeper ROUGE\-L degradation curve than all evaluated baselines \([§˜4\.2](https://arxiv.org/html/2607.01002#S4.SS2)\)\. On Qwen3\-8B, ablating the top\-50LOCOSheads collapses ROUGE\-L from0\.4010\.401to0\.0000\.000, while the strongest attention\-based baseline retains0\.2920\.292at the same depth\. Control experiments confirm retrieval specificity: parametric recall and arithmetic reasoning remain intact under the same ablation \([§˜4\.6](https://arxiv.org/html/2607.01002#S4.SS6)\)\. The same ablation also degrades downstream long\-context performance, most strongly on the Qwen3 family \([§˜4\.8](https://arxiv.org/html/2607.01002#S4.SS8)\): on Qwen3\-8B, ablating the top\-5050LOCOSheads drops MuSiQue accuracy from0\.550\.55to0\.080\.08and BABILong from0\.620\.62to0\.200\.20; transfer is most consistent on the Qwen3 family, and the ranking against attention\-based baselines is benchmark\-dependent on Gemma\-3 and OLMo\-3\.1 \([§˜4\.8](https://arxiv.org/html/2607.01002#S4.SS8)\)\.

## 2Background

Notation\.Consider a transformer\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.01002#bib.bib1)\)withLLlayers andHHattention heads per layer, head dimensiondhd\_\{h\}, and model dimensiond=H⋅dhd=H\\cdot d\_\{h\}\. Head\(l,h\)\(l,h\)attends to source positions with weightsαt,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}, reads value vectors𝐯t,j\(l,h\)∈ℝdh\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}\\in\\mathbb\{R\}^\{d\_\{h\}\}, and writes to the residual stream via its output projectionWO\(l,h\)∈ℝd×dhW\_\{O\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{h\}\}\. The per\-position output of head\(l,h\)\(l,h\)from source positionjjis:

𝐨t,j\(l,h\)=αt,j\(l,h\)⋅WO\(l,h\)​𝐯t,j\(l,h\)∈ℝd\.\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\\;=\\;\\alpha^\{\(l,h\)\}\_\{t,j\}\\cdot W\_\{O\}^\{\(l,h\)\}\\,\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}\\;\\in\\;\\mathbb\{R\}^\{d\}\\,\.\(1\)The unembedding matrixWU∈ℝ\|𝒱\|×dW\_\{U\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}maps the residual stream to logits;𝐮y∈ℝd\\mathbf\{u\}\_\{y\}\\in\\mathbb\{R\}^\{d\}is itsyy\-th row\.

Attention heads as read\-and\-write circuits\.An attention head decomposes into a*QK circuit*that determinesαt,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}\(where the head reads\) and an*OV circuit*that maps each source value throughWOW\_\{O\}to the residual stream \(what the head writes\)\(Elhageet al\.,[2021](https://arxiv.org/html/2607.01002#bib.bib24)\)\. The full output of head\(l,h\)\(l,h\)at stepttis∑j𝐨t,j\(l,h\)\\sum\_\{j\}\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}, and the next\-token logits come from mapping the final\-layer residual stream throughWUW\_\{U\}\. A head is useful for retrieval only when both stages agree: the QK circuit must select the right source positions*and*the OV circuit must write an answer\-aligned update\.

Induction heads and retrieval heads\.Induction heads implement the literal copy pattern\[A\]​\[B\]​…​\[A\]↦\[B\]\[A\]\[B\]\\ldots\[A\]\\mapsto\[B\]\(Olssonet al\.,[2022](https://arxiv.org/html/2607.01002#bib.bib25)\): they place highαt,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}on a previously matching position and write an output whose projection onto𝐮yt\\mathbf\{u\}\_\{y\_\{t\}\}is large because the attended token*is*the next token\. Retrieval heads, identified on literal needle\-in\-a\-haystack \(NIAH\) prompts byWuet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib17)\), generalize this picture to long\-context factuality\. Their detection procedure rewards heads whose argmax attention falls inside the needle and the attended token matches the generated token, a literal\-copy criterion\.

Literal versus non\-literal retrieval\.In a NIAH setup\(Kamradt,[2023](https://arxiv.org/html/2607.01002#bib.bib2)\), a*needle*is a short answer\-bearing span inserted into a longer distractor context \(the*haystack*\); we index it as\[sτ,eτ\)\[s\_\{\\tau\},e\_\{\\tau\}\)for trialτ\\tauand refer to all other source positions as*off\-needle*\. In literal NIAH, the answer token appears in the needle, so large needle attentionαt,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}usually coincides with a large logit\-relevant write𝐮yt⊤​𝐨t,j\(l,h\)\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\. NoLiMa breaks this equivalence\(Modarressiet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib20)\): the answer must be recovered from the meaning of the needle and may share no lexical overlap with it\. A head can then retrieve non\-literal information by attending to a semantically relevant phrase inside\[sτ,eτ\)\[s\_\{\\tau\},e\_\{\\tau\}\)and writing an answer\-aligned direction even when no attended token matchesyty\_\{t\}\. This is the regime targeted byLOCOS\.

## 3Logit\-Contribution Scoring

![Refer to caption](https://arxiv.org/html/2607.01002v1/x2.png)Figure 2:An attention head has two circuits: where it reads \(QK\) and what it writes \(OV\)\. Logit\-Contribution Scoring uses the OV circuit to identify*non\-literal retrieval heads*\.\(a\)Anatomy of a head’s per\-position output: the QK circuit produces attention weightαt,j\\alpha\_\{t,j\}; the OV circuit producesWO​𝐯jW\_\{O\}\\mathbf\{v\}\_\{j\}\. Attention\-based methods measure onlyα\\alpha\. Logit\-contribution scoring \(LOCOS\) measuresϕ=𝐮yt⊤​\(α⋅WO​𝐯j\)\\phi=\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\(\\alpha\\cdot W\_\{O\}\\mathbf\{v\}\_\{j\}\), capturing the full pipeline\.\(b\)Consequence for non\-literal retrieval: two heads read from “Eiffel Tower” to answer “Yuki\.” Head A attends strongly \(α=0\.30\\alpha\{=\}0\.30\), but its OV output is orthogonal to the answer direction \(ϕ≈0\\phi\{\\approx\}0\)\. Head B attends moderately \(α=0\.08\\alpha\{=\}0\.08\) but writes toward the answer \(ϕ=1\.3\\phi\{=\}1\.3\)\. Attention\-based methods select Head A;LOCOSselects Head B\.We introduceLOCOS, which scores each head by what it*writes*toward the answer rather than where it allocates attention, because non\-literal retrieval transforms attended content through the OV circuit before it becomes an answer\. We define a three\-step procedure:

Per\-Position Logit Contribution\.Consider a probing trialτ\\tauwith needle span\[sτ,eτ\)\[s\_\{\\tau\},e\_\{\\tau\}\)embedded in a context ofNτN\_\{\\tau\}tokens\. Let𝒜τ\\mathcal\{A\}^\{\\tau\}denote the set of decoding steps at which the model generates a correct answer token \(identified by matching against the tokenized gold answer\),yt∈𝒱y\_\{t\}\\in\\mathcal\{V\}the correct token at steptt, andNtN\_\{t\}the total number of key positions available at steptt\. The contribution of source positionjjthrough head\(l,h\)\(l,h\)to the logit ofyty\_\{t\}is:

ϕt,j\(l,h\)=𝐮yt⊤​𝐨t,j\(l,h\)=αt,j\(l,h\)⋅𝐮yt⊤​WO\(l,h\)​𝐯t,j\(l,h\)∈ℝ\.\\phi^\{\(l,h\)\}\_\{t,j\}\\;=\\;\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\\,\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\\;=\\;\\alpha^\{\(l,h\)\}\_\{t,j\}\\cdot\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}W\_\{O\}^\{\(l,h\)\}\\,\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}\\;\\in\\;\\mathbb\{R\}\\,\.\(2\)The scalarϕt,j\(l,h\)\\phi^\{\(l,h\)\}\_\{t,j\}depends on both*where*the head reads \(viaαt,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}\) and*what*it extracts \(viaWO\(l,h\)​𝐯t,j\(l,h\)W\_\{O\}^\{\(l,h\)\}\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}\)\. A head that attends strongly to a position whose OV output is orthogonal to𝐮yt\\mathbf\{u\}\_\{y\_\{t\}\}receivesϕ≈0\\phi\\approx 0despite high attention; conversely, a head performing non\-literal retrieval \(e\.g\.,attending to “Paris” to produce “France”\) receives largeϕ\\phibecause its OV circuit transforms the attended representation into an answer\-aligned output\.

Spatial Contrast\.For each head\(l,h\)\(l,h\)at answer stepttin trialτ\\tau, we define the logit contribution from needle and off\-needle positions:

Φt\(l,h\),\+=∑j=sτeτ−1ϕt,j\(l,h\),Φt\(l,h\),−=eτ−sτNt−\(eτ−sτ\)​∑j∉\[sτ,eτ\)ϕt,j\(l,h\),\\Phi^\{\(l,h\),\+\}\_\{t\}=\\sum\_\{j=s\_\{\\tau\}\}^\{e\_\{\\tau\}\-1\}\\phi^\{\(l,h\)\}\_\{t,j\}\\,,\\qquad\\Phi^\{\(l,h\),\-\}\_\{t\}=\\tfrac\{e\_\{\\tau\}\-s\_\{\\tau\}\}\{N\_\{t\}\-\(e\_\{\\tau\}\-s\_\{\\tau\}\)\}\\sum\_\{j\\notin\[s\_\{\\tau\},\\,e\_\{\\tau\}\)\}\\phi^\{\(l,h\)\}\_\{t,j\}\\,,\(3\)where the rescaling factor\(eτ−sτ\)/\(Nt−\(eτ−sτ\)\)\(e\_\{\\tau\}\{\-\}s\_\{\\tau\}\)/\(N\_\{t\}\{\-\}\(e\_\{\\tau\}\{\-\}s\_\{\\tau\}\)\)makesΦt\(l,h\),\+\\Phi^\{\(l,h\),\+\}\_\{t\}andΦt\(l,h\),−\\Phi^\{\(l,h\),\-\}\_\{t\}comparable: both represent the logit contribution of a region of length\(eτ−sτ\)\(e\_\{\\tau\}\{\-\}s\_\{\\tau\}\)\. The contrastΦt\(l,h\),\+−Φt\(l,h\),−\\Phi^\{\(l,h\),\+\}\_\{t\}\-\\Phi^\{\(l,h\),\-\}\_\{t\}is*spatial*: it compares needle and off\-needle positions within a single decoding step, rather than the*temporal*contrast \(answer vs\. non\-answer steps\) used by attention\-based methods\. Spatial contrast yields a score from a single answer step, identifies heads that attend to the needle persistently but write answer\-relevant content only from needle positions, and cancels uniform contributors such as token\-frequency priors whoseϕt,j\(l,h\)\\phi^\{\(l,h\)\}\_\{t,j\}is large but position\-independent\.

Aggregation\.We pool over all answer steps across all trials passing a correctness filter \(ROUGE\-1 recall\>ρ\>\\rho, defaultρ=0\.5\\rho=0\.5\), without per\-trial normalization\. Let𝒟pass⊆\{1,…,T\}\\mathcal\{D\}\_\{\\mathrm\{pass\}\}\\subseteq\\\{1,\\ldots,T\\\}denote the set of passing trials\. The final score is:

Sl,h=1∑τ∈𝒟pass\|𝒜τ\|​∑τ∈𝒟pass∑t∈𝒜τ\(Φt\(l,h\),\+−Φt\(l,h\),−\)\.S\_\{l,h\}=\\frac\{1\}\{\\sum\_\{\\tau\\in\\mathcal\{D\}\_\{\\mathrm\{pass\}\}\}\|\\mathcal\{A\}^\{\\tau\}\|\}\\sum\_\{\\tau\\in\\mathcal\{D\}\_\{\\mathrm\{pass\}\}\}\\sum\_\{t\\in\\mathcal\{A\}^\{\\tau\}\}\\Big\(\\Phi^\{\(l,h\),\+\}\_\{t\}\-\\Phi^\{\(l,h\),\-\}\_\{t\}\\Big\)\\,\.\(4\)The scoreSl,hS\_\{l,h\}is the mean over all \(trial, answer\-step\) pairs, with each answer step weighted equally\. Unlike attention\-based methods, we do not clampSl,hS\_\{l,h\}at zero: negative values indicate heads whose logit contribution toward the correct answer originates predominantly from off\-needle positions, which points to factors other than needle\-specific retrieval \(e\.g\.,parametric or contextual associations\)\. A worked example with Per\-trial consistency diagnostics is reported in[Appx\.˜G](https://arxiv.org/html/2607.01002#A7)\.LOCOSrecovers attention\-based scoring as a special case when the OV circuit contributes no position\-dependent signal\.111[Appx\.O](https://arxiv.org/html/2607.01002#A15)works through the reduction\.

## 4Experiments

### 4\.1Experimental Setup

Probing benchmark\.NoLiMa\(Modarressiet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib20)\)is aneedle\-in\-a\-haystackbenchmark in which each trial embeds a factual statement in a long context and poses a question whose answer requires understanding the needle’s meaning rather than copying a literal token from it\. Its non\-literal retrieval property expose weaknesses of attention\-based scoring\. We use theonehopreasoning, 10 context lengths×\\times10 insertion depths, and 3 characters per entry \(context range 1,000–5,000 tokens\)\. We evaluateLOCOSon six models from three families: Qwen3 \(8B, 14B, 32B\)\(Team,[2025b](https://arxiv.org/html/2607.01002#bib.bib33)\), Gemma\-3 \(12B, 27B\)\(Team,[2025a](https://arxiv.org/html/2607.01002#bib.bib34)\), and OLMo\-3\.1 \(32B\)\(Olmoet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib35)\)\. Trials with ROUGE\-1 recall\>0\.5\>0\.5against the gold answer are retained, followingWuet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib17)\)\.

Causal validation via mean\-ablation\.For each selected head\(l,h\)\(l,h\)and at every decode steptt, we replace the post\-Q\-projection, pre\-RoPE query vector with a head\-specific calibration vector𝐪¯\(l,h\)\\bar\{\\mathbf\{q\}\}^\{\(l,h\)\}\(computed once over a 50\-trial sample; see[Appx\.˜B](https://arxiv.org/html/2607.01002#A2)\); we then evaluate ROUGE\-L on a disjoint held\-out set of 800 NoLiMa trials\. Mean replacement keeps downstream\-layer activations in\-distribution\(Nandaet al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib44); Wanget al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib26)\)\. Architecture\-specific implementation details are given in[Appx\.˜H](https://arxiv.org/html/2607.01002#A8)\.

Baselines\.We compareLOCOSagainst:\(i\)Random: uniformly sampled heads, used to check that non\-trivial ablation effects require targeted selection\.\(ii\)Wu/NIAH\-scored: the token\-matching retrieval score ofWuet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib17)\)computed on NIAH dataset; and\(iii\)Wu/NoLiMa\-scored: the same token\-matching criterion computed on NoLiMa probing trials;The suffix on*Wu/X\-scored*denotes the dataset used to detect the retrieval heads\.

### 4\.2Ablation Comparison Across Scoring Methods

![Refer to caption](https://arxiv.org/html/2607.01002v1/x3.png)Figure 3:LOCOSheads produce steeper ROUGE\-L degradation under mean\-ablation across all six models\.Each panel shows NoLiMa ROUGE\-L \(800 trials\) as a function of the number of ablated headskkfor four scoring methods across three model families at two scales each: Qwen3 \(8B, 14B, 32B\), OLMo\-3\.1 \(32B\), and Gemma\-3 \(12B, 27B\)\.LOCOS\(blue\) produces the steepest degradation curve in every model, reaching near\-zero ROUGE\-L byk=50k\{=\}50in five of six configurations and severe degradation \(≈0\.1\{\\approx\}\\,0\.1\) in Qwen3\-32B\.[Fig\.˜3](https://arxiv.org/html/2607.01002#S4.F3)compares the four methods under mean\-ablation of the top\-kkheads\. On Qwen3\-8B, ablating the top\-55LOCOSheads already reduces ROUGE\-L to0\.3210\.321\(baseline=0\.401=0\.401\), while ablating Wu/NIAH heads remains at0\.4060\.406\. Byk=50k\{=\}50,LOCOSreaches0\.0000\.000, whereas Wu/NIAH\-scored still achieves0\.2920\.292and Wu/NoLiMa\-scored0\.3370\.337\. Random\-head ablation remains near baseline throughout \(0\.3580\.358–0\.4020\.402\); the degradation is therefore specific to head selection\. Wu/NoLiMa\-scored\-ranked heads ablation raises ROUGE\-L above baseline at intermediatekk\(0\.4280\.428atk=5k\{=\}5\), which indicates that its token\-matching criterion selects causally irrelevant heads\.

*Takeaway 1:*LOCOSheads produce the steepest ROUGE\-L ablation curve on NoLiMa, reaching near\-zero output at lowerkkthan all evaluated baselines\.

### 4\.3Isolating the OV Contribution: An Attention\-Only Spatial\-Contrast Control

LOCOSdiffers fromWuet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib17)\)in both the per\-position observable \(OV projectionϕ\\phiv\.s\.attention\-based token matching\) and aggregation \(spatialv\.s\.temporal contrast\)\. To isolate the OV contribution, we substituteα\\alphaforϕ\\phiin[Equations˜3](https://arxiv.org/html/2607.01002#S3.E3)and[4](https://arxiv.org/html/2607.01002#S3.E4), yielding an*attention\-only spatial\-contrast*scoreSl,hattS^\{\\text\{att\}\}\_\{l,h\}that matchesLOCOS’s aggregation but removes the OV projection \([Propositions˜1](https://arxiv.org/html/2607.01002#Thmproposition1)and[O\.3](https://arxiv.org/html/2607.01002#A15.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x4.png)Figure 4:OV projections improve causal head selection on most models\.Each panel shows NoLiMa ROUGE\-L \(800 held\-out trials\) under mean\-ablation of the top\-kkheads ranked byLOCOS\(blue\) and the attention\-only control \(cyan\)\. Both scorers use identical spatial\-contrast aggregation; only the per\-position observable differs\.LOCOSis stronger on Qwen3\-8B, Qwen3\-32B, and Gemma\-3\-12B, comparable on Qwen3\-14B and OLMo\-3\.1\-32B, and weaker at largekkon Gemma\-3\-27B\.[Fig\.˜4](https://arxiv.org/html/2607.01002#S4.F4)shows thatLOCOSis more damaging on Qwen3\-8B, Qwen3\-32B, and Gemma\-3\-12B: atk=50k\{=\}50, it reaches0\.0000\.000versus0\.1480\.148on Qwen3\-8B and0\.0770\.077versus0\.3460\.346on Qwen3\-32B\. The scorers are comparable on Qwen3\-14B and OLMo\-3\.1\-32B, while Gemma\-3\-27B inverts at depth: the attention\-only control is more damaging atk∈\{20,50\}k\{\\in\}\\\{20,50\\\}\(0\.1180\.118vs\.0\.3690\.369atk=20k\{=\}20\)\.222We analyze the Gemma\-3\-27B inversion in[§N\.5](https://arxiv.org/html/2607.01002#A14.SS5), where tuned\-lens\-corrected projections test whether the direct\-path assumption explains this inversion\.Thus, OV projection primarily improves reliability rather than uniformly increasing effect size:LOCOSis the only scorer in our evaluation that produces severe or total collapse across configurations\.

*Takeaway 2:*Under identical spatial\-contrast aggregation, an attention\-only control is competitive on average but substantially less reliable across models\. The OV projection enablesLOCOSto consistently identify highly causal heads across all six configurations\.

### 4\.4Bottom\-kkControl: Spatial Source Matters

A potential objection to the ablation result is circularity: the method selects heads whose OV output projects onto𝐮yt\\mathbf\{u\}\_\{y\_\{t\}\}, so ablating them mechanically reduces theyty\_\{t\}logit regardless of whether the heads perform retrieval\. If this were the case,*any*set of heads with large answer\-aligned logit contribution should be equally causal when ablated,*irrespective of whether that contribution originates from the needle or from unrelated context*\. Heads with the most negative spatial contrast scores \(Sl,h≪0S\_\{l,h\}\\ll 0\) \(i\.e\.,*bottom\-kkheads*\) provide a direct test: they contribute strongly to the correct\-answer logit but predominantly from off\-needle positions\. The absolute logit contribution of these bottom\-kkheads is large, so the comparison with top\-kkheads is not confounded by magnitude\. The full score distribution \([Appx\.˜I](https://arxiv.org/html/2607.01002#A9)\) confirms that all bottom\-50 heads have strictly negativeSl,hS\_\{l,h\}in every model, so the bottom\-kkexperiments exclusively target heads with off\-needle\-dominant contributions\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x5.png)Figure 5:Bottom\-kkablation does not degrade retrieval\.Each panel shows NoLiMa ROUGE\-L as a function of ablation depthkkfor top\-kk\(blue\), bottom\-kk\(cyan\), and random heads \(orange\) for three representative models \(one per family\); the full six\-model version is in[Appx\.˜L](https://arxiv.org/html/2607.01002#A12)\. Top\-kkheads produce steep degradation; bottom\-kkheads track the random baseline despite having equally large absolute logit contribution, ruling out the circularity objection\.As shown in[Fig\.˜5](https://arxiv.org/html/2607.01002#S4.F5), mean\-ablating the top\-kkLOCOSheads produces steep ROUGE\-L degradation, reaching near\-zero atk=50k\{=\}50in most models\. By contrast, ablating the bottom\-kkheads leaves ROUGE\-L near baseline, tracking the random\-head control at all ablation depths\.

*Takeaway 3:*Ablating bottom\-kkheads \(those contributing to the answer logit from off\-needle positions\) does not degrade NoLiMa ROUGE\-L\. This indicates that the ablation result \([§˜4\.2](https://arxiv.org/html/2607.01002#S4.SS2)\) is not merely an artifact of removing answer\-aligned signal\.

### 4\.5Layer Distribution

![Refer to caption](https://arxiv.org/html/2607.01002v1/x6.png)Figure 6:LOCOSheads are more concentrated in late layers than Wu/NIAH\-scored scores\.Layer×\\timesHead heatmaps on NoLiMa for Gemma\-3\-27B \(left\) and Qwen3\-32B \(right\)\. The left\-hand panel of each model showsLOCOS; the right shows Wu/NIAH\-scored token\-matching\. Red squares mark top\-10 heads\. BothLOCOSand Wu/NIAH\-scored assign high scores predominantly to late layers, but Wu/NIAH\-scored additionally identifies heads in early\-to\-middle layers\.[Fig\.˜6](https://arxiv.org/html/2607.01002#S4.F6)visualizes score distributions across layers for Gemma\-3\-27B and Qwen3\-32B\.LOCOSis most concentrated in the Qwen3 family and Gemma\-3\-27B; at KV\-group granularity, Gemma\-3\-12B and OLMo\-3\.1\-32B span broader layer ranges \([Appx\.˜K](https://arxiv.org/html/2607.01002#A11)\)\. One possible explanation is thatLOCOSrelies on a*direct\-path assumption*that is more accurate near the output, biasing it toward late layers\.

A tuned\-lens check on Gemma\-3\-27B preserves the late\-layer band, and causal activation patching on Qwen3\-8B and Gemma\-3\-12B also concentrates top\-10 heads in upper layers \([§§˜N\.4](https://arxiv.org/html/2607.01002#A14.SS4)and[N\.6](https://arxiv.org/html/2607.01002#A14.SS6)\)\. The top\-10 sets overlap only marginally between the two scores \(2/10 on Qwen3\-8B, 3/10 on Gemma\-3\-12B\); the top\-kkLOCOSset should therefore be read as a collectively causal retrieval circuit, validated by the group ablations of[§˜4\.2](https://arxiv.org/html/2607.01002#S4.SS2), rather than a list of individually load\-bearing heads\.

*Takeaway 4:*LOCOSscores concentrate in late layers in the Qwen3 family and Gemma\-3\-27B; the layer pattern is family\-dependent \([Appx\.˜K](https://arxiv.org/html/2607.01002#A11)\)\. A tuned\-lens variant \([§˜N\.4](https://arxiv.org/html/2607.01002#A14.SS4)\) and a causal\-attribution probe \([§˜N\.6](https://arxiv.org/html/2607.01002#A14.SS6)\) confirm the late\-layer concentration is not a direct\-path artifact on the models examined; the top\-kkLOCOSset should be read as a collectively causal retrieval circuit rather than a universal architectural pattern across all families\.

### 4\.6Retrieval Specificity

A potential concern is thatLOCOSidentifies generically important heads for model output, not retrieval\-specific\. We test this by ablatingLOCOSheads and measuring performance on tasks that do not require retrieval:\(i\)City–country associations: parametric factual recall \(e\.g\.,“Which country does Paris belong to?”\)\.\(ii\)PopQA\(Mallenet al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib31)\): top 100 popular QA drawn from a knowledge base \(e\.g\.,“Who is the mother of Jesus?”\)\.\(iii\)Arithmetic: two\-operand addition and subtraction \(e\.g\.,“What is 47 \+ 23?”\)\.333Full dataset available at[![[Uncaptioned image]](https://arxiv.org/html/2607.01002v1/hf-logo.png)aryopg/parametric\-arithmetic\-eval](https://huggingface.co/datasets/aryopg/parametric-arithmetic-eval)

To compare specificity across methods, we define the*Dissociation Score*at ablation depthkkasDS​\(k\)=Δ​R​\(k\)−Δ​P​\(k\)\\mathrm\{DS\}\(k\)=\\Delta R\(k\)\-\\Delta P\(k\), whereΔ​R​\(k\)=\(R0−R​\(k\)\)/R0\\Delta R\(k\)=\(R\_\{0\}\-R\(k\)\)/R\_\{0\}andΔ​P​\(k\)=\(P0−P​\(k\)\)/P0\\Delta P\(k\)=\(P\_\{0\}\-P\(k\)\)/P\_\{0\}are the relative drops in NoLiMa ROUGE\-L and aggregate parametric accuracy \(mean of city–country, PopQA, and arithmetic\), respectively\.R0,P0R\_\{0\},P\_\{0\}are baselines\.DS=1\\mathrm\{DS\}\{=\}1means retrieval is fully destroyed with zero parametric damage;k∗=arg⁡maxk⁡DS​\(k\)k^\{\*\}=\\arg\\max\_\{k\}\\mathrm\{DS\}\(k\)identifies the most\-specific ablation depth\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x7.png)Figure 7:LOCOSheads exhibit the strongest functional dissociation between retrieval and parametric capabilities\.Each panel shows DS\(k\)\(k\)\(lines, right axis\) and parametric accuracy \(bars, left axis\) as a function of ablation depthkkfor four scoring methods, on three representative models \(one per family\); the full six\-model version is in[Appx\.˜L](https://arxiv.org/html/2607.01002#A12)\. Higher DS indicates that ablation degrades retrieval far more than parametric tasks\.LOCOS\(blue\) achieves the highest DS in every model configuration; the enlarged marker indicatesk∗k^\{\*\}, the point of maximum dissociation\.[Fig\.˜7](https://arxiv.org/html/2607.01002#S4.F7)shows DS\(k\)\(k\)for each scoring method, withLOCOSachieving the highest peak in every model\. Parametric accuracy likewise remains stable under bottom\-kkablation \([Appx\.˜J](https://arxiv.org/html/2607.01002#A10)\); these heads are therefore not responsible for parametric recall either\.

*Takeaway 5:*Retrieval heads identified byLOCOSare retrieval\-specific: ablating them severely degrades contextual retrieval while leaving parametric accuracy near baseline \([Fig\.˜7](https://arxiv.org/html/2607.01002#S4.F7)\), andLOCOSachieves the highest dissociation\-score peak in every configuration\.

### 4\.7Literal vs\. Non\-Literal Retrieval Specificity

[§˜4\.6](https://arxiv.org/html/2607.01002#S4.SS6)established that ablatingLOCOSheads degrades non\-literal retrieval far more than parametric tasks\. A write\-aware detector that measures the full OV pipeline should, in principle, identify retrieval heads in both literal and non\-literal regimes; the novel contribution over attention\-based scoring is access to the non\-literal subset\. We test this directly by mean\-ablating the same top\-kkLOCOSheads \(selected on NoLiMa probing trials\) and evaluating on both NoLiMa and standard NIAH\(Kamradt,[2023](https://arxiv.org/html/2607.01002#bib.bib2)\), treating the two benchmarks as probes for the non\-literal and literal retrieval circuits respectively, using the protocol of[§˜4\.1](https://arxiv.org/html/2607.01002#S4.SS1)for both benchmarks\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x8.png)Figure 8:AblatingLOCOSheads damages non\-literal retrieval more than literal retrieval\.Each panel shows ROUGE\-L on NoLiMa \(solid\) and standard NIAH \(dashed\) under mean\-ablation of the same top\-kkLOCOSheads, with the NoLiMa and NIAH baselines marked by solid and dashed gray lines\. Three representative models are shown here; the full six\-model version is in[Appx\.˜L](https://arxiv.org/html/2607.01002#A12)\. The NoLiMa curve declines more steeply in every configuration, reaching near\-zero atk=50k\{=\}50in five of six models, and the NoLiMa–NIAH gap widens withkk\.[Fig\.˜8](https://arxiv.org/html/2607.01002#S4.F8)shows that the non\-literal curve drops faster than the literal curve in every configuration, but the literal curve also drops in five of six models, so the top\-kkLOCOSset is not strictly non\-literal\-specific\. Both retrieval regimes require heads that read the answer span; they differ only in what the OV circuit writes\. The top\-kkset contains both kinds of heads: those whose OV write is answer\-aligned in both regimes damage both benchmarks when ablated, while those aligned only in the non\-literal regime explain the larger NoLiMa drop\.LOCOStherefore identifies retrieval\-relevant heads in both regimes; its novelty relative to attention\-based scoring lies in identifying the non\-literal subset\.

*Takeaway 6:*Ablating top\-kkLOCOSheads degrades both non\-literal \(NoLiMa\) and literal \(NIAH\) retrieval, with a steeper drop on NoLiMa\. The dissociation indicates thatLOCOScaptures retrieval\-relevant heads beyond those identified by attention\-based scoring, including heads that contribute to non\-literal retrieval which prior detectors miss\.

### 4\.8Downstream Long\-Context Evaluation

We test transfer by re\-running the head ablation on two downstream long\-context benchmarks that were not used to score heads\.MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2607.01002#bib.bib51)\)is a multi\-hop QA benchmark whose questions require composing facts across multiple paragraphs of a long supporting context\.BABILong\(qa2 and qa3 subsets\)\(Kuratovet al\.,[2024](https://arxiv.org/html/2607.01002#bib.bib50)\)requires tracing an object’s trajectory through movements interleaved with distractor narrative; literal token copying is insufficient because each candidate location is mentioned many times for many entities \(see[Appx\.˜M](https://arxiv.org/html/2607.01002#A13)for an example\)\. We fixk=50k\{=\}50for direct comparison with the headline NoLiMa result and retain the random\-heads control and the Wu/NIAH\-scored baseline\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x9.png)Figure 9:Mean\-ablating top\-5050LOCOSheads degrades downstream long\-context performance, most strongly on the Qwen3 family\.Accuracy on MuSiQue \(top\) and BABILong qa2\+qa3 \(bottom\) for six models\. Bars show the unablated baseline \(gray\) and the three ablation conditions: random heads \(orange\), Wu/NIAH\-scored heads \(pink\), andLOCOS\(blue\)\. Error bars are standard deviations across three independent runs\.LOCOSproduces the largest drop in66of1212model–benchmark cells; Wu/NIAH\-scored ablation is more damaging on MuSiQue for Gemma\-3 and OLMo\-3\.1, and slightly raises BABILong accuracy on Qwen3\-14B, Gemma\-3\-27B, and OLMo\-3\.1\-32B\.[Fig\.˜9](https://arxiv.org/html/2607.01002#S4.F9)shows that ablating the top\-5050LOCOSheads on Qwen3\-8B drops MuSiQue accuracy from0\.550\.55to0\.080\.08and BABILong from0\.620\.62to0\.200\.20\. Across the twelve model–benchmark cells,LOCOSis the most damaging ablation in six and produces a drop of at least0\.100\.10below baseline in eight, while the random\-heads control stays within0\.050\.05of baseline in eleven; the degradation is therefore specific to the headsLOCOSidentifies\. The ranking against attention\-based scoring is benchmark\-dependent: on MuSiQue, Wu/NIAH\-scored ablation is more damaging on Gemma\-3 \(12B, 27B\) and OLMo\-3\.1\-32B, consistent with multi\-hop QA recruiting both literal and non\-literal retrieval; on BABILong, Wu/NIAH\-scored ablation slightly*raises*accuracy on Qwen3\-14B, Gemma\-3\-27B, and OLMo\-3\.1\-32B, whileLOCOSablation drops accuracy in every model\.

*Takeaway 7:*Ablating top\-5050LOCOSheads degrades downstream long\-context performance, most strongly on the Qwen3 family \(Qwen3\-8B MuSiQue:0\.55→0\.080\.55\{\\to\}0\.08, BABILong:0\.62→0\.200\.62\{\\to\}0\.20\)

## 5Related Work

Attention is not explanation\.The faithfulness of attention weights as an explanation has been contested\(Jain and Wallace,[2019](https://arxiv.org/html/2607.01002#bib.bib47); Serrano and Smith,[2019](https://arxiv.org/html/2607.01002#bib.bib9); Wiegreffe and Pinter,[2019](https://arxiv.org/html/2607.01002#bib.bib48)\), withBastings and Filippova \([2020](https://arxiv.org/html/2607.01002#bib.bib10)\)arguing that gradient\-based saliency methods provide more reliable attribution than attention alone\. Our method,LOCOS, measures the head’s direct write onto the answer logit rather than its attention distribution, aligning with the*task\-grounded faithfulness*criteria\(Wiegreffe and Pinter,[2019](https://arxiv.org/html/2607.01002#bib.bib48)\)\.

OV/QK circuit analysis and attribution\.Our method builds on the QK/OV circuit decomposition ofElhageet al\.\([2021](https://arxiv.org/html/2607.01002#bib.bib24)\)and generalizes the literal copying mechanism of induction heads\(Olssonet al\.,[2022](https://arxiv.org/html/2607.01002#bib.bib25)\)to non\-literal retrieval\. The logit lens\(nostalgebraist,[2020](https://arxiv.org/html/2607.01002#bib.bib22)\)and tuned lens\(Belroseet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib23)\)project residual\-stream states onto the unembedding matrix at per\-layer granularity;LOCOSapplies an analogous projection at per\-head, per\-position granularity with spatial contrast, building on direct logit attribution\.McDougallet al\.\([2023](https://arxiv.org/html/2607.01002#bib.bib12)\)characterize copy suppression heads whose OV circuits actively inhibit direct token copying, which may be a complementary mechanism to the non\-literal retrieval heads identified here\.

Retrieval head identification\.Existing methods identify retrieval heads through attention\-weight observables\.Wuet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib17)\)define retrieval heads via token matching on needle\-in\-a\-haystack tasks;Fuet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib19)\),Linet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib18)\), andXiaoet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib30)\)extend attention\-based head scoring to KV cache allocation, semantic attention\-mass criteria, and retrieval/streaming\-head separation\. The broader KV cache compression literature\(Zhanget al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib5); Liet al\.,[2024](https://arxiv.org/html/2607.01002#bib.bib6); Xiaoet al\.,[2024](https://arxiv.org/html/2607.01002#bib.bib7); Caiet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib8)\)motivates per\-head retrieval scoring by showing that uniform token eviction misses structured head and layer importance\. These methods measure the QK circuit, but not the OV circuit\. For literal retrieval, this suffices; for non\-literal retrieval, the OV circuit transforms attended content, and attention\-based methods miss these heads\.Sunet al\.\([2025](https://arxiv.org/html/2607.01002#bib.bib45)\)analyze OV circuits for hallucination detection in RAG, decomposing contributions into copying heads \(attention\) and knowledge FFNs; our method differs by scoring individual heads at per\-position granularity with spatial contrast, targeting retrieval head identification\.

## 6Conclusion

Existing retrieval\-head detectors\(Wuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib17); Fuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib19); Linet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib18)\)score attention heads by where they read\. This identifies the heads that copy literal tokens, but misses the heads that synthesize answers from the meaning of an attended span\. We presentedLOCOS\(Logit\-Contribution Scoring\), a detector that scores each head by the projection of its OV\-circuit output onto the answer\-token unembedding direction, and uses a needle\-versus\-off\-needle spatial contrast to isolate position\-specific writes within a single forward pass per probing trial\. Across the six configurations evaluated, mean\-ablating the top\-kkheads selected byLOCOScollapses NoLiMa ROUGE\-L at lowerkkthan every attention\-based baseline we evaluate\. The selected heads are retrieval\-specific: under the same ablation, parametric recall and arithmetic reasoning remain at baseline\. The top\-kkLOCOSset captures retrieval\-relevant heads in both literal and non\-literal regimes; its advantage over attention\-based scoring is in additionally identifying the non\-literal subset that the token\-matching criterion misses \([§˜4\.7](https://arxiv.org/html/2607.01002#S4.SS7)\)\.

Limitations\.\(i\)Off\-needle baseline:If the context contains distractor information related to the answer,Φ−\\Phi^\{\-\}rises and scores drop for heads performing broad semantic matching rather than targeted needle retrieval—desirable for span\-specific retrieval, but may miss heads performing more diffuse contextual integration;\(ii\)Architecture coverage: Our model selection does not include Mixture\-of\-experts routing, encoder–decoder stacks, and state\-space hybrids\. The per\-head OV decomposition still applies in principle, but the causal head\-ablation magnitudes and late\-layer concentration we report should not be assumed to transfer without verification\.

## Acknowledgements

APG was supported by the United Kingdom Research and Innovation \(grant EP/S02431X/1\), UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics\. BA is a Fellow at and has been supported by the Generative AI Lab \(GAIL\) at the University of Edinburgh\. PM was supported by the Engineering and Physical Sciences Research Council \(EPSRC\) through the AI Hub in Generative Models \(grant number EP/Y028805/1\)\. This work was supported by the Edinburgh International Data Facility \(EIDF\) and the Data\-Driven Innovation Programme at the University of Edinburgh\. We also thank Neel Rajani, Raj Bhalwankar, Matteo Attimonelli, and Federico Tiblias for their helpful comments and suggestions\.

## References

- GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6–10, 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 4895–4901\.External Links:[Link](https://doi.org/10.18653/v1/2023.emnlp-main.298),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.298)Cited by:[Appendix K](https://arxiv.org/html/2607.01002#A11.p1.1)\.
- J\. Bastings and K\. Filippova \(2020\)The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?\.InProceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,Online,pp\. 149–155\.External Links:[Link](https://aclanthology.org/2020.blackboxnlp-1.14/),[Document](https://dx.doi.org/10.18653/v1/2020.blackboxnlp-1.14)Cited by:[§5](https://arxiv.org/html/2607.01002#S5.p1.1)\.
- N\. Belrose, I\. Ostrovsky, L\. McKinney, Z\. Furman, L\. Smith, D\. Halawi, S\. Biderman, and J\. Steinhardt \(2025\)Eliciting latent predictions from transformers with the tuned lens\.arXiv preprint arXiv:2303\.08112\.Cited by:[§N\.3](https://arxiv.org/html/2607.01002#A14.SS3.p1.7),[§5](https://arxiv.org/html/2607.01002#S5.p2.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: dynamic KV cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.External Links:[Link](https://arxiv.org/abs/2406.02069)Cited by:[§5](https://arxiv.org/html/2607.01002#S5.p3.1)\.
- N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, N\. DasSarma, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah \(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2021/framework/index\.htmlCited by:[§O\.1](https://arxiv.org/html/2607.01002#A15.SS1.p1.6),[§1](https://arxiv.org/html/2607.01002#S1.p1.1),[§2](https://arxiv.org/html/2607.01002#S2.p2.6),[§5](https://arxiv.org/html/2607.01002#S5.p2.1)\.
- Y\. Fu, Z\. Cai, A\. Asi, W\. Xiong, Y\. Dong, and W\. Xiao \(2025\)Not all heads matter: a head\-level KV cache compression method with integrated retrieval and reasoning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=FJFVmeXusW)Cited by:[§O\.5](https://arxiv.org/html/2607.01002#A15.SS5.p1.4),[Appendix O](https://arxiv.org/html/2607.01002#A15.p1.1),[Appendix P](https://arxiv.org/html/2607.01002#A16.p1.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p1.1),[§1](https://arxiv.org/html/2607.01002#S1.p1.1),[§5](https://arxiv.org/html/2607.01002#S5.p3.1),[§6](https://arxiv.org/html/2607.01002#S6.p1.3)\.
- A\. P\. Gema, C\. Jin, A\. Abdulaal, T\. Diethe, P\. A\. Teare, B\. Alex, P\. Minervini, and A\. Saseendran \(2025\)DeCoRe: decoding by contrasting retrieval heads to mitigate hallucinations\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 10003–10039\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.531/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.531),ISBN 979\-8\-89176\-335\-7Cited by:[Appendix P](https://arxiv.org/html/2607.01002#A16.p1.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p1.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p2.1)\.
- S\. Jain and B\. C\. Wallace \(2019\)Attention is not Explanation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 3543–3556\.External Links:[Link](https://aclanthology.org/N19-1357/),[Document](https://dx.doi.org/10.18653/v1/N19-1357)Cited by:[§5](https://arxiv.org/html/2607.01002#S5.p1.1)\.
- G\. Kamradt \(2023\)LLMTest\_NeedleInAHaystack: pressure testing LLMs\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)GitHub repositoryCited by:[Table 1](https://arxiv.org/html/2607.01002#A1.T1.1.1.12.1),[§2](https://arxiv.org/html/2607.01002#S2.p4.6),[§4\.7](https://arxiv.org/html/2607.01002#S4.SS7.p1.1)\.
- Y\. Kuratov, A\. Bulatov, P\. Anokhin, I\. Rodkin, D\. I\. Sorokin, A\. Sorokin, and M\. Burtsev \(2024\)BABILong: testing the limits of LLMs with long context reasoning\-in\-a\-haystack\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=u7m2CG84BQ)Cited by:[§4\.8](https://arxiv.org/html/2607.01002#S4.SS8.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10–15, 2024,pp\. 22947–22970\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/28ab418242603e0f7323e54185d19bde-Abstract-Conference.html)Cited by:[Appendix K](https://arxiv.org/html/2607.01002#A11.p4.1),[Appendix P](https://arxiv.org/html/2607.01002#A16.p1.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p1.1),[§5](https://arxiv.org/html/2607.01002#S5.p3.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[Table 1](https://arxiv.org/html/2607.01002#A1.T1.1.1.19.1),[Appendix B](https://arxiv.org/html/2607.01002#A2.p3.1)\.
- X\. Lin, J\. Wang, O\. Kondrateva, Y\. Shi, B\. Li, and G\. L\. Zhang \(2025\)CompressKV: semantic retrieval heads know what tokens are not important before generation\.arXiv preprint arXiv:2508\.02401\.Cited by:[§O\.5](https://arxiv.org/html/2607.01002#A15.SS5.p1.4),[Appendix O](https://arxiv.org/html/2607.01002#A15.p1.1),[Appendix P](https://arxiv.org/html/2607.01002#A16.p1.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p1.1),[§1](https://arxiv.org/html/2607.01002#S1.p1.1),[§5](https://arxiv.org/html/2607.01002#S5.p3.1),[§6](https://arxiv.org/html/2607.01002#S6.p1.3)\.
- Y\. Ma and N\. Okazaki \(2026\)From interpretability to performance: optimizing retrieval heads for long\-context language models\.ArXivabs/2601\.11020\.External Links:[Link](https://api.semanticscholar.org/CorpusID:284860903)Cited by:[Appendix P](https://arxiv.org/html/2607.01002#A16.p1.1)\.
- A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. Hajishirzi \(2023\)When not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 9802–9822\.External Links:[Link](https://aclanthology.org/2023.acl-long.546/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546)Cited by:[Table 1](https://arxiv.org/html/2607.01002#A1.T1.1.1.13.1),[item \(ii\)](https://arxiv.org/html/2607.01002#S4.I2.i2)\.
- C\. McDougall, A\. Conmy, C\. Rushing, T\. McGrath, and N\. Nanda \(2023\)Copy suppression: comprehensively understanding an attention head\.External Links:2310\.04625,[Link](https://arxiv.org/abs/2310.04625)Cited by:[§5](https://arxiv.org/html/2607.01002#S5.p2.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in GPT\.Advances in Neural Information Processing Systems36\.Note:arXiv:2202\.05262Cited by:[§N\.3](https://arxiv.org/html/2607.01002#A14.SS3.SSS0.Px2.p1.1)\.
- A\. Modarressi, H\. Deilamsalehy, F\. Dernoncourt, T\. Bui, R\. A\. Rossi, S\. Yoon, and H\. Schuetze \(2025\)NoLiMa: long\-context evaluation beyond literal matching\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=0OshX1hiSa)Cited by:[Table 1](https://arxiv.org/html/2607.01002#A1.T1.1.1.11.1),[Table 2](https://arxiv.org/html/2607.01002#A2.T2.8.8.11.3),[§1](https://arxiv.org/html/2607.01002#S1.p3.8),[§2](https://arxiv.org/html/2607.01002#S2.p4.6),[§4\.1](https://arxiv.org/html/2607.01002#S4.SS1.p1.2)\.
- N\. Nanda and J\. Bloom \(2022\)TransformerLens\.Note:[https://github\.com/TransformerLensOrg/TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)Cited by:[Table 1](https://arxiv.org/html/2607.01002#A1.T1.1.1.20.1)\.
- N\. Nanda, L\. Chan, T\. Lieberum, J\. Smith, and J\. Steinhardt \(2023\)Progress measures for grokking via mechanistic interpretability\.arXiv preprint arXiv:2301\.05217\.Cited by:[§4\.1](https://arxiv.org/html/2607.01002#S4.SS1.p2.3)\.
- nostalgebraist \(2020\)Interpreting GPT: the logit lens\.Note:[https://www\.lesswrong\.com/posts/AcKRB8wDpdaN6v6ru/interpreting\-gpt\-the\-logit\-lens](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)LessWrong blog postCited by:[§5](https://arxiv.org/html/2607.01002#S5.p2.1)\.
- T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2025\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[§1](https://arxiv.org/html/2607.01002#S1.p3.8),[§4\.1](https://arxiv.org/html/2607.01002#S4.SS1.p1.2)\.
- C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen,et al\.\(2022\)In\-context learning and induction heads\.arXiv preprint arXiv:2209\.11895\.Cited by:[§1](https://arxiv.org/html/2607.01002#S1.p1.1),[§2](https://arxiv.org/html/2607.01002#S2.p3.3),[§5](https://arxiv.org/html/2607.01002#S5.p2.1)\.
- S\. Serrano and N\. A\. Smith \(2019\)Is attention interpretable?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 2931–2951\.External Links:[Link](https://aclanthology.org/P19-1282/),[Document](https://dx.doi.org/10.18653/v1/P19-1282)Cited by:[§5](https://arxiv.org/html/2607.01002#S5.p1.1)\.
- Z\. Sun, X\. Zang, K\. Zheng, J\. Xu, X\. Zhang, W\. Yu, Y\. Song, and H\. Li \(2025\)ReDeEP: detecting hallucination in retrieval\-augmented generation via mechanistic interpretability\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ztzZDzgfrh)Cited by:[§5](https://arxiv.org/html/2607.01002#S5.p3.1)\.
- G\. Team \(2025a\)Gemma 3 technical report\.ArXivabs/2503\.19786\.External Links:[Link](https://goo.gle/Gemma3Report)Cited by:[§1](https://arxiv.org/html/2607.01002#S1.p3.8),[§4\.1](https://arxiv.org/html/2607.01002#S4.SS1.p1.2)\.
- Q\. Team \(2025b\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2607.01002#S1.p3.8),[§4\.1](https://arxiv.org/html/2607.01002#S4.SS1.p1.2)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)♫ MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Link](https://aclanthology.org/2022.tacl-1.31/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475)Cited by:[§4\.8](https://arxiv.org/html/2607.01002#S4.SS8.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2607.01002#S2.p1.10)\.
- K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. Steinhardt \(2023\)Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NpsVSN6o4ul)Cited by:[§N\.3](https://arxiv.org/html/2607.01002#A14.SS3.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.01002#S4.SS1.p2.3)\.
- S\. Wiegreffe and Y\. Pinter \(2019\)Attention is not not explanation\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 11–20\.External Links:[Link](https://aclanthology.org/D19-1002/),[Document](https://dx.doi.org/10.18653/v1/D19-1002)Cited by:[§5](https://arxiv.org/html/2607.01002#S5.p1.1)\.
- W\. Wu, Y\. Wang, G\. Xiao, H\. Peng, and Y\. Fu \(2025\)Retrieval head mechanistically explains long\-context factuality\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=EytBpUGB1Z)Cited by:[§O\.4](https://arxiv.org/html/2607.01002#A15.SS4.p1.5),[§O\.6](https://arxiv.org/html/2607.01002#A15.SS6.p1.1),[Appendix O](https://arxiv.org/html/2607.01002#A15.p1.1),[Table 2](https://arxiv.org/html/2607.01002#A2.T2.1.1.1.3),[§1](https://arxiv.org/html/2607.01002#S1.p1.1),[§2](https://arxiv.org/html/2607.01002#S2.p3.3),[item \(ii\)](https://arxiv.org/html/2607.01002#S4.I1.i2.2),[§4\.1](https://arxiv.org/html/2607.01002#S4.SS1.p1.2),[§4\.3](https://arxiv.org/html/2607.01002#S4.SS3.p1.4),[§5](https://arxiv.org/html/2607.01002#S5.p3.1),[§6](https://arxiv.org/html/2607.01002#S6.p1.3)\.
- G\. Xiao, J\. Tang, J\. Zuo, J\. Guo, S\. Yang, H\. Tang, Y\. Fu, and S\. Han \(2025\)DuoAttention: efficient long\-context LLM inference with retrieval and streaming heads\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QHROe7Skvh)Cited by:[Appendix K](https://arxiv.org/html/2607.01002#A11.p4.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p1.1),[§5](https://arxiv.org/html/2607.01002#S5.p3.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7–11, 2024,External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[Appendix K](https://arxiv.org/html/2607.01002#A11.p4.1),[Appendix P](https://arxiv.org/html/2607.01002#A16.p1.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p1.1),[§5](https://arxiv.org/html/2607.01002#S5.p3.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. W\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10–16, 2023,pp\. 34661–34710\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-Conference.html)Cited by:[Appendix K](https://arxiv.org/html/2607.01002#A11.p4.1),[Appendix P](https://arxiv.org/html/2607.01002#A16.p1.1),[Appendix E](https://arxiv.org/html/2607.01002#A5.p1.1),[§5](https://arxiv.org/html/2607.01002#S5.p3.1)\.

Appendix

Table of Contents

Reproducibility\.

- [Appendix A \- Datasets, Models, and Licenses](https://arxiv.org/html/2607.01002#A1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[A](https://arxiv.org/html/2607.01002#A1)
- [Appendix B \- Experimental Setup Details](https://arxiv.org/html/2607.01002#A2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[B](https://arxiv.org/html/2607.01002#A2)
- [Appendix C \- Compute Resources](https://arxiv.org/html/2607.01002#A3)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[C](https://arxiv.org/html/2607.01002#A3)
- [Appendix D \- Code and Data Availability](https://arxiv.org/html/2607.01002#A4)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D](https://arxiv.org/html/2607.01002#A4)
- [Appendix E \- Broader Impacts](https://arxiv.org/html/2607.01002#A5)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E](https://arxiv.org/html/2607.01002#A5)
- [Appendix F \- Declaration of LLM Usage](https://arxiv.org/html/2607.01002#A6)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[F](https://arxiv.org/html/2607.01002#A6)

Method walk\-through\.

- [Appendix G \- Worked Example](https://arxiv.org/html/2607.01002#A7)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[G](https://arxiv.org/html/2607.01002#A7)
- [Appendix H \- Architecture Adaptations](https://arxiv.org/html/2607.01002#A8)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[H](https://arxiv.org/html/2607.01002#A8)

Additional empirical analyses\.

- [Appendix I \- Logit\-Contribution Score Distribution](https://arxiv.org/html/2607.01002#A9)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[I](https://arxiv.org/html/2607.01002#A9)
- [Appendix J \- Bottom\-kkDissociation Score](https://arxiv.org/html/2607.01002#A10)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[J](https://arxiv.org/html/2607.01002#A10)
- [Appendix K \- KV\-Group×\\timesLayer View](https://arxiv.org/html/2607.01002#A11)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[K](https://arxiv.org/html/2607.01002#A11)
- [Appendix L \- Six\-Model Versions of Main\-Text Ablation Figures](https://arxiv.org/html/2607.01002#A12)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[L](https://arxiv.org/html/2607.01002#A12)
- [Appendix M \- Downstream Benchmark Example](https://arxiv.org/html/2607.01002#A13)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[M](https://arxiv.org/html/2607.01002#A13)

Theoretical analysis\.

- [Appendix N \- Direct\-Path Robustness via Tuned Lens](https://arxiv.org/html/2607.01002#A14)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[N](https://arxiv.org/html/2607.01002#A14)
- [Appendix O \- Relationship to Attention\-Based Scoring](https://arxiv.org/html/2607.01002#A15)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[O](https://arxiv.org/html/2607.01002#A15)

Outlook\.

- [Appendix P \- Future Work](https://arxiv.org/html/2607.01002#A16)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[P](https://arxiv.org/html/2607.01002#A16)

## Appendix ADatasets, Models, and Licenses

[Tab\.˜1](https://arxiv.org/html/2607.01002#A1.T1)lists every external asset used in this work, together with the version we used and its license\. All assets are used within the terms of their respective licenses; we do not redistribute model weights or dataset content\.

Table 1:External assets used in this work\.Model checkpoints are accessed via the HuggingFace Hub; dataset and library versions are the ones used in our experiments\.AssetIdentifier / versionUseLicenseModelsQwen3\-8BQwen/Qwen3\-8Bdetection, ablationApache\-2\.0Qwen3\-14BQwen/Qwen3\-14Bdetection, ablationApache\-2\.0Qwen3\-32BQwen/Qwen3\-32Bdetection, ablationApache\-2\.0Gemma\-3\-12Bgoogle/gemma\-3\-12b\-itdetection, ablationGemma Terms of Use†Gemma\-3\-27Bgoogle/gemma\-3\-27b\-itdetection, ablationGemma Terms of Use†OLMo\-3\.1\-32Ballenai/OLMo\-3\.1\-32B\-Instructdetection, ablationApache\-2\.0DatasetsNoLiMa\[Modarressiet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib20)\]v1 \(onehopreasoning type\)probing benchmarkCC\-BY\-4\.0NIAH\[Kamradt,[2023](https://arxiv.org/html/2607.01002#bib.bib2)\]standard configurationbaseline scoring \(Wu/NIAH\)MITPopQA\[Mallenet al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib31)\]v1parametric specificity controlMITCity–country associationsauthors’ constructionparametric specificity controlreleased with this paperArithmetic \(two\-operand±\\pm\)authors’ constructionparametric specificity controlreleased with this paperSoftwarevLLMv0\.10\.0model loading, weight shardingApache\-2\.0PyTorch2\.4numerical backendBSD\-3\-ClauseTransformers \(HuggingFace\)4\.45tokenizers, configsApache\-2\.0rouge\-score\[Lin,[2004](https://arxiv.org/html/2607.01002#bib.bib3)\]0\.1\.2ROUGE\-L evaluationApache\-2\.0TransformerLens\[Nanda and Bloom,[2022](https://arxiv.org/html/2607.01002#bib.bib11)\]2\.7inspiration for direct logit attributionMIT

†Gemma Terms of Use permit research and commercial use subject to a prohibited\-use policy and an attribution requirement\.

## Appendix BExperimental Setup Details

This appendix consolidates the procedural details that allow the experiments in[§˜4](https://arxiv.org/html/2607.01002#S4)to be reproduced\.[Tab\.˜2](https://arxiv.org/html/2607.01002#A2.T2)summarizes the hyperparameters used throughout; the paragraphs below give the rationale and the few cases where defaults were varied\.

Table 2:Hyperparameters used throughout the paper\.ComponentValueNoteProbingNoLiMa reasoning typeonehopfollowingModarressiet al\.\[[2025](https://arxiv.org/html/2607.01002#bib.bib20)\]Context lengths10 settings, 1k–5k tokensstandard NoLiMa protocolInsertion depths10 per lengthstandard NoLiMa protocolCharacters per entry3standard NoLiMa protocolTrial filterROUGE\-1 recall\>0\.5\>0\.5followingWuet al\.\[[2025](https://arxiv.org/html/2607.01002#bib.bib17)\]Decoding \(probing and evaluation\)Strategygreedy \(temperature=0=0\)deterministicMax new tokens50sufficient for NoLiMa answers \(2–5 tokens\)AggregationBootstrap resamples \(BB\)1,000for 95% CI onSl,hS\_\{l,h\}Confidence interval2\.5th–97\.5thpercentilenon\-parametricAblationCalibration trials50 \(sampled from passing pool\)for mean\-activationHeld\-out evaluation trials800disjoint from probing poolAblation depthskk\{0,5,10,20,30,40,50\}\\\{0,5,10,20,30,40,50\\\}for top\-kkand bottom\-kk

Calibration for mean\-ablation\.Mean activations for the ablation intervention are computed from 50 NoLiMa trials sampled uniformly from the pool of ROUGE\-passing trials\. For each head\(l,h\)\(l,h\), the calibration vector𝐪¯\(l,h\)∈ℝdh\\bar\{\\mathbf\{q\}\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\}is a*mean of trial\-means*: we first compute the token\-average of the post\-Q\-projection query𝐪t\(l,h\)=WQ\(l,h\)​𝐱t\(l\)\\mathbf\{q\}^\{\(l,h\)\}\_\{t\}=W\_\{Q\}^\{\(l,h\)\}\\,\\mathbf\{x\}^\{\(l\)\}\_\{t\}within each calibration trial, then average these per\-trial vectors across the 50 trials\. Each trial therefore contributes a single equal\-weight entry irrespective of its sequence length; a flat mean over all \(token, trial\) pairs would up\-weight longer trials\. During ablation, the hook intercepts the post\-Q\-projection vector*before*the rotary position embedding is applied and replaces it with𝐪¯\(l,h\)\\bar\{\\mathbf\{q\}\}^\{\(l,h\)\}at every token position; RoPE then rotates𝐪¯\(l,h\)\\bar\{\\mathbf\{q\}\}^\{\(l,h\)\}position\-by\-position in the usual way\. The value projectionWV\(l,h\)W\_\{V\}^\{\(l,h\)\}and the output projectionWO\(l,h\)W\_\{O\}^\{\(l,h\)\}are unchanged\. Because𝐪¯\(l,h\)\\bar\{\\mathbf\{q\}\}^\{\(l,h\)\}is a single content\-independent vector, the resulting attention logits depend only on the keys \(and on the position\-dependent RoPE rotation of𝐪¯\(l,h\)\\bar\{\\mathbf\{q\}\}^\{\(l,h\)\}\), so the head’s attention distribution becomes content\-independent \(approximately uniform across positions\)\.

ROUGE\-L evaluation\.ROUGE\-L is computed at the summary level \(longest common subsequence between the generated text and gold answer, normalized by gold length\) using therouge\-scorelibrary\[Lin,[2004](https://arxiv.org/html/2607.01002#bib.bib3)\]\. Generation uses greedy decoding \(temperature=0=0\) with a maximum of 50 new tokens\.

Bootstrap confidence intervals\.For each head’s global scoreSl,hS\_\{l,h\}, we resample the set of ROUGE\-passing trialsB=1,000B=1\{,\}000times \(with replacement\), recomputeSl,hS\_\{l,h\}for each bootstrap sample, and report the 2\.5thand 97\.5thpercentiles as the 95% confidence interval\.

Wu et al\. scoring on NoLiMa and NIAH\.The Wu et al\. retrieval score assigns credit at decode stepttif \(i\) the head’s argmax attention position falls within the needle span, and \(ii\) the token at that position matches the generated token\. On NoLiMa, this criterion undercounts retrieval heads because of token\-identity mismatch: the question is asked non\-literally, so the attended needle token rarely matches the generated answer token\. The top head’s mean score drops from0\.970\.97\(NIAH\) to0\.030\.03\(NoLiMa\), a30×30\{\\times\}gap driven by the scoring criterion rather than by differences in retrieval behavior\.

## Appendix CCompute Resources

Hardware\.All experiments were run on 2×\\timesNVIDIA H100 80GB\. Tensor parallelism was used for the larger checkpoints:TP=2\\mathrm\{TP\}=2for the 14B/12B/27B/32B models,TP=1\\mathrm\{TP\}=1for Qwen3\-8B\.

Per\-experiment cost\.Detection requires one forward pass per probing trial, generating tokens autoregressively while recording attention weights and the value cache \(the primary memory overhead\)\. Ablation evaluation is similarly one forward pass per trial per ablation depthkk, plus the 50\-trial calibration pass\.

## Appendix DCode and Data Availability

The detection, ablation, and evaluation code is released at[https://github\.com/aryopg/locos](https://github.com/aryopg/locos)The repository contains:

- •Detection scripts producing the per\-headSl,hS\_\{l,h\}scores for each model in[§˜4\.1](https://arxiv.org/html/2607.01002#S4.SS1)\.
- •The vLLM\-based ablation driver described in[Appx\.˜H](https://arxiv.org/html/2607.01002#A8), including the monkey\-patches for each supported attention class\.
- •Evaluation scripts for NoLiMa ROUGE\-L \([§˜4\.2](https://arxiv.org/html/2607.01002#S4.SS2)\), the parametric controls \([§˜4\.6](https://arxiv.org/html/2607.01002#S4.SS6)\), and the bottom\-kkanalyses \([§˜4\.4](https://arxiv.org/html/2607.01002#S4.SS4)\)\.
- •AREADMEwith exact commands, environment specification \(environment\.yml, pinned package versions matching[Tab\.˜1](https://arxiv.org/html/2607.01002#A1.T1)\), and the random seeds used for all sampling steps\.

NoLiMa probing inputs are generated from the public NoLiMa release via the included generation script; we do not redistribute the underlying haystack content\. The author\-released parametric control sets \(city–country associations, two\-operand arithmetic\) are included as JSON files in the repository\.

## Appendix EBroader Impacts

LOCOSis a diagnostic tool: it identifies attention heads that contribute to non\-literal retrieval, without modifying model weights or behavior\. Its primary intended uses are interpretability research and downstream applications that benefit from head\-level retrieval signal, such as KV cache compression\[Zhanget al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib5), Liet al\.,[2024](https://arxiv.org/html/2607.01002#bib.bib6), Xiaoet al\.,[2024](https://arxiv.org/html/2607.01002#bib.bib7), Fuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib19), Linet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib18), Xiaoet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib30)\]and retrieval\-aware decoding\[Gemaet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib21)\]\. We do not release a new model, and the method does not enable new generative capabilities\.

Potential positive impacts\.A more accurate map of which heads carry non\-literal retrieval may enable \(i\) more aggressive KV cache compression at fixed quality, lowering inference cost and energy use; \(ii\) more targeted interventions for hallucination mitigation in retrieval\-augmented generation\[Gemaet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib21)\]; and \(iii\) cleaner mechanistic accounts of long\-context behavior in production\-scale models\.

Potential negative impacts and mitigations\.The method exposes which heads are causally critical for retrieval, which in principle could be used to construct adversarial inputs that suppress retrieval \(a denial\-of\-capability attack\)\. We judge this risk to be low because: \(i\) the same information is recoverable from any open\-weight model with a few hours of compute, so the marginal uplift from publication is small; \(ii\) the method requires white\-box access to attention weights and value caches, which is not available through standard inference APIs; and \(iii\) the attack surface \(degrading long\-context retrieval\) is qualitatively similar to risks already present in any KV cache compression method\. We therefore do not impose access restrictions on the released code\.

Fairness and demographic considerations\.Our evaluation uses only the English NoLiMa benchmark and English parametric controls; we do not evaluate retrieval\-head identification across languages, dialects, or demographic axes, and we do not claim that the identified heads generalize to non\-English text \([§˜6](https://arxiv.org/html/2607.01002#S6)\)\. Practitioners deployingLOCOS\-derived KV\-compression policies on multilingual systems should re\-validate on their target languages\.

## Appendix FDeclaration of LLM Usage

We used LLMs as writing assistants for paper polishing and for routine refactoring of plotting scripts\. Specifically: \(i\) prose passes for clarity and concision, particularly on appendix sections, \(ii\) LaTeX\-formatting suggestions, and \(iii\) minor refactoring of plotting scripts\. The LLM was not used to design experiments, derive the theoretical results, or generate numerical results\. All experimental claims and numerical values reported in the paper are produced by the analysis pipeline released with the code\.

## Appendix GWorked Example

The following example uses head\(16,1\)\(16,1\)from Qwen3\-8B on a NoLiMa trial where the answer is “Yuki” and the needle span\[200,212\)\[200,212\)\(12 tokens\) is embedded in a context ofN=3,000N=3\{,\}000tokens\. Numerical values below are illustrative, chosen to show the typical scale ofα\\alpha,φ\\varphi,Φ\+\\Phi^\{\+\}, andΦ−\\Phi^\{\-\}for a strongly\-retrieving head; they are not drawn from a specific logged trial\.

Step 1: Per\-position logit contribution \([Equation˜2](https://arxiv.org/html/2607.01002#S3.E2)\)\.At decode stept=1t\{=\}1producing “Yuki” \(y1y\_\{1\}\), consider needle positionj=205j=205containing the token “Kiasma”\. The head’s attention weight isα1,205\(16,1\)=0\.08\\alpha^\{\(16,1\)\}\_\{1,205\}=0\.08\. After the OV circuit, the per\-position output𝐨1,205\(16,1\)\\mathbf\{o\}^\{\(16,1\)\}\_\{1,205\}has a large component aligned with𝐮Yuki\\mathbf\{u\}\_\{\\text\{Yuki\}\}:

ϕ1,205\(16,1\)=𝐮Yuki⊤​𝐨1,205\(16,1\)=1\.3\.\\phi^\{\(16,1\)\}\_\{1,205\}=\\mathbf\{u\}\_\{\\text\{Yuki\}\}^\{\\top\}\\,\\mathbf\{o\}^\{\(16,1\)\}\_\{1,205\}=1\.3\\,\.Despite moderate attention \(α=0\.08\\alpha=0\.08\), this position contributes substantially to the “Yuki” logit because the OV circuit transforms the “Kiasma” representation into an output aligned with the answer direction\. This is exactly the non\-literal retrieval that attention\-based scoring cannot detect\.

Step 2: Spatial contrast \([Equation˜3](https://arxiv.org/html/2607.01002#S3.E3)\)\.WithN1=3,000N\_\{1\}=3\{,\}000key positions at stept=1t\{=\}1:

Φ\+\\displaystyle\\Phi^\{\+\}=∑j=200211ϕ1,j\(16,1\)=4\.7\(total logit contribution from needle\)\\displaystyle=\\textstyle\\sum\_\{j=200\}^\{211\}\\phi^\{\(16,1\)\}\_\{1,j\}=4\.7\\quad\\text\{\(total logit contribution from needle\)\}Φ−\\displaystyle\\Phi^\{\-\}=122988​∑j∉\[200,212\)ϕ1,j\(16,1\)=122988×58\.3=0\.23\(length\-normalized off\-needle\)\\displaystyle=\\tfrac\{12\}\{2988\}\\textstyle\\sum\_\{j\\notin\[200,212\)\}\\phi^\{\(16,1\)\}\_\{1,j\}=\\tfrac\{12\}\{2988\}\\times 58\.3=0\.23\\quad\\text\{\(length\-normalized off\-needle\)\}The 12 needle tokens contribute4\.74\.7to the “Yuki” logit; a comparable 12\-token span from the remaining context contributes only0\.230\.23\.

Step 3: Aggregation \([Equation˜4](https://arxiv.org/html/2607.01002#S3.E4)\)\.After 200 ROUGE\-passing trials contributing a total of 247 answer steps, head\(16,1\)\(16,1\)has pooled scoreS16,1=3\.8S\_\{16,1\}=3\.8with 95% bootstrap CI\[3\.1,4\.5\]\[3\.1,4\.5\]and per\-trial consistency0\.940\.94\(positive in 188/200 trials\)\. A head withS=0\.2S=0\.2, CI\[−0\.1,0\.5\]\[\-0\.1,0\.5\], and consistency0\.530\.53is not reliably retrieval\-active\.

## Appendix HArchitecture Adaptations

The logit\-contribution equation \([Equation˜2](https://arxiv.org/html/2607.01002#S3.E2)\) is architecture\-invariant: any transformer decomposing into attention weightsα\\alpha, value vectors𝐯\\mathbf\{v\}, an output projectionWOW\_\{O\}, and an unembedding matrixWUW\_\{U\}provides the four inputs the method requires\. The adaptations needed for different architectures are mechanical, not algorithmic\.

Grouped\-query attention \(GQA\)\.In GQA models \(Qwen3, Gemma\-3, OLMo\-3\.1\), each KV group servesG=HQ/HK​VG=H\_\{Q\}/H\_\{KV\}query heads\. The value cache is stored atHK​VH\_\{KV\}granularity and expanded toHQH\_\{Q\}heads before the per\-position projection:

𝐕~h\(l\)=𝐕⌊h⋅HK​V/HQ⌋\(l\)for​h=0,…,HQ−1\.\\tilde\{\\mathbf\{V\}\}^\{\(l\)\}\_\{h\}=\\mathbf\{V\}^\{\(l\)\}\_\{\\lfloor h\\cdot H\_\{KV\}/H\_\{Q\}\\rfloor\}\\quad\\text\{for \}h=0,\\ldots,H\_\{Q\}\-1\\,\.Query heads within the same group share value vectors but differ inWO\(l,h\)W\_\{O\}^\{\(l,h\)\}, so their logit\-contribution scores can differ\.

Vision–language models \(Gemma\-3\)\.The decoder layers and unembedding matrix reside at a nested path \(language\_model\.model\.layersandlanguage\_model\.lm\_head\)\. Per\-head QK normalization affects attention distributions but does not change the value vectors or output projections\. Detection operates on text\-only self\-attention; extending to multimodal inputs is left for future work\.

#### Implementing head ablation in vLLM\.

Our ablation intervention requires a masked pass that zeros the queries of a designated head subset while sharing the same prefix history\. vLLM’s default generation path is built around paged attention and a continuous\-batch scheduler, neither of which exposes the per\-head query manipulation the method needs\. We therefore use vLLM specifically for model loading, tensor\-parallel weight sharding, and tokenization, and replace its generation loop with a manual autoregressive driver\. At load time, we monkey\-patch each supported attention class \(Gemma3Attention,Qwen3Attention,Olmo2Attention\) so that, when the driver is active, the layer routes throughF\.scaled\_dot\_product\_attentionbacked by two independent sequential KV caches—one per pass\. Each decode step runs the forward with retrieval\-head queries zeroed\. For tensor\-parallel execution \(TP\>1\\mathrm\{TP\}\>1\), the patching and KV bookkeeping run inside every worker, and the two passes are dispatched via vLLM’scollective\_rpc; global head indices are remapped to local shard indices per rank\. The trade\-off is anO​\(n2\)O\(n^\{2\}\)KV cache and single\-request decoding rather than continuous batching; the benefit is that any model loadable by vLLM under one of the supported attention classes is supported without modifying vLLM internals\.

## Appendix ILogit\-Contribution Score Distribution

[Fig\.˜10](https://arxiv.org/html/2607.01002#A9.F10)shows the distribution ofLOCOSscoresSl,hS\_\{l,h\}across all attention heads for each of the six models, with the top\-50 and bottom\-50 heads highlighted\. In every model, the score distribution is heavily right\-skewed: a small number of heads have large positive scores, the vast majority cluster near zero, and the bottom\-50 heads all have strictly negative scores\. This confirms that the bottom\-kkablation experiments in[§˜4\.4](https://arxiv.org/html/2607.01002#S4.SS4)—which ablate up tok=50k\{=\}50heads ranked from most negative—exclusively target heads withSl,h<0S\_\{l,h\}<0,i\.e\.,heads whose logit contribution toward the correct answer originates predominantly from off\-needle positions\. The clear separation between positive and negative tails across all six models supports the validity of the spatial contrast as a discriminative criterion\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x10.png)Figure 10:Distribution ofLOCOSscores across all heads for each model\.Heads are sorted bySl,hS\_\{l,h\}; the top\-50 \(blue, left\) and bottom\-50 \(red, right\) are highlighted\. In every model, the bottom\-50 heads have strictly negative scores, confirming that the bottom\-kkexperiments \([§˜4\.4](https://arxiv.org/html/2607.01002#S4.SS4)\) exclusively ablate heads whose answer\-aligned logit contribution originates from off\-needle positions\.
## Appendix JBottom\-kkDissociation Score

[Fig\.˜11](https://arxiv.org/html/2607.01002#A10.F11)shows the dissociation score under bottom\-kkablation alongside parametric accuracy\. While top\-kkablation produces high dissociation scores—retrieval degrades steeply with parametric accuracy intact—bottom\-kkablation yields near\-zero dissociation at all depths, confirming that neither retrieval nor parametric performance is affected\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x11.png)Figure 11:Bottom\-kkablation produces near\-zero dissociation\.Dissociation score DS\(k\)\(k\)and parametric accuracy as a function of ablation depthkkfor bottom\-kkheads across six models\. Unlike top\-kkablation \([Fig\.˜7](https://arxiv.org/html/2607.01002#S4.F7)\), bottom\-kkablation leaves both retrieval and parametric performance near baseline\.
## Appendix KKV\-Group×\\timesLayer View of Logit\-Contribution Scores

In GQA models\[Ainslieet al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib4)\], query heads sharing a KV group consume identical key and value vectors, so their per\-headLOCOSscores are correlated\. The cache unit in GQA is the KV group, not the individual query head, so KV cache footprint is also reasoned about at \(layer, KV\-group\) granularity\. This appendix asks one spatial question that the head\-level analyses in[§§˜4\.2](https://arxiv.org/html/2607.01002#S4.SS2)and[4\.4](https://arxiv.org/html/2607.01002#S4.SS4)do not: at which layers do the highest\-scoring KV groups appear?

![Refer to caption](https://arxiv.org/html/2607.01002v1/x12.png)Figure 12:Top\-10LOCOScells concentrate in late layers in the Qwen3 family on NoLiMa, but span broader layer ranges in Gemma\-3\-12B and OLMo\-3\.1\-32B\.Per\-\(layer, KV\-group\) meanLOCOSscore on NoLiMa for Qwen3\-8B, Qwen3\-14B, Qwen3\-32B, OLMo\-3\.1\-32B, Gemma\-3\-12B, and Gemma\-3\-27B\. Layer is on thexx\-axis, KV group on theyy\-axis, color encodes the mean score across passing trials\. Red boxes mark the top\-10 \(layer, KV\-group\) cells per model\. Per\-panel color scales differ; the layer pattern is read from the spatial location of red boxes, not from absolute magnitude\. Per\-model layer spans and KV\-group counts are reported in[Tab\.˜3](https://arxiv.org/html/2607.01002#A11.T3)\.Observations\.[Tab\.˜3](https://arxiv.org/html/2607.01002#A11.T3)summarizes the top\-10 \(layer, KV\-group\) cells across all six GQA models\. Two patterns emerge\. First, layer concentration is family\-dependent\. The Qwen3 family places every top\-10 cell in the upper quartile of layers across all three scales \(29–34 of 36, 30–38 of 40, and 54–62 of 64\), and Gemma\-3\-27B is concentrated in the upper third \(41–53 of 62\)\. Gemma\-3\-12B and OLMo\-3\.1\-32B do not show comparable upper\-layer concentration: their top\-10 cells span 0–47 of 48 and 27–57 of 64 respectively\. Second, KV\-group spread is roughly at chance\. The number of distinct KV groups containing top\-10 cells \(6–8\) is at or just above the uniform\-allocation expectation \(≈5\.9\\approx 5\.9for 8\-group models,≈7\.6\\approx 7\.6for 16\-group models\), so the KV\-group dimension does not concentrate underLOCOSin any of the six models\. The evidence is six models from three families on one detector and one benchmark; the family\-dependent layer pattern in particular warrants replication on additional benchmarks before generalization\.

Table 3:Spatial summary of the top\-10 \(layer, KV\-group\)LOCOScells on NoLiMa\.Listed at KV\-group granularity, since query heads in a group share keys and values\. The rightmost column reports the number of distinct KV groups containing at least one top\-10 cell; under uniform allocation of 10 cells the expected count is≈5\.9\\approx 5\.9for 8\-group models and≈7\.6\\approx 7\.6for 16\-group models\.ModelLayers×\\timesKV\-groupsTop\-10 layer spanTop\-10 KV groupsDistinct KV groups in top\-10Qwen3\-8B36×836\\times 8layers 29–34\{0,1,2,3,4,7\}\\\{0,1,2,3,4,7\\\}6/86/8Qwen3\-14B40×840\\times 8layers 30–38\{1,2,3,5,6,7\}\\\{1,2,3,5,6,7\\\}6/86/8Qwen3\-32B64×864\\times 8layers 54–62\{1,2,3,5,6,7\}\\\{1,2,3,5,6,7\\\}6/86/8OLMo\-3\.1\-32B64×864\\times 8layers 27–57\{0,1,2,3,4,5,6\}\\\{0,1,2,3,4,5,6\\\}7/87/8Gemma\-3\-12B48×848\\times 8layers 0–47\{0,1,2,4,5,6\}\\\{0,1,2,4,5,6\\\}6/86/8Gemma\-3\-27B62×1662\\times 16layers 41–53\{0,3,4,5,6,7,10,13\}\\\{0,3,4,5,6,7,10,13\\\}8/168/16

The layer pattern is detector\- and metric\-conditional\.The late\-layer concentration in[Fig\.˜12](https://arxiv.org/html/2607.01002#A11.F12)is not evidence that retrieval circuitry lives only in late layers\. Two alternative explanations remain unaddressed by the figure\. First, the per\-position score \([Equation˜2](https://arxiv.org/html/2607.01002#S3.E2)\) is a direct\-path projection onto the answer\-token unembedding\. Late\-layer heads sit closer to the unembedding with fewer downstream non\-linearities to redirect their output, so their direct\-path projections are systematically larger as a property of the metric, irrespective of whether they implement retrieval\. The tuned\-lens variant in[Fig\.˜16](https://arxiv.org/html/2607.01002#A14.F16)addresses this concern at head granularity for Gemma\-3\-27B; we have not re\-run it at KV\-group granularity for the two Qwen3 models\. Second, the same KV\-group×\\timeslayer view computed from Wu \(NIAH\) attention scores places top cells across a wider layer range, so a “where retrieval lives” claim made from[Fig\.˜12](https://arxiv.org/html/2607.01002#A11.F12)alone would not survive a different detector\.[Fig\.˜12](https://arxiv.org/html/2607.01002#A11.F12)is therefore descriptive ofLOCOSscores at KV\-group granularity, not a localization of retrieval circuitry\. Whether these cells implement retrieval rather than some other late\-layer behavior is settled by the head\-level NoLiMa ablation in[§˜4\.2](https://arxiv.org/html/2607.01002#S4.SS2)and the specificity controls in[§˜4\.6](https://arxiv.org/html/2607.01002#S4.SS6), not by this figure\.

Implication for KV cache compression\.The unit of KV cache memory in GQA is the \(layer, KV\-group\) cell, so concentration at this granularity is what saves cache memory; head\-level concentration does not, because query heads in a group share keys and values\. A compression policy could keep full\-resolution K/V for the small set of top\-scoring cells \(10 of 288 in Qwen3\-8B and 10 of 320 in Qwen3\-14B\) and apply heavier quantisation, eviction, or window truncation to the rest\. The lower three quarters of layers contain no top\-10 cell in either model\. This gives a selection criterion at finer granularity than the per\-layer eviction policies ofZhanget al\.\[[2023](https://arxiv.org/html/2607.01002#bib.bib5)\],Liet al\.\[[2024](https://arxiv.org/html/2607.01002#bib.bib6)\], andXiaoet al\.\[[2024](https://arxiv.org/html/2607.01002#bib.bib7)\]; whether it produces a better quality–budget trade\-off is the experiment we do not run\.Xiaoet al\.\[[2025](https://arxiv.org/html/2607.01002#bib.bib30)\]make the closely related argument that a small set of retrieval heads should retain full KV cache while the rest use streaming attention; the KV\-group view sharpens this argument for GQA models, where retention saves cache only at the group level\.

Caveat for compression\.The late\-layer pattern is detector\- and metric\-conditional, so a policy derived fromLOCOSalone risks over\-compressing mid\-layer KV that other detectors flag as relevant\. A practical heuristic should pool signal from multiple retrieval\-head detectors rather than rely on a single score\. We do not run a KV\-compression experiment in this paper, and promoting the implication above to a claim would require, at minimum, top\-kkversus random KV\-group eviction on long\-context perplexity, NIAH, and NoLiMa, with a held\-out suite \(e\.g\.,parametric recall, arithmetic\) to check that the policy leaves non\-retrieval capabilities intact\.

## Appendix LSix\-Model Versions of Main\-Text Ablation Figures

The ablation figures in[§§˜4\.4](https://arxiv.org/html/2607.01002#S4.SS4),[4\.6](https://arxiv.org/html/2607.01002#S4.SS6)and[4\.7](https://arxiv.org/html/2607.01002#S4.SS7)show three representative models \(one per family\)\. This appendix reports the full six\-model versions\. The conclusions stated in the main text are based on all six configurations\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x13.png)Figure 13:Bottom\-kkablation does not degrade retrieval \(six\-model version of[Fig\.˜5](https://arxiv.org/html/2607.01002#S4.F5)\)\.Each panel shows NoLiMa ROUGE\-L as a function of ablation depthkkfor top\-kk\(blue\), bottom\-kk\(purple\), and random heads \(red\) across all six models\.![Refer to caption](https://arxiv.org/html/2607.01002v1/x14.png)Figure 14:Functional dissociation between retrieval and parametric capabilities across all six models \(six\-model version of[Fig\.˜7](https://arxiv.org/html/2607.01002#S4.F7)\)\.Each panel shows DS\(k\)\(k\)\(lines, right axis\) and parametric accuracy \(bars, left axis\) as a function of ablation depthkkfor four scoring methods\.LOCOS\(blue\) achieves the highest DS in every model configuration; the enlarged marker indicatesk∗k^\{\*\}\.![Refer to caption](https://arxiv.org/html/2607.01002v1/x15.png)Figure 15:Non\-literal vs\. literal retrieval damage across all six models \(six\-model version of[Fig\.˜8](https://arxiv.org/html/2607.01002#S4.F8)\)\.Each panel shows ROUGE\-L on NoLiMa \(solid blue\) and standard NIAH \(dashed blue\) under mean\-ablation of the same top\-kkLOCOSheads, with the NoLiMa and NIAH baselines marked by solid and dashed gray lines\. The NoLiMa curve declines more steeply than the NIAH curve in every configuration\.
## Appendix MDownstream Benchmark Example

*BABILong qa2 \(truncated\):*Context:Mary journeyed to the bathroom\. Sandra went to the garden\. \[…\] Daniel journeyed to the bedroom\. \[…\] Daniel took the football there\. \[…\] Daniel dropped the football\. \[…\] Daniel grabbed the football there\. \[…\] Daniel went to the kitchen\.Question:Where is the football?Answer:kitchen

The example above illustrates why literal token copying is insufficient for BABILong: the correct location \(“kitchen”\) is the*most recent*of many positions where the entity and object appear, requiring the model to trace a trajectory across interleaved narrative rather than match a unique token\.

## Appendix NDirect\-Path Robustness via Tuned Lens

This appendix derives the direct\-path identity that motivates the per\-position scoreϕt,j\(l,h\)\\phi^\{\(l,h\)\}\_\{t,j\}in[Equation˜2](https://arxiv.org/html/2607.01002#S3.E2), characterizes what the projection misses, and shows how a tuned\-lens variant addresses the bias that affects early\-layer heads\.

### N\.1The direct\-path identity

Let𝐱t\(l\)∈ℝd\\mathbf\{x\}^\{\(l\)\}\_\{t\}\\in\\mathbb\{R\}^\{d\}denote the residual stream at layerlland decode steptt\. A standard transformer block updates the residual stream additively,

𝐱t\(l\)=𝐱t\(l−1\)\+∑h=1H𝐚t\(l,h\)\+𝐦t\(l\),\\mathbf\{x\}^\{\(l\)\}\_\{t\}\\;=\\;\\mathbf\{x\}^\{\(l\-1\)\}\_\{t\}\\;\+\\;\\sum\_\{h=1\}^\{H\}\\mathbf\{a\}^\{\(l,h\)\}\_\{t\}\\;\+\\;\\mathbf\{m\}^\{\(l\)\}\_\{t\},\(5\)where𝐚t\(l,h\)=∑j𝐨t,j\(l,h\)\\mathbf\{a\}^\{\(l,h\)\}\_\{t\}=\\sum\_\{j\}\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}is the head\-\(l,h\)\(l,h\)output \([Equation˜1](https://arxiv.org/html/2607.01002#S2.E1)\) and𝐦t\(l\)\\mathbf\{m\}^\{\(l\)\}\_\{t\}is the MLP output of layerll\. We writeσl​\(⋅\)\\sigma\_\{l\}\(\\cdot\)for the layer\-norm rescaling that precedes the unembedding \(e\.g\.,the final RMSNorm\)\. The next\-token logits at stepttare

ℓt=WU​σL​\(𝐱t\(L\)\)∈ℝ\|𝒱\|\.\\boldsymbol\{\\ell\}\_\{t\}\\;=\\;W\_\{U\}\\,\\sigma\_\{L\}\\\!\\bigl\(\\mathbf\{x\}^\{\(L\)\}\_\{t\}\\bigr\)\\;\\in\\;\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}\.\(6\)
Unrolling[Equation˜5](https://arxiv.org/html/2607.01002#A14.E5)from layer11toLLand substituting into[Equation˜6](https://arxiv.org/html/2607.01002#A14.E6),

ℓt=WU​σL​\(𝐱t\(0\)\+∑l=1L\[∑h=1H∑j=1Nt𝐨t,j\(l,h\)\+𝐦t\(l\)\]\)\.\\boldsymbol\{\\ell\}\_\{t\}\\;=\\;W\_\{U\}\\,\\sigma\_\{L\}\\\!\\Bigl\(\\mathbf\{x\}^\{\(0\)\}\_\{t\}\+\\sum\_\{l=1\}^\{L\}\\Bigl\[\\sum\_\{h=1\}^\{H\}\\sum\_\{j=1\}^\{N\_\{t\}\}\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\+\\mathbf\{m\}^\{\(l\)\}\_\{t\}\\Bigr\]\\Bigr\)\.\(7\)[Equation˜7](https://arxiv.org/html/2607.01002#A14.E7)is exact\. BecauseσL\\sigma\_\{L\}is non\-linear in general, the contribution of any single𝐨t,j\(l,h\)\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}toℓt\\boldsymbol\{\\ell\}\_\{t\}is not additively separable\.

#### The direct\-path approximation\.

LinearizingσL\\sigma\_\{L\}at𝐱t\(L\)\\mathbf\{x\}^\{\(L\)\}\_\{t\}givesσL​\(𝐱t\(L\)\+𝐨t,j\(l,h\)\)−σL​\(𝐱t\(L\)\)≈JσL​\(𝐱t\(L\)\)​𝐨t,j\(l,h\)\\sigma\_\{L\}\(\\mathbf\{x\}^\{\(L\)\}\_\{t\}\+\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\)\-\\sigma\_\{L\}\(\\mathbf\{x\}^\{\(L\)\}\_\{t\}\)\\approx J\_\{\\sigma\_\{L\}\}\(\\mathbf\{x\}^\{\(L\)\}\_\{t\}\)\\,\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}, so the contribution of𝐨t,j\(l,h\)\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}to the logit of tokenyyis approximately𝐮y⊤​JσL​\(𝐱t\(L\)\)​𝐨t,j\(l,h\)\\mathbf\{u\}\_\{y\}^\{\\top\}J\_\{\\sigma\_\{L\}\}\(\\mathbf\{x\}^\{\(L\)\}\_\{t\}\)\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\. The*direct\-path score*replacesJσLJ\_\{\\sigma\_\{L\}\}with the identity, yielding

ϕt,j\(l,h\)=def𝐮yt⊤​𝐨t,j\(l,h\)\.\\phi^\{\(l,h\)\}\_\{t,j\}\\;\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\\;\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\\,\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\.\(8\)This is the score used throughout the main text \([Equation˜2](https://arxiv.org/html/2607.01002#S3.E2)\)\. The omitted Jacobian acts close to a per\-step rescaling for the RMSNorm models we evaluate \(Qwen3, Gemma\-3, OLMo\-3 are all RMSNorm\); we treat the substitution as an approximation, not an identity, and validate it empirically via the tuned\-lens variant in[§§˜N\.3](https://arxiv.org/html/2607.01002#A14.SS3)and[N\.4](https://arxiv.org/html/2607.01002#A14.SS4)\.

### N\.2What the direct path misses

The error in the direct\-path score \([Equation˜8](https://arxiv.org/html/2607.01002#A14.E8)\) relative to the exact contribution decomposes into three sources, in increasing order of severity for early\-layer heads:

1. 1\.LayerNorm rescaling\.The finalσL\\sigma\_\{L\}rescales each direction byγ/ρ​\(𝐱t\(L\)\)\\gamma/\\rho\(\\mathbf\{x\}^\{\(L\)\}\_\{t\}\)\. This is approximately constant across heads at a given step but varies across steps; it does not change rankings within a step but inflates magnitudes\.
2. 2\.Downstream attention re\-mixing\.A head\(l,h\)\(l,h\)at layerl<Ll<Lwrites into the residual stream that is read by all heads at layersl\+1,…,Ll\+1,\\ldots,L\. If a downstream head\(l′,h′\)\(l^\{\\prime\},h^\{\\prime\}\)withl′\>ll^\{\\prime\}\>lamplifies or cancels the answer\-aligned component of𝐨t,j\(l,h\)\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}, the direct\-path score under\-counts \(or, in the cancellation case, can sign\-flip relative to\) the true causal contribution\.
3. 3\.MLP composition\.MLP sublayers between layerllandLLapply non\-linear transformations \(gated activations, GELU/SwiGLU\)\. A retrieval head whose output only becomes answer\-aligned after a downstream MLP read–write pair is invisible to any linear probe of𝐱t\(l\)\\mathbf\{x\}^\{\(l\)\}\_\{t\}\.

Sources \(1\) and \(2\) are linear in𝐨t,j\(l,h\)\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}and are therefore correctable in principle by a learned linear lens\. Source \(3\) is non\-linear and is fundamentally outside the reach of any linear probe\.

### N\.3Tuned\-lens formalism

The tuned lens ofBelroseet al\.\[[2025](https://arxiv.org/html/2607.01002#bib.bib23)\]learns, for each layerll, an affine mapTl:ℝd→ℝdT\_\{l\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\}such thatWU​Tl​\(𝐱t\(l\)\)W\_\{U\}\\,T\_\{l\}\(\\mathbf\{x\}^\{\(l\)\}\_\{t\}\)approximates the model’s true next\-token logitsℓt\\boldsymbol\{\\ell\}\_\{t\}\. Concretely,Tl​\(𝐱\)=Al​𝐱\+blT\_\{l\}\(\\mathbf\{x\}\)=A\_\{l\}\\mathbf\{x\}\+b\_\{l\}, with\(Al,bl\)\(A\_\{l\},b\_\{l\}\)trained by minimizing the KL divergence betweensoftmax​\(WU​Tl​\(𝐱t\(l\)\)\)\\mathrm\{softmax\}\(W\_\{U\}T\_\{l\}\(\\mathbf\{x\}^\{\(l\)\}\_\{t\}\)\)and the true output distribution on a held\-out corpus\. The tuned lens absorbs sources \(1\) and \(2\) of[§˜N\.2](https://arxiv.org/html/2607.01002#A14.SS2)into the learned linear map; source \(3\) remains uncaptured\.

#### Lens\-corrected score\.

Substituting the tuned lens for the identity in[Equation˜8](https://arxiv.org/html/2607.01002#A14.E8)yields a layer\-aware score

ϕt,j\(l,h\),TL=def𝐮yt⊤​Al​𝐨t,j\(l,h\)\+constant in​j,\\phi^\{\(l,h\),\\mathrm\{TL\}\}\_\{t,j\}\\;\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\\;\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\\,A\_\{l\}\\,\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\\;\+\\;\\text\{constant in \}j,\(9\)where the biasblb\_\{l\}contributes ajj\-independent term that cancels under the spatial contrast \([Equation˜3](https://arxiv.org/html/2607.01002#S3.E3)\)\. Writing𝐩~\(l,h\)=\(WO\(l,h\)\)⊤​Al⊤​𝐮yt\\tilde\{\\mathbf\{p\}\}^\{\(l,h\)\}=\(W\_\{O\}^\{\(l,h\)\}\)^\{\\top\}A\_\{l\}^\{\\top\}\\mathbf\{u\}\_\{y\_\{t\}\}, the lens\-corrected per\-position score takes the same factored form as the direct\-path score:

ϕt,j\(l,h\),TL=αt,j\(l,h\)⋅\(𝐯t,j\(l,h\)\)⊤​𝐩~\(l,h\)\+const\.,\\phi^\{\(l,h\),\\mathrm\{TL\}\}\_\{t,j\}\\;=\\;\\alpha^\{\(l,h\)\}\_\{t,j\}\\cdot\\bigl\(\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}\\bigr\)^\{\\top\}\\tilde\{\\mathbf\{p\}\}^\{\(l,h\)\}\\;\+\\;\\text\{const\.\},so the same precomputation\-and\-batched\-inner\-product implementation applies; only the projection vector𝐩~\\tilde\{\\mathbf\{p\}\}changes\.

#### What the lens does not fix\.

[Equation˜9](https://arxiv.org/html/2607.01002#A14.E9)replaces the direct\-path Jacobian by a layer\-aware linear approximation, but it cannot capture any non\-linear composition \(source \(3\)\)\. A head whose write only becomes answer\-aligned after passing through a downstream MLP scores low under both the direct\-path and tuned\-lens variants\. Distinguishing a genuinely causally inert head from one whose contribution is hidden by downstream non\-linearity requires a non\-linear probe, of which the canonical example is causal activation patching\[Wanget al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib26), Menget al\.,[2022](https://arxiv.org/html/2607.01002#bib.bib46)\]\.

### N\.4Empirical comparison

![Refer to caption](https://arxiv.org/html/2607.01002v1/x16.png)Figure 16:Late\-layer concentration persists under tuned\-lens projection\.Heatmaps for Gemma\-3\-27B: direct\-pathLOCOS\(left\) vs\. tuned\-lens variant \(right\)\. Both methods concentrate high\-scoring heads in layers 35–60; the layer\-marginal distributions peak in the same band\. The tuned\-lens variant surfaces two additional heads at layer 11 \(heads 26 and 27\) that do not appear in the direct\-path top\-kkset, but does not produce a broader redistribution toward earlier layers\. The score magnitudes differ \(±75\\pm 75vs\.±0\.75\\pm 0\.75\) because the learned affine map amplifies contributions by accounting for downstream transformations\.[Fig\.˜16](https://arxiv.org/html/2607.01002#A14.F16)compares layer×\\timeshead score heatmaps for Gemma\-3\-27B under[Equation˜8](https://arxiv.org/html/2607.01002#A14.E8)and[Equation˜9](https://arxiv.org/html/2607.01002#A14.E9)\. The agreement on the late\-layer band \(l∈\[35,60\]l\\in\[35,60\]\) suggests that the late\-layer concentration reflects retrieval structure rather than a systematic underestimation of early\-layer heads by the direct\-path projection\. Two layer\-11 heads \(26 and 27\) surface only under the tuned lens; we treat these as candidate early\-layer retrieval heads that the direct\-path score under\-ranks but the linear correction recovers\.

### N\.5Gemma\-3\-27B inversion

![Refer to caption](https://arxiv.org/html/2607.01002v1/x17.png)Figure 17:Tuned\-lens correction only partly resolves the Gemma\-3\-27B inversion\.NoLiMa ROUGE\-L under mean\-ablation of top\-kkheads ranked by directLOCOS, the attention\-only spatial\-contrast control, and the tuned\-lens\-correctedLOCOSvariant on Gemma\-3\-27B\. The tuned\-lens variant closes much of the gap with attention\-only scoring at largekk, but directLOCOSselects the most damaging heads at smallkk\.Replacing the direct\-path projection with the tuned\-lens readout largely closes the gap with the attention\-only control at largekkon Gemma\-3\-27B \([Fig\.˜17](https://arxiv.org/html/2607.01002#A14.F17)\)\. At smallkk, however, directLOCOSselects more damaging heads than eitherα\\alpha\-spatial scoring or the tuned\-lens variant\. Thus, the Gemma\-3\-27B anomaly is not cleanly explained as a direct\-path readout artifact: correcting the readout helps only at the deepest sweep and worsens the top\-ranked heads\. This suggests that, for this model, spatial attention contrast captures a causal signal that the evaluated linear write\-projections do not rank consistently \([§˜N\.2](https://arxiv.org/html/2607.01002#A14.SS2)\)\. For Gemma\-3\-27B, attention placement under spatial contrast carries more causal signal than any linear write\-projection we evaluate\.

### N\.6Beyond linear probes

A linear probe \(direct path or tuned lens\) cannot detect a head whose contribution toyty\_\{t\}materializes only through MLP composition\. Per\-head causal activation patching is the canonical gradient\-free, non\-linear\-aware test: for each head\(l,h\)\(l,h\), run a clean forward pass on the needle prompt and a corrupt forward pass with the needle removed or scrambled, then patch only the\(l,h\)\(l,h\)activation from the clean run into the corrupt run and measure the change in logit difference at the answer position,

CA​\(l,h\)=\[ℓy∗patch−ℓycfpatch\]−\[ℓy∗corr−ℓycfcorr\],\\mathrm\{CA\}\(l,h\)\\;=\\;\\bigl\[\\ell^\{\\mathrm\{patch\}\}\_\{y^\{\*\}\}\-\\ell^\{\\mathrm\{patch\}\}\_\{y^\{\\mathrm\{cf\}\}\}\\bigr\]\-\\bigl\[\\ell^\{\\mathrm\{corr\}\}\_\{y^\{\*\}\}\-\\ell^\{\\mathrm\{corr\}\}\_\{y^\{\\mathrm\{cf\}\}\}\\bigr\],\(10\)wherey∗y^\{\*\}is the gold first answer token andycf=arg⁡maxy≠y∗⁡ℓycorry^\{\\mathrm\{cf\}\}=\\arg\\max\_\{y\\neq y^\{\*\}\}\\ell^\{\\mathrm\{corr\}\}\_\{y\}is the strongest non\-gold competitor under the corrupt baseline\. We refer to this score as*causal attribution*\(CA\)\.

![Refer to caption](https://arxiv.org/html/2607.01002v1/x18.png)Figure 18:Causal attribution vs\.LOCOStop\-10 heads on Qwen3\-8B and Gemma\-3\-12B\.Per\-\(layer, head\) score heatmaps with red boxes marking each method’s top\-10 cells; layer\-marginal kernel densities on the right of each panel\. Both methods concentrate top\-10 heads in the upper layers in both models, but the top\-10 sets overlap only marginally \(2/10 for Qwen3\-8B, 3/10 for Gemma\-3\-12B\)\. On Gemma\-3\-12B,LOCOSsurfaces several layer\-0 heads that causal attribution does not flag\.Empirical comparison on two models\.[Fig\.˜18](https://arxiv.org/html/2607.01002#A14.F18)reports causal attribution andLOCOSside by side for Qwen3\-8B and Gemma\-3\-12B\. The shared finding is that the late\-layer concentration is not a direct\-path artifact: causal attribution—which is non\-linear\-aware and gradient\-free—also places its top\-10 heads predominantly in the upper layers in both models \(layers 22–35 for Qwen3\-8B, concentrated in 33–35; layers 35–45 for Gemma\-3\-12B, concentrated near 41\)\.LOCOS’s top\-10 sets sit in similar bands \(29–34 for Qwen3\-8B; layers spanning 0, 35, 41, 47 for Gemma\-3\-12B\)\. Top\-10 overlap is small \(2/10 and 3/10\), which is consistent with the two scores capturing different aspects of retrieval\-related computation rather than one being a strict refinement of the other\.

Where the methods diverge, and why\.The notable disagreement on Gemma\-3\-12B is thatLOCOSsurfaces heads at layer 0 that causal attribution does not\. Two explanations are consistent with the figure and we cannot distinguish them here\. First, layer\-0 heads may be a direct\-path artifact: their write enters the residual stream before any downstream MLP can either amplify or cancel the answer\-aligned component, so the linear projection onto𝐮yt\\mathbf\{u\}\_\{y\_\{t\}\}records a magnitude that the model itself does not realize at the output \(sources \(2\)–\(3\) of[§˜N\.2](https://arxiv.org/html/2607.01002#A14.SS2)\)\. Second, layer\-0 heads may participate in distributed circuits where ablating any single head leaves performance intact \(because parallel paths compensate\), so causal attribution—a single\-head intervention—under\-counts them whileLOCOScorrectly registers each head’s contribution to the linear readout\. The bottom\-kkcontrol \([§˜4\.4](https://arxiv.org/html/2607.01002#S4.SS4)\) and the top\-kkablation curves \([Fig\.˜7](https://arxiv.org/html/2607.01002#S4.F7)\) bound the issue at the level of method validation:LOCOS\-selected top heads, including any artifactual ones, are causally critical*collectively*under group ablation even if individual heads are not under single\-head patching\.

Scope\.The comparison is two models from two families on one benchmark with one alternative detector\. Causal attribution is also expensive: it requires one additional forward pass per head per trial \(a clean–corrupt patching pair scaled by the number of heads,HQ⋅LH\_\{Q\}\\cdot Lper trial\), so the cost grows linearly with the head count and is roughly two orders of magnitude above a singleLOCOSpass for the models we evaluate\. We therefore do not run it on the other four models or on NIAH/parametric controls; running it across the full evaluation matrix is left to future work\. The takeaway we draw is narrow: the late\-layer concentration ofLOCOStop heads is corroborated by a non\-linear\-aware detector on the two models for which we have both scores, and the residual disagreement \(notably the Gemma\-3\-12B layer\-0 heads\) is a candidate site for future single\-head causal validation rather than a refutation of the present paper’s group\-ablation results\.

## Appendix ORelationship to Attention\-Based Scoring

This appendix gives a self\-contained derivation of howLOCOSrelates to the attention\-pattern observable used by prior detection methods\[Wuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib17), Fuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib19), Linet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib18)\]\.[§˜O\.1](https://arxiv.org/html/2607.01002#A15.SS1)fixes notation;[§˜O\.2](https://arxiv.org/html/2607.01002#A15.SS2)decomposes the per\-position OV output into answer\-aligned and answer\-orthogonal components;[§˜O\.3](https://arxiv.org/html/2607.01002#A15.SS3)states and proves the reduction ofLOCOSto attention\-based scoring under a position\-independent OV output;[§§˜O\.4](https://arxiv.org/html/2607.01002#A15.SS4)and[O\.5](https://arxiv.org/html/2607.01002#A15.SS5)cast the Wu and HeadKV/CompressKV scores as special cases under additional assumptions;[§˜O\.6](https://arxiv.org/html/2607.01002#A15.SS6)contrasts spatial and temporal aggregation;[§˜O\.7](https://arxiv.org/html/2607.01002#A15.SS7)gives the gradient interpretation; and[§˜O\.8](https://arxiv.org/html/2607.01002#A15.SS8)consolidates the results\.

### O\.1Setup

We work with the standard pre\-softmax decomposition of a transformer attention head\[Elhageet al\.,[2021](https://arxiv.org/html/2607.01002#bib.bib24)\]\. At decode steptt, head\(l,h\)\(l,h\)produces query𝐪t\(l,h\)\\mathbf\{q\}^\{\(l,h\)\}\_\{t\}and, for each source positionj∈\{1,…,Nt\}j\\in\\\{1,\\ldots,N\_\{t\}\\\}, key𝐤t,j\(l,h\)\\mathbf\{k\}^\{\(l,h\)\}\_\{t,j\}and value𝐯t,j\(l,h\)∈ℝdh\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}\\in\\mathbb\{R\}^\{d\_\{h\}\}\. The QK circuit produces

αt,j\(l,h\)=exp⁡\(𝐪t\(l,h\)⊤​𝐤t,j\(l,h\)/dh\)∑j′=1Ntexp⁡\(𝐪t\(l,h\)⊤​𝐤t,j′\(l,h\)/dh\),\\alpha^\{\(l,h\)\}\_\{t,j\}\\;=\\;\\frac\{\\exp\\bigl\(\\mathbf\{q\}^\{\(l,h\)\{\\top\}\}\_\{t\}\\mathbf\{k\}^\{\(l,h\)\}\_\{t,j\}/\\sqrt\{d\_\{h\}\}\\bigr\)\}\{\\sum\_\{j^\{\\prime\}=1\}^\{N\_\{t\}\}\\exp\\bigl\(\\mathbf\{q\}^\{\(l,h\)\{\\top\}\}\_\{t\}\\mathbf\{k\}^\{\(l,h\)\}\_\{t,j^\{\\prime\}\}/\\sqrt\{d\_\{h\}\}\\bigr\)\},\(11\)and the OV circuit writes𝐨t,j\(l,h\)=αt,j\(l,h\)⋅WO\(l,h\)​𝐯t,j\(l,h\)\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}=\\alpha^\{\(l,h\)\}\_\{t,j\}\\cdot W\_\{O\}^\{\(l,h\)\}\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}to the residual stream \([Equation˜1](https://arxiv.org/html/2607.01002#S2.E1)\)\. The full head output is𝐚t\(l,h\)=∑j=1Nt𝐨t,j\(l,h\)\\mathbf\{a\}^\{\(l,h\)\}\_\{t\}=\\sum\_\{j=1\}^\{N\_\{t\}\}\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\.

For brevity, we drop the\(l,h\)\(l,h\)superscripts in this appendix when no ambiguity arises and writeαj\\alpha\_\{j\},𝐯j\\mathbf\{v\}\_\{j\},𝐰j=defWO​𝐯j\\mathbf\{w\}\_\{j\}\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}W\_\{O\}\\mathbf\{v\}\_\{j\}, and𝐮=def𝐮yt\\mathbf\{u\}\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\\mathbf\{u\}\_\{y\_\{t\}\}\. With this notation the per\-position score[Equation˜2](https://arxiv.org/html/2607.01002#S3.E2)isϕj=αj⋅𝐮⊤​𝐰j\\phi\_\{j\}=\\alpha\_\{j\}\\cdot\\mathbf\{u\}^\{\\top\}\\mathbf\{w\}\_\{j\}\.

### O\.2Parallel–orthogonal decomposition of the OV output

Decompose each unweighted OV output𝐰j∈ℝd\\mathbf\{w\}\_\{j\}\\in\\mathbb\{R\}^\{d\}along𝐮\\mathbf\{u\}and orthogonal to it:

𝐰j=cj⋅𝐮\+𝐰j⟂,cj=𝐮⊤​𝐰j‖𝐮‖2,𝐮⊤​𝐰j⟂=0\.\\mathbf\{w\}\_\{j\}\\;=\\;c\_\{j\}\\cdot\\mathbf\{u\}\\;\+\\;\\mathbf\{w\}^\{\\perp\}\_\{j\},\\qquad c\_\{j\}\\;=\\;\\frac\{\\mathbf\{u\}^\{\\top\}\\mathbf\{w\}\_\{j\}\}\{\\\|\\mathbf\{u\}\\\|^\{2\}\},\\qquad\\mathbf\{u\}^\{\\top\}\\mathbf\{w\}^\{\\perp\}\_\{j\}=0\.\(12\)The scalarcjc\_\{j\}is the answer\-aligned write magnitude at source positionjj; the residual𝐰j⟂\\mathbf\{w\}^\{\\perp\}\_\{j\}writes into directions orthogonal to the answer token\. Substituting[Equation˜12](https://arxiv.org/html/2607.01002#A15.E12)into the per\-position score gives the identity

ϕj=αj⋅‖𝐮‖2⋅cj,\\phi\_\{j\}\\;=\\;\\alpha\_\{j\}\\cdot\\\|\\mathbf\{u\}\\\|^\{2\}\\cdot c\_\{j\},\(13\)which factorises cleanly into*where*the head reads \(αj\\alpha\_\{j\}\) and*what answer\-aligned content*it writes from that position \(cjc\_\{j\}\)\. The factor‖𝐮‖2\\\|\\mathbf\{u\}\\\|^\{2\}is constant across heads and positions and cancels under any cross\-head ranking, so we may treatϕj∝αj​cj\\phi\_\{j\}\\propto\\alpha\_\{j\}c\_\{j\}for the purposes of head selection\.

###### Lemma 1\(Sufficient statistic\)\.

For any score that aggregates per\-position OV writes against the answer direction𝐮\\mathbf\{u\}, the pair\(αj,cj\)\(\\alpha\_\{j\},c\_\{j\}\)is sufficient: any answer\-orthogonal component𝐰j⟂\\mathbf\{w\}^\{\\perp\}\_\{j\}contributes zero\.

###### Proof\.

Immediate from𝐮⊤​𝐰j⟂=0\\mathbf\{u\}^\{\\top\}\\mathbf\{w\}^\{\\perp\}\_\{j\}=0and the linearity of𝐮⊤​𝐨j=αj​𝐮⊤​𝐰j\\mathbf\{u\}^\{\\top\}\\mathbf\{o\}\_\{j\}=\\alpha\_\{j\}\\mathbf\{u\}^\{\\top\}\\mathbf\{w\}\_\{j\}\. ∎

[Lemma˜1](https://arxiv.org/html/2607.01002#Thmlemma1)shows that any modification ofϕ\\phithat is linear in the residual stream and projects onto𝐮\\mathbf\{u\}—for instance the tuned\-lens variant of[Appx\.˜N](https://arxiv.org/html/2607.01002#A14)—inherits the same factorisation, withcjc\_\{j\}replaced by a lens\-corrected scalar\.

### O\.3Reduction to attention\-based scoring

###### Proposition 1\(Reduction under position\-independent OV write\)\.

Suppose head\(l,h\)\(l,h\)writes a position\-independent answer\-aligned magnitude at steptt, i\.e\., there exists a scalarctc\_\{t\}such that

cj=ctfor every​j∈\{1,…,Nt\}\.c\_\{j\}\\;=\\;c\_\{t\}\\quad\\text\{for every \}j\\in\\\{1,\\ldots,N\_\{t\}\\\}\.\(14\)Then the spatial\-contrast score \([Equation˜3](https://arxiv.org/html/2607.01002#S3.E3)\) reduces to

Φt\+−Φt−=‖𝐮‖2⋅ct⋅\(Mt\+−eτ−sτNt−\(eτ−sτ\)⋅Mt−\),\\Phi^\{\+\}\_\{t\}\-\\Phi^\{\-\}\_\{t\}\\;=\\;\\\|\\mathbf\{u\}\\\|^\{2\}\\cdot c\_\{t\}\\cdot\\Bigl\(M^\{\+\}\_\{t\}\\;\-\\;\\tfrac\{e\_\{\\tau\}\-s\_\{\\tau\}\}\{N\_\{t\}\-\(e\_\{\\tau\}\-s\_\{\\tau\}\)\}\\cdot M^\{\-\}\_\{t\}\\Bigr\),\(15\)whereMt\+=∑j∈\[sτ,eτ\)αjM^\{\+\}\_\{t\}=\\sum\_\{j\\in\[s\_\{\\tau\},e\_\{\\tau\}\)\}\\alpha\_\{j\}andMt−=∑j∉\[sτ,eτ\)αjM^\{\-\}\_\{t\}=\\sum\_\{j\\notin\[s\_\{\\tau\},e\_\{\\tau\}\)\}\\alpha\_\{j\}are the needle and off\-needle attention masses respectively\. Up to the per\-step scalar‖𝐮‖2​ct\\\|\\mathbf\{u\}\\\|^\{2\}c\_\{t\},LOCOScoincides with the length\-normalized attention\-mass contrast between needle and off\-needle positions\.

###### Proof\.

Under[Equation˜14](https://arxiv.org/html/2607.01002#A15.E14),[Equation˜13](https://arxiv.org/html/2607.01002#A15.E13)readsϕj=‖𝐮‖2​ct⋅αj\\phi\_\{j\}=\\\|\\mathbf\{u\}\\\|^\{2\}c\_\{t\}\\cdot\\alpha\_\{j\}with the scalar‖𝐮‖2​ct\\\|\\mathbf\{u\}\\\|^\{2\}c\_\{t\}independent ofjj\. Substituting into the definitions ofΦt±\\Phi^\{\\pm\}\_\{t\}from[Equation˜3](https://arxiv.org/html/2607.01002#S3.E3),

Φt\+\\displaystyle\\Phi^\{\+\}\_\{t\}=‖𝐮‖2​ct​∑j∈\[sτ,eτ\)αj=‖𝐮‖2​ct​Mt\+,\\displaystyle=\\\|\\mathbf\{u\}\\\|^\{2\}c\_\{t\}\\sum\_\{j\\in\[s\_\{\\tau\},e\_\{\\tau\}\)\}\\alpha\_\{j\}\\;=\\;\\\|\\mathbf\{u\}\\\|^\{2\}c\_\{t\}\\,M^\{\+\}\_\{t\},Φt−\\displaystyle\\Phi^\{\-\}\_\{t\}=eτ−sτNt−\(eτ−sτ\)​‖𝐮‖2​ct​∑j∉\[sτ,eτ\)αj=eτ−sτNt−\(eτ−sτ\)​‖𝐮‖2​ct​Mt−\.\\displaystyle=\\tfrac\{e\_\{\\tau\}\-s\_\{\\tau\}\}\{N\_\{t\}\-\(e\_\{\\tau\}\-s\_\{\\tau\}\)\}\\\|\\mathbf\{u\}\\\|^\{2\}c\_\{t\}\\sum\_\{j\\notin\[s\_\{\\tau\},e\_\{\\tau\}\)\}\\alpha\_\{j\}\\;=\\;\\tfrac\{e\_\{\\tau\}\-s\_\{\\tau\}\}\{N\_\{t\}\-\(e\_\{\\tau\}\-s\_\{\\tau\}\)\}\\\|\\mathbf\{u\}\\\|^\{2\}c\_\{t\}\\,M^\{\-\}\_\{t\}\.Subtracting yields[Equation˜15](https://arxiv.org/html/2607.01002#A15.E15)\. ∎

###### Corollary 1\(Sign of the reduction\)\.

Under[Proposition˜1](https://arxiv.org/html/2607.01002#Thmproposition1),sign​\(Φt\+−Φt−\)=sign​\(ct\)⋅sign​\(Mt\+−eτ−sτNt−\(eτ−sτ\)​Mt−\)\\mathrm\{sign\}\(\\Phi^\{\+\}\_\{t\}\-\\Phi^\{\-\}\_\{t\}\)=\\mathrm\{sign\}\(c\_\{t\}\)\\cdot\\mathrm\{sign\}\\bigl\(M^\{\+\}\_\{t\}\-\\tfrac\{e\_\{\\tau\}\-s\_\{\\tau\}\}\{N\_\{t\}\-\(e\_\{\\tau\}\-s\_\{\\tau\}\)\}M^\{\-\}\_\{t\}\\bigr\)\. A head that reads from the needle \(Mt\+M^\{\+\}\_\{t\}exceeds the length\-normalized off\-needle mass\) and writes a positive answer\-aligned direction \(ct\>0c\_\{t\}\>0\) is correctly assigned a positiveLOCOSscore; a head that suppresses the answer logit \(ct<0c\_\{t\}<0\) receives a negative score\.

#### Where the reduction holds\.

[Equation˜14](https://arxiv.org/html/2607.01002#A15.E14)requires the head to write the same answer\-aligned magnitudectc\_\{t\}at every source position\. This is the defining property of a literal\-copy head: an induction head\[A\]​\[B\]​…​\[A\]↦\[B\]\[A\]\[B\]\\ldots\[A\]\\mapsto\[B\]writes the “advance the previous token by one step” direction independently of which\[A\]\[A\]position it attended to\. In our notation,𝐰j\\mathbf\{w\}\_\{j\}depends onjjonly through𝐯j\\mathbf\{v\}\_\{j\}, and for a literal\-copy head𝐮⊤​WO​𝐯j\\mathbf\{u\}^\{\\top\}W\_\{O\}\\mathbf\{v\}\_\{j\}takes the same value whenever𝐯j\\mathbf\{v\}\_\{j\}encodes the answer token\.

#### Where the reduction fails\.

For a non\-literal retrieval head,cjc\_\{j\}depends on whetherjjlies inside the needle \(where the value vector encodes the semantic concept that the head must transform into the answer direction\) or outside \(where the value encodes unrelated context\)\.[Equation˜14](https://arxiv.org/html/2607.01002#A15.E14)breaks, and[Proposition˜1](https://arxiv.org/html/2607.01002#Thmproposition1)no longer holds:LOCOSsees this position dependence through the per\-positioncjc\_\{j\}in[Equation˜13](https://arxiv.org/html/2607.01002#A15.E13), while attention\-based scoring observes onlyαj\\alpha\_\{j\}and is invariant tocjc\_\{j\}\.

### O\.4Wu’s token\-matching score as a special case

The score ofWuet al\.\[[2025](https://arxiv.org/html/2607.01002#bib.bib17)\]assigns credit at decode stepttto head\(l,h\)\(l,h\)on a literal\-NIAH trial when \(i\) the head’s argmax attention position falls within the needle span and \(ii\) the token at that position matches the generated token\. Formally, withj∗=arg⁡maxj⁡αt,j\(l,h\)j^\{\*\}=\\arg\\max\_\{j\}\\alpha^\{\(l,h\)\}\_\{t,j\}andxjx\_\{j\}the input token at positionjj,

Wut\(l,h\)=1​\[j∗∈\[sτ,eτ\)\]⋅𝟏​\[xj∗=yt\]\.\\mathrm\{Wu\}^\{\(l,h\)\}\_\{t\}\\;=\\;\\mathbf\{1\}\\\!\\bigl\[\\,j^\{\*\}\\in\[s\_\{\\tau\},e\_\{\\tau\}\)\\,\\bigr\]\\cdot\\mathbf\{1\}\\\!\\bigl\[\\,x\_\{j^\{\*\}\}=y\_\{t\}\\,\\bigr\]\.\(16\)
###### Proposition 2\(Wu’s score as a hard\-thresholded special case\)\.

Suppose three conditions hold for head\(l,h\)\(l,h\)at steptton a literal\-NIAH trial:

- \(W1\)The head is a literal\-copy head:WO​𝐯j≈𝐮xjW\_\{O\}\\mathbf\{v\}\_\{j\}\\approx\\mathbf\{u\}\_\{x\_\{j\}\}, the unembedding row of the attended token, so𝐮yt⊤​WO​𝐯j∝𝐮yt⊤​𝐮xj\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}W\_\{O\}\\mathbf\{v\}\_\{j\}\\propto\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\\mathbf\{u\}\_\{x\_\{j\}\}\.
- \(W2\)The unembedding rows are approximately orthogonal across distinct tokens:𝐮yt⊤​𝐮xj≈‖𝐮yt‖2⋅𝟏​\[xj=yt\]\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\\mathbf\{u\}\_\{x\_\{j\}\}\\approx\\\|\\mathbf\{u\}\_\{y\_\{t\}\}\\\|^\{2\}\\cdot\\mathbf\{1\}\[x\_\{j\}=y\_\{t\}\]\.
- \(W3\)Attention is concentrated:αj∗≫αj\\alpha\_\{j^\{\*\}\}\\gg\\alpha\_\{j\}forj≠j∗j\\neq j^\{\*\}, so the spatial\-contrast score is dominated by the argmax position\.

ThenWut\(l,h\)=1⇔Φt\+−Φt−\>0\\mathrm\{Wu\}^\{\(l,h\)\}\_\{t\}=1\\iff\\Phi^\{\+\}\_\{t\}\-\\Phi^\{\-\}\_\{t\}\>0to leading order, and the rankings induced by Wu’s score andLOCOScoincide on the trial\.

###### Proof\.

Under \(W1\) and \(W2\),cj≈𝟏​\[xj=yt\]c\_\{j\}\\approx\\mathbf\{1\}\[x\_\{j\}=y\_\{t\}\]\. The needle contribution becomesΦt\+≈‖𝐮‖2​∑j∈\[sτ,eτ\),xj=ytαj\\Phi^\{\+\}\_\{t\}\\approx\\\|\\mathbf\{u\}\\\|^\{2\}\\sum\_\{j\\in\[s\_\{\\tau\},e\_\{\\tau\}\),\\,x\_\{j\}=y\_\{t\}\}\\alpha\_\{j\}and the off\-needle contributionΦt−≈eτ−sτNt−\(eτ−sτ\)​‖𝐮‖2​∑j∉\[sτ,eτ\),xj=ytαj\\Phi^\{\-\}\_\{t\}\\approx\\tfrac\{e\_\{\\tau\}\-s\_\{\\tau\}\}\{N\_\{t\}\-\(e\_\{\\tau\}\-s\_\{\\tau\}\)\}\\\|\\mathbf\{u\}\\\|^\{2\}\\sum\_\{j\\notin\[s\_\{\\tau\},e\_\{\\tau\}\),\\,x\_\{j\}=y\_\{t\}\}\\alpha\_\{j\}\. On a literal\-NIAH trial the answer tokenyty\_\{t\}appears predominantly at one position inside the needle, soΦt−\\Phi^\{\-\}\_\{t\}is negligible\. Under \(W3\),Φt\+≈‖𝐮‖2​αj∗⋅𝟏​\[j∗∈\[sτ,eτ\)\]⋅𝟏​\[xj∗=yt\]=‖𝐮‖2​αj∗⋅Wut\(l,h\)\\Phi^\{\+\}\_\{t\}\\approx\\\|\\mathbf\{u\}\\\|^\{2\}\\alpha\_\{j^\{\*\}\}\\cdot\\mathbf\{1\}\[j^\{\*\}\\in\[s\_\{\\tau\},e\_\{\\tau\}\)\]\\cdot\\mathbf\{1\}\[x\_\{j^\{\*\}\}=y\_\{t\}\]=\\\|\\mathbf\{u\}\\\|^\{2\}\\alpha\_\{j^\{\*\}\}\\cdot\\mathrm\{Wu\}^\{\(l,h\)\}\_\{t\}, which is positive iffWut\(l,h\)=1\\mathrm\{Wu\}^\{\(l,h\)\}\_\{t\}=1\. ∎

[Proposition˜2](https://arxiv.org/html/2607.01002#Thmproposition2)clarifies why Wu’s score is a strong baseline on literal NIAH: under \(W1\)–\(W3\), it is a hard\-thresholded version ofLOCOS\. It also clarifies the failure mode on NoLiMa: assumption \(W1\) collapses, since a non\-literal retrieval head transforms the value vector through a learnedWO​𝐯jW\_\{O\}\\mathbf\{v\}\_\{j\}that does not factor through𝐮xj\\mathbf\{u\}\_\{x\_\{j\}\}\. The Wu indicator drops to near zero \(the attended token is not the answer token\) whilecjc\_\{j\}remains large, producing the0\.97→0\.030\.97\\to 0\.03gap reported in[Appx\.˜B](https://arxiv.org/html/2607.01002#A2)\.

### O\.5HeadKV/CompressKV as weighted attention accumulation

Fuet al\.\[[2025](https://arxiv.org/html/2607.01002#bib.bib19)\]\(HeadKV\) andLinet al\.\[[2025](https://arxiv.org/html/2607.01002#bib.bib18)\]\(CompressKV\) score heads by accumulating attention mass on retrieval\-relevant positions, optionally weighted\. A schematic form, abstracting over implementation details, is

HeadKV\(l,h\)∝∑t∈𝒜τ∑j∈ℛtwj⋅αt,j\(l,h\),\\mathrm\{HeadKV\}^\{\(l,h\)\}\\;\\propto\\;\\sum\_\{t\\in\\mathcal\{A\}^\{\\tau\}\}\\sum\_\{j\\in\\mathcal\{R\}\_\{t\}\}w\_\{j\}\\cdot\\alpha^\{\(l,h\)\}\_\{t,j\},\(17\)whereℛt\\mathcal\{R\}\_\{t\}is the set of retrieval\-relevant source positions at stepttandwj≥0w\_\{j\}\\geq 0are non\-negative weights\.

###### Corollary 2\(HeadKV/CompressKV are non\-negative reweightings of attention mass\)\.

[Equation˜17](https://arxiv.org/html/2607.01002#A15.E17)is non\-decreasing in eachαt,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}forj∈ℛtj\\in\\mathcal\{R\}\_\{t\}and depends on the OV circuit only through the choice ofℛt\\mathcal\{R\}\_\{t\}andwjw\_\{j\}\. In particular, two heads with identical attention patternαt,⋅\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,\\cdot\}but different output projectionsWO\(l,h\)W\_\{O\}^\{\(l,h\)\}receive the same HeadKV/CompressKV score;LOCOSdistinguishes them viacjc\_\{j\}\.

###### Proof\.

The first claim is immediate\. The second follows because[Equation˜17](https://arxiv.org/html/2607.01002#A15.E17)is a function ofα\\alphaalone and theℛt,wj\\mathcal\{R\}\_\{t\},w\_\{j\}choices are tied to the attention pattern, while[Equation˜13](https://arxiv.org/html/2607.01002#A15.E13)is a function of bothα\\alphaandcjc\_\{j\}\. ∎

[Corollary˜2](https://arxiv.org/html/2607.01002#Thmcorollary2)formalises the central conceptual claim of the paper: any score that observes only the attention pattern, however weighted, is blind to differences inWO​𝐯jW\_\{O\}\\mathbf\{v\}\_\{j\}and therefore cannot distinguish the two heads sketched in[Fig\.˜2](https://arxiv.org/html/2607.01002#S3.F2)\(b\)\.

### O\.6Spatial vs\. temporal contrast

Attention\-based methods commonly compute a*temporal*contrast: attention mass accumulated at answer\-token decode steps, optionally minus mass at non\-answer steps\[Wuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib17)\]\.LOCOSuses a*spatial*contrast \([Equation˜3](https://arxiv.org/html/2607.01002#S3.E3)\)\. The two are not equivalent\.

###### Proposition 3\(Spatial contrast dominates under decode\-step\-stationary attention\)\.

Suppose head\(l,h\)\(l,h\)exhibits a constant attention pattern across decode steps,αt,j\(l,h\)=αj\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}=\\alpha^\{\(l,h\)\}\_\{j\}for allt∈𝒜τ∪𝒜τ,negt\\in\\mathcal\{A\}^\{\\tau\}\\cup\\mathcal\{A\}^\{\\tau,\\mathrm\{neg\}\}, where𝒜τ,neg\\mathcal\{A\}^\{\\tau,\\mathrm\{neg\}\}is a set of non\-answer steps and\|𝒜τ\|=\|𝒜τ,neg\|\|\\mathcal\{A\}^\{\\tau\}\|=\|\\mathcal\{A\}^\{\\tau,\\mathrm\{neg\}\}\|\. Then the temporal contrast at any single source position is zero, while the spatial contrast is generally non\-zero whenever the head allocates more mass to the needle than the length\-normalized off\-needle expectation\.

###### Proof\.

The temporal contrast at positionjjis∑t∈𝒜ταt,j\(l,h\)−∑t∈𝒜τ,negαt,j\(l,h\)=\(\|𝒜τ\|−\|𝒜τ,neg\|\)⋅αj\(l,h\)=0\\sum\_\{t\\in\\mathcal\{A\}^\{\\tau\}\}\\alpha^\{\(l,h\)\}\_\{t,j\}\-\\sum\_\{t\\in\\mathcal\{A\}^\{\\tau,\\mathrm\{neg\}\}\}\\alpha^\{\(l,h\)\}\_\{t,j\}=\(\|\\mathcal\{A\}^\{\\tau\}\|\-\|\\mathcal\{A\}^\{\\tau,\\mathrm\{neg\}\}\|\)\\cdot\\alpha^\{\(l,h\)\}\_\{j\}=0by the cardinality assumption\. The spatial contrast at any single stepttisMt\+−eτ−sτNt−\(eτ−sτ\)​Mt−M^\{\+\}\_\{t\}\-\\tfrac\{e\_\{\\tau\}\-s\_\{\\tau\}\}\{N\_\{t\}\-\(e\_\{\\tau\}\-s\_\{\\tau\}\)\}M^\{\-\}\_\{t\}, which is non\-zero iff the per\-position attention mass exceeds the off\-needle average\. ∎

[Proposition˜3](https://arxiv.org/html/2607.01002#Thmproposition3)captures the second motivation for the spatial contrast in[§˜3](https://arxiv.org/html/2607.01002#S3): a head that persistently reads the needle—e\.g\., as “context” rather than “answer”—receives no credit under temporal contrast but is correctly identified by spatial contrast when its OV write is answer\-aligned at needle positions\.

### O\.7Gradient interpretation

We give the standard derivation ofϕ\\phias a leading\-order gradient\. Letℒt=−log⁡p​\(yt∣context\)\\mathcal\{L\}\_\{t\}=\-\\log p\(y\_\{t\}\\mid\\mathrm\{context\}\)denote the cross\-entropy at steptt, withp\(⋅∣context\)=softmax\(ℓt\)p\(\\cdot\\mid\\mathrm\{context\}\)=\\mathrm\{softmax\}\(\\boldsymbol\{\\ell\}\_\{t\}\)\. Under the direct\-path approximationℓt≈WU​𝐱t\(L\)\\boldsymbol\{\\ell\}\_\{t\}\\approx W\_\{U\}\\mathbf\{x\}^\{\(L\)\}\_\{t\}\([§˜N\.1](https://arxiv.org/html/2607.01002#A14.SS1)\), the chain rule gives

∇𝐨t,j\(l,h\)ℒt=WU⊤​∇ℓtℒt=WU⊤​\(𝐩t−𝐞yt\)=−\(1−p​\(yt\)\)​𝐮yt\+∑v≠ytp​\(v\)​𝐮v,\\nabla\_\{\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}\}\\mathcal\{L\}\_\{t\}\\;=\\;W\_\{U\}^\{\\top\}\\nabla\_\{\\boldsymbol\{\\ell\}\_\{t\}\}\\mathcal\{L\}\_\{t\}\\;=\\;W\_\{U\}^\{\\top\}\\bigl\(\\mathbf\{p\}\_\{t\}\-\\mathbf\{e\}\_\{y\_\{t\}\}\\bigr\)\\;=\\;\-\(1\-p\(y\_\{t\}\)\)\\,\\mathbf\{u\}\_\{y\_\{t\}\}\\;\+\\;\\\!\\sum\_\{v\\neq y\_\{t\}\}\\\!p\(v\)\\,\\mathbf\{u\}\_\{v\},\(18\)where𝐩t\\mathbf\{p\}\_\{t\}is the predicted distribution and𝐞yt\\mathbf\{e\}\_\{y\_\{t\}\}is the one\-hot target\. The dominant term is−\(1−p​\(yt\)\)​𝐮yt\-\(1\-p\(y\_\{t\}\)\)\\mathbf\{u\}\_\{y\_\{t\}\}, a positive scalar multiple of𝐮yt\\mathbf\{u\}\_\{y\_\{t\}\}wheneverp​\(yt\)<1p\(y\_\{t\}\)<1\. Therefore the projection𝐮yt⊤​𝐨t,j\(l,h\)=ϕt,j\(l,h\)\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}=\\phi^\{\(l,h\)\}\_\{t,j\}captures the leading\-order direction of steepest descent ofℒt\\mathcal\{L\}\_\{t\}in the𝐨t,j\(l,h\)\\mathbf\{o\}^\{\(l,h\)\}\_\{t,j\}subspace\.

###### Corollary 3\(Gradient\-faithful sign\)\.

Under the direct\-path approximation andp​\(yt\)<1p\(y\_\{t\}\)<1, increasingϕt,j\(l,h\)\\phi^\{\(l,h\)\}\_\{t,j\}decreasesℒt\\mathcal\{L\}\_\{t\}to first order; ablating a head with largeϕt,j\(l,h\)\\phi^\{\(l,h\)\}\_\{t,j\}should therefore raise the cross\-entropy atyty\_\{t\}by an amount proportional toϕt,j\(l,h\)\\phi^\{\(l,h\)\}\_\{t,j\}\.

The corollary motivates the use ofϕ\\phifor head selection: heads with large positiveϕ\\phiare heads whose ablation should hurt the answer logit most, which is exactly the head set we wantLOCOSto identify\. Attention\-based scoring, by contrast, projects the per\-position output onto a uniform direction injj\(the all\-ones combiner\) and is gradient\-free with respect to the answer token—which is why it generalizes poorly when the answer\-aligned direction varies across attended positions\.

### O\.8Summary

[Tab\.˜4](https://arxiv.org/html/2607.01002#A15.T4)consolidates the structural differences derived above\.

Table 4:Attention\-based scoring is a special case ofLOCOS, recovered when the OV circuit contributes no position\-dependent answer\-aligned signal \([Proposition˜1](https://arxiv.org/html/2607.01002#Thmproposition1)\)\.PropertyAttention\-basedLogit contribution \(LOCOS\)Per\-position observableαt,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}αt,j\(l,h\)⋅𝐮yt⊤​WO\(l,h\)​𝐯t,j\(l,h\)\\alpha^\{\(l,h\)\}\_\{t,j\}\\cdot\\mathbf\{u\}\_\{y\_\{t\}\}^\{\\top\}W\_\{O\}^\{\(l,h\)\}\\mathbf\{v\}^\{\(l,h\)\}\_\{t,j\}Includes OV circuitnoyesDistinguishes heads with equalα\\alphano \(Cor\.[2](https://arxiv.org/html/2607.01002#Thmcorollary2)\)yesContrast axistemporal \(answer vs\. non\-answer steps\)spatial \(needle vs\. off\-needle positions\)Requires non\-answer stepsyesno \(Prop\.[3](https://arxiv.org/html/2607.01002#Thmproposition3)\)Score signnon\-negative \(clamped\)unclamped \(negative⇒\\Rightarrowoff\-needle\-dominant; Cor\.[1](https://arxiv.org/html/2607.01002#Thmcorollary1)\)Reduction—equals attention mass whenWO​𝐯jW\_\{O\}\\mathbf\{v\}\_\{j\}is position\-independentWu/NIAH scorenativespecial case \(Prop\.[2](https://arxiv.org/html/2607.01002#Thmproposition2), conditions W1–W3\)HeadKV/CompressKVnativesubsumed \(Cor\.[2](https://arxiv.org/html/2607.01002#Thmcorollary2)\)Gradient interpretationprojects onto𝟏\\mathbf\{1\}injjprojects onto𝐮yt\\mathbf\{u\}\_\{y\_\{t\}\}inℝd\\mathbb\{R\}^\{d\}\(Eq\.[18](https://arxiv.org/html/2607.01002#A15.E18)\)

The reductions above predict three empirical patterns:

- •On literal NIAH \(assumptions W1–W3 hold\), Wu/NIAH andLOCOSshould rank heads similarly\.
- •On NoLiMa \(assumption W1 fails\), the rankings should diverge, withLOCOSassigning credit to heads whoseWO​𝐯jW\_\{O\}\\mathbf\{v\}\_\{j\}is answer\-aligned even when the attended token is not the answer token\.
- •Under causal validation, the divergent rankings should manifest as different ablation curves: heads identified only byLOCOSshould be more causally critical on NoLiMa than heads identified only by Wu\.

The Wu/NIAH\-vs\-NoLiMa gap reported in[Appx\.˜B](https://arxiv.org/html/2607.01002#A2)\(top head:0\.970\.97on NIAH,0\.030\.03on NoLiMa, no causal effect under ablation\) and the ablation\-curve separation in[Fig\.˜3](https://arxiv.org/html/2607.01002#S4.F3)are consistent with all three predictions\. The bottom\-kkcontrol in[§˜4\.4](https://arxiv.org/html/2607.01002#S4.SS4)additionally rules out the trivial alternative that any answer\-aligned signal would suffice for causal effect: heads with large\|cj\|\|c\_\{j\}\|but off\-needle\-dominant attention mass leave NoLiMa ROUGE\-L near baseline under ablation, exactly as[Corollary˜1](https://arxiv.org/html/2607.01002#Thmcorollary1)predicts\.

## Appendix PFuture Work

Head\-aware KV cache compression methods\[Zhanget al\.,[2023](https://arxiv.org/html/2607.01002#bib.bib5), Liet al\.,[2024](https://arxiv.org/html/2607.01002#bib.bib6), Xiaoet al\.,[2024](https://arxiv.org/html/2607.01002#bib.bib7), Fuet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib19), Linet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib18)\]allocate per\-head budgets from attention\-based retrieval scores\. If the heads that matter for non\-literal retrieval write rather than attend, then attention\-based budgets will under\-allocate cache to the heads doing the work and over\-allocate it to heads whose attention is bookkeeping\. SubstitutingLOCOSas the scoring function is a direct test of this prediction at fixed cache budget, and[Appx\.˜K](https://arxiv.org/html/2607.01002#A11)sketches a per\-\(layer, KV\-group\) variant for grouped\-query attention\. Context\-faithful decoding methods\[Gemaet al\.,[2025](https://arxiv.org/html/2607.01002#bib.bib21), Ma and Okazaki,[2026](https://arxiv.org/html/2607.01002#bib.bib49)\]contrast a base model with one whose attention\-identified retrieval heads are masked; maskingLOCOSheads instead changes which circuit is suppressed and is a candidate for a stronger contrastive signal\.

## NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact\. Do not remove the checklist:The papers not including the checklist will be desk rejected\.The checklist should follow the references and follow the \(optional\) supplemental material\. The checklist does NOT count towards the page limit\.

Please read the checklist guidelines carefully for information on how to answer these questions\. For each question in the checklist:

- •You should answer\[Yes\],\[No\], or\[N/A\]\.
- •\[N/A\]means either that the question is Not Applicable for that particular paper or the relevant information is Not Available\.
- •Please provide a short \(1–2 sentence\) justification right after your answer \(even for\[N/A\]\)\.

The checklist answers are an integral part of your paper submission\.They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers\. You will also be asked to include it \(after eventual revisions\) with the final version of your paper, and its final version will be published with the paper\.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation\. While\[Yes\]is generally preferable to\[No\], it is perfectly acceptable to answer\[No\]provided a proper justification is given \(e\.g\., error bars are not reported because it would be too computationally expensive” or “we were unable to find the license for the dataset we used”\)\. In general, answering\[No\]or\[N/A\]is not grounds for rejection\. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate\. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix\. If you answer\[Yes\]to a question, in the justification please point to the section\(s\) where related material for the question can be found\.

IMPORTANT, please:

- •Delete this instruction block, but keep the section heading “NeurIPS Paper Checklist",
- •Keep the checklist subsection headings, questions/answers and guidelines below\.
- •Do not modify the questions and only use the provided macros for your answers\.

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The abstract and[§˜1](https://arxiv.org/html/2607.01002#S1)state three scoped claims — non\-literal retrieval head detection via OV\-circuit projection, causal validation across six model configurations, and retrieval specificity — each backed by experiments in[§§˜4\.2](https://arxiv.org/html/2607.01002#S4.SS2),[4\.4](https://arxiv.org/html/2607.01002#S4.SS4)and[4\.6](https://arxiv.org/html/2607.01002#S4.SS6)\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification:[§˜6](https://arxiv.org/html/2607.01002#S6)discusses the off\-needle\-baseline sensitivity to distractor content, the restriction to attention heads \(FFN sublayers not scored\), and the coverage of architecture variations\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[Yes\]
14. Justification: All propositions, lemmas, and corollaries are stated and proved in[Appx\.˜O](https://arxiv.org/html/2607.01002#A15), with assumptions made explicit in each statement \(e\.g\., conditions W1–W3 in[Proposition˜2](https://arxiv.org/html/2607.01002#Thmproposition2)\); supporting derivations for the direct\-path approximation appear in[Appx\.˜N](https://arxiv.org/html/2607.01002#A14)\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification:[§§˜4\.1](https://arxiv.org/html/2607.01002#S4.SS1)and[B](https://arxiv.org/html/2607.01002#A2)specify the probing protocol, model checkpoints, decoding configuration, ablation procedure, and aggregation;[Appx\.˜H](https://arxiv.org/html/2607.01002#A8)details the architecture\-specific implementation;[Appx\.˜A](https://arxiv.org/html/2607.01002#A1)lists exact model versions and datasets\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
24. Justification:[Appx\.˜D](https://arxiv.org/html/2607.01002#A4)describes the released repository, including detection scripts, the vLLM\-based ablation driver, evaluation scripts, pre\-computed head\-score files, and a pinned environment specification\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: All hyperparameters \(probing protocol, decoding, aggregation, ablation depths\) are consolidated in[Tab\.˜2](https://arxiv.org/html/2607.01002#A2.T2)of[Appx\.˜B](https://arxiv.org/html/2607.01002#A2); trial filtering and held\-out splits are specified in[§˜4\.1](https://arxiv.org/html/2607.01002#S4.SS1)\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: 95% non\-parametric bootstrap confidence intervals \(B=1,000B\{=\}1\{,\}000resamples over passing trials\) are reported for per\-head scoresSl,hS\_\{l,h\}, as specified in[Appx\.˜B](https://arxiv.org/html/2607.01002#A2); ablation curves are evaluated on a held\-out set of 800 NoLiMa trials per condition\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification:[Appx\.˜C](https://arxiv.org/html/2607.01002#A3)reports GPU type, tensor\-parallel configuration, and approximate wall\-clock for detection, calibration, and ablation, plus a total\-project estimate that includes preliminary experiments not in the paper\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: The work uses publicly available models and datasets within their licenses \([Appx\.˜A](https://arxiv.org/html/2607.01002#A1)\), involves no human subjects, and the broader\-impact considerations are discussed in[Appx\.˜E](https://arxiv.org/html/2607.01002#A5)\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification:[Appx\.˜E](https://arxiv.org/html/2607.01002#A5)discusses positive impacts \(KV cache compression, hallucination mitigation, mechanistic interpretability\), potential misuse \(denial\-of\-capability attacks via head suppression\) with mitigations, and fairness considerations regarding monolingual evaluation\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: We release no new pre\-trained models or scraped datasets; only diagnostic code and small author\-constructed control sets \(city–country, arithmetic\) are released, none of which carry high misuse risk \([Appxs\.˜E](https://arxiv.org/html/2607.01002#A5)and[D](https://arxiv.org/html/2607.01002#A4)\)\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification:[Tab\.˜1](https://arxiv.org/html/2607.01002#A1.T1)in[Appx\.˜A](https://arxiv.org/html/2607.01002#A1)lists every model, dataset, and software library used, with version, identifier, and license; original papers are cited and terms of use are respected\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://arxiv.org/html/2607.01002v1/paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[Yes\]
64. Justification: The released code, pre\-computed head\-score files, and author\-constructed control sets are documented in[Appx\.˜D](https://arxiv.org/html/2607.01002#A4), with a README and pinned environment included in the repository\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: The work involves no crowdsourcing or human subjects\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: The work involves no human subjects, so no IRB review was required\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[N/A\]
79. Justification: LLMs are the object of study but are not part of the core methodology in any non\-standard way; LLM use for writing assistance is declared in[Appx\.˜F](https://arxiv.org/html/2607.01002#A6)\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.