Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Summary
This paper introduces LOCOS, a method for identifying attention heads responsible for non-literal context synthesis in large language models, outperforming existing techniques on retrieval benchmarks.
View Cached Full Text
Cached at: 07/03/26, 03:55 PM
Paper page - Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Source: https://huggingface.co/papers/2607.01002
Abstract
Logit-Contribution Scoring (LOCOS) identifies attention heads responsible for non-literal context synthesis in large language models by measuring their output-value circuit’s contribution to answer tokens, outperforming existing methods on retrieval benchmarks.
In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying whichattention headsperform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carriesnon-literal retrieval. We introduceLogit-Contribution Scoring(LOCOS), a write-aware detector that scores each head by the projection of itsOV-circuitoutput onto theanswer-token unembeddingdirection, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiManon-literal retrievalbenchmark collapsesROUGE-Lat lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drivesROUGE-Lfrom 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific:parametric recallandarithmetic reasoningstay at baseline under the same ablation. On Qwen3-8B, the same ablation also dropsMuSiQuefrom 0.55 to 0.08 andBABI-Longfrom 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.
View arXiv pageView PDFProject pageGitHub0Add to collection
Get this paper in your agent:
hf papers read 2607\.01002
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.01002 in a model README.md to link it from this page.
Datasets citing this paper2
#### aryopg/locos-results #### aryopg/parametric-arithmetic-eval Viewer• Updated1 day ago • 600 • 38
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.01002 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
The paper introduces LOGOS, a write-aware detector that identifies attention heads responsible for non-literal retrieval in LLMs by scoring the projection of their OV-circuit output onto the answer-token unembedding direction, outperforming prior attention-based methods across multiple model families.
Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
This paper systematically studies in-context retrieval at million-token scale, introducing BlockSearch, a 0.6B LM retriever, and analyzing attention dilution. The model matches or outperforms dense retrieval on benchmarks like MS MARCO and NQ, and significantly outperforms on tasks requiring different similarity notions, highlighting the potential of in-context retrieval while emphasizing attention control under extreme context growth.
Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
This paper identifies a specialized subset of attention heads called CoRe heads in multimodal LLMs that exhibit functional sparsity in cross-modal retrieval. Causal interventions show these heads are crucial for multimodal reasoning, and leveraging this sparsity can accelerate inference.
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Researchers apply contrastive LRP-based attribution to analyze why LLMs fail on realistic benchmarks, finding the method gives useful signals in some cases but is not universally reliable.
A local attention-based retrieval with SOTA results on LongMemEval, LoCoMo, and code search benchmarks
Attemory is an open-source local memory retrieval engine that uses attention-based retrieval over KV cache instead of traditional embedding or BM25 methods, achieving state-of-the-art results on LongMemEval, LoCoMo, and code search benchmarks.