Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models

# Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
Source: [https://arxiv.org/html/2607.20524](https://arxiv.org/html/2607.20524)
Sagar DangalManoj ShakyaDepartment of Computer Science and Engineering, Kathmandu University, Dhulikhel, Nepal

###### Abstract

Mean cross\-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested\. We present six coordinated experiments across GPT\-2, LLaMA\-3\.2\-1B/3B, OPT\-1\.3B, and distilgpt2\. We first characterise short\-term \(5–100 token\) attention degradation, finding a universal exponential\-then\-plateau pattern whose rate is inversely correlated with depth, with distinct layer\-wise entropy signatures per architecture\. Function token anchoring proves architecture\-dependent: OPT\-1\.3B \(absolute positional encoding\) shows distance\-dependent preposition specificity, GPT\-2 shows uniform non\-specific dependence, and LLaMA \(RoPE\) shows reversal at long distances\. Strategic comma insertion at clause boundaries causally reduces prediction degradation in the 40–80 token range, with the benefit tied to syntactic boundary alignment rather than token density\. We then test the mechanism causally: Relay\-Aware Attention \(RAA\), which biases attention logits toward function token positions, verifiably increases attention mass by 16–24% yet yields null effects on GPT\-2 and LLaMA\-1B, preliminary harm on LLaMA\-3B, and a mixed effect on OPT\-1\.3B that nets to approximately zero\. Multi\-fact retrieval probes further show that degradation rate does not predict retrieval accuracy across models\. We conclude that mean attention degradation is largely descriptive rather than prescriptive: function tokens contribute through what their hidden states compute, not through the attention they receive — with implications for interpretability methodology and attention\-score\-based inference optimisations such as KV\-cache eviction\.

## 1Introduction

The self\-attention mechanism in transformer\-based language models\[[16](https://arxiv.org/html/2607.20524#bib.bib16)\]allows each token to attend to every preceding position, weighted by learned query\-key similarity\. In principle this enables arbitrary long\-range dependency modelling; in practice, large language models \(LLMs\) exhibit systematic limitations in utilising contextual information as input length increases\.

The “Lost in the Middle” phenomenon\[[9](https://arxiv.org/html/2607.20524#bib.bib9)\]demonstrates that information placed in the middle of long contexts is retrieved substantially less accurately than information at the beginning or end—a primacy\-recency bias observed across models with context windows from 4K to 128K tokens\.Hong et al\. \[[6](https://arxiv.org/html/2607.20524#bib.bib6)\]documented “context rot” across 18 frontier models, with performance degrading well before nominal context limits\.Yang et al\. \[[20](https://arxiv.org/html/2607.20524#bib.bib20)\]identified analogous fading focus patterns in vision\-language models\. These studies converge on the conclusion that transformer attention does not distribute uniformly across input\.

Short\-term degradation—over spans of 5 to 100 tokens—has been comparatively underexplored, yet this range corresponds to the typical span of a clause or short sentence: the fundamental unit of syntactic composition\.Zhang et al\. \[[22](https://arxiv.org/html/2607.20524#bib.bib22)\]proposed the*function token hypothesis*: that articles, prepositions, and punctuation marks activate predictive features from context during inference\. We develop a complementary*relay\-chain*interpretation: if each function token has a bounded radius of contextual influence, the high density of function tokens in natural prose may produce a chain of overlapping anchors supporting clause\-length context retention\.

Three questions motivate this work: \(1\) How does short\-term attention degradation vary across architectures with different positional encoding schemes and depths? \(2\) Do function tokens act as structural anchors for contextual retention, and is this role architecture\-dependent? \(3\) Is mean attention degradation a causal bottleneck limiting retrieval, or merely a descriptive correlate of computations that occur elsewhere?

The first two questions are addressed through four correlational and interventional experiments \(Experiments 1–4\): we characterise the degradation profile, probe function token specificity via cloze substitution, test causal benefit via strategic token insertion, and verify corpus\-level relay\-chain coverage\. The third question is addressed through two causal experiments \(Experiments 5–6\): we operationalise the relay chain as an attention\-logit bias toward function token positions \(Relay\-Aware Attention, RAA\) and test whether verified attention redistribution improves behaviour, and we test whether degradation rate predicts multi\-fact retrieval performance across models\.

The answer to the third question is decisively negative, and constitutes our central methodological finding:mean cross\-positional attention degradation is largely descriptive rather than prescriptive\. A verified 16–24% increase in attention mass at function token positions produces null effects on GPT\-2 and LLaMA\-1B, preliminary evidence of harm on LLaMA\-3B, and a distance\-dependent mixed effect on OPT\-1\.3B that nets to approximately zero; and degradation rate fails to predict retrieval accuracy across models\. Where function tokens contribute to contextual processing—and Experiments 2–3 show they genuinely do, in architecture\-specific ways—they do so through the contextual information accumulated in their hidden states, not through the attention routing that mean attention analysis captures\.

### 1\.1Research Questions

RQ1How does attention strength degrade with token distance in the 5–100 token range, and does the rate vary across architectures?

RQ2Do function tokens specifically contribute to contextual memory retention, or is replacement damage comparable to removing any token?

RQ3Can inserting function tokens at strategic positions reduce prediction degradation, specifically for function tokens over matched controls?

RQ4Are function tokens dense enough in natural text for their effective radii to form a continuous relay chain?

RQ5Can the relay\-chain mechanism be operationalised via attention\-logit biasing toward function token positions, and does this improve contextual retrieval?

RQ6Does mean attention degradation rate predict contextual retrieval performance across architectures?

### 1\.2Contributions

1. 1\.The first systematic cross\-architecture characterisation of short\-term attention degradation \(5–100 tokens\) across four architecturally diverse models, establishing an exponential\-then\-plateau pattern universal in shape but architecture\-dependent in rate, together with layer\-wise entropy signatures that explain the cross\-model differences\.
2. 2\.First systematic evidence that function token specificity is architecture\-dependent: OPT\-1\.3B shows distance\-dependent preposition specificity; GPT\-2 shows uniform non\-specific dependence; LLaMA shows reversal at long distances consistent with RoPE redundancy\.
3. 3\.Causal evidence that comma insertion at clause boundaries reduces prediction degradation \(40–80 tokens\), specific to function tokens over matched content\-token controls, including rank\-based evidence in LLaMA\-3\.2\-3B and a comparison of four insertion strategies showing that syntactic boundary alignment, not token density, drives the benefit\.
4. 4\.Corpus\-level confirmation that function token density in natural English provides 83–89% positional coverage per category \(96–98% combined\), stable across three tokenizers\.
5. 5\.A direct causal test of the relay\-chain hypothesis via Relay\-Aware Attention \(RAA\), demonstrating that biasing attention toward function token positions produces null, negative, or marginal effects despite verified attention mass redistribution\.
6. 6\.Multi\-fact retrieval probes demonstrating that mean degradation rate does not predict retrieval accuracy: model capacity, not degradation rate, is the dividing line\.
7. 7\.The methodological conclusion that mean cross\-positional attention is descriptive rather than prescriptive, extending the attention\-as\-explanation debate to autoregressive LMs with a verified interventional design, with practical implications for attention\-score\-based inference optimisations such as KV\-cache eviction\.

## 2Related Work

### 2\.1Attention Mechanisms and Positional Encoding

The transformer\[[16](https://arxiv.org/html/2607.20524#bib.bib16)\]employs multi\-head scaled dot\-product attention\.Clark et al\. \[[2](https://arxiv.org/html/2607.20524#bib.bib2)\]showed BERT heads develop distinct specialisations;Voita et al\. \[[17](https://arxiv.org/html/2607.20524#bib.bib17)\]showed most heads are prunable\. GPT\-2\[[14](https://arxiv.org/html/2607.20524#bib.bib14)\]and OPT\[[23](https://arxiv.org/html/2607.20524#bib.bib23)\]use learned absolute positional embeddings; LLaMA\[[4](https://arxiv.org/html/2607.20524#bib.bib4)\]uses Rotary Position Embedding \(RoPE;Su et al\.[15](https://arxiv.org/html/2607.20524#bib.bib15)\), encoding relative position directly into the attention computation\. LLaMA\-3\.2 additionally uses Grouped\-Query Attention \(GQA\); we treat query\-level attention weights as the unit of analysis, consistent with prior interpretability work\.

### 2\.2Attention Degradation

Liu et al\. \[[9](https://arxiv.org/html/2607.20524#bib.bib9)\]demonstrated U\-shaped performance curves on multi\-document QA tasks at the 4K–128K token scale\.Hong et al\. \[[6](https://arxiv.org/html/2607.20524#bib.bib6)\]documented systematic degradation across 18 frontier LLMs\.Yang et al\. \[[20](https://arxiv.org/html/2607.20524#bib.bib20)\]extended findings to vision\-language models\.Gupta et al\. \[[5](https://arxiv.org/html/2607.20524#bib.bib5)\]found LLMs use depth non\-uniformly, with function words among the earliest correctly predicted\. None of this prior work examines the 5–100 token clause\-level range, which is the regime addressed here\.

### 2\.3Attention as Explanation: Descriptive vs\. Causal

Whether attention weights constitute mechanistic evidence of model behaviour is contested\.Jain and Wallace \[[7](https://arxiv.org/html/2607.20524#bib.bib7)\]showed that attention distributions are not faithful explanations in encoder classifiers: perturbing attention often has minimal effect on predictions\.Wiegreffe and Pinter \[[18](https://arxiv.org/html/2607.20524#bib.bib18)\]argued that this does not fully refute an explanatory role, and the debate has continued\[[1](https://arxiv.org/html/2607.20524#bib.bib1)\]\. These studies were largely conducted on encoder models in classification settings, relying primarily on sensitivity analysis\. Our work extends this question to autoregressive LMs and contributes a direct interventional test: we verify that attention mass is redistributed \(16–24%\), then ask whether behaviour changes in the predicted direction\. The interventional framing addresses the key limitation of correlation\-based faithfulness analyses\.

### 2\.4The Function Token Hypothesis

Xiao et al\. \[[19](https://arxiv.org/html/2607.20524#bib.bib19)\]identified the attention sink: a disproportionate fraction of attention mass allocated to the first few tokens regardless of semantic relevance\.Zhang et al\. \[[22](https://arxiv.org/html/2607.20524#bib.bib22)\]proposed that function tokens activate predictive features from context and drive next\-token prediction\. We extend this into a relay\-chain interpretation and, critically, test it causally\.Qian et al\. \[[13](https://arxiv.org/html/2607.20524#bib.bib13)\]find thinking tokens in chain\-of\-thought correspond to mutual\-information peaks with the final answer—an analogous but distinct category from our syntactic function tokens \(see alsoDing et al\.[3](https://arxiv.org/html/2607.20524#bib.bib3)\)\.Zhang et al\. \[[21](https://arxiv.org/html/2607.20524#bib.bib21)\]showed reinforced functional token tuning improves multi\-step reasoning without architectural modification\.

### 2\.5Attention Modification and Attention\-Based Selection

Press et al\. \[[12](https://arxiv.org/html/2607.20524#bib.bib12)\]introduced ALiBi, subtracting linear distance penalties from attention logits to enable length extrapolation\. Our RAA inverts this logic, rewarding structural positions rather than penalising distance\.Meng et al\. \[[10](https://arxiv.org/html/2607.20524#bib.bib10)\]showed factual associations in GPT are primarily localised in MLP layers, not attention heads—motivating our hypothesis that mean attention degradation may not capture the mechanisms responsible for retrieval\. A separate line of work uses accumulated attention scores to select which tokens to retain during inference, including H2O\[[24](https://arxiv.org/html/2607.20524#bib.bib24)\]and SnapKV\[[8](https://arxiv.org/html/2607.20524#bib.bib8)\]; these methods implicitly assume that attention mass tracks causal importance, an assumption our experiments test directly\.

## 3Methodology

### 3\.1Models

Table 1:Models used across experiments\. L=layers, QH=query heads, KVH=KV heads\. All are base pretrained models\. distilgpt2 \(6 layers, 82M\) is additionally used in Experiment 6\. LLaMA\-3\.2 uses GQA; attention is analysed at the query\-head level\.Table[1](https://arxiv.org/html/2607.20524#S3.T1)summarises the models\. The set spans a 26×\\timesparameter range \(124M–3\.21B\), a 2\.3×\\timesdepth range \(12–28 layers\), and both major positional encoding families \(learned absolute embeddings vs\. RoPE\), enabling the architecture\-comparative analysis that is central to this work\.

### 3\.2Datasets

WikiText\-103\-raw\-v1\[[11](https://arxiv.org/html/2607.20524#bib.bib11)\]was used for Experiments 1 and 5; WikiText\-2\-raw\-v1 for Experiments 2, 3, and 4\. Texts shorter than 50 characters were filtered; all texts were tokenised with each model’s native tokenizer and truncated to 512 tokens\. 100 samples were drawn from the test split, with 5–10 random seeds per model\. Experiment 6 uses a synthetic multi\-fact interference design \(§[3\.9](https://arxiv.org/html/2607.20524#S3.SS9)\)\.

### 3\.3Function Token Categories

FollowingZhang et al\. \[[22](https://arxiv.org/html/2607.20524#bib.bib22)\]:Articles\(*the, a, an*\);Prepositions\(*of, to, in, for, on, at, by, with, from, as*\);Punctuation\(,\.;:\!?\)\. Categories are operationalised at the token level using each model’s native tokenizer\.

### 3\.4Experiment 1: Baseline Attention Degradation

A full forward pass was performed with attention outputs retained\. For each of 30 randomly sampled query positions per text and each target distanced∈\{5,10,…,100\}d\\in\\\{5,10,\\ldots,100\\\}, the attention score is the mean across all layers and heads of the attention weight from positionqqtoq−dq\-d\. Degradation at distanceddis defined as

Degradation​\(d\)=\(1−Attn​\(d\)Attn​\(5\)\)×100%\.\\text\{Degradation\}\(d\)=\\left\(1\-\\frac\{\\text\{Attn\}\(d\)\}\{\\text\{Attn\}\(5\)\}\\right\)\\times 100\\%\.\(1\)Statistical significance was assessed via pairedtt\-tests across 10 seed\-level means\. Measurements were taken atd∈\{5,10,…,95\}d\\in\\\{5,10,\\ldots,95\\\};d=100d=100was excluded due to an attention\-sink boundary artefact at exactly this distance from position 0 in 512\-token sequences \(§[4\.1\.1](https://arxiv.org/html/2607.20524#S4.SS1.SSS1)\)\. Layer\-wise attention entropy \(Shannon entropy of the attention distribution over prior positions, averaged across query positions and texts\) was additionally computed for each model\.

### 3\.5Experiment 2: Cloze Substitution

A content word is selected as prediction target; baseline probability and rank are measured\. For each distanceddand function token categorycc, all function tokens of categoryccwithin the window\[target−d,target\]\[\\text\{target\}\-d,\\,\\text\{target\}\]are replaced with a neutral substitution token \(a newline character\)\.111Replacing function tokens with a newline introduces a potential confound, as newline is itself structurally meaningful in some tokenizers\. A rare out\-of\-vocabulary token would be a cleaner substitute; we treat this as a limitation \(§[6](https://arxiv.org/html/2607.20524#S6)\)\.A matched control replaces an equal number of randomly selected content tokens\. Specificity is defined as function token damage systematically exceeding control damage at a given distance\. Bonferroni correction was applied across≈\\approx63 distance×\\timescategory tests\.

##### Inter\-run stability\.

Repeat runs of Experiment 2 with identical configuration produced mean function\-token replacementΔ​p\\Delta pestimates varying by up to2×2\\timesacross runs, suggesting results in the 5–100 sample range are sensitive to sample composition\. Specificity percentages should be interpreted with this variance in mind\.

### 3\.6Experiment 3: Function Token Insertion

Four insertion strategies were tested: \(1\)*Boundary*: inserts a comma after content words directly followed by other content words, approximating natural clause\-boundary positions; \(2\)*Gap\-fill*: inserts at positions where no function token exists within the effective radius; \(3\)*Interval\-10*and \(4\)*Interval\-20*: inserts every 10 or 20 tokens regardless of syntactic structure\. A matched control inserts an equal number of random content tokens at random positions\. Significance was assessed via pairedtt\-tests on per\-seed mean probability and rank changes\. Because the control inserts at random positions rather than at the positions used by the function\-token strategies, observed differences partially reflect insertion position as well as token identity; we treat this as a limitation \(§[6](https://arxiv.org/html/2607.20524#S6)\)\.

### 3\.7Experiment 4: Relay Chain Coverage

For each text and function token category, gaps between consecutive function tokens are measured\. Coverage is the percentage of non\-function\-token positions falling within the effective radius of at least one function token\. Effective radii are derived empirically from Experiment 2 results \(articles: 60 tokens; prepositions: 25; punctuation: 35\)\. Note that article and punctuation radii are derived from conditions showing weak or absent specificity in most models \(§[4\.2](https://arxiv.org/html/2607.20524#S4.SS2)\); coverage figures for these categories thus primarily reflect corpus density rather than model\-specific anchoring, and should be interpreted accordingly\. Three tokenizers \(GPT\-2 BPE, LLaMA SentencePiece, OPT BPE\) are compared\.

### 3\.8Experiment 5: Relay\-Aware Attention \(RAA\)

Standard scaled dot\-product attention is

A​\(q,j\)=softmax⁡\(Q​K⊤dk\)​\[q,j\]\.A\(q,j\)=\\operatorname\{softmax\}\\\!\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\[q,j\]\.\(2\)RAA adds a position\-dependent bias before the softmax:

A′​\(q,j\)=softmax⁡\(Q​K⊤dk\+B​\(j\)\)​\[q,j\],A^\{\\prime\}\(q,j\)=\\operatorname\{softmax\}\\\!\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+B\(j\)\\right\)\[q,j\],\(3\)whereB​\(j\)=δB\(j\)=\\deltaif tokenjjis a function token andB​\(j\)=0B\(j\)=0otherwise\. The scalarδ\\deltais tested at\{0\.0,0\.1,0\.5,1\.0,2\.0\}\\\{0\.0,0\.1,0\.5,1\.0,2\.0\\\}, applied uniformly across all layers and heads\. RAA was tested on GPT\-2 \(10 seeds, 100 samples\), OPT\-1\.3B \(5 seeds, 50 samples\), LLaMA\-1B \(2 seeds\), and LLaMA\-3B \(2 seeds\)\.

##### Attention verification\.

Before interpreting effect sizes, we verify that RAA materially changes attention distributions\. Atδ=2\.0\\delta=2\.0, mean attention at function\-token positions increases by 16–24% atd=5d=5\(OPT:0\.0113→0\.01310\.0113\\to 0\.0131; LLaMA\-1B:0\.0112→0\.01390\.0112\\to 0\.0139\), ruling out the interpretation that RAA failed to redirect attention\. This redistribution was not separately verified at long distances \(d=40d=40–9090\) where behavioural nulls are reported; we treat this as a limitation \(§[6](https://arxiv.org/html/2607.20524#S6)\)\.

### 3\.9Experiment 6: Multi\-Fact Interference Probe

Each trial presents four fact sentences with fabricated rare names sharing the same syntactic structure, drawn from eight distinct topic sets to prevent results depending on a single pattern\. Facts are separated by WikiText filler text; the layout places two distractor facts before the target, the target fact, a variable\-length filler gap, one distractor after the target, then the cue\. The primary metric is*beats\-distractors*: the percentage of trials where the target name ranks higher than all three distractor names\. 200 trials were run per distance acrossd∈\{10,20,…,80\}d\\in\\\{10,20,\\ldots,80\\\}\.

##### Methodological note\.

Varying target distance simultaneously varies inter\-fact spacing; central conclusions rely on cross\-model comparisons at matched distance, where this confound is held constant\. With four candidates, random ranking yields≈\\approx25% beats\-distractors; models scoring substantially below this threshold should be considered at floor on this task\.

## 4Results

### 4\.1Experiment 1: Baseline Attention Degradation

All models show statistically significant attention degradation following a consistent exponential\-then\-plateau pattern \(Figure[1](https://arxiv.org/html/2607.20524#S4.F1)\): a steep decline fromd=5d=5to approximatelyd=40d=40–5050, followed by an asymptotic plateau\. All distances beyondd=10d=10are significant atp<10−13p<10^\{\-13\}\. The degradation rate is inversely correlated with model depth: GPT\-2 \(12 layers\) degrades fastest, LLaMA\-1B \(16 layers\) at an intermediate rate, and OPT\-1\.3B \(24 layers\) most gradually\. LLaMA\-3B follows the same qualitative pattern, consistent with the depth\-rate relationship\. Full per\-distance values, including attention means and standard deviations for each model, are provided in Appendix[A](https://arxiv.org/html/2607.20524#A1)\.

Table 2:Cross\-model attention degradation at selected distances\. pp = percentage points\. Spread = GPT\-2 minus OPT\-1\.3B\. Measurements spand=5d=5–9595;d=100d=100is excluded \(§[4\.1\.1](https://arxiv.org/html/2607.20524#S4.SS1.SSS1)\)\.1010252540405555707085850252550507575100100Token distanceAttention degradation \(%\)GPT\-2 \(12L\)LLaMA\-1B \(16L\)OPT\-1\.3B \(24L\)Figure 1:Attention degradation by token distance across architectures \(data from Appendix[A](https://arxiv.org/html/2607.20524#A1)\)\. All models follow an exponential\-then\-plateau pattern; degradation rate is inversely correlated with depth\. Alld\>10d\>10significant atp<10−13p<10^\{\-13\}\.Two further observations are noteworthy\. First, the plateau levels differ systematically: GPT\-2 and LLaMA\-1B plateau at 88–92%, while OPT\-1\.3B plateaus at 80–82%, indicating that OPT retains more absolute attention at long distances, not merely a slower approach to the same asymptote\. Second, absolute attention values atd=80d=80converge to the same order of magnitude across models \(≈\\approx0\.0011–0\.0021\) despite very different baselines, far below the1/80=0\.01251/80=0\.0125a uniform distribution would assign—consistent with models concentrating residual long\-range attention at specific positions \(cf\. the attention sink,Xiao et al\.[19](https://arxiv.org/html/2607.20524#bib.bib19)\) rather than distributing it uniformly\.

#### 4\.1\.1The Distance\-100 Anomaly

All models show a pronounced anomaly at exactlyd=100d=100: GPT\-2’s apparent degradation collapses to\+15\.18%\+15\.18\\%\(attention=0\.0156=0\.0156\), while LLaMA\-1B and OPT\-1\.3B show*negative*degradation of−266\.75%\-266\.75\\%and−157\.49%\-157\.49\\%respectively—attention far above thed=5d=5baseline\. This reflects the attention sink: when the target position at distance 100 coincides with the beginning\-of\-sequence position in the sampling scheme used, attention weights are dramatically inflated\. The distance is excluded from all substantive analyses\.

#### 4\.1\.2Layer\-wise Entropy Analysis

Attention entropy across layers reveals distinct architectural signatures\.OPT\-1\.3Bshows monotonically decreasing entropy across its 24 layers \(3\.91 nats at layer 1 to 0\.79 nats at layer 23\), consistent with gradual, progressive refinement in which no single layer resolves long\-range dependencies alone\.LLaMA\-1Bshows a W\-shaped profile: focused early \(minimum 0\.91 nats at layer 3\), diffuse in the middle \(peak 2\.25 nats at layer 8\), refocused late \(1\.02 nats at layer 14\)—suggesting two\-phase processing in which early layers perform local syntactic integration and middle layers broad contextual integration\.GPT\-2shows an oscillating pattern with terminal diffusion \(2\.32 nats at layer 12\); its diffuse final\-layer attention may explain its sensitivity to inserted structural markers \(§[4\.3](https://arxiv.org/html/2607.20524#S4.SS3)\): when attention is not strongly focused, any token that concentrates probability mass is disproportionately influential\.

### 4\.2Experiment 2: Cloze Substitution

The cloze experiment reveals highly architecture\-dependent patterns of function token reliance, summarised in Table[3](https://arxiv.org/html/2607.20524#S4.T3)\.

Table 3:Cross\-model cloze specificity \(10 seeds per model\)\. FN = function token replaced with neutral token\. Specificity = proportion of distances where\|FN​Δ​p\|\>\|Control​Δ​p\|\|\\text\{FN\}\\,\\Delta p\|\>\|\\text\{Control\}\\,\\Delta p\|\. GPT\-2 results are reported over 20 distances; LLaMA\-1B and OPT\-1\.3B over 21, reflecting a one\-token tokenizer boundary difference\. Inter\-runΔ​p\\Delta pestimates vary by up to2×2\\timesacross identical\-configuration runs \(§[3\.5](https://arxiv.org/html/2607.20524#S3.SS5)\)\.##### OPT\-1\.3B: distance\-dependent preposition specificity\.

Replacing prepositions causes probability drops increasing approximately monotonically with distance:−0\.064\-0\.064atd=5d=5, deepening to−0\.078\-0\.078atd=20d=20,−0\.088\-0\.088atd=35d=35, and plateauing at≈−0\.095\\approx\-0\.095byd=60d=60\. The per\-distance comparison \(Table[4](https://arxiv.org/html/2607.20524#S4.T4)\) shows the specificity gap widening with distance: atd=5d=5control damage actually exceeds preposition damage, the crossover occurs at≈d=15\\approx d=15, and fromd=30d=30onward preposition damage consistently exceeds control damage by0\.0130\.013–0\.0300\.030\. After Bonferroni correction, specificity is significant \(p<0\.01p<0\.01\) atd∈\{20,25,30,35,40\}d\\in\\\{20,25,30,35,40\\\}and observed at 16 of 21 distances\. This widening gap is the quantitative signature predicted by the relay\-chain hypothesis: as distance removes the target from the range of direct attention, the absence of relay anchors becomes proportionally more costly\.

Table 4:OPT\-1\.3B per\-distance preposition substitution results \(10 seeds, 100 samples\)\. FN = prepositions replaced with neutral token; Control = equal number of random content tokens replaced\. The anomalous distanced=100d=100is omitted \(§[4\.1\.1](https://arxiv.org/html/2607.20524#S4.SS1.SSS1)\); specificity is observed at 16 of the 21 measured distances\.In contrast, article and punctuation replacement in OPT causes substantial probability drops \(−0\.054\-0\.054and−0\.026\-0\.026\) but the control condition consistently causes comparable or larger damage \(specificity 10% and 0% respectively\)\. Articles primarily signal definiteness—a semantic property—whereas prepositions explicitly encode argument structure; this difference in structural function maps directly onto the difference in relay\-chain specificity\.

##### GPT\-2: uniform non\-specific dependence\.

Function token replacement causes consistent but remarkably flat damage across distances \(−0\.022\-0\.022articles,−0\.028\-0\.028prepositions,−0\.009\-0\.009punctuation; cross\-distance SD an order of magnitude smaller than OPT’s variation\)\. Content tokens cause comparable damage, and specificity is observed at only≈\\approx35% of distances\. Weak positive correlations between distance and probability change \(articles:r=0\.51r=0\.51,p=0\.02p=0\.02; punctuation:r=0\.80r=0\.80,p<0\.001p<0\.001\) indicate slightly*less*damage at longer distances—the opposite of the relay\-chain prediction—though the absolute magnitudes are too small to be practically meaningful\. The relay\-chain hypothesis is not supported for GPT\-2\.

##### LLaMA\-1B: reversal at long distances\.

Preposition replacement initially harms prediction \(−0\.017\-0\.017atd=10d=10\) but the effect diminishes monotonically to−0\.006\-0\.006byd=100d=100; rank\-based measures show progressive*improvement*when prepositions are replaced, from−19\-19rank positions atd=30d=30to−372\-372byd=105d=105\. This reversal is consistent with RoPE encoding relative position directly in the attention computation, making function tokens redundant as positional cues and potentially introducing competing signals at long distances\. Per\-sample variance is, however, very large \(SD\>\>600 rank positions\), and none of the LLaMA\-1B cloze results survive Bonferroni correction; we report this pattern as an exploratory observation that subsequent experiments \(§[4\.3](https://arxiv.org/html/2607.20524#S4.SS3), §[4\.5](https://arxiv.org/html/2607.20524#S4.SS5)\) independently corroborate\.

### 4\.3Experiment 3: Function Token Insertion

The boundary strategy is consistently the most effective, with strongest effects in the 40–80 token transition zone where attention has partially but not fully degraded \(Table[5](https://arxiv.org/html/2607.20524#S4.T5)\)\.

Table 5:Boundary insertion pairedtt\-test results \(function token vs\. content\-token control\)\. \* =p<0\.05p<0\.05\.Δ​p\\Delta p= mean probability change for the function\-token condition \(negative = less damage than baseline\)\. 4–5 seeds per condition\.Atd=80d=80, comma insertion causes4\.1×4\.1\\timesless probability damage than random token insertion in GPT\-2 \(−0\.007\-0\.007vs\.−0\.029\-0\.029\)\. Significant distances in OPT\-1\.3B \(20–80\) converge with the distances showing preposition specificity in Experiment 2, strengthening a causal interpretation for this model\. In LLaMA\-1B, the probability changes atd=70d=70–8080turn slightly*positive*, indicating marginal harm from insertion at the longest distances—consistent with the reversal in Experiment 2\.

##### LLaMA\-3\.2\-3B: rank\-based evidence\.

LLaMA\-3B’s 128K\-token vocabulary spreads probability mass so thinly that per\-token probability effects are on the order of10−510^\{\-5\}and difficult to interpret; rank\-based effects, however, are substantial \(Table[6](https://arxiv.org/html/2607.20524#S4.T6)\)\. Boundary insertion atd=50d=50improves the target word’s rank by−3,294\-3\{,\}294positions in the function\-token condition versus−280\-280for the control—a 7\.5\-fold difference—with significant rank effects atd=30d=30–8080\(p=0\.014p=0\.014–0\.0450\.045\)\. Function token insertion thus genuinely reorders LLaMA\-3B’s predictions even when probability changes are negligible\.

Table 6:LLaMA\-3\.2\-3B boundary insertion, rank\-based results \(seeds 1–4\)\. RankΔ\\Delta= mean change in target\-word rank \(negative = improved prediction\)\. \* =p<0\.05p<0\.05\(one\-samplett\-test vs\. 0\)\.
##### Strategy comparison\.

The boundary strategy outperforms gap\-fill and both interval strategies across all models\. Gap\-fill helps in LLaMA\-1B fromd=30d=30onward but mostly causes harm in GPT\-2 and OPT; interval\-10 shows significant rank improvements only in LLaMA\-3B atd=40d=40–8080; interval\-20 is least effective overall\. The superiority of syntactically aligned insertion over density\-matched uniform insertion indicates that the benefit is tied to alignment with syntactic structure, not to function\-token density per se\. Practically, this means uniform\-interval punctuation injection is a poor substitute for even coarse clause\-boundary heuristics\.

##### Per\-seed consistency\.

For GPT\-2 atd=40d=40–8080, comma insertion beat the control in 4/5 seeds atd=40d=40and 5/5 seeds atd=50d=50,6060, and8080; for OPT at significant distances, 4–5 of 5 seeds; for LLaMA\-3B, 4/4 completed seeds at all distances\. The significantpp\-values are therefore driven by consistent cross\-seed patterns rather than outlier seeds\.

### 4\.4Experiment 4: Relay Chain Coverage

Table 7:Relay chain coverage by function token category and tokenizer\. Coverage ranges from 83\.4% to 88\.9%; combined coverage across all three categories is≈\\approx96–98%\. Radii for articles and punctuation derive from Experiment 2 conditions showing weak specificity in most models; these figures primarily reflect corpus density \(§[3\.7](https://arxiv.org/html/2607.20524#S3.SS7)\)\.Coverage ranges from 83\.4% to 88\.9% across all category\-tokenizer combinations \(Table[7](https://arxiv.org/html/2607.20524#S4.T7)\), with<<2 pp difference across tokenizers, indicating the analysis reflects a property of the English text distribution rather than a tokenisation artefact\. Gap distributions are right\-skewed with modal gaps of 2–5 tokens and tails to 40–50 tokens\. The three categories are complementary: positions uncovered by one category are typically covered by another, raising combined coverage to≈\\approx96–98%\. Uncovered positions concentrate in long spans of uninterrupted content words \(multi\-modifier noun phrases, dense technical passages, numerical sequences\), which are comparatively rare in encyclopaedic prose\.

### 4\.5Experiment 5: Relay\-Aware Attention — The Central Causal Test

Table 8:RAA effects across architectures\. L=layers\. Best/Worst = largest observed improvement or harm at any distance for the strongest\-effectδ\\delta\.†\\dagger= significant atp<0\.05p<0\.05\. GPT\-2 and LLaMA\-1B show no significant improvement at any distance\. The LLaMA\-3B result is based on 2 seeds and should be treated as indicative only \(∗\)\.##### GPT\-2 and LLaMA\-1B: null effects\.

No significant improvement at any distance survives Bonferroni correction\.

##### LLaMA\-3B: preliminary negative effect\.

Allδ\\deltavalues increase degradation \(−5\.03\-5\.03pp atd=10d=10,δ=0\.1\\delta=0\.1\), extending the reversal pattern from Experiments 2–3\. This result rests on 2 seeds and should be treated as a preliminary indication of harm rather than a robust finding\.

##### OPT\-1\.3B: distance\-dependent mixed effect\.

Short distances \(d=10d=10–3030\) show mean\+1\.6\+1\.6pp improvement \(significant atd=15d=15:\+2\.82\+2\.82pp,p=0\.003p=0\.003;d=25d=25:\+1\.73\+1\.73pp,p=0\.013p=0\.013\)\. Long distances \(d=40d=40–9090\) show mean−0\.3\-0\.3pp harm \(significant atd=40d=40:−1\.53\-1\.53pp,p=0\.019p=0\.019\)\. The net effect across all distances is approximately zero: RAA shifts*where*degradation occurs without reducing its overall magnitude \(Figure[2](https://arxiv.org/html/2607.20524#S4.F2)\)\.

101020203030404050506060707080809090−4\-4−2\-2022440p=0\.003p\{=\}0\.003p=0\.013p\{=\}0\.013p=0\.019p\{=\}0\.019Token distanceDegradation change \(pp\)OPT\-1\.3B,δ=2\.0\\delta=2\.0Figure 2:RAA effect on OPT\-1\.3B atδ=2\.0\\delta=2\.0\(positive = improvement, negative = harm\)\. Improvements at short distances are offset by harm at longer distances; the net effect is approximately zero\.This is the central finding: a sound intervention that verifiably increases attention to function token positions by 16–24% does not produce the improvements that the relay\-chain interpretation of mean attention would predict in three of four models\. Even in the most favourable case \(OPT\-1\.3B\)—the model whose correlational profile most strongly supports the relay chain—the redistribution nets to approximately zero\.

### 4\.6Experiment 6: Multi\-Fact Interference Probe

Table 9:Beats\-distractors % across models and distances \(200 trials per distance\)\. Chance level≈\\approx25%; the GPT\-2 family scores below chance at all distances\.##### distilgpt2 vs\. GPT\-2\.

Pairedtt\-tests yieldp=0\.41p=0\.41\(beats\-distractors, primary metric\) andp=0\.44p=0\.44\(chose\-distractor\) across eight distance points\. However, both models score 10–17\.5% beats\-distractors at every distance—below the≈\\approx25% chance level for four candidates—indicating both are at floor on this task\. Statistical indistinguishability at floor is weaker evidence than a mid\-range comparison, and we caution against over\-interpreting this comparison\.

##### LLaMA\-1B vs\. OPT\-1\.3B: the stronger comparison\.

Both models substantially outperform the GPT\-2 family\. Atd=10d=10, performance is nearly identical \(35\.5% vs\. 36\.0%\); OPT’s advantage then grows markedly with distance, reaching 84\.0% vs\. 58\.5% atd=80d=80—despite LLaMA\-1B having a*steeper*mean attention degradation curve \(Table[2](https://arxiv.org/html/2607.20524#S4.T2)\)\. This divergence, in the direction opposite to what degradation rate would predict, provides the most compelling evidence that mean degradation rate does not determine retrieval capacity\. The dividing line is model capacity \(∼\\sim1\.2–1\.3B parameters\), not degradation rate\.

## 5Discussion

### 5\.1The Architecture\-Dependent Relay Chain

The correlational and interventional evidence \(Experiments 1–4\) establishes that the function token relay\-chain mechanism is real but architecture\-dependent, interpretable through three interacting factors: positional encoding scheme, depth, and entropy profile\.

OPT\-1\.3Bis most consistent with the relay\-chain interpretation\. Its absolute positional encoding does not encode relative distance in the attention computation, so the model must learn to use token content as a positional substitute at long distances; its 24\-layer depth and monotonically decreasing entropy profile create the conditions for function tokens to serve as persistent anchors\. The convergence of Experiments 2 and 3—specific preposition dependence and significant boundary insertion effects at the same distance ranges—provides compelling evidence for a causal rather than merely correlational mechanism in this model\.

GPT\-2presents an apparent paradox: strong boundary insertion effects \(Experiment 3,p<0\.001p<0\.001atd=80d=80\) but no cloze specificity \(Experiment 2\)\. This resolves by recognising that Experiment 2 tests*removing*existing function tokens while Experiment 3 tests*adding*structural tokens\. GPT\-2 can utilise structural signals when provided but does not route attention specifically through existing function tokens; in its shallow, capacity\-limited architecture, content tokens carry as much predictive information as function tokens\.

LLaMAprovides the most theoretically informative results\. LLaMA\-1B’s monotonic reversal with distance is consistent with RoPE encoding relative position directly via query\-key rotations: even without structural tokens at intermediate positions, attention scores already contain relative\-distance information, making function tokens redundant in the positional dimension and—at long distances—potentially a source of attention dilution\. LLaMA\-3B’s RAA result extends this: function token emphasis is actively harmful for RoPE models \(though this rests on 2 seeds and warrants replication\)\.

This architectural contingency yields a testable prediction: models with absolute positional encoding and greater depth should show stronger function token dependence than shallow or RoPE\-based models, all else being equal\.

### 5\.2Mean Attention Is Largely Descriptive, Not Prescriptive

Experiments 5 and 6 jointly establish that mean cross\-positional attention degradation is at best a weak, architecture\-specific contributor to contextual processing\. This conclusion extends the attention\-as\-explanation debate\[[7](https://arxiv.org/html/2607.20524#bib.bib7),[18](https://arxiv.org/html/2607.20524#bib.bib18),[1](https://arxiv.org/html/2607.20524#bib.bib1)\]from encoder classifiers to autoregressive LMs, and from correlation\-based faithfulness analysis to direct intervention: we verify that attention mass moves before asking whether behaviour changes\.

The RAA experiments verify a 16–24% redistribution of attention mass without improvement in three of four models; even in OPT\-1\.3B the net effect is approximately zero\. The retrieval probes show that degradation rate does not predict task performance, with the LLaMA\-1B/OPT\-1\.3B divergence running opposite to the degradation\-rate prediction\. This is exactly the pattern expected if mean attention is a descriptive summary of a downstream computation rather than the computational variable itself\.

### 5\.3Resolving the Comma Insertion Paradox

Experiment 3 shows comma insertion reduces prediction degradation \(causal benefit\), while Experiment 5 shows biasing attention toward*existing*function tokens does not help\. The resolution: comma insertion adds a*new*token to the sequence, creating a fresh computational node whose hidden state is shaped by the full forward pass\. RAA merely increases attention to positions whose representations are already fixed\. The benefit of function tokens lies in what they compute—the contextual information accumulated in their hidden states—not in how much attention they receive\. This is consistent withZhang et al\. \[[22](https://arxiv.org/html/2607.20524#bib.bib22)\]’s proposal that function tokens activate predictive features, and withMeng et al\. \[[10](https://arxiv.org/html/2607.20524#bib.bib10)\]: factual associations live in MLP layers, not attention heads\.

### 5\.4Theoretical Contributions

Beyond the specific empirical findings, this work makes two broader theoretical contributions\. First, it introduces*architecture\-conditioned function token dependence*: the degree to which a model relies on function tokens as relay nodes is a predictable function of its positional encoding scheme and depth, providing a framework for anticipating model behaviour under function\-token manipulation without per\-model experimentation\. Second, it establishes the short\-term window \(5–100 tokens\) as a distinct degradation regime, parameterisable by initial decay rate \(varying with depth\) and plateau level \(varying with depth and encoding\), separate from the long\-context regime studied in prior work\.

### 5\.5Practical Implications

##### Prompt engineering\.

Strategic comma placement at clause boundaries reduces prediction degradation in the 40–80 token range for the GPT\-2\- and OPT\-family*base*models tested\. For RoPE\-based models \(LLaMA\), the benefit is less consistent and may be negative at long distances\. Conversely, compression strategies that strip articles, prepositions, or punctuation to save tokens may degrade relay\-chain coverage for absolute\-encoding models\. These findings derive from base pretrained models below 3\.5B parameters and should not be extrapolated to instruction\-tuned or frontier\-scale models without further investigation\.

##### Retrieval\-augmented generation\.

Natural function token density in standard English prose suffices for corpus\-level coverage, but heavily extractive or telegraphic content—product fragments, JSON, code—lacks this density and may be more susceptible to contextual degradation; restoring function token density before context insertion is a candidate preprocessing step\.

##### Interpretability methodology\.

Studies using mean attention weights as evidence for contextual processing may be measuring a correlate rather than a cause\. The correlation\-to\-intervention gap documented here suggests head\-level analysis, activation patching\[[10](https://arxiv.org/html/2607.20524#bib.bib10)\], or residual stream decomposition provide more causally informative views than aggregate attention statistics\.

##### KV\-cache eviction\.

Our findings carry a caution for inference\-efficiency methods that select or evict cached tokens based on accumulated attention scores, such as H2O\[[24](https://arxiv.org/html/2607.20524#bib.bib24)\]and SnapKV\[[8](https://arxiv.org/html/2607.20524#bib.bib8)\]\. These methods implicitly assume that attention mass to a position tracks that position’s causal contribution to downstream prediction—precisely the assumption our intervention tests\. The dissociation we observe between attention mass and behavioural contribution, together with the comma\-paradox resolution \(§[5\.3](https://arxiv.org/html/2607.20524#S5.SS3)\), suggests that a token’s value lies in what its hidden state has computed rather than in how much attention it subsequently receives; eviction policies keyed to attention scores may therefore discard tokens whose representations remain causally important\. Evaluating representation\-aware eviction criteria against attention\-score baselines is a natural follow\-up\.

##### Architecture design and evaluation\.

Naive attention\-bias toward structural tokens is not the right operationalisation of the relay chain; layer\-specific biasing or representation\-level interventions targeting what function tokens*compute*are more promising\. The degradation curve of Experiment 1 also offers a task\-independent diagnostic of effective contextual reach that could complement standard benchmarks: models with similar benchmark scores but different degradation profiles may fail in different contexts\.

### 5\.6Relationship to Mechanistic Interpretability

The head\-specialisation findings ofClark et al\. \[[2](https://arxiv.org/html/2607.20524#bib.bib2)\]andVoita et al\. \[[17](https://arxiv.org/html/2607.20524#bib.bib17)\]suggest specific heads may track function token positions\. Our contribution is behavioural and architecture\-comparative: it establishes that the function token mechanism exists, characterises its architecture\-dependence, and shows that its causal pathway is representational rather than attention\-routing\. A natural next step is to use our perturbations \(substitution, insertion, logit biasing\) as causal probes within an activation\-patching framework to localise the responsible components\.

### 5\.7Generalisability

WikiText is formal English encyclopaedic prose; function token density is likely lower in conversational, telegraphic, or code\-heavy text and the relay chain correspondingly sparser\. The function token hypothesis is implicitly language\-specific: agglutinative languages encoding grammatical relations morphologically may show fundamentally different relay properties, making cross\-linguistic replication a priority\. All models are base pretrained models below 3\.5B parameters; instruction\-tuned or RLHF\-aligned variants are known to alter attention distributions, and whether alignment strengthens or weakens the relay\-chain mechanism is an open empirical question\.

## 6Limitations

##### Statistical power\.

Experiment 3 uses 4–5 seeds \(3–4 df\); LLaMA RAA experiments used 2 complete seeds due to computational constraints \(consumer Apple Silicon hardware\)\. The convergent pattern across distances and conditions provides more robust support than any single test, but the LLaMA\-3B negative result in particular should be treated as preliminary; a properly powered replication with≥\\geq5 seeds is needed\.

##### RAA verification at long distances\.

Attention redistribution was verified atd=5d=5only\. At the long distances where behavioural nulls matter most \(d=40d=40–9090\), softmax saturation or attention\-sink absorption could in principle have absorbed the bias, making the null ambiguous at those distances\.

##### Multiple comparisons\.

Bonferroni correction across≈\\approx63 tests in Experiment 2 is conservative; Benjamini\-Hochberg FDR control would be more appropriate for this exploratory design and would likely identify additional significant effects\.

##### Substitution token confound\.

The newline substitution token is itself structurally meaningful in some tokenizers; a rare out\-of\-vocabulary token would be cleaner\.

##### Control position matching\.

Experiment 3’s control inserts at random positions rather than at the boundary positions used by the function\-token strategies, so observed differences partially reflect insertion position as well as token identity; position\-matched controls would isolate type\-specific effects\.

##### Dataset, language, and model scope\.

All experiments use English WikiText and base pretrained models of 82M–3\.21B parameters truncated to 512\-token sequences\. Results may not generalise to other registers, languages, instruction\-tuned variants, larger models, or longer sequence regimes \(1K–128K tokens\)\.

##### Attention metric\.

Mean attention across all layers and heads masks head\-level heterogeneity; syntactically specialised heads may show different distance\-dependence than the average\.

##### Experiment 4 effective radii\.

Radii for articles and punctuation derive from Experiment 2 conditions showing weak or absent specificity in most models; coverage figures for these categories primarily reflect corpus token density rather than model\-specific anchoring\.

##### RAA scope\.

RAA is one operationalisation of the relay chain, using uniform logit biasing across all layers and heads\. Layer\-specific biasing, head\-specific biasing, and value\-pathway interventions remain unexplored\.

##### Softmax normalisation artefact\.

Mean attention to a positionddtokens back must decline withddto some degree because softmax mass is conserved over a growing context; part of the exponential\-then\-plateau shape is therefore arithmetic rather than mechanistic—which independently supports the interpretation of mean attention degradation as a descriptive summary statistic rather than a causal variable\.

## 7Conclusion

We presented six coordinated experiments investigating short\-term attention degradation, function token anchoring, and the causal status of mean cross\-positional attention across transformer language models\. The function token relay\-chain mechanism is real but architecture\-dependent: OPT\-1\.3B shows distance\-dependent preposition specificity and responds to boundary insertion; GPT\-2 shows non\-specific uniform dependence but benefits from inserted structural tokens; LLaMA shows reversal effects consistent with redundant positional information encoded by RoPE\.

Attempts to amplify this mechanism through Relay\-Aware Attention reveal its limits: null effects on GPT\-2 and LLaMA\-1B, preliminary indication of active harm on LLaMA\-3B, and a small distance\-dependent mixed effect on OPT\-1\.3B that nets to approximately zero, despite verified 16–24% attention redistribution\. Multi\-fact interference probes confirm that mean degradation rate does not predict retrieval accuracy, with the LLaMA\-1B vs\. OPT\-1\.3B comparison providing the strongest evidence\.

These findings establish that mean cross\-positional attention degradation is largely descriptive rather than prescriptive\. Where function tokens contribute to contextual processing, they do so primarily through token representations—the contextual information accumulated in their hidden states—rather than through the attention routing that mean cross\-positional analysis captures\. This refines the function token hypothesis from a universal claim to an architecture\-conditioned mechanism, extends the attention\-as\-explanation debate to autoregressive models with a verified interventional design, and cautions against treating aggregate attention statistics—in interpretability analyses or in attention\-score\-based inference optimisations—as proxies for causal importance\.

## Appendix AFull Experiment 1 Degradation Tables

Table 10:GPT\-2 attention degradation \(10 seeds, 100 samples\)\. Selected distances shown; alld≥10d\\geq 10significant atp<5×10−17p<5\\times 10^\{\-17\}\.Table 11:LLaMA\-3\.2\-1B attention degradation \(10 seeds, 100 samples\)\.Table 12:OPT\-1\.3B attention degradation \(10 seeds, 100 samples\)\.
## Code and Data Availability

Code, per\-experiment configuration files, and result tables supporting the findings of this paper are available from the authors on reasonable request\.

## Acknowledgements

This work derives from the first author’s Level 7 project at London Metropolitan University / Islington College, supervised by the second author\.

## References

- Bibal et al\. \[2022\]Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, and Pierre Watrin\.Is attention explanation? An introduction to the debate\.*arXiv preprint arXiv:2206\.09775*, 2022\.
- Clark et al\. \[2019\]Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning\.What does BERT look at? An analysis of BERT’s attention\.In*Proceedings of the 2019 ACL Workshop BlackboxNLP*, pages 276–286, 2019\.
- Ding et al\. \[2025\]Bowen Ding, Yuhan Chen, Futing Wang, Lingfeng Ming, and Tao Lin\.Do thinking tokens help or trap? Towards more efficient large reasoning models\.*arXiv preprint arXiv:2506\.23840*, 2025\.
- Dubey et al\. \[2024\]Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Gupta et al\. \[2025\]Arjun Gupta, Jimmy Yeung, Gopala Anumanchipalli, and Anna Ivanova\.How do LLMs use their depth?*arXiv preprint arXiv:2510\.18871*, 2025\.
- Hong et al\. \[2025\]Kevin Hong, Anton Troynikov, and James Huber\.Context rot: How increasing input tokens impacts LLM performance\.Technical report, Chroma Research, 2025\.
- Jain and Wallace \[2019\]Sarthak Jain and Byron C Wallace\.Attention is not Explanation\.In*Proceedings of NAACL\-HLT 2019*, pages 3543–3556, 2019\.
- Li et al\. \[2024\]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen\.SnapKV: LLM knows what you are looking for before generation\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.
- Liu et al\. \[2024\]Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\.Lost in the middle: How language models use long contexts\.*Transactions of the Association for Computational Linguistics*, 12:157–173, 2024\.
- Meng et al\. \[2022\]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov\.Locating and editing factual associations in GPT\.In*Advances in Neural Information Processing Systems*, volume 35, pages 17359–17372, 2022\.
- Merity et al\. \[2017\]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models\.In*Proceedings of the 5th ICLR*, 2017\.
- Press et al\. \[2022\]Ofir Press, Noah A Smith, and Mike Lewis\.Train short, test long: Attention with linear biases enables input length extrapolation\.In*Proceedings of the 10th ICLR*, 2022\.
- Qian et al\. \[2025\]Chen Qian, Dong Liu, Hao Wen, Zhen Bai, Yang Liu, and Jing Shao\.Demystifying reasoning dynamics with mutual information: Thinking tokens are information peaks in LLM reasoning\.*arXiv preprint arXiv:2506\.02867*, 2025\.
- Radford et al\. \[2019\]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language models are unsupervised multitask learners\.*OpenAI Blog*, 1\(8\), 2019\.
- Su et al\. \[2024\]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, 568:127063, 2024\.
- Vaswani et al\. \[2017\]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.In*Advances in Neural Information Processing Systems*, volume 30, pages 5998–6008, 2017\.
- Voita et al\. \[2019\]Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov\.Analyzing multi\-head self\-attention: Specialized heads do the heavy lifting, the rest can be pruned\.In*Proceedings of the 57th Annual Meeting of the ACL*, pages 5797–5808, 2019\.
- Wiegreffe and Pinter \[2019\]Sarah Wiegreffe and Yuval Pinter\.Attention is not not Explanation\.In*Proceedings of EMNLP\-IJCNLP 2019*, pages 11–20, 2019\.
- Xiao et al\. \[2023\]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient streaming language models with attention sinks\.*arXiv preprint arXiv:2309\.17453*, 2023\.
- Yang et al\. \[2025\]Jian Yang, Chen Cui, Yi Zhou, Yang Chen, Peng Xia, Yue Wei, Tao Yu, Yue Huang, and Lin Wang\.IKOD: Mitigating visual attention degradation in large vision\-language models\.*arXiv preprint arXiv:2508\.03469*, 2025\.
- Zhang et al\. \[2025a\]Kang Zhang, Qi Yao, Bo Lai, Jian Huang, Wei Fang, Dacheng Tao, Mingli Song, and Si Liu\.Reasoning with reinforced functional token tuning\.*arXiv preprint arXiv:2502\.13389*, 2025a\.
- Zhang et al\. \[2025b\]Shaohua Zhang, Yuan Lin, and Hang Li\.Memory retrieval and consolidation in large language models through function tokens\.*arXiv preprint arXiv:2510\.08203*, 2025b\.
- Zhang et al\. \[2022\]Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer\.OPT: Open pre\-trained transformer language models\.*arXiv preprint arXiv:2205\.01068*, 2022\.
- Zhang et al\. \[2023\]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\.H2O: Heavy\-hitter oracle for efficient generative inference of large language models\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.