Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

# Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Source: [https://arxiv.org/html/2608.06849](https://arxiv.org/html/2608.06849)
Yehan Yang1,2, Junyuan Shang4,†\{\}^\{4,^\{\\dagger\}\}, Yang Li1,2, Guanqun Zhao3,4, Shuohuan Wang4,Dianhai Yu4 1Institute of Computing Technology, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Beijing University of Posts and Telecommunications4Baidu Inc\. \{yangyehan25z,liyang23s\}@ict\.ac\.cn,zhao\-guanqun@bupt\.edu\.cn \{shangjunyuan, wangshuohuan, yudianhai\}@baidu\.com

###### Abstract

Long\-context LLM inference is bottlenecked by quadratic attention computation and growing KV\-cache costs\. Existing sparse attention and KV\-compression methods typically decide which tokens or heads to preserve from runtime attention scores, observation windows, calibration prompts, or learned gates, making head diagnosis input\-dependent and costly to deploy\. We propose Autonomy\-of\-Heads \(AoH\), a data\-free method that identifies retrieval and streaming heads from the spectral geometry of query\-key projections\. AoH defines the kernel attention operatorMh=WKh⊤​WQhM\_\{h\}=W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}and uses its effective\-rank as a weight\-space measure of head function: concentrated spectra indicate a small number of dominant query\-key matching directions and are associated with retrieval heads, whereas diffuse spectra indicate the absence of a dominant global matching direction and are associated with streaming heads\. We further derive an efficientdheadd\_\{\\text\{head\}\}\-dimensional computation that avoids constructing the fulldmodel×dmodeld\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}matrix\. We conducted extensive experiments across models demonstrating that at 50% sparsity, AoH retains 96\.5% of Full Attention performance on average while reducing prefill and decode latency by up to 41\.4% and 66\.0%, respectively, and KV\-cache memory by 50\.0% at 256K tokens\.

Autonomy\-of\-Heads: Data\-Free Sparse Attention from Frozen Query\-Key Geometry

Yehan Yang1,2, Junyuan Shang4,†\{\}^\{4,^\{\\dagger\}\}, Yang Li1,2, Guanqun Zhao3,4,Shuohuan Wang4,Dianhai Yu41Institute of Computing Technology, Chinese Academy of Sciences2University of Chinese Academy of Sciences3Beijing University of Posts and Telecommunications4Baidu Inc\.\{yangyehan25z,liyang23s\}@ict\.ac\.cn,zhao\-guanqun@bupt\.edu\.cn\{shangjunyuan, wangshuohuan, yudianhai\}@baidu\.com

††footnotetext:†Corresponding author\.††footnotetext:Project: https://undground\.fun/aoh/![Refer to caption](https://arxiv.org/html/2608.06849v1/x1.png)Figure 1:Input\-Dependent Methods vs\. AoH\.![Refer to caption](https://arxiv.org/html/2608.06849v1/x2.png)

Figure 2:Visualization of theMhM\_\{h\}singular value spectrum and attention maps in the Qwen2\.5\-7B model for the sentence*“The best fruit is orange\. What is the best fruit? Orange\.”*, showing that concentrated spectra correspond to retrieval heads while uniform spectra correspond to streaming heads\. Left: Retrieval heads \(e\.g\., Layer 15, Head 20\) attend selectively to contextually relevant tokens, requiring full attention\.Center:MhM\_\{h\}singular value spectrum heatmap;orange\-highlighted rows \(concentrated spectrum\) are retrieval heads,red\-highlighted rows \(uniform spectrum\) are streaming heads\. Right: Streaming heads \(e\.g\., Layer 15, Head 11\) focus on sink and recent tokens, sufficient with sliding\-window attention\.## 1Introduction

Long\-context inference is increasingly central to LLM applications, particularly in agentic workflowsteam at Anthropic \([2024](https://arxiv.org/html/2608.06849#bib.bib11)\); Teamet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib10)\)and reasoning scenarios\. But standard attention incurs quadratic score computation and a linearly growing KV cache\. Existing sparse\-attention and KV\-compression methods usually decide what to keep from runtime behavior: heavy\-hitter attention massZhanget al\.\([2023](https://arxiv.org/html/2608.06849#bib.bib12)\); Wanet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib13)\), local windows or selectorsXiaoet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib24)\); Fuet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib14)\); Mohtashami and Jaggi \([2023](https://arxiv.org/html/2608.06849#bib.bib15)\); DeepSeek\-AIet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib7)\), cross\-layer reuseDeshmukhet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib4)\); Gaoet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib1)\); Baiet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib6)\); Brandonet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib16)\), or learned head gatesXiaoet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib2)\); Linet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib3)\)\. As shown in Figure[1](https://arxiv.org/html/2608.06849#S0.F1), these approaches are effective but input\-dependent or rely on learned gates, and calibration procedures\. We ask a question:*Can frozen weights alone provide a useful prior for attention\-head function, independent of runtime attention scores, calibration prompts, or additional training?*

As illustrated in Figure[2](https://arxiv.org/html/2608.06849#S0.F2), we observe that heads with concentrated spectra tend to exhibit retrieval\-style attention, selectively attending to contextually relevant tokens, whereas heads with diffuse spectra mainly focus on sink and recent tokens\.

This observation leads to our key insight: heads know what they know\!The head\-specific attention operator is already encoded in the frozen query\-key projections\. During decoding, the score of headhhcan be written asscoresh,i=Xctx​WKh⊤​WQh​xi\\text\{scores\}\_\{h,i\}=X\_\{\\text\{ctx\}\}W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}x\_\{i\}, where the middle operator is head\-specific\. We therefore define thekernel attention matrixMh=WKh⊤​WQhM\_\{h\}=W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}, whose spectral geometry characterizes the query\-key matching directions used by headhh\.

We proposeAutonomy\-of\-Heads\(AoH\), a data\-free head\-selection criterion based on the effective rank ofMhM\_\{h\}Roy and Vetterli \([2007](https://arxiv.org/html/2608.06849#bib.bib42)\)\. A low effective rank indicates a few dominant matching directions and suggests a retrieval role requiring global context; a high effective rank indicates a diffuse spectrum and suggests a streaming role that can use sink and recent\-window attention\. Because AoH depends only on frozen weights, head labels are computed once before any prompt is processed, enabling sparse attention from prefill rather than after runtime observation\. For efficient deployment, we show that the nonzero singular values ofMhM\_\{h\}can be computed from adhead×dheadd\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}proxy, avoiding construction of the fulldmodel×dmodeld\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}matrix\. We then use the resulting head labels to build AoH\-guided sparse attention: retrieval heads retain global attention, while streaming heads use bounded sink and recent\-window caches\. This keeps the sparse policy simple, training\-free, and compatible with GQA and FlashAttention\-style implementations\.

We evaluate AoH on LongBench across Models\. At 50% sparsity111We define sparsity assparsity=1−NfullNtotal\\mathrm\{sparsity\}=1\-\\frac\{N\_\{\\mathrm\{full\}\}\}\{N\_\{\\mathrm\{total\}\}\}, whereNfullN\_\{\\mathrm\{full\}\}is the number of full\-attention heads andNtotalN\_\{\\mathrm\{total\}\}is the total number of attention heads\. Since streaming heads retain only a small sink\-plus\-recent cache \(128\+256128\+256tokens in this paper\), this cache is negligible at long contexts such as 32K, 64K, and 128K\. We therefore use the equivalent KV\-cache budget approximationKV​budget≈1−sparsity\\mathrm\{KV\\ budget\}\\approx 1\-\\mathrm\{sparsity\}in the following analysis\., AoH remains close to Full Attention and consistently outperforms baselines, random and reversed head selection, showing that the effective\-rank ordering captures meaningful head\-function structure rather than an arbitrary sparse subset\. Efficiency results further show that AoH reduces prefill/decode latency and KV\-cache memory at long context lengths\. Our contributions are summarized as follows: \(1\) We introduce AoH, a data\-free and training\-free method for identifying retrieval and streaming heads directly from frozen query\-key geometry\.\(2\) We develop an effective\-rank classifier forMhM\_\{h\}and an efficientdheadd\_\{\\text\{head\}\}\-dimensional computation of its nonzero spectrum\. \(3\)We instantiate the AoH as a simple sparse\-attention policy and evaluate it on three long\-context LLMs, where it preserves strong accuracy and improves inference efficiency at 50% sparsity\.

![Refer to caption](https://arxiv.org/html/2608.06849v1/x3.png)

Figure 3:Empirical relationship between effective\-rank and head attention distance on Qwen2\.5\-7B\. Left: Per\-head average attention distance heatmaps under context lengths from 4K to 100K\. The layer–head distance patterns remain largely consistent across context lengths\. Right: Scatter plot of effective\-rank versus average attention distance, with points colored by layer\. Heads with lower effective\-rank tend to attend farther into the context, while high\-ER heads concentrate on recent tokens\.![Refer to caption](https://arxiv.org/html/2608.06849v1/x4.png)Figure 4:Passkey retrieval under progressive head\-to\-streaming conversion\. Restricting low\-ER heads to sink\-and\-recent attention rapidly collapses retrieval accuracy, while restricting high\-ER heads has little impact\.
## 2Related Work

### 2\.1Layer\-wise Sparse Attention

A major line of efficient long\-context work sparsifies attention at the token or layer level\. Early sparse\-attention architectures such as LongformerBeltagyet al\.\([2020](https://arxiv.org/html/2608.06849#bib.bib22)\)and BigBirdZaheeret al\.\([2020](https://arxiv.org/html/2608.06849#bib.bib23)\)replace dense attention with fixed local, random, and global patterns, reducing the quadratic cost of self\-attention\. For pretrained LLM inference, StreamingLLM\(Xiaoet al\.,[2024](https://arxiv.org/html/2608.06849#bib.bib24)\)combines attention sinks with a recent window, while token\-eviction and KV\-selection methods such asH2​OH\_\{2\}OZhanget al\.\([2023](https://arxiv.org/html/2608.06849#bib.bib12)\),D2​OD\_\{2\}OWanet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib13)\), NACLChenet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib36)\), SnapKVLiet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib33)\), and QuestTanget al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib34)\)retain tokens or KV caches using observed attention statistics, proxy/observation tokens, randomized eviction, or query\-aware runtime estimates\. These methods are effective, but they primarily decide*which tokens*to keep from input\-dependent signals\. AoH instead asks which heads should retain global access before seeing any input, using only frozen query\-key weights\.

### 2\.2Head\-wise Sparse Attention

A second line of work exploits the functional heterogeneity of attention heads\. RazorAttentionTanget al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib35)\)observes that only a small number of retrieval heads need long\-range cache access, while most heads focus on local context; it keeps full cache for retrieval heads and compresses non\-retrieval heads\. DuoAttentionXiaoet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib2)\)similarly separates retrieval and streaming heads, using full KV cache for retrieval heads and lightweight cache for streaming heads, but obtains head labels through a trained gate\. LycheeDecodeLinet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib3)\)uses HardKuma\-based routing for head classification and also studies cross\-layer reuse\. These works motivate head\-aware sparse attention, but their head diagnosis is tied to attention observations, learned gates, or task\-dependent procedures\. AoH contributes a complementary data\-free classifier: it assigns head or KV\-group roles from the spectral geometry of frozen query\-key weights\.

### 2\.3Cross\-layer Sharing

Cross\-layer methods reduce redundancy by sharing KV states, representations, or sparse indices across adjacent layers\. CLABrandonet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib16)\)shares key/value heads between layers, and YOCOSunet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib25)\)redesigns the decoder so that global KV cache is stored once\. KASCADE\(Deshmukhet al\.,[2025](https://arxiv.org/html/2608.06849#bib.bib4)\), HySparseGaoet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib1)\), and IndexCache\(Baiet al\.,[2026](https://arxiv.org/html/2608.06849#bib.bib6)\)exploit cross\-layer stability for sparse index or KV reuse; HySparse in particular derives both sparse\-layer token selection and KV cache from a preceding full\-attention layer\. These methods are largely orthogonal to AoH: they exploit redundancy across layers, while AoH identifies which heads should retain global access before observing any input\. As a result, AoH can serve as a data\-free head prior for cross\-layer or token\-selection systems without replacing their runtime selectors\.

## 3Observations

Before introducing Autonomy\-of\-Heads \(AoH\), we first askwhether frozen query\-key geometry reflects stable and functional differences among attention heads?We present two empirical observations that motivate our work and answer this question\.

Observation 1: Long\-range attention behavior is stable and negatively associated with effective\-rank \(ER\)\.In Figure[3](https://arxiv.org/html/2608.06849#S1.F3), the left heatmaps show that per\-head average attention distance on Qwen2\.5\-7B remains structurally stable across 4K–100K contexts, suggesting that long\-range attention is a persistent head\-level property rather than a prompt\-length artifact\. The right panel further shows that lower\-ER heads tend to attend farther into the context, whereas higher\-ER heads are more local, suggesting an overall negative association between ER and long\-range attention behavior\.

Observation 2: Low\-ER heads are functionally important\.Figure[4](https://arxiv.org/html/2608.06849#S1.F4)evaluates passkey retrieval when different KV heads are progressively converted to streaming attention\.222We conduct the study on Llama3\.1\-8B\-Instruct at 32K context length\. Converted KV heads retain only the first 128 sink tokens and the most recent 256 tokens, while the remaining heads keep full\-context KV Cache\.We compare three conversion orders: Low\-ER first, High\-ER first, and Random\. For each Streaming Attention ratio, we evaluate exact\-match passkey retrieval accuracy over 100 samples, with passkeys inserted at 20%, 40%, 60%, and 80% depths of the 32K context\. The results show a clear functional separation\. Restricting high\-ER heads to sink\-plus\-recent attention has little effect on retrieval accuracy\. In contrast, converting low\-ER heads causes accuracy to collapse rapidly, indicating that they are essential for accessing remote information\. Random conversion lies between the two strategies\.

Together, these observations motivate Autonomy\-of\-Heads: using ER as a data\-free weight\-space criterion to assign low\-ER heads to full attention and high\-ER heads to sink\-and\-recent attention\.

## 4Autonomy\-of\-Heads

In this section, we will introduce the AoH method in detail\. Prior work commonly distinguishes retrieval heads, which support long\-range content lookup, from streaming heads, which mainly rely on sink and recent tokensXiaoet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib2)\); Linet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib3)\); Shaikhet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib8)\)\. AoH asks whether this distinction can be approximated without calibration prompts, runtime attention traces, or trained gates\. During decoding, for theii\-th query token with contextXctx∈ℝT×dmodelX\_\{\\text\{ctx\}\}\\in\\mathbb\{R\}^\{T\\times d\_\{\\text\{model\}\}\}, the attention scores for headhhare:333Here, we consider only the case without additions such as RoPE\. The ablation study in Section[6\.4](https://arxiv.org/html/2608.06849#S6.SS4.SSS0.Px5)shows that RoPE\-aware AoH produces highly consistent head rankings with vanilla AoH\.

scoresh,i⏟T×batch\-size=Xctx⏟T×dmodel⋅WKh⊤⏟dmodel×dhead⋅WQh⏟dhead×dmodel⋅xi⏟dmodel×batch\-size\\underbrace\{\\text\{scores\}\_\{h,i\}\}\_\{T\\times\\text\{batch\-size\}\}=\\underbrace\{X\_\{\\text\{ctx\}\}\}\_\{T\\times d\_\{\\text\{model\}\}\}\\cdot\\underbrace\{W\_\{K\}^\{h\\top\}\}\_\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{head\}\}\}\\cdot\\underbrace\{W\_\{Q\}^\{h\}\}\_\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{model\}\}\}\\cdot\\underbrace\{x\_\{i\}\}\_\{d\_\{\\text\{model\}\}\\times\\text\{batch\-size\}\}

\(1\)For the same input,XctxX\_\{\\text\{ctx\}\}andxix\_\{i\}are shared across heads; the head\-specific component is the middle operator\. We therefore define the kernel attention matrix:

Mh=WKh⊤​WQh∈ℝdmodel×dmodelM\_\{h\}=W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}\}\(2\)which acts as a frozen query\-key matching operator for headhh\.

The matrixMhM\_\{h\}connects query\-side information demand to key\-side information supply\. For each attention headhh, we compute the singular value decomposition

Mh=Uh​Σh​Vh⊤,M\_\{h\}=U\_\{h\}\\Sigma\_\{h\}V\_\{h\}^\{\\top\},\(3\)whereMh∈ℝdmodel×dmodelM\_\{h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}\}, andUh,Vh∈ℝdmodel×dmodelU\_\{h\},V\_\{h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}\}are orthogonal matrices\.Σh\\Sigma\_\{h\}is a diagonal matrix containing the singular values ofMhM\_\{h\}in descending order\. Letσh,1≥⋯≥σh,rh\>0\\sigma\_\{h,1\}\\geq\\cdots\\geq\\sigma\_\{h,r\_\{h\}\}\>0denote the nonzero singular values, whererh=rank⁡\(Mh\)r\_\{h\}=\\operatorname\{rank\}\(M\_\{h\}\)\. The right singular directions inVhV\_\{h\}represent query\-side directions that activate the head, whereas the left singular directions inUhU\_\{h\}represent key\-side directions that can be matched in the context\. The singular values inΣh\\Sigma\_\{h\}quantify the strength of these query\-key matching directions\. Letrhr\_\{h\}be the number of nonzero singular values of headhh, letσ^k\\hat\{\\sigma\}\_\{k\}denote the normalized weight of itskk\-th singular direction, and leteff​\_​rank⁡\(h\)\\operatorname\{eff\\\_rank\}\(h\)denote the effective rank of the head:

σ^k=σh,k∑j=1rhσh,j,k=1,…,rh,eff​\_​rank⁡\(h\)=exp⁡\(−∑k=1rhσ^k​log⁡σ^k\)∈\[1,rh\]\\begin\{split\}\\hat\{\\sigma\}\_\{k\}&=\\frac\{\\sigma\_\{h,k\}\}\{\\sum\_\{j=1\}^\{r\_\{h\}\}\\sigma\_\{h,j\}\},\\qquad k=1,\\ldots,r\_\{h\},\\\\ \\operatorname\{eff\\\_rank\}\(h\)&=\\exp\\left\(\-\\sum\_\{k=1\}^\{r\_\{h\}\}\\hat\{\\sigma\}\_\{k\}\\log\\hat\{\\sigma\}\_\{k\}\\right\)\\in\[1,r\_\{h\}\]\\end\{split\}\(4\)Here,eff​\_​rank⁡\(h\)∈\[1,rh\]\\operatorname\{eff\\\_rank\}\(h\)\\in\[1,r\_\{h\}\]is the exponential of the Shannon entropy of the normalized singular\-value distribution\. A low ER indicates a concentrated spectrum dominated by a few query\-key matching directions, whereas a high ER indicates a diffuse spectrum spread across many directions\. We therefore classify low\-ER heads as retrieval heads and high\-ER heads as streaming heads:

- •Retrieval Heads: low ER heads whose attention can be driven by a small number of content\-matching directions and may need to search globally over the context\.
- •Streaming Heads: high ER heads with more diffuse spectra and no small set of dominant global matching directions\. They are less likely to require content\-specific global retrieval\.

Optimized Implementation\. Directly constructingMh∈ℝdmodel×dmodelM\_\{h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}\}and performing SVD is unnecessary and expensive\. From Eq\. \([1](https://arxiv.org/html/2608.06849#S4.E1)\),

rank⁡\(Mh\)≤min⁡\(rank⁡\(WKh⊤\),rank⁡\(WQh\)\)≤dhead\\operatorname\{rank\}\(M\_\{h\}\)\\leq\\min\\\!\\left\(\\operatorname\{rank\}\(W\_\{K\}^\{h\\top\}\),\\operatorname\{rank\}\(W\_\{Q\}^\{h\}\)\\right\)\\leq d\_\{\\text\{head\}\}

AlthoughMhM\_\{h\}is admodel×dmodeld\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}matrix, its nonzero spectrum is limited by thedheadd\_\{\\text\{head\}\}bottleneck\. Using Sylvester’s determinant theoremKarapiperiet al\.\([2015](https://arxiv.org/html/2608.06849#bib.bib20)\),A​BABandB​ABAshare the same nonzero eigenvalues\. Therefore, the nonzero eigenvalues ofMh⊤​MhM\_\{h\}^\{\\top\}M\_\{h\}can be obtained from the smaller proxy

Ch=\(WQh​WQh⊤\)​\(WKh​WKh⊤\)∈ℝdhead×dhead,C\_\{h\}=\(W\_\{Q\}^\{h\}W\_\{Q\}^\{h\\top\}\)\(W\_\{K\}^\{h\}W\_\{K\}^\{h\\top\}\)\\in\\mathbb\{R\}^\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}\},\(5\)with

σk​\(Mh\)=λk​\(Ch\)\\sigma\_\{k\}\(M\_\{h\}\)=\\sqrt\{\\lambda\_\{k\}\(C\_\{h\}\)\}\(6\)The detailed derivation is presented in Appendix[L](https://arxiv.org/html/2608.06849#A12)\. This reduces the computation from operating on admodel×dmodeld\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}matrix to adhead×dheadd\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}eigenvalue problem, with costO​\(dhead2⋅dmodel\)O\(d\_\{\\text\{head\}\}^\{2\}\\cdot d\_\{\\text\{model\}\}\)\. For Qwen2\.5\-7B \(dhead=128d\_\{\\text\{head\}\}=128,dmodel=3584d\_\{\\text\{model\}\}=3584\), this is approximately4\.6×1074\.6\\times 10^\{7\}operations versus5\.9×10105\.9\\times 10^\{10\}for the full matrix construction\.

Group\-level Classification\. For each KV Groupgg, every constituent Query Headhhindependently evaluates its effective\-rank from its per\-head Kernel Attention MatrixMhM\_\{h\}, computed using the shared group key projectionWKgW\_\{K\}^\{g\}and its own query projectionWQhW\_\{Q\}^\{h\}\. We aggregate these per\-head ranks into a single group\-level score by taking the*mean*across all member heads in the group\. Within each layer, KV groups are then ranked by this score, and thek=⌈\(1−s\),G⌉k=\\lceil\(1\-s\),G\\rceilgroups with the lowest ER are labelled Retrieval Groups \(wheressis the target sparsity andGGthe number of KV groups\), while the remainder are Streaming Groups\. All Query Heads within a group inherit the same label, so the classification maps cleanly back to the Q\-Head level without ambiguity\.

Algorithm 1AoH Head Classification0:Frozen weights

\{WQ\(l\)​h,WK\(l\)​h\}\\\{W\_\{Q\}^\{\(l\)h\},W\_\{K\}^\{\(l\)h\}\\\}, layers

l∈\[L\]l\\in\[L\], heads

h∈\[H\]h\\in\[H\], budget

kk
0:Retrieval / Streaming head sets

\{ℛ\(l\),𝒮\(l\)\}l=1L\\\{\\mathcal\{R\}^\{\(l\)\},\\mathcal\{S\}^\{\(l\)\}\\\}\_\{l=1\}^\{L\}
1:for

l=1l=1to

LLdo

2:for

h=1h=1to

HHdo

3:

Q​Q=WQ\(l\)​h​\(WQ\(l\)​h\)⊤∈ℝdhead×dheadQQ=W\_\{Q\}^\{\(l\)h\}\(W\_\{Q\}^\{\(l\)h\}\)^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}\}
4:

K​K=WK\(l\)​h​\(WK\(l\)​h\)⊤∈ℝdhead×dheadKK=W\_\{K\}^\{\(l\)h\}\(W\_\{K\}^\{\(l\)h\}\)^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}\}
5:

Ch\(l\)=Q​Q⋅K​KC\_\{h\}^\{\(l\)\}=QQ\\cdot KK⊳λk​\(Ch\(l\)\)=σk2​\(Mh\(l\)\)\\triangleright\\ \\lambda\_\{k\}\(C\_\{h\}^\{\(l\)\}\)=\\sigma\_\{k\}^\{2\}\(M\_\{h\}^\{\(l\)\}\)

6:

σk=max⁡\(eig⁡\(Ch\(l\)\),0\)\\sigma\_\{k\}=\\sqrt\{\\max\(\\operatorname\{eig\}\(C\_\{h\}^\{\(l\)\}\),\\;0\)\}
7:

σ^k=σk∑jσj\\hat\{\\sigma\}\_\{k\}=\\frac\{\\sigma\_\{k\}\}\{\\sum\_\{j\}\\sigma\_\{j\}\}
8:

eff​\_​rank\(l\)⁡\(h\)=exp⁡\(−∑kσ^k​log⁡σ^k\)\\operatorname\{eff\\\_rank\}^\{\(l\)\}\(h\)=\\exp\\\!\\left\(\-\\sum\_\{k\}\\hat\{\\sigma\}\_\{k\}\\log\\hat\{\\sigma\}\_\{k\}\\right\)
9:endfor

10:

π\(l\)=argsorth⁡\[eff​\_​rank\(l\)⁡\(h\)\]\\pi^\{\(l\)\}=\\operatorname\{argsort\}\_\{h\}\[\\operatorname\{eff\\\_rank\}^\{\(l\)\}\(h\)\]⊳\\trianglerightascending order

11:

ℛ\(l\)=\{h∈\[H\]\|eff​\_​rank\(l\)⁡\(h\)≤eff​\_​rank\(l\)⁡\(πk\(l\)\)\}\\mathcal\{R\}^\{\(l\)\}=\\bigl\\\{h\\in\[H\]\\;\\big\|\\;\\operatorname\{eff\\\_rank\}^\{\(l\)\}\(h\)\\leq\\operatorname\{eff\\\_rank\}^\{\(l\)\}\(\\pi^\{\(l\)\}\_\{k\}\)\\bigr\\\}⊳\\trianglerightlowest\-kkeff\_rank⇒\\RightarrowRetrieval

12:

𝒮\(l\)=\[H\]∖ℛ\(l\)\\mathcal\{S\}^\{\(l\)\}=\[H\]\\setminus\\mathcal\{R\}^\{\(l\)\}⊳\\trianglerightremaining⇒\\RightarrowStreaming

13:endfor

## 5Deploying LLMs With AoH

Head Classification: Algorithm[1](https://arxiv.org/html/2608.06849#alg1)classifies each head as retrieval or streaming using the ER of its kernel attention matrix\. For headhhin layerll, we compute thedheadd\_\{\\text\{head\}\}\-dimensional proxyCh\(l\)=\(WQ\(l\)​h​WQ\(l\)​h⊤\)​\(WK\(l\)​h​WK\(l\)​h⊤\)C\_\{h\}^\{\(l\)\}=\(W\_\{Q\}^\{\(l\)h\}W\_\{Q\}^\{\(l\)h\\top\}\)\(W\_\{K\}^\{\(l\)h\}W\_\{K\}^\{\(l\)h\\top\}\), whose eigenvalues equal the squared nonzero singular values ofMh\(l\)M\_\{h\}^\{\(l\)\}, and evaluate

eff​\_​rank\(l\)⁡\(h\)=exp⁡\(−∑kλk​\(Ch\(l\)\)∑jλj​\(Ch\(l\)\)​log⁡λk​\(Ch\(l\)\)∑jλj​\(Ch\(l\)\)\)\\operatorname\{eff\\\_rank\}^\{\(l\)\}\(h\)=\\exp\\\!\\left\(\-\\sum\_\{k\}\\frac\{\\sqrt\{\\lambda\_\{k\}\(C\_\{h\}^\{\(l\)\}\)\}\}\{\\sum\_\{j\}\\sqrt\{\\lambda\_\{j\}\(C\_\{h\}^\{\(l\)\}\)\}\}\\log\\frac\{\\sqrt\{\\lambda\_\{k\}\(C\_\{h\}^\{\(l\)\}\)\}\}\{\\sum\_\{j\}\\sqrt\{\\lambda\_\{j\}\(C\_\{h\}^\{\(l\)\}\)\}\}\\right\)

\(7\)
Reordering: Before deployment, we preprocess the model by reordering the output channels of the Query, Key, and Value projection weights according to the attention heads, which are sorted by ER in ascending order and the lowest\-kkunits are designated retrieval heads\. The budgetkkcontrols the retrieval/streaming ratio\. This reordering groups retrieval heads and streaming heads, allowing for efficient slicing and concatenation operations when managing the KV cache for these two types of heads within a layer, rather than relying on scattering and gathering operations\. For GQA models, the same decision is lifted to KV\-group granularity to preserve grouped\-cache efficiency; details are in Section[K](https://arxiv.org/html/2608.06849#A11)\.

Decode Stage: Each layer applies its assigned strategy based on its head type, which uses layer\-specific full K/V states for Retrieval Heads and only a fixed\-size window cache plus sink cache for Streaming Heads\. Each head type is computed independently; their outputs are concatenated along the head dimension and projected through a shared output matrix:

o\(l\)=Concat⁡\(Attn⁡\(qℛ\(l\),KVℛ\(l\)\)⏟Full Attn\.,Attn⁡\(q𝒮\(l\),KV𝒮\(l\)\)⏟SWA\)​WO\(l\)o^\{\(l\)\}=\\operatorname\{Concat\}\\\!\\left\(\\underbrace\{\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{R\}\}^\{\(l\)\},\\mathrm\{KV\}\_\{\\mathcal\{R\}\}^\{\(l\)\}\\right\)\}\_\{\\text\{Full Attn\.\}\},\\underbrace\{\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{S\}\}^\{\(l\)\},\\mathrm\{KV\}\_\{\\mathcal\{S\}\}^\{\(l\)\}\\right\)\}\_\{\\text\{SWA\}\}\\right\)W\_\{O\}^\{\(l\)\}

\(8\)whereConcat⁡\(⋅\)\\operatorname\{Concat\}\(\\cdot\)denotes concatenation along the head axis, followed by reshaping todmodeld\_\{\\text\{model\}\}\.

Chunked\-Prefill Stage: AoH is compatible with standard chunked prefillAgrawalet al\.\([2023](https://arxiv.org/html/2608.06849#bib.bib38)\); Kwonet al\.\([2023](https://arxiv.org/html/2608.06849#bib.bib39)\)using FlashAttention\-2Dao \([2023](https://arxiv.org/html/2608.06849#bib.bib28)\)\. Retrieval Heads attend to the accumulated prefix and retain their layer\-specificO​\(T\)O\(T\)KV states\. For Streaming Heads, after processing each chunk of sizeKK, we evict all butssinks\_\{\\text\{sink\}\}sink tokens andsrecents\_\{\\text\{recent\}\}recent tokens\. Thus, each subsequent chunk attends to at mostssink\+srecents\_\{\\text\{sink\}\}\+s\_\{\\text\{recent\}\}tokens\. For a sequence of lengthTT, this reduces the prefill cost fromO​\(T2\)O\(T^\{2\}\)toO​\(T​K\)O\(TK\)for fixed cache sizes, while bounding persistent KV storage byO​\(ssink\+srecent\)O\(s\_\{\\text\{sink\}\}\+s\_\{\\text\{recent\}\}\)and peak working memory byO​\(K\)O\(K\)\.

This design requires no specialized kernels beyond standard chunked FlashAttention and remains compatible with batched serving, which can further enhance LLM efficiency in serving scenarios with large sizes\.

## 6Experiment

### 6\.1Setups

#### Datasets and models

We evaluate on the LongBenchBaiet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib18)\), a comprehensive long\-context understanding suite covering 21 tasks across six categories; Appendix[D](https://arxiv.org/html/2608.06849#A4)lists the task names\. Each task is evaluated using its official metric, and we report the arithmetic mean over all 21 tasks as the primary aggregate score\. The main models are Qwen2\.5\-7BYanget al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib9)\)\(28 layers, 28 Q\-heads, 4 KV\-heads, GQA group size 7\) and Qwen3\-8BYanget al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib30)\)\(36 layers, 32 Q\-heads, 8 KV\-heads\)\.

Table 1:KV Budget\-matched setting for LongBench evaluation\. H denotes Heads\. RH denotes retrieval heads, and SH denotes streaming heads\. Storage KV denotes the KV\-cache footprint, while attended KV denotes the KV entries participating in attention computation\.Table 2:LongBench results on Qwen3\-8B and Llama3\.1\-8B\-Instruct\. All AoH variants use a retrieval/streaming ratio of 50%\. Randomavgrandomly selects the same number of retrieval heads, and Reverse keeps the highest ER heads as retrieval heads\. AoH\-RoPE uses a RoPE\-aware ER score computed with relative RoPE rotations up toΔ=32​K\\Delta=32\\mathrm\{K\}\.![Refer to caption](https://arxiv.org/html/2608.06849v1/x5.png)Figure 5:AoH Prefill and Decoding Efficiency Across Context Lengths vs\. Full Attention\. Gray and red bars report AoH gains in the prefill and decode stages\. The top row shows latency speedup, and the bottom row shows KV\-cache memory reduction under 50% and 75% sparsity\.![Refer to caption](https://arxiv.org/html/2608.06849v1/x6.png)Figure 6:Performance comparison of three LLMs under different AoH sparsity\.Table 3:LongBench\-Qasper results of AoH sink\-size and recent\-window sensitivity on Llama\-3\.1\-8B\-Instruct\. Bold indicates the settings for our main experiment\.
#### Implementation Details and baseline

We implement AoH in PyTorchPaszkeet al\.\([2019](https://arxiv.org/html/2608.06849#bib.bib19)\)with RoPESuet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib32)\), RMSNormZhang and Sennrich \([2019](https://arxiv.org/html/2608.06849#bib.bib21)\), Eager AttentionVaswaniet al\.\([2017](https://arxiv.org/html/2608.06849#bib.bib40)\); Wolfet al\.\([2020](https://arxiv.org/html/2608.06849#bib.bib41)\)\. Unless otherwise stated, streaming heads use sink sizessink=128s\_\{\\text\{sink\}\}=128and recent\-window sizesrecent=256s\_\{\\text\{recent\}\}=256\. We compare AoH with Full Attention,H2​OH\_\{2\}OZhanget al\.\([2023](https://arxiv.org/html/2608.06849#bib.bib12)\), SnapKVLiet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib33)\), QuestTanget al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib34)\), DuoAttentionXiaoet al\.\([2025](https://arxiv.org/html/2608.06849#bib.bib2)\), and two controlled head\-selection baselines: Randomavgand Reverse\. For all baseline methods, we ensure a fair comparison under comparable KV\-cache costs\. Table[1](https://arxiv.org/html/2608.06849#S6.T1)summarizes the KV\-cache storage and attended\-KV budgets used by each method\. Except for Quest, which uses a small attended\-token budget but still stores the full KV cache, the sparse baselines are configured with comparable storage and attention budgets\. All LongBench evaluations use a maximum sequence length of 32,768 and greedy decoding\.

Table 4:Ablation of spectral metrics for head classification on LongBench with Llama\-3\.1\-8B\-Instruct at 50% sparsity\. ER denotes effective\-rank, the metric used by AoH\. The best result in each row isbolded\.Table 5:Ablation of GQA group\-level aggregation strategies on LongBench with Llama\-3\.1\-8B\-Instruct at 50% sparsity\. The best result isbolded\.Table 6:RoPE\-aware AoH ranking stability\. We report the mean layer\-wise Spearman correlation between vanilla AoH scores and RoPE\-aware AoH scores under different maximum relative distances\.

### 6\.2Main Results

Table[2](https://arxiv.org/html/2608.06849#S6.T2)reports LongBench results on Qwen3\-8B444[https://huggingface\.co/Qwen/Qwen3\-8B](https://huggingface.co/Qwen/Qwen3-8B)and Llama3\.1\-8B\-Instruct555[https://huggingface\.co/meta\-llama/Llama\-3\.1\-8B\-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), with Qwen2\.5\-7B666[https://huggingface\.co/Qwen/Qwen2\.5\-7B](https://huggingface.co/Qwen/Qwen2.5-7B)results provided in Appendix[F](https://arxiv.org/html/2608.06849#A6)\. With 50% sparsity, AoH achieves average scores of 38\.78 and 47\.55 on the two main models, respectively, remaining close to Full Attention while reducing KV budgets by half\. The gap to dense attention is small: AoH trails Full Attention by only 0\.32 points on Qwen3\-8B and 1\.10 points on Llama3\.1\-8B\. AoH also outperforms the sparse baselines in average score across all models\. The Random and Reverse ablations further validate the ER criterion: under the same sparsity budget, AoH substantially outperforms both random head selection and reversed ER selection\. These results show that low\-rank query\-key geometry identifies heads that are especially important for preserving long\-context performance\.

### 6\.3Efficiency Results

AoH reduces KV\-cache memory nearly proportionally to the fraction of streaming heads, and its latency benefits become more pronounced at longer contexts where attention and cache access dominate runtime\. As shown in Figure[5](https://arxiv.org/html/2608.06849#S6.F5)and Table[11](https://arxiv.org/html/2608.06849#A9.T11), at 75% sparsity, AoH achieves up to3\.24×3\.24\\timesprefill speedup and9\.14×9\.14\\timesdecode speedup at 256K context, while reducing KV\-cache memory by up to3\.98×3\.98\\timeson Llama3\.1\-8B\. Additional efficiency results on Qwen2\.5\-7B and Qwen3\-8B are provided in Appendix[H](https://arxiv.org/html/2608.06849#A8)\.

### 6\.4Ablation Studies

#### Ablation of head ordering\.

We first isolate whether AoH’s ER ordering provides a meaningful head\-function prior\. We compare AoH with two controlled alternatives under the same sparsity budget\. Randomavgselects the same number of retrieval heads uniformly at random, while Reverse flips the AoH ordering by treating high ER heads as retrieval heads\. Table[2](https://arxiv.org/html/2608.06849#S6.T2)shows the quality of the head\-ranking criterion\. We also show the detailed results in Section[I](https://arxiv.org/html/2608.06849#A9)\.

#### Ablation of hyperparameters\.

We ablate the main hyperparameters: the sparsity ratio and the streaming\-cache size\. As shown in Figure[6](https://arxiv.org/html/2608.06849#S6.F6), AoH achieves the best trade\-off between performance and computational overhead at around 50% sparsity\. We further study the sensitivity to sink size and recent\-window size\. As shown in Table[3](https://arxiv.org/html/2608.06849#S6.T3), AoH is relatively robust to these streaming\-cache hyperparameters across a range of sparsity levels\. We choose a sink size of 128 and a recent\-window size of 256, which provides a good balance between performance and KV\-cache reduction\.

#### Ablation of Kernel spectral metric\.

We further compare effective\-rankRoy and Vetterli \([2007](https://arxiv.org/html/2608.06849#bib.bib42)\)with three data\-free spectral baselines computed from the same frozen query\-key kernel matrix\. For each attention head, we formMh=WKh⊤​WQhM\_\{h\}=W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}, or equivalently compute its singular spectrum from the low\-dimensional Gram proxy\(WQh​WQh⊤\)​\(WKh​WKh⊤\)\(W\_\{Q\}^\{h\}W\_\{Q\}^\{h\\top\}\)\(W\_\{K\}^\{h\}W\_\{K\}^\{h\\top\}\)\. We then derive three alternative head\-selection scores: the Frobenius norm‖Mh‖F\\\|M\_\{h\}\\\|\_\{F\}, which measures the overall kernel energy; the spectral norm‖Mh‖2=σ1\\\|M\_\{h\}\\\|\_\{2\}=\\sigma\_\{1\}, which measures the strongest query\-key matching direction; and the stable rank‖Mh‖F2‖Mh‖22\\frac\{\\\|M\_\{h\}\\\|\_\{F\}^\{2\}\}\{\\\|M\_\{h\}\\\|\_\{2\}^\{2\}\}Rudelson and Vershynin \([2007](https://arxiv.org/html/2608.06849#bib.bib43)\), which estimates spectral dimensionality while emphasizing dominance by the largest singular value\. Under the same retrieval\-head budgets as AoH, we rank heads within each layer according to each metric and generate the corresponding retrieval/streaming heads\. As shown in Table[4](https://arxiv.org/html/2608.06849#S6.T4), effective\-rank \(Ours\) achieves the best average score\. This indicates that AoH benefits from measuring spectral concentration rather than simply selecting heads with large kernel magnitude or a single dominant singular direction\.

#### Ablation of GQA group\-level aggregation\.

We also ablate how query\-head scores are aggregated into GQA group\-level decisions\. As shown in Table[5](https://arxiv.org/html/2608.06849#S6.T5), mean and min aggregation perform nearly identically, with average scores of 47\.55 and 47\.60, respectively\. The small difference suggests that AoH is not sensitive to this choice\. In contrast, max aggregation is noticeably worse, dropping to 45\.82 on average\. This indicates that max aggregation is too coarse: a single high\-rank query head can dominate the group score and obscure lower\-rank retrieval\-oriented heads within the same KV group\. We therefore use mean aggregation as the default because it is stable, simple, and less sensitive to individual outlier heads\.

#### Ablation of AoH\-RoPE\.

Vanilla AoH computes the ER of each head from the frozen query\-key kernel without explicitly inserting positional rotations\. To examine whether RoPE changes the head\-level ordering used by AoH, we construct a RoPE\-aware variant\. For a relative distanceΔ\\Delta, we insert the RoPE relative rotationRΔR\_\{\\Delta\}into the head\-specific QK kernel and compute

Ch,Δ\(l\)=\(RΔ​WQ\(l\)​h​WQ\(l\)​h⊤​RΔ⊤\)​\(WK\(l\)​g​\(h\)​WK\(l\)​g​\(h\)⊤\)\\displaystyle C\_\{h,\\Delta\}^\{\(l\)\}=\\left\(R\_\{\\Delta\}W\_\{Q\}^\{\(l\)h\}W\_\{Q\}^\{\(l\)h\\top\}R\_\{\\Delta\}^\{\\top\}\\right\)\\left\(W\_\{K\}^\{\(l\)g\(h\)\}W\_\{K\}^\{\(l\)g\(h\)\\top\}\\right\)

\(9\)wherehhdenotes a query head in layerllandg​\(h\)g\(h\)denotes its corresponding KV group\. We compute the ER ofCh,Δ\(l\)C\_\{h,\\Delta\}^\{\(l\)\}as the RoPE\-aware AoH score\. For each maximum relative distanceDD, we average scores over logarithmically spacedΔ≤D\\Delta\\leq D, compare the resulting head ordering with vanilla AoH using layer\-wise Spearman correlation, and report the mean across layers\.

As shown in Table[6](https://arxiv.org/html/2608.06849#S6.T6), vanilla AoH and RoPE\-aware AoH produce highly consistent head rankings across all tested models and distance ranges\. The mean layer\-wise Spearman correlation remains above0\.980\.98even at 128K relative distance\. As shown in Table[2](https://arxiv.org/html/2608.06849#S6.T2), vanilla AoH performs comparably to AoH\-RoPE on both Qwen3\-8B and Llama\-3\.1\-8B\-Instruct\. This suggests that although RoPE changes the relative\-position phase of query\-key interactions, it largely preserves the head\-level spectral ordering exploited by AoH\.

## 7Conclusion

We presented Autonomy\-of\-Heads \(AoH\), a data\-free and training\-free method for identifying retrieval and streaming heads from frozen query\-key geometry\. By analyzing the effective\-rank of the kernel attention matrixMh=WKh⊤​WQhM\_\{h\}=W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}, AoH replaces calibration data and learned gates with a one\-time weight\-space classifier\. Across multiple long\-context LLMs, AoH preserves 96\.5% Full Attention performance on average at 50% sparsity and consistently outperforms random, reversed head selection and trainging\-free baselines\. On the hardware side, AoH achieves up to3\.24×3\.24\\timesprefill speedup and9\.14×9\.14\\timesdecode speedup at 256K context on Llama3\.1\-8B, while reducing KV\-cache memory by up to3\.98×3\.98\\timesunder 75% sparsity\. More broadly, AoH provides a simple and extensible head prior for sparse attention, and can be combined with existing KV\-cache compression or token\-selection methods to improve long\-context inference without additional training\.

## Limitations

Our evaluation focuses on decoder\-only LLMs with standard attention variants, including GQA\-based models\. Although the effective\-rank criterion is simple and architecture\-agnostic at the level of query\-key projections, we have not exhaustively tested it on encoder\-decoder models, multimodal LLMs, or models with heavily modified attention mechanisms\.

AoH uses a fixed sparsity budget to split heads into retrieval and streaming groups\. In this work, the budget is selected manually and kept constant across layers for simplicity\. A more adaptive budget, potentially varying by layer, model, or deployment constraint, may further improve the accuracy–efficiency trade\-off\. We leave automatic budget selection and finer\-grained head policies for future work\.

## References

- Sarathi: efficient llm inference by piggybacking decodes with chunked prefills\.arXiv preprint arXiv:2308\.16369\.Cited by:[§5](https://arxiv.org/html/2608.06849#S5.p4.10)\.
- J\. Ainslie, J\. Lee\-Thorp, M\. De Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. Sanghai \(2023\)Gqa: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4895–4901\.Cited by:[Appendix K](https://arxiv.org/html/2608.06849#A11.p1.3)\.
- Y\. Bai, Q\. Dong, T\. Jiang, X\. Lv, Z\. Du, A\. Zeng, J\. Tang, and J\. Li \(2026\)IndexCache: accelerating sparse attention via cross\-layer index reuse\.arXiv preprint arXiv:2603\.12201\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.06849#S2.SS3.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou,et al\.\(2024\)Longbench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 3119–3137\.Cited by:[Appendix D](https://arxiv.org/html/2608.06849#A4.p1.1),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px1.p1.1)\.
- I\. Beltagy, M\. E\. Peters, and A\. Cohan \(2020\)Longformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.Cited by:[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2)\.
- W\. Brandon, M\. Mishra, A\. Nrusimha, R\. Panda, and J\. Ragan\-Kelley \(2024\)Reducing transformer key\-value cache size with cross\-layer attention\.Advances in Neural Information Processing Systems37,pp\. 86927–86957\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.06849#S2.SS3.p1.1)\.
- Y\. Chen, G\. Wang, J\. Shang, S\. Cui, Z\. Zhang, T\. Liu, S\. Wang, Y\. Sun, D\. Yu, and H\. Wu \(2024\)NACL: a general and effective KV cache eviction framework for LLM at inference time\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7913–7926\.External Links:[Link](https://aclanthology.org/2024.acl-long.428/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.428)Cited by:[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2)\.
- J\. Cordonnier, A\. Loukas, and M\. Jaggi \(2020\)Multi\-head attention: collaborate instead of concatenate\.arXiv preprint arXiv:2006\.16362\.Cited by:[Appendix K](https://arxiv.org/html/2608.06849#A11.p1.3)\.
- T\. Dao \(2023\)Flashattention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[§J\.3](https://arxiv.org/html/2608.06849#A10.SS3.p1.3),[Appendix K](https://arxiv.org/html/2608.06849#A11.p4.4),[§5](https://arxiv.org/html/2608.06849#S5.p4.10)\.
- A\. L\. DeepSeek\-AI, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.\(2025\)DeepSeek\-v3\. 2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1)\.
- D\. Deshmukh, S\. Goyal, N\. Kwatra, and R\. Ramjee \(2025\)Kascade: a practical sparse attention method for long\-context llm inference\.arXiv preprint arXiv:2512\.16391\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.06849#S2.SS3.p1.1)\.
- Z\. Fu, W\. Song, Y\. Wang, X\. Wu, Y\. Zheng, Y\. Zhang, D\. Xu, X\. Wei, T\. Xu, and X\. Zhao \(2025\)Sliding window attention training for efficient large language models\.arXiv preprint arXiv:2502\.18845\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1)\.
- Y\. Gao, J\. Wei, Q\. Zhang, Y\. Cheng, S\. Chen, Z\. Tang, Z\. Jiang, Y\. Song, H\. Zhang, L\. Zhao,et al\.\(2026\)HySparse: a hybrid sparse attention architecture with oracle token selection and kv cache sharing\.arXiv preprint arXiv:2602\.03560\.Cited by:[§J\.2](https://arxiv.org/html/2608.06849#A10.SS2.p2.1),[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.06849#S2.SS3.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Appendix K](https://arxiv.org/html/2608.06849#A11.p1.3)\.
- A\. Karapiperi, M\. Redivo\-Zaglia, and M\. R\. Russo \(2015\)Generalizations of sylvester’s determinantal identity\.arXiv preprint arXiv:1503\.00519\.Cited by:[§4](https://arxiv.org/html/2608.06849#S4.p2.24)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§5](https://arxiv.org/html/2608.06849#S5.p4.10)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: llm knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 22947–22970\.External Links:[Document](https://dx.doi.org/10.52202/079017-0722),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/28ab418242603e0f7323e54185d19bde-Paper-Conference.pdf)Cited by:[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- G\. Lin, D\. Li, Z\. Chen, Y\. Shi, X\. Chen, B\. Hu, and M\. Zhang \(2026\)LycheeDecode: accelerating long\-context llm inference via hybrid\-head sparse decoding\.arXiv preprint arXiv:2602\.04541\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.06849#S2.SS2.p1.1),[§4](https://arxiv.org/html/2608.06849#S4.p1.3)\.
- A\. Mohtashami and M\. Jaggi \(2023\)Random\-access infinite context length for transformers\.Advances in Neural Information Processing Systems36,pp\. 54567–54585\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1)\.
- A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga,et al\.\(2019\)Pytorch: an imperative style, high\-performance deep learning library\.Advances in neural information processing systems32\.Cited by:[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- O\. Roy and M\. Vetterli \(2007\)The effective rank: a measure of effective dimensionality\.In2007 15th European signal processing conference,pp\. 606–610\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p4.4),[§6\.4](https://arxiv.org/html/2608.06849#S6.SS4.SSS0.Px3.p1.5)\.
- M\. Rudelson and R\. Vershynin \(2007\)Sampling from large matrices: an approach through geometric functional analysis\.Journal of the ACM \(JACM\)54\(4\),pp\. 21–es\.Cited by:[§6\.4](https://arxiv.org/html/2608.06849#S6.SS4.SSS0.Px3.p1.5)\.
- K\. Shaikh, A\. K\. Singh, R\. C\. Dsouza, and S\. Shiromani \(2026\)Linear predictability of attention heads in large language models\.arXiv preprint arXiv:2603\.13314\.Cited by:[§4](https://arxiv.org/html/2608.06849#S4.p1.3)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)Roformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- Y\. Sun, L\. Dong, Y\. Zhu, S\. Huang, W\. Wang, S\. Ma, Q\. Zhang, J\. Wang, and F\. Wei \(2024\)You only cache once: decoder\-decoder architectures for language models\.Advances in Neural Information Processing Systems37,pp\. 7339–7361\.Cited by:[§2\.3](https://arxiv.org/html/2608.06849#S2.SS3.p1.1)\.
- H\. Tang, Y\. Lin, J\. Lin, Q\. Han, D\. Ke, S\. Hong, Y\. Yao, and G\. Wang \(2025\)RazorAttention: efficient KV cache compression through retrieval heads\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tkiZQlL04w)Cited by:[§2\.2](https://arxiv.org/html/2608.06849#S2.SS2.p1.1)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)QUEST: query\-aware sparsity for efficient long\-context llm inference\.InICML,External Links:[Link](https://openreview.net/forum?id=KzACYw0MTV)Cited by:[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- E\. team at Anthropic \(2024\)Building effective agents\.https://www\.anthropic\.com/engineering/building\-effective\-agents\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1)\.
- K\. Team, Y\. Zhang, Z\. Lin, X\. Yao, J\. Hu, F\. Meng, C\. Liu, X\. Men, S\. Yang, Z\. Li,et al\.\(2025\)Kimi linear: an expressive, efficient attention architecture\.arXiv preprint arXiv:2510\.26692\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[Appendix H](https://arxiv.org/html/2608.06849#A8.SS0.SSS0.Px1.p1.2),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- Z\. Wan, X\. Wu, Y\. Zhang, Y\. Xin, C\. Tao, Z\. Zhu, X\. Wang, S\. Luo, J\. Xiong, L\. Wang,et al\.\(2024\)D2o: dynamic discriminative operations for efficient long\-context inference of large language models\.arXiv preprint arXiv:2406\.13035\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. Rush \(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Q\. Liu and D\. Schlangen \(Eds\.\),Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.emnlp-demos.6/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[Appendix H](https://arxiv.org/html/2608.06849#A8.SS0.SSS0.Px1.p1.2),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- G\. Xiao, J\. Tang, J\. Zuo, J\. Guo, S\. Yang, H\. Tang, Y\. Fu, and S\. Han \(2025\)DuoAttention: efficient long\-context llm inference with retrieval and streaming heads\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 37228–37253\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/5c1ddd2e59df46fd2aa85c833b1b36ed-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.06849#S2.SS2.p1.1),[§4](https://arxiv.org/html/2608.06849#S4.p1.3),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 21875–21895\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5e5fd18f863cbe6d8ae392a93fd271c9-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2)\.
- Xiaomi MiMo Team \(2026\)MiMo\-v2\.5\.Note:[https://huggingface\.co/collections/XiaomiMiMo/mimo\-v25](https://huggingface.co/collections/XiaomiMiMo/mimo-v25)Cited by:[Appendix K](https://arxiv.org/html/2608.06849#A11.p1.3)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix K](https://arxiv.org/html/2608.06849#A11.p1.3),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px1.p1.1)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[Appendix K](https://arxiv.org/html/2608.06849#A11.p1.3),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px1.p1.1)\.
- M\. Zaheer, G\. Guruganesh, A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontanon, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang, and A\. Ahmed \(2020\)Big bird: transformers for longer sequences\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 17283–17297\.Cited by:[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2)\.
- B\. Zhang and R\. Sennrich \(2019\)Root mean square layer normalization\.Advances in neural information processing systems32\.Cited by:[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.Cited by:[§1](https://arxiv.org/html/2608.06849#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06849#S2.SS1.p1.2),[§6\.1](https://arxiv.org/html/2608.06849#S6.SS1.SSS0.Px2.p1.4)\.

### 7\.1Appendices

## Appendix AFurther discussion on the kernel attention matrixMhM\_\{h\}

The head\-specific part of attention is encoded in the frozen query\-key projections\. During decoding, the attention score of headhhfor a query token can be written asscoresh,i=Xctx​WKh⊤​WQh​xi\\text\{scores\}\_\{h,i\}=X\_\{\\text\{ctx\}\}W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}x\_\{i\}, where the context matrixXctxX\_\{\\text\{ctx\}\}and query vectorxix\_\{i\}are shared across heads, while the middle operatorWKh⊤​WQhW\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}is head\-specific\. We therefore define thekernel attention matrixMh=WKh⊤​WQhM\_\{h\}=W\_\{K\}^\{h\\top\}W\_\{Q\}^\{h\}, a frozen query\-key matching operator whose spectral geometry summarizes how headhhmaps query\-side information demand to key\-side information supply\. Intuitively, the spectrum ofMhM\_\{h\}describes how many dominant query\-key matching directions a head relies on\. A concentrated spectrum indicates that a few stable matching directions dominate the head’s behavior, which is consistent with retrieval heads that search globally for relevant content\. A diffuse spectrum indicates that no small set of global matching directions dominates, which is consistent with streaming heads that mainly rely on sink and recent tokens\.

![Refer to caption](https://arxiv.org/html/2608.06849v1/x7.png)

Figure 7:Correlation between effective\-rank and empirical attention behavior\. Negative correlations indicate that higher\-ER heads are more local, while lower\-ER heads are more associated with long\-range attention\.
## Appendix BAdditional Observations

To test whether this association also appears beyond Qwen2\.5\-7B, we quantify it on Qwen3\-8B in Figure[7](https://arxiv.org/html/2608.06849#A1.F7)\. For each layer, we compute Spearman correlations across heads between effective\-rank and three empirical behavior metrics: average attended distance, attention entropy, and far\-token mass\. Distance and entropy exclude sink tokens, and far\-token mass measures attention probability assigned to tokens farther than 8K positions\. On Qwen3\-8B under 16K–64K contexts, effective\-rank is consistently negatively correlated with long\-range attention behavior, indicating that frozen query\-key geometry provides a meaningful signal of head function\.

## Appendix CAoH Forms the Upper Envelope

Shown in Figure[8](https://arxiv.org/html/2608.06849#A3.F8), AoH forms the upper envelope of the accuracy–sparsity trade\-off, with all baseline methods lying below the AoH curve under comparable sparsity budgets\.

![Refer to caption](https://arxiv.org/html/2608.06849v1/x8.png)Figure 8:Accuracy\-efficiency trade\-off of AoH\.
## Appendix DLongBench Introduction

We evaluate AoH on the LongBenchBaiet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib18)\), a comprehensive long\-context understanding suite covering 21 tasks across six categories: single\-document QA \(narrativeqa, qasper, multifieldqa\_en, multifieldqa\_zh\), multi\-document QA \(hotpotqa, 2wikimqa, musique, dureader\), summarization \(gov\_report, qmsum, multi\_news, vcsum\), few\-shot learning \(trec, triviaqa, lsht, samsum\), synthetic retrieval \(passage\_count, passage\_retrieval\_en, passage\_retrieval\_zh\), and code completion \(lcc, repobench\-p\)\.

## Appendix EDetails on Baseline Budgets

Table[7](https://arxiv.org/html/2608.06849#A5.T7)reports the persistent KV\-cache memory used by each baseline at the maximum 32K LongBench context length with BF16 KV caches\. Full Attention stores all KV states and defines the 100% reference budget\. Quest has the same KV storage footprint because it retains the full cache, although only a subset of entries participates in attention computation\. SnapKV andH2H\_\{2\}O allocate a 16K\-token cache for all heads, corresponding to roughly 50% of the full KV storage\. For head\-wise methods, DuoAttention and all AoH variants keep full 32K caches for 50% retrieval heads and only a 512\-token sink/recent cache for the remaining streaming heads, giving a storage budget of approximately 50\.6%\. Since AoH, AoH\-Random, and AoH\-Reverse use identical KV budgets, their differences in LongBench accuracy reflect the quality of the retrieval\-head assignment\.

Table 7:KV\-cache memory footprint under the 32K LongBench setting\. KV memory is computed for the maximum 32K context using BF16 KV caches\.
## Appendix FAdditional LongBench Results

Table[9](https://arxiv.org/html/2608.06849#A6.T9)shows the additional LongBench results on Qwen2\.5\-7B\. We also include a complementary MoE evaluation to examine whether AoH remains applicable beyond dense transformer models\.

#### Experimental setup for MoE models\.

We further evaluate AoH on LongBench using Qwen3\-30B\-A3B\-Instruct\-2507, a Mixture\-of\-Experts \(MoE\) model with 30B total parameters and 3B activated parameters per token\. Compared with dense models, MoE architectures introduce an additional architectural component through expert routing; however, AoH operates only on the attention layers, where it compresses and allocates the KV cache at the attention\-head level\. We keep all hyperparameters, including attention sink tokens and window size, identical to the default AoH configuration used in the main LongBench table, without MoE\-specific tuning\.

#### Reults for MoE models\.

As shown in Table[8](https://arxiv.org/html/2608.06849#A6.T8), AoH remains effective on the MoE model: with only 50% KV heads kept, it achieves an average LongBench score of 48\.79, close to the Full Attention score of 50\.95, while preserving comparable performance on the code category\.

![Refer to caption](https://arxiv.org/html/2608.06849v1/x9.png)Figure 9:Single\-needle passkey retrieval accuracy of AoH at 50% sparsity across context lengths and insertion depths\.![Refer to caption](https://arxiv.org/html/2608.06849v1/x10.png)Figure 10:Ablation of AoH head classification under different KV\-cache budgets on LongBench\. We compare AoH with Full Attention, Randomavg, and Reverse\.Table 8:LongBench results on Qwen3\-30B\-A3B\-Instruct\-2507 \(MoE\)\.Table 9:LongBench results on Qwen2\.5\-7B\. All AoH variants use a retrieval/streaming ratio of 0\.5 unless otherwise stated\.R​a​n​d​o​mavgRandom\_\{\\mathrm\{avg\}\}randomly selects the same number of retrieval heads, and Reverse keeps the highest effective\-rank heads as retrieval heads\. AoH\-RoPE\-aware uses a RoPE\-aware effective\-rank score computed with relative RoPE rotations up toΔ=32​K\\Delta=32\\mathrm\{K\}\. sp refers to the sparsity ratio:sparsity=1−NfullNtotal\\mathrm\{sparsity\}=1\-\\frac\{N\_\{\\mathrm\{full\}\}\}\{N\_\{\\mathrm\{total\}\}\}\.

## Appendix GAdditional PassKeyRetrieval Results

Figure[9](https://arxiv.org/html/2608.06849#A6.F9)shows the single\-needle passkey retrieval accuracy of AoH at 50% sparsity across context lengths from 1K to 128K and insertion depths from 0% to 100%\. AoH maintains near\-perfect retrieval accuracy across almost all settings, with degradation only at the most challenging 128K context and 100% insertion depth\.

## Appendix HAdditional Efficiency Results

#### Measurement protocol\.

To isolate the effect of head\-level sparsity from low\-level kernel engineering, the decode latencies reported here are measured under a deliberately simple attention backend\. Both Full and AoH use*eager*attentionVaswaniet al\.\([2017](https://arxiv.org/html/2608.06849#bib.bib40)\); Wolfet al\.\([2020](https://arxiv.org/html/2608.06849#bib.bib41)\), i\.e\. vanilla PyTorchsoftmax​\(Q​K⊤d\)​V\\mathrm\{softmax\}\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\}\}\)V\(with the softmax accumulated in FP32 and cast back to bf16\)\. All numbers are single\-sequence \(batchsize=1\\mathrm\{batchsize\}=1\), single\-token decode latencies in bf16\.

#### Results

Tables[11](https://arxiv.org/html/2608.06849#A9.T11),[13](https://arxiv.org/html/2608.06849#A9.T13), and[12](https://arxiv.org/html/2608.06849#A9.T12)provide the full efficiency results for AoH on LLaMA3\.1\-8B, Qwen3\-8B, and Qwen2\.5\-7B, including prefill latency, decode latency, and KV\-cache memory\.

## Appendix IAdditional Ablation Results

#### Ablation of head ordering\.

Figure[10](https://arxiv.org/html/2608.06849#A6.F10)shows that AoH consistently preserves stronger LongBench performance across task categories and KV\-cache budgets\. Randomavgperforms substantially worse, showing that sparse attention is sensitive to which heads retain global context\. Reverse is usually the weakest variant, confirming that the direction of the effective\-rank criterion matters: low\-rank query\-key geometry identifies retrieval\-oriented heads, whereas high\-rank heads are better treated as streaming heads\. Together with the aggregate results in Table[2](https://arxiv.org/html/2608.06849#S6.T2), this ablation shows the quality of the head\-ranking criterion\.

#### Hyperparameter sensitivity on ShortBench\.

We also use MMLU as a representative short\-context knowledge task\. Shown in Tables[10](https://arxiv.org/html/2608.06849#A9.T10), AoH is not sensitive to these hyperparameters\. On MMLU, accuracy is almost unchanged across all combinations and sparsity levels, showing clear robustness in the short\-context setting\.

Table 10:ShortBench\-MMLU accuracy of AoH sink\-size and recent\-window sensitivity on Llama\-3\.1\-8B\-Instruct\.Table 11:Prefill/decode latency and KV memory of AoH on LLaMA\-3\.1\-8B across sparsity levels and context lengths\. Speedups and memory reduction are relative to Full Attention\.Ctx LenPrefill Latency \(s, total\)Prefill Mem \(GB\)Decode Latency \(ms/tok\)Decode Mem \(GB\)FullAoHSpd↑\\uparrowFullAoHMem↑\\uparrowFullAoHSpd↑\\uparrowFullAoHMem↑\\uparrowLLaMA\-3\.1\-8B \(sparsity=25%\)4K1\.3820\.8701\.59×\\times0\.5370\.4151\.29×\\times22\.0011\.551\.90×\\times0\.5400\.4201\.29×\\times8K2\.6322\.1651\.22×\\times1\.0740\.8181\.31×\\times23\.6514\.451\.64×\\times1\.0760\.8201\.31×\\times16K6\.3075\.7031\.11×\\times2\.1481\.6231\.32×\\times31\.5520\.871\.51×\\times2\.1501\.6251\.32×\\times32K19\.59218\.0971\.08×\\times4\.2953\.2341\.33×\\times53\.2035\.241\.51×\\times4\.2983\.2361\.33×\\times64K68\.44762\.1241\.10×\\times8\.5906\.4551\.33×\\times99\.5362\.261\.60×\\times8\.5936\.4571\.33×\\times128K259\.748219\.3581\.18×\\times17\.18012\.8981\.33×\\times199\.55122\.471\.63×\\times17\.18312\.9001\.33×\\times256K1073\.530882\.5201\.22×\\times34\.36025\.7801\.33×\\times838\.80237\.203\.54×\\times34\.36225\.7841\.33×\\timesLLaMA\-3\.1\-8B \(sparsity=50%\)4K1\.3820\.8001\.73×\\times0\.5370\.2941\.83×\\times22\.0010\.602\.08×\\times0\.5400\.2901\.83×\\times8K2\.6321\.8611\.41×\\times1\.0740\.5621\.91×\\times23\.6512\.531\.89×\\times1\.0760\.5631\.91×\\times16K6\.3074\.6981\.34×\\times2\.1481\.0991\.95×\\times31\.5516\.911\.87×\\times2\.1501\.1001\.95×\\times32K19\.59213\.2501\.48×\\times4\.2952\.1731\.98×\\times53\.2026\.761\.99×\\times4\.2982\.1741\.98×\\times64K68\.44743\.5871\.57×\\times8\.5904\.3201\.99×\\times99\.5345\.212\.20×\\times8\.5934\.3211\.99×\\times128K259\.748154\.2251\.68×\\times17\.1808\.6151\.99×\\times199\.5585\.492\.33×\\times17\.1838\.6161\.99×\\times256K1073\.530615\.0201\.75×\\times34\.36017\.2002\.00×\\times838\.80162\.275\.17×\\times34\.36217\.2062\.00×\\timesLLaMA\-3\.1\-8B \(sparsity=75%\)4K1\.3820\.7591\.82×\\times0\.5370\.1723\.12×\\times22\.009\.642\.28×\\times0\.5400\.1703\.13×\\times8K2\.6321\.6111\.63×\\times1\.0740\.3063\.51×\\times23\.6510\.652\.22×\\times1\.0760\.3073\.51×\\times16K6\.3073\.7301\.69×\\times2\.1480\.5753\.74×\\times31\.5513\.002\.43×\\times2\.1500\.5753\.74×\\times32K19\.5929\.4272\.08×\\times4\.2951\.1123\.86×\\times53\.2018\.492\.88×\\times4\.2981\.1123\.86×\\times64K68\.44727\.1842\.52×\\times8\.5902\.1853\.93×\\times99\.5328\.503\.49×\\times8\.5932\.1863\.93×\\times128K259\.74890\.0842\.88×\\times17\.1804\.3333\.97×\\times199\.5549\.904\.00×\\times17\.1834\.3333\.97×\\times256K1073\.530331\.8403\.24×\\times34\.3608\.6303\.98×\\times838\.8091\.739\.14×\\times34\.3628\.6283\.98×\\times

Table 12:Prefill/decode latency and KV memory of AoH on Qwen2\.5\-7B across sparsity levels and context lengths\. Speedups and memory reduction are relative to Full Attention\. AoH decode latency uses eager two\-path attention with whole\-step CUDA\-graph capture\.Ctx LenPrefill Latency \(s, total\)Prefill Mem \(GB\)Decode Latency \(ms/tok\)Decode Mem \(GB\)FullAoHSpd↑\\uparrowFullAoHMem↓\\downarrowFullAoHSpd↑\\uparrowFullAoHMem↓\\downarrowQwen2\.5\-7B \(sparsity=25%\)4K1\.2730\.9471\.35×\\times0\.2350\.1821\.29×\\times19\.929\.912\.01×\\times0\.2360\.1831\.29×\\times8K2\.1151\.9751\.07×\\times0\.4700\.3581\.31×\\times21\.5212\.041\.79×\\times0\.4710\.3591\.31×\\times16K5\.0565\.1290\.99×\\times0\.9400\.7101\.32×\\times26\.6217\.091\.56×\\times0\.9410\.7111\.32×\\times32K16\.03114\.6731\.09×\\times1\.8791\.4151\.33×\\times43\.8327\.811\.58×\\times1\.8801\.4161\.33×\\times64K50\.91446\.7501\.09×\\times3\.7582\.8241\.33×\\times78\.8348\.381\.63×\\times3\.7592\.8251\.33×\\times128K191\.469169\.3051\.13×\\times7\.5165\.6431\.33×\\times160\.4494\.631\.70×\\times7\.5175\.6441\.33×\\times256K776\.550642\.6931\.21×\\times15\.03211\.2801\.33×\\times312\.31181\.921\.72×\\times15\.03411\.2811\.33×\\timesQwen2\.5\-7B \(sparsity=50%\)4K1\.2730\.7651\.66×\\times0\.2350\.1281\.83×\\times19\.929\.292\.14×\\times0\.2360\.1291\.83×\\times8K2\.1151\.7371\.22×\\times0\.4700\.2461\.91×\\times21\.5210\.861\.98×\\times0\.4710\.2461\.91×\\times16K5\.0565\.1090\.99×\\times0\.9400\.4811\.95×\\times26\.6214\.101\.89×\\times0\.9410\.4811\.95×\\times32K16\.03111\.3621\.41×\\times1\.8790\.9511\.98×\\times43\.8321\.752\.02×\\times1\.8800\.9511\.98×\\times64K50\.91434\.6701\.47×\\times3\.7581\.8901\.99×\\times78\.8335\.942\.19×\\times3\.7591\.8911\.99×\\times128K191\.469134\.3451\.43×\\times7\.5163\.7691\.99×\\times160\.4466\.982\.40×\\times7\.5173\.7701\.99×\\times256K776\.550456\.7761\.70×\\times15\.0327\.5272\.00×\\times312\.31127\.322\.45×\\times15\.0347\.5282\.00×\\timesQwen2\.5\-7B \(sparsity=75%\)4K1\.2730\.7681\.66×\\times0\.2350\.0753\.12×\\times19\.927\.612\.62×\\times0\.2360\.0763\.13×\\times8K2\.1151\.5431\.37×\\times0\.4700\.1343\.51×\\times21\.527\.682\.80×\\times0\.4710\.1343\.51×\\times16K5\.0563\.4401\.47×\\times0\.9400\.2513\.74×\\times26\.628\.323\.20×\\times0\.9410\.2523\.74×\\times32K16\.0318\.1611\.96×\\times1\.8790\.4863\.86×\\times43\.839\.384\.67×\\times1\.8800\.4873\.86×\\times64K50\.91421\.6752\.35×\\times3\.7580\.9563\.93×\\times78\.8310\.807\.30×\\times3\.7590\.9563\.93×\\times128K191\.46966\.2672\.89×\\times7\.5161\.8963\.96×\\times160\.4413\.6311\.77×\\times7\.5171\.8963\.97×\\times256K776\.550222\.1893\.50×\\times15\.0323\.7753\.98×\\times312\.3121\.5314\.50×\\times15\.0343\.7753\.98×\\times

Table 13:Prefill/decode latency and KV memory of AoH on Qwen3\-8B across sparsity levels and context lengths\. Speedups and memory reduction are relative to Full Attention\.Ctx LenPrefill Latency \(s, total\)Prefill Mem \(GB\)Decode Latency \(ms/tok\)Decode Mem \(GB\)FullAoHSpd↑\\uparrowFullAoHMem↓\\downarrowFullAoHSpd↑\\uparrowFullAoHMem↓\\downarrowQwen3\-8B \(sparsity=25%\)4K1\.3051\.0751\.21×\\times0\.6040\.4671\.29×\\times28\.8214\.052\.05×\\times0\.6070\.4691\.29×\\times8K2\.5112\.4921\.01×\\times1\.2080\.9201\.31×\\times28\.9017\.861\.62×\\times1\.2110\.9221\.31×\\times16K7\.0646\.6941\.06×\\times2\.4161\.8261\.32×\\times38\.4225\.461\.51×\\times2\.4191\.8281\.32×\\times32K22\.45820\.7011\.09×\\times4\.8323\.6381\.33×\\times64\.4040\.651\.58×\\times4\.8353\.6401\.33×\\times64K76\.03469\.0771\.10×\\times9\.6647\.2621\.33×\\times116\.4371\.291\.63×\\times9\.6677\.2641\.33×\\times128K292\.425257\.5931\.14×\\times19\.32714\.5101\.33×\\times235\.80144\.211\.64×\\times19\.33014\.5121\.33×\\times256K1155\.8981016\.4601\.14×\\times38\.65529\.0051\.33×\\times459\.90277\.591\.66×\\times38\.65829\.0071\.33×\\timesQwen3\-8B \(sparsity=50%\)4K1\.3051\.4730\.89×\\times0\.6040\.3301\.83×\\times28\.8212\.992\.22×\\times0\.6070\.3321\.83×\\times8K2\.5112\.3131\.09×\\times1\.2080\.6321\.91×\\times28\.9015\.621\.85×\\times1\.2110\.6341\.91×\\times16K7\.0645\.5811\.27×\\times2\.4161\.2361\.95×\\times38\.4220\.781\.85×\\times2\.4191\.2381\.95×\\times32K22\.45815\.7621\.42×\\times4\.8322\.4441\.98×\\times64\.4031\.112\.07×\\times4\.8352\.4461\.98×\\times64K76\.03451\.4371\.48×\\times9\.6644\.8601\.99×\\times116\.4351\.992\.24×\\times9\.6674\.8621\.99×\\times128K292\.425181\.2801\.61×\\times19\.3279\.6921\.99×\\times235\.80101\.452\.32×\\times19\.3309\.6941\.99×\\times256K1155\.898691\.5311\.67×\\times38\.65519\.3562\.00×\\times459\.90192\.292\.39×\\times38\.65819\.3572\.00×\\timesQwen3\-8B \(sparsity=75%\)4K1\.3050\.8731\.50×\\times0\.6040\.1933\.12×\\times28\.8211\.892\.42×\\times0\.6070\.1943\.13×\\times8K2\.5111\.9001\.32×\\times1\.2080\.3453\.51×\\times28\.9013\.342\.17×\\times1\.2110\.3453\.51×\\times16K7\.0644\.4081\.60×\\times2\.4160\.6463\.74×\\times38\.4216\.122\.38×\\times2\.4190\.6473\.74×\\times32K22\.45811\.4781\.96×\\times4\.8321\.2503\.86×\\times64\.4021\.822\.95×\\times4\.8351\.2513\.86×\\times64K76\.03432\.8132\.32×\\times9\.6642\.4583\.93×\\times116\.4333\.133\.51×\\times9\.6672\.4593\.93×\\times128K292\.425105\.4512\.77×\\times19\.3274\.8743\.96×\\times235\.8059\.713\.95×\\times19\.3304\.8753\.97×\\times256K1155\.898384\.7653\.00×\\times38\.6559\.7063\.98×\\times459\.90108\.874\.22×\\times38\.6589\.7073\.98×\\times

## Appendix JAoH\-Guided Sparse Attention: H2Share

![Refer to caption](https://arxiv.org/html/2608.06849v1/x11.png)

Figure 11:Overview of H2Share\.Left: Offline preprocessing classifies heads with AoH, partitions layers into anchor–reuse blocks, and builds cross\-layer head mappings\.Right: During decoding, anchor retrieval heads compute full attention and select important\-token indices; reuse retrieval heads borrow only these indices and attend to their own layer\-specific keys and values at the mapped positions; streaming heads use sliding\-window attention\.AoH has the strongest scalability and configurability; therefore, in this section, we introduce an AoH\-guided cross\-layer shared sparse Attention structure, H2Share\. Building on AoH, H2Share exploits both head heterogeneity and adjacent\-layer stability through three offline steps: head or KV\-group classification, anchor–reuse block assignment, and cross\-layer head/group mapping\. The online policy is then determined by the layer role and head type\. All offline steps use frozen weights only; no training or calibration prompts are required\. Figure[11](https://arxiv.org/html/2608.06849#A10.F11)summarizes the architecture\. Importantly, H2Share is an*index\-only*sharing method: the only object transferred across layers is a set of integer token positions, never key/value vectors\.

### J\.1Head Classification

Algorithm[1](https://arxiv.org/html/2608.06849#alg1)classifies each head as retrieval or streaming using the effective\-rank of its kernel attention matrix\. For headhhin layerll, we compute thedheadd\_\{\\text\{head\}\}\-dimensional proxyCh\(l\)=\(WQ\(l\)​h​WQ\(l\)​h⊤\)​\(WK\(l\)​h​WK\(l\)​h⊤\)C\_\{h\}^\{\(l\)\}=\(W\_\{Q\}^\{\(l\)h\}W\_\{Q\}^\{\(l\)h\\top\}\)\(W\_\{K\}^\{\(l\)h\}W\_\{K\}^\{\(l\)h\\top\}\), whose eigenvalues equal the squared nonzero singular values ofMh\(l\)M\_\{h\}^\{\(l\)\}, and evaluate

eff​\_​rank\(l\)⁡\(h\)=exp⁡\(−∑k=1rhσ^k​log⁡σ^k\),where​σ^k=λk​\(Ch\(l\)\)∑j=1rhλj​\(Ch\(l\)\),k=1,…,rh\.\\begin\{split\}\\operatorname\{eff\\\_rank\}^\{\(l\)\}\(h\)&=\\exp\\\!\\left\(\-\\sum\_\{k=1\}^\{r\_\{h\}\}\\hat\{\\sigma\}\_\{k\}\\log\\hat\{\\sigma\}\_\{k\}\\right\),\\\\ \\text\{where \}\\hat\{\\sigma\}\_\{k\}&=\\frac\{\\sqrt\{\\lambda\_\{k\}\(C\_\{h\}^\{\(l\)\}\)\}\}\{\\sum\_\{j=1\}^\{r\_\{h\}\}\\sqrt\{\\lambda\_\{j\}\(C\_\{h\}^\{\(l\)\}\)\}\},\\quad k=1,\\ldots,r\_\{h\}\.\\end\{split\}\(10\)For each layer, heads are sorted by effective\-rank in ascending order and the lowest\-kkunits are designated retrieval heads\. The budgetkkcontrols the retrieval/streaming ratio\. For GQA models, the same decision is lifted to KV\-group granularity to preserve grouped\-cache efficiency; details are in Section[K](https://arxiv.org/html/2608.06849#A11)\.

### J\.2Anchor–Reuse Blocks and Head\-Mapped Index Reuse

Anchor–reuse blocks\.We partition theLLtransformer layers into consecutive, non\-overlapping blocks of sizeBB\. The first layer of each block is theAnchor Layer; the remainingB−1B\{\-\}1layers areReuse Layers\. In an Anchor Layer, AoH\-identified retrieval heads compute global attention and select top\-kkimportant\-token indices according to their attention scores\. These indices are made available to Reuse Layers in the same block\. Streaming heads do not participate in cross\-layer reuse and are served by sink tokens plus a recent\-window cache\.

Index\-only reuse\.Unlike HySparseGaoet al\.\([2026](https://arxiv.org/html/2608.06849#bib.bib1)\), which shares both token selection and KV cache from a preceding full\-attention layer, H2Share transfers only selected token indices\. Each Reuse Layer computes its own keys and values, then its retrieval heads attend sparsely to the layer\-specific K/V states at the borrowed positions\. This design preserves the pretrained layer\-specific projection geometry while still reducing attention computation\. The shared\-KV variant is used only as a negative ablation in Figure[12](https://arxiv.org/html/2608.06849#A10.F12), where it collapses due to cross\-layer projection\-space mismatch\.

![Refer to caption](https://arxiv.org/html/2608.06849v1/x12.png)Figure 12:Ablation Experiments on LongBench\. H2Share offers a better KV budget\-accuracy trade\-off\.Head mapping\.Naively giving every Reuse Retrieval Head the indices selected by the same\-indexed Anchor Head assumes that head indices have identical functions across adjacent layers\. This assumption is unreliable\. We instead map each Reuse Retrieval Headh′∈ℛ\(l′\)h^\{\\prime\}\\in\\mathcal\{R\}^\{\(l^\{\\prime\}\)\}to the functionally closest Retrieval Headh∈ℛ\(l\)h\\in\\mathcal\{R\}^\{\(l\)\}in the Anchor Layer using the normalized Frobenius inner product between their kernel attention matrices:

ℳh,h′\(l→l′\)=⟨Mh\(l\),Mh′\(l′\)⟩F‖Mh\(l\)‖F⋅‖Mh′\(l′\)‖F,h∈\{1,…,H\}\(l\),h′∈\{1,…,H\}\(l′\),ℳ\(l→l′\)∈ℝH×Hℳh,h′\(l→l′\)=tr⁡\(WQ\(l\)​h​WQ\(l′\)​h′⊤⋅WK\(l′\)​h′​WK\(l\)​h⊤\)tr⁡\(WQ\(l\)​h​WQ\(l\)​h⊤⋅WK\(l\)​h​WK\(l\)​h⊤\)⋅tr⁡\(WQ\(l′\)​h′​WQ\(l′\)​h′⊤⋅WK\(l′\)​h′​WK\(l′\)​h′⊤\)\.\\begin\{split\}\\mathcal\{M\}^\{\(l\\to l^\{\\prime\}\)\}\_\{h,h^\{\\prime\}\}&=\\frac\{\\left\\langle M\_\{h\}^\{\(l\)\},\\,M\_\{h^\{\\prime\}\}^\{\(l^\{\\prime\}\)\}\\right\\rangle\_\{F\}\}\{\\left\\\|M\_\{h\}^\{\(l\)\}\\right\\\|\_\{F\}\\cdot\\left\\\|M\_\{h^\{\\prime\}\}^\{\(l^\{\\prime\}\)\}\\right\\\|\_\{F\}\},\\quad h\\in\\\{1,\\ldots,H\\\}^\{\(l\)\},\\;h^\{\\prime\}\\in\\\{1,\\ldots,H\\\}^\{\(l^\{\\prime\}\)\},\\quad\\mathcal\{M\}^\{\(l\\to l^\{\\prime\}\)\}\\in\\mathbb\{R\}^\{H\\times H\}\\\\\[8\.0pt\] \\mathcal\{M\}^\{\(l\\to l^\{\\prime\}\)\}\_\{h,h^\{\\prime\}\}&=\\frac\{\\operatorname\{tr\}\\\!\\left\(W\_\{Q\}^\{\(l\)h\}\\,W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\\cdot W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}\\,W\_\{K\}^\{\(l\)h\\top\}\\right\)\}\{\\sqrt\{\\operatorname\{tr\}\\\!\\left\(W\_\{Q\}^\{\(l\)h\}\\,W\_\{Q\}^\{\(l\)h\\top\}\\cdot W\_\{K\}^\{\(l\)h\}\\,W\_\{K\}^\{\(l\)h\\top\}\\right\)\}\\cdot\\sqrt\{\\operatorname\{tr\}\\\!\\left\(W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}\\,W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\\cdot W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}\\,W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\\right\)\}\}\.\\end\{split\}\(11\)By the cyclic invariance of the trace, all quantities in Eq\. \([11](https://arxiv.org/html/2608.06849#A10.E11)\) are computed efficiently indheadd\_\{\\text\{head\}\}space without ever forming thedmodel×dmodeld\_\{\\text\{model\}\}\{\\times\}d\_\{\\text\{model\}\}matrixMhM\_\{h\}; the detailed proof is in Appendix[L](https://arxiv.org/html/2608.06849#A12)\. Each Reuse Retrieval Head uses the important\-token indices selected by its best\-matched Anchor Retrieval Head\.

Inference\.Letℛ\(l\)\\mathcal\{R\}^\{\(l\)\}and𝒮\(l\)\\mathcal\{S\}^\{\(l\)\}denote the retrieval and streaming heads in layerll\. An Anchor Layer computes global attention for retrieval heads and sliding\-window attention for streaming heads:

o\(l\)=\[Attn⁡\(qℛ\(l\),Kℛ\(l\),Vℛ\(l\)\)\|SWA\(q𝒮\(l\),K𝒮\(l\),V𝒮\(l\)\)\]WO\(l\)\.\\begin\{split\}o^\{\(l\)\}=\\Big\[&\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{R\}\}^\{\(l\)\},K\_\{\\mathcal\{R\}\}^\{\(l\)\},V\_\{\\mathcal\{R\}\}^\{\(l\)\}\\right\)\\\\ &\\Big\|\\;\\operatorname\{SWA\}\\\!\\left\(q\_\{\\mathcal\{S\}\}^\{\(l\)\},K\_\{\\mathcal\{S\}\}^\{\(l\)\},V\_\{\\mathcal\{S\}\}^\{\(l\)\}\\right\)\\Big\]W\_\{O\}^\{\(l\)\}\.\\end\{split\}\(12\)It records the selected indicesℐh\(l\)\\mathcal\{I\}\_\{h\}^\{\(l\)\}for each Anchor Retrieval Head\. For a Reuse Layerl′l^\{\\prime\}, each Retrieval Headh′h^\{\\prime\}obtains indices from its mapped Anchor Headm​\(h′\)m\(h^\{\\prime\}\)and attends only to its own layer\-specific K/V states at those positions:

o\(l′\)=\[Attn⁡\(qℛ\(l′\),Kℛ\(l′\)​\[ℐm​\(⋅\)\(l\)\],Vℛ\(l′\)​\[ℐm​\(⋅\)\(l\)\]\)\|SWA\(q𝒮\(l′\),K𝒮\(l′\),V𝒮\(l′\)\)\]WO\(l′\)\.\\begin\{split\}o^\{\(l^\{\\prime\}\)\}=\\Big\[&\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{R\}\}^\{\(l^\{\\prime\}\)\},K\_\{\\mathcal\{R\}\}^\{\(l^\{\\prime\}\)\}\[\\mathcal\{I\}\_\{m\(\\cdot\)\}^\{\(l\)\}\],V\_\{\\mathcal\{R\}\}^\{\(l^\{\\prime\}\)\}\[\\mathcal\{I\}\_\{m\(\\cdot\)\}^\{\(l\)\}\]\\right\)\\\\ &\\Big\|\\;\\operatorname\{SWA\}\\\!\\left\(q\_\{\\mathcal\{S\}\}^\{\(l^\{\\prime\}\)\},K\_\{\\mathcal\{S\}\}^\{\(l^\{\\prime\}\)\},V\_\{\\mathcal\{S\}\}^\{\(l^\{\\prime\}\)\}\\right\)\\Big\]W\_\{O\}^\{\(l^\{\\prime\}\)\}\.\\end\{split\}\(13\)Thus, H2Share reuses cross\-layer information only at the level of discrete important\-token positions; all key/value vectors remain layer\-specific\.

### J\.3Inference Phase

Prefill Stage:WQ\(l\),WK\(l\),WV\(l\)W\_\{Q\}^\{\(l\)\},W\_\{K\}^\{\(l\)\},W\_\{V\}^\{\(l\)\}are reordered offline along the head dimension so that Retrieval and Streaming Heads form contiguous slices, making all head\-type splits simple tensor indexing with no gather overhead\. Retrieval Heads process the full prompt via standard FlashAttention\-2Dao \([2023](https://arxiv.org/html/2608.06849#bib.bib28)\), maintaining layer\-specificO​\(T\)O\(T\)KV states\. Streaming Heads adopt chunked prefilling: the prompt is divided into fixed\-size chunks, and after each chunk, the KV cache is immediately trimmed to retain only sink tokens and the most recent window, bounding memory toO​\(ssink\+srecent\)O\(s\_\{\\text\{sink\}\}\+s\_\{\\text\{recent\}\}\)regardless of text length\.

Decode Stage: Each layer applies its assigned strategy based on its role \(Anchor or Reuse\) and head type \(Retrieval or Streaming\)\. An Anchor Layerlluses layer\-specific full K/V states for Retrieval Heads and a fixed\-size window cache for Streaming Heads\. Each head type is computed independently; their outputs are concatenated along the head dimension and projected through a shared output matrix:

o\(l\)=\[Attn⁡\(qℛ\(l\),KVℛ\(l\)\)⏟Full Attention,Attn⁡\(q𝒮\(l\),KV𝒮\(l\)\)⏟SlidingWindowAttention\]WO\(l\)\\begin\{split\}o^\{\(l\)\}=\\left\[\\\!\\vphantom\{\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{R\}\}^\{\(l\)\},\\;\\mathrm\{KV\}\_\{\\mathcal\{R\}\}^\{\(l\)\}\\right\)\}\\right\.&\\underbrace\{\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{R\}\}^\{\(l\)\},\\;\\mathrm\{KV\}\_\{\\mathcal\{R\}\}^\{\(l\)\}\\right\)\}\_\{\\text\{Full Attention\}\},\\\\ &\\left\.\\underbrace\{\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{S\}\}^\{\(l\)\},\\;\\mathrm\{KV\}\_\{\\mathcal\{S\}\}^\{\(l\)\}\\right\)\}\_\{\\text\{SlidingWindowAttention\}\}\\\!\\right\]W\_\{O\}^\{\(l\)\}\\end\{split\}\(14\)whereConcat⁡\(⋅\)\\operatorname\{Concat\}\(\\cdot\)denotes concatenation along the head axis, followed by reshaping todmodeld\_\{\\text\{model\}\}\.

After processing the full context, the Anchor Layer records the top\-kktoken indicesℐh\(l\)\\mathcal\{I\}\_\{h\}^\{\(l\)\}per Retrieval Headhhfor reuse within the block\. A Reuse Layerl′l^\{\\prime\}borrows only these integer indices via the head mappingℳ\(l→l′\)\\mathcal\{M\}^\{\(l\\to l^\{\\prime\}\)\}and computes its own K/V states independently: each Retrieval Head computes sparse attention over the mappedkkpositions, while Streaming Heads compute sliding\-window attention independently\.

o\(l′\)=\[Attn⁡\(qℛ\(l′\),KVℛ\(l′\)​\[ℐℳ\(l→l′\)​\(⋅\)\(l\)\]\)⏟Sparse Attention \(borrowed indices, own KV\),Attn⁡\(q𝒮\(l′\),KV𝒮\(l′\)\)⏟SWA\]WO\(l′\)\\begin\{split\}o^\{\(l^\{\\prime\}\)\}=\\left\[\\\!\\vphantom\{\\left\[\\mathcal\{I\}\_\{\\mathcal\{M\}^\{\(l\\to l^\{\\prime\}\)\}\(\\cdot\)\}^\{\(l\)\}\\right\]\}\\right\.&\\underbrace\{\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{R\}\}^\{\(l^\{\\prime\}\)\},\\;\\mathrm\{KV\}\_\{\\mathcal\{R\}\}^\{\(l^\{\\prime\}\)\}\\\!\\left\[\\mathcal\{I\}\_\{\\mathcal\{M\}^\{\(l\\to l^\{\\prime\}\)\}\(\\cdot\)\}^\{\(l\)\}\\right\]\\right\)\}\_\{\\text\{Sparse Attention \(borrowed indices, own KV\)\}\},\\\\ &\\left\.\\underbrace\{\\operatorname\{Attn\}\\\!\\left\(q\_\{\\mathcal\{S\}\}^\{\(l^\{\\prime\}\)\},\\;\\mathrm\{KV\}\_\{\\mathcal\{S\}\}^\{\(l^\{\\prime\}\)\}\\right\)\}\_\{\\text\{SWA\}\}\\\!\\right\]W\_\{O\}^\{\(l^\{\\prime\}\)\}\\end\{split\}\(15\)whereKVℛ\(l′\)​\[ℐℳ\(l→l′\)​\(⋅\)\(l\)\]\\mathrm\{KV\}\_\{\\mathcal\{R\}\}^\{\(l^\{\\prime\}\)\}\\\!\\left\[\\mathcal\{I\}\_\{\\mathcal\{M\}^\{\(l\\to l^\{\\prime\}\)\}\(\\cdot\)\}^\{\(l\)\}\\right\]denotes the Reuse Layer’s own K/V states indexed by the top\-kkpositions borrowed from the mapped Anchor Head\. The borrowed object is only the discrete index set; no Anchor\-Layer K/V vector is transferred\. For Retrieval Heads, the attention scan is restricted toO​\(k\)O\(k\)positions; for Streaming Heads, a fixed\-size sliding\-window KV cache is dynamically maintained\.

## Appendix KGQA Extension

Our proposed solutions are all based on Multi\-head Attention \(MHA\)Cordonnieret al\.\([2020](https://arxiv.org/html/2608.06849#bib.bib37)\), but modern LLMsXiaomi MiMo Team \([2026](https://arxiv.org/html/2608.06849#bib.bib31)\); Grattafioriet al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib17)\); Yanget al\.\([2024](https://arxiv.org/html/2608.06849#bib.bib9),[2025](https://arxiv.org/html/2608.06849#bib.bib30)\)increasingly adopt Grouped Query Attention \(GQA\)Ainslieet al\.\([2023](https://arxiv.org/html/2608.06849#bib.bib29)\), whereHHQuery Heads are grouped intoGGKV Groups, withHG\\frac\{H\}\{G\}Query Heads sharing a single KV Head within each group\. Therefore, the retention decision must also be made at the KV\-Group level\. If a classification strategy targeting individual query heads were adopted, partitioning the KV cache within a group would lose the storage efficiency of GQA\.

Group\-level Classification: For each KV Groupgg, every constituent Query Headhhindependently evaluates its effective\-rank from its per\-head Kernel Attention MatrixMhM\_\{h\}, computed using the shared group key projectionWKgW\_\{K\}^\{g\}and its own query projectionWQhW\_\{Q\}^\{h\}\. We aggregate these per\-head ranks into a single group\-level score by taking the*mean*across all member heads in the group\. Within each layer, KV groups are then ranked by this score, and thek=⌈\(1−s\),G⌉k=\\lceil\(1\-s\),G\\rceilgroups with the lowest effective\-rank are labelled Retrieval Groups \(wheressis the target sparsity andGGthe number of KV groups\), while the remainder are Streaming Groups\. All Query Heads within a group inherit the same label, so the classification maps cleanly back to the Q\-Head level without ambiguity\.

Group\-level Head Mapping: The cross\-layer semantic similarity matrix reduced fromH×HH\\times H\(MHA\) toG×GG\\times G\(GQA\), with one entry per KV Group pair\. For each group pair, the pairwise Q\-Head similarities are aggregated \(mean\) to produce a single group\-to\-group score\. The mapping then assigns each Reuse Retrieval Group to its most similar Anchor Retrieval Group, and all Query Heads within that group borrow the same set of top\-kktoken indices\.

Weight Reordering: Offline permutation operates at two granularities simultaneously:WK\(l\)W\_\{K\}^\{\(l\)\}andWV\(l\)W\_\{V\}^\{\(l\)\}are reordered along the KV\-Group axis according to the group classification permutation, whileWQ\(l\)W\_\{Q\}^\{\(l\)\}is reordered along the Q\-Head axis by expanding each group index into itsHG\\frac\{H\}\{G\}constituent Q\-Head indices\. After reordering, Retrieval and Streaming Groups remain contiguous in memory, preserving full compatibility with FlashAttention ’sDao \([2023](https://arxiv.org/html/2608.06849#bib.bib28)\)native GQA kernel support with no additional gather overhead\.

## Appendix LProof: Trace Reduction todheadd\_\{\\text\{head\}\}Space

We prove that the Frobenius inner product⟨Mh\(l\),Mh′\(l′\)⟩F\\langle M\_\{h\}^\{\(l\)\},\\,M\_\{h^\{\\prime\}\}^\{\(l^\{\\prime\}\)\}\\rangle\_\{F\}and the Frobenius norms‖Mh\(l\)‖F\\\|M\_\{h\}^\{\(l\)\}\\\|\_\{F\}in Eq\. \([11](https://arxiv.org/html/2608.06849#A10.E11)\) can be computed entirely indheadd\_\{\\text\{head\}\}space, without ever constructing thedmodel×dmodeld\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}kernel attention matrixMhM\_\{h\}\.

### Setup

Recall the kernel attention matrix for headhhat layerll:

Mh\(l\)=WK\(l\)​h⊤​WQ\(l\)​h∈ℝdmodel×dmodel,WQ\(l\)​h,WK\(l\)​h∈ℝdhead×dmodel\.M\_\{h\}^\{\(l\)\}=W\_\{K\}^\{\(l\)h\\top\}W\_\{Q\}^\{\(l\)h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}\},\\qquad W\_\{Q\}^\{\(l\)h\},\\,W\_\{K\}^\{\(l\)h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{model\}\}\}\.\(16\)Sincerank⁡\(Mh\(l\)\)≤dhead≪dmodel\\operatorname\{rank\}\(M\_\{h\}^\{\(l\)\}\)\\leq d\_\{\\text\{head\}\}\\ll d\_\{\\text\{model\}\}, direct construction ofMh\(l\)M\_\{h\}^\{\(l\)\}requiresO​\(dmodel2⋅dhead\)O\(d\_\{\\text\{model\}\}^\{2\}\\cdot d\_\{\\text\{head\}\}\)operations\. We show that both quantities reduce to traces ofdhead×dheadd\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}matrices\.

### Frobenius Inner Product

###### Proposition 1\.

⟨Mh\(l\),Mh′\(l′\)⟩F=tr⁡\(WQ\(l\)​h​WQ\(l′\)​h′⊤⋅WK\(l′\)​h′​WK\(l\)​h⊤\)\.\\displaystyle\\langle M\_\{h\}^\{\(l\)\},\\,M\_\{h^\{\\prime\}\}^\{\(l^\{\\prime\}\)\}\\rangle\_\{F\}=\\operatorname\{tr\}\\\!\\left\(W\_\{Q\}^\{\(l\)h\}\\,W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\\cdot W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}\\,W\_\{K\}^\{\(l\)h\\top\}\\right\)\.

###### Proof\.

By definition of the Frobenius inner product and the trace identity⟨A,B⟩F=tr⁡\(A⊤​B\)\\langle A,B\\rangle\_\{F\}=\\operatorname\{tr\}\(A^\{\\top\}B\):

⟨Mh\(l\),Mh′\(l′\)⟩F\\displaystyle\\langle M\_\{h\}^\{\(l\)\},\\,M\_\{h^\{\\prime\}\}^\{\(l^\{\\prime\}\)\}\\rangle\_\{F\}=tr⁡\(Mh\(l\)⊤​Mh′\(l′\)\)\\displaystyle=\\operatorname\{tr\}\\\!\\left\(\{M\_\{h\}^\{\(l\)\}\}^\{\\top\}M\_\{h^\{\\prime\}\}^\{\(l^\{\\prime\}\)\}\\right\)=tr⁡\(WQ\(l\)​h⊤⏟dmodel×dhead​WK\(l\)​h⏟dhead×dmodel​WK\(l′\)​h′⊤⏟dmodel×dhead​WQ\(l′\)​h′⏟dhead×dmodel\)\.\\displaystyle=\\operatorname\{tr\}\\\!\\left\(\\underbrace\{W\_\{Q\}^\{\(l\)h\\top\}\}\_\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{head\}\}\}\\underbrace\{W\_\{K\}^\{\(l\)h\}\}\_\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{model\}\}\}\\underbrace\{W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\}\_\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{head\}\}\}\\underbrace\{W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}\}\_\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{model\}\}\}\\right\)\.\(17\)The trace in \([17](https://arxiv.org/html/2608.06849#A12.E17)\) operates on admodel×dmodeld\_\{\\text\{model\}\}\\times d\_\{\\text\{model\}\}matrix, which is expensive\. We apply the*cyclic invariance*of the trace,tr⁡\(A​B​C​D\)=tr⁡\(D​A​B​C\)\\operatorname\{tr\}\(ABCD\)=\\operatorname\{tr\}\(DABC\), with

A=WQ\(l\)​h⊤,B=WK\(l\)​h,C=WK\(l′\)​h′⊤,D=WQ\(l′\)​h′\.A=W\_\{Q\}^\{\(l\)h\\top\},\\quad B=W\_\{K\}^\{\(l\)h\},\\quad C=W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\},\\quad D=W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}\.CyclingDDto the front:

tr⁡\(A​B​C​D\)=tr⁡\(D​A​B​C\)\\displaystyle\\operatorname\{tr\}\(ABCD\)=\\operatorname\{tr\}\(DABC\)=tr⁡\(WQ\(l′\)​h′​WQ\(l\)​h⊤⏟dhead×dhead⋅WK\(l\)​h​WK\(l′\)​h′⊤⏟dhead×dhead\)\.\\displaystyle=\\operatorname\{tr\}\\\!\\left\(\\underbrace\{W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}W\_\{Q\}^\{\(l\)h\\top\}\}\_\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}\}\\cdot\\underbrace\{W\_\{K\}^\{\(l\)h\}W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\}\_\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}\}\\right\)\.\(18\)Finally, usingtr⁡\(A\)=tr⁡\(A⊤\)\\operatorname\{tr\}\(A\)=\\operatorname\{tr\}\(A^\{\\top\}\)on \([18](https://arxiv.org/html/2608.06849#A12.E18)\):

=tr⁡\(WQ\(l\)​h​WQ\(l′\)​h′⊤⋅WK\(l′\)​h′​WK\(l\)​h⊤\),=\\operatorname\{tr\}\\\!\\left\(W\_\{Q\}^\{\(l\)h\}W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\\cdot W\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}W\_\{K\}^\{\(l\)h\\top\}\\right\),which is the trace of adhead×dheadd\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}matrix product\. This completes the proof\. ∎

### Frobenius Norm

###### Corollary 1\.

‖Mh\(l\)‖F2=tr⁡\(WQ\(l\)​h​WQ\(l\)​h⊤⋅WK\(l\)​h​WK\(l\)​h⊤\)\.\\displaystyle\\\|M\_\{h\}^\{\(l\)\}\\\|\_\{F\}^\{2\}=\\operatorname\{tr\}\\\!\\left\(W\_\{Q\}^\{\(l\)h\}\\,W\_\{Q\}^\{\(l\)h\\top\}\\cdot W\_\{K\}^\{\(l\)h\}\\,W\_\{K\}^\{\(l\)h\\top\}\\right\)\.

###### Proof\.

Settingl′=ll^\{\\prime\}=landh′=hh^\{\\prime\}=hin Proposition[1](https://arxiv.org/html/2608.06849#Thmproposition1)gives

‖Mh\(l\)‖F2=⟨Mh\(l\),Mh\(l\)⟩F=tr⁡\(WQ\(l\)​h​WQ\(l\)​h⊤⋅WK\(l\)​h​WK\(l\)​h⊤\)\.∎\\\|M\_\{h\}^\{\(l\)\}\\\|\_\{F\}^\{2\}=\\langle M\_\{h\}^\{\(l\)\},\\,M\_\{h\}^\{\(l\)\}\\rangle\_\{F\}=\\operatorname\{tr\}\\\!\\left\(W\_\{Q\}^\{\(l\)h\}W\_\{Q\}^\{\(l\)h\\top\}\\cdot W\_\{K\}^\{\(l\)h\}W\_\{K\}^\{\(l\)h\\top\}\\right\)\.\\qed

### Complexity

Both quantities now require computing twodhead×dheadd\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}matrices and taking their trace:

- •WQ\(l\)​h​WQ\(l′\)​h′⊤∈ℝdhead×dheadW\_\{Q\}^\{\(l\)h\}W\_\{Q\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}\}: costO​\(dhead2⋅dmodel\)O\(d\_\{\\text\{head\}\}^\{2\}\\cdot d\_\{\\text\{model\}\}\)\.
- •WK\(l′\)​h′​WK\(l\)​h⊤∈ℝdhead×dheadW\_\{K\}^\{\(l^\{\\prime\}\)h^\{\\prime\}\}W\_\{K\}^\{\(l\)h\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}\}: costO​\(dhead2⋅dmodel\)O\(d\_\{\\text\{head\}\}^\{2\}\\cdot d\_\{\\text\{model\}\}\)\.
- •Trace ofdhead×dheadd\_\{\\text\{head\}\}\\times d\_\{\\text\{head\}\}product: costO​\(dhead3\)O\(d\_\{\\text\{head\}\}^\{3\}\)\.

This is a reduction fromO​\(dmodel3\)O\(d\_\{\\text\{model\}\}^\{3\}\)\(direct construction \+ inner product ofMhM\_\{h\}\) toO​\(dhead2⋅dmodel\)O\(d\_\{\\text\{head\}\}^\{2\}\\cdot d\_\{\\text\{model\}\}\)\. For Qwen3\-8B \(dhead=128d\_\{\\text\{head\}\}=128,dmodel=4096d\_\{\\text\{model\}\}=4096\), this yields a factor of≈dmodel/dhead=32×\\approx\\\!d\_\{\\text\{model\}\}/d\_\{\\text\{head\}\}=32\\timesspeedup\.