Recall Before You Rank: Similarity-Guided Top-$K$ Reuse for Efficient Long-Context Attention

# Similarity-Guided Top-𝐾 Reuse for Efficient Long-Context Attention
Source: [https://arxiv.org/html/2607.27692](https://arxiv.org/html/2607.27692)
## Recall Before You Rank: Similarity\-Guided Top\-KKReuse for Efficient Long\-Context Attention

Wenshuai Yao1\\equalcontrib, Wenyong Zhou2\\equalcontrib, Hanyong Shao1, Yizhe Chen3, Zhiyuan Ning1, Yuannuo Feng3, Ru Huang1\\corresponding, Kechao Tang1\\corresponding

###### Abstract

Top\-KKsparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key–value \(KV\) entries\. However, identifying this subset still requires scoring the current query against the full KV cache and performing global Top\-KKselection, leaving selector cost linear in context length and limiting the practical efficiency of sparse attention for long\-context decoding\. In this paper, we introduceReTopK, a training\-free method that accelerates dynamic Top\-KKattention by reusing historical retrieval decisions\. ReTopK builds on the observation that similar queries often attend to overlapping supports and that partially overlapping supports can still preserve most of the Exact Top\-KKattention mass\. For each attention head, it maintains a bounded cache of historical query–support pairs, retrieves the most similar cached queries for each new query, unions their stored supports with a recent window, and reranks only the resulting compact candidate set using exact current\-query scores\. A similarity\-based fallback invokes full\-history Exact Top\-KKwhen reuse is unreliable, while periodic exact refreshes limit cache drift\. ReTopK retains the complete KV cache and reuses only selected indices, rather than historical scores, attention weights, or outputs\. Across 16K–128K contexts, ReTopK achieves the lowest PG19 perplexity and the highest NIAH and LongBench scores among the evaluated approximate methods\. At 128K withK=512K=512, ReTopK incurs only a 0\.50% perplexity increase over Exact Top\-KKwhile accelerating attention computation by3\.07×3\.07\\times\.

## Introduction

Large language models \(LLMs\) increasingly process long documents, code repositories, and extended interaction histories\. During autoregressive decoding, however, the cost of attention grows with the key–value \(KV\) cache: each new query must compare against a growing history of keys and aggregate the corresponding values\. Although optimized kernels such as FlashAttention\(Daoet al\.[2022](https://arxiv.org/html/2607.27692#bib.bib1); Dao[2024](https://arxiv.org/html/2607.27692#bib.bib15)\)improve hardware efficiency, they do not remove the linear dependence of decoding attention on context length\.

Top\-KKsparse attention offers a promising alternative by restricting Softmax and value aggregation to theKKhighest\-scoring KV entries\. Yet, conventional Exact Top\-KKattention must still score the current query against the entire KV cache and perform global Top\-KKselection before sparse aggregation can begin\. Thus, it sparsifies attention aggregation but not index discovery\. As shown in Figure[1](https://arxiv.org/html/2607.27692#Sx1.F1), full\-history query–key \(QK\) scoring and Top\-KKselection dominate the latency of Exact Top\-KKattention, accounting for an increasingly large fraction of runtime as the context grows\. Consequently, efficient long\-context sparse attention requires reducing not only the cost of attending to selected entries, but also the cost of identifying them\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x1.png)Figure 1:Long\-context attention latency: \(a\) Exact Top\-KKbreakdown atK=512K=512, with full\-history QK and Top\-KKselection outlined; \(b\) per\-head BF16 latency\.Existing approaches address this challenge from complementary directions\. KV\-cache reduction and streaming methods bound attention cost by retaining a limited subset of tokens\(Xiaoet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib2); Zhanget al\.[2023](https://arxiv.org/html/2607.27692#bib.bib3)\), but discarded positions are unavailable if they become relevant later\. Query\-aware methods preserve more dynamic access to the context through page\-level pruning, compressed representations, or approximate retrieval\(Tanget al\.[2024](https://arxiv.org/html/2607.27692#bib.bib6); Singhaniaet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib16)\); however, their selection procedures typically still inspect a search space that grows with context length\. This leaves an important question:*can a dynamic selector recover query\-specific attention supports without searching the full history at every decoding step?*

Our answer is motivated by a simple observation: similar queries often induce overlapping Top\-KKsupports, and even incomplete support overlap can preserve most of the attention mass\. Historical query–support pairs therefore provide a compact record of prior retrieval decisions\. Rather than recomputing a full\-history Top\-KKsupport for every query, the selector can first recall supports associated with similar historical queries and then rerank only their union\.

In this paper, we introduceReTopK, a training\-free method that accelerates dynamic Top\-KKattention through retrieval\-decision reuse\. ReTopK maintains a bounded query–support cache independently for each attention head\. For a new query, it retrieves the most similar cached queries, unions their stored Top\-KKsupports with a recent\-token window, and reranks this compact candidate set using exact current\-query scores\. The resulting support is then used for sparse Softmax–value aggregation\. ReTopK uses historical index selections only as candidates: it does not reuse historical attention scores, weights, or outputs\. To avoid unreliable reuse, it falls back to full\-history Exact Top\-KKwhen cached\-query similarity is low and periodically refreshes the cache with exact supports\. Our contributions are as follows:

- •We identify full\-history index discovery as a central bottleneck of Exact Top\-KKattention and show that query similarity provides an effective signal for reusing sparse attention supports\.
- •We propose ReTopK, a training\-free recall\-before\-rerank method that combines a per\-head query–support cache, similarity\-guided candidate construction, exact candidate reranking, similarity\-based fallback, and periodic refresh\.
- •We develop a fused GPU implementation and evaluate ReTopK across multiple models and long\-context tasks\. At 128K withK=512K=512, ReTopK incurs only a 0\.50% perplexity increase over Exact Top\-KKwhile accelerating attention computation by3\.07×3\.07\\times\.

## Related Work

#### Efficient attention and Exact Top\-KKattention\.

IO\-aware attention kernels, including FlashAttention and FlashAttention\-2, improve the efficiency of dense attention through tiling, fusion, and reduced memory traffic\(Daoet al\.[2022](https://arxiv.org/html/2607.27692#bib.bib1); Dao[2024](https://arxiv.org/html/2607.27692#bib.bib15)\)\. However, autoregressive decoding still requires each query to interact with a KV cache whose length grows with the context\. Exact Top\-KKattention reduces the cost of Softmax–value aggregation by retaining only the globally highest\-scoringKKentries, but it must first compute QK scores over the full history and perform global Top\-KKselection\. ReTopK targets this remaining index\-discovery cost: it approximates the support\-selection process while retaining exact current\-query scoring within the recalled candidate set\.

#### KV\-cache reduction and structured sparsity\.

A complementary line of work bounds attention cost by reducing or organizing the active KV cache\. Streaming and eviction methods retain attention sinks, recent tokens, or tokens selected by accumulated importance\(Xiaoet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib2); Liuet al\.[2023](https://arxiv.org/html/2607.27692#bib.bib11); Zhanget al\.[2023](https://arxiv.org/html/2607.27692#bib.bib3)\)\. Other approaches compress the cache using observation\-window signals, layer\-dependent budgets, or head\-specific policies\(Liet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib4); Caiet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib5); Xiaoet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib18)\)\. Structured sparse methods further exploit recurring attention patterns to accelerate long\-context computation\(Jianget al\.[2024](https://arxiv.org/html/2607.27692#bib.bib12)\)\. These approaches provide bounded memory or predictable computation, but may restrict access to positions excluded from the active cache or sparse pattern\. In contrast, ReTopK preserves the complete KV cache without eviction, enabling query\-dependent retrieval from the full history\.

#### Query\-aware sparse selection\.

Query\-aware methods reduce attention cost by estimating relevant context from the current query\. SparQ estimates key relevance from selected query dimensions, Quest prunes KV pages with query\-dependent bounds, and Loki searches a low\-rank key space\(Ribaret al\.[2024](https://arxiv.org/html/2607.27692#bib.bib7); Tanget al\.[2024](https://arxiv.org/html/2607.27692#bib.bib6); Singhaniaet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib16)\)\. RetrievalAttention indexes KV entries, Squeezed Attention retrieves key clusters, and SeerAttention predicts sparse structures with learned gates\(Liuet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib13); Hooperet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib17); Gaoet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib19)\)\. TokenSelect caches a selection for similar consecutive queries\(Wuet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib14)\)\. Recent methods reuse sparse selections across layers or decoding steps: Kascade and IndexCache share Top\-KKindices across layers, while PRR overlaps temporal reuse with exact selection and repairs missed blocks\(Deshmukhet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib23); Baiet al\.[2026](https://arxiv.org/html/2607.27692#bib.bib24); Wanget al\.[2026](https://arxiv.org/html/2607.27692#bib.bib25)\)\. ReTopK instead retrieves multiple supports from a bounded per\-head query cache and reranks their union with current\-query scores, avoiding a full\-history scan on reuse steps; fallback and refresh maintain reliability without training or KV eviction\.

## Methodology

### Motivation: Query Similarity and Support Reuse

We begin by examining whether historical attention decisions can help identify the important positions for a new query\. We consider one attention layer and one query head, and omit these indices throughout this section\. Letqtq\_\{t\}denote the query at decoding steptt, and let\{ki\}i=1Lt\\\{k\_\{i\}\\\}\_\{i=1\}^\{L\_\{t\}\}be the keys currently available in the KV cache\. The Exact Top\-KKsupport is

St=TopKi∈\{1,…,Lt\}⁡\(qt⊤​kid,K\),S\_\{t\}=\\operatorname\{TopK\}\_\{i\\in\\\{1,\\ldots,L\_\{t\}\\\}\}\\left\(\\frac\{q\_\{t\}^\{\\top\}k\_\{i\}\}\{\\sqrt\{d\}\},K\\right\),\(1\)whereStS\_\{t\}contains the indices of theKKlargest scores\.

For two queries at positionsttandjj, we measure their normalized support overlap as

Overlap⁡\(t,j\)=\|St∩Sj\|K\.\\operatorname\{Overlap\}\(t,j\)=\\frac\{\\left\|S\_\{t\}\\cap S\_\{j\}\\right\|\}\{K\}\.\(2\)Given theRRmost similar preceding queries\{j1,…,jR\}\\\{j\_\{1\},\\ldots,j\_\{R\}\\\}, we further define the union of their supports and its coverage of the current Exact Top\-KKsupport as

Ut​\(R\)=⋃r=1RSjr,Coveraget⁡\(R\)=\|St∩Ut​\(R\)\|K\.U\_\{t\}\(R\)=\\bigcup\_\{r=1\}^\{R\}S\_\{j\_\{r\}\},\\;\\operatorname\{Coverage\}\_\{t\}\(R\)=\\frac\{\\left\|S\_\{t\}\\cap U\_\{t\}\(R\)\\right\|\}\{K\}\.\(3\)
![Refer to caption](https://arxiv.org/html/2607.27692v1/x2.png)Figure 2:Query similarity versus Exact Top\-KKsupport overlap: \(a\) all historical pairs; \(b\) each query’s most similar predecessor; \(c\) coverage by the union ofRRretrieved supports\.Figure[2](https://arxiv.org/html/2607.27692#Sx3.F2)shows that query cosine similarity is strongly correlated with Exact Top\-KKsupport overlap\. This relation holds both across historical query pairs and when each query is matched to its most similar preceding query\. Although the support of a single historical query is incomplete, unioning supports from multiple similar queries substantially improves coverage\.

Reuse quality is not uniform across attention heads\. As shown in Figure[3](https://arxiv.org/html/2607.27692#Sx3.F3), some heads achieve near\-complete coverage from retrieved supports, whereas others exhibit substantially lower coverage\. Within each head, however, coverage remains relatively stable over consecutive decoding steps\. This head\-dependent behavior motivates maintaining retrieval decisions independently for each query head\. It also shows that reuse should not be forced universally: a practical method requires a mechanism to identify and recover from unreliable reuse\.

These observations lead to three design principles\. First, retrieval decisions should be stored together with their originating queries, so that query similarity can serve as a lightweight retrieval signal\. Second, recalled supports should be treated as candidates rather than final selections, and should be reranked under the current query\. Third, reuse must be paired with safeguards for low\-confidence and stale retrieval decisions\. ReTopK implements these principles through a bounded per\-head query–support cache, exact reranking over recalled candidates, similarity\-based fallback, and periodic refresh\.

### Problem Setup and ReTopK Overview

At decoding steptt, letqt∈ℝdq\_\{t\}\\in\\mathbb\{R\}^\{d\}denote the current query, and let\{\(ki,vi\)\}i=1Lt\\\{\(k\_\{i\},v\_\{i\}\)\\\}\_\{i=1\}^\{L\_\{t\}\}denote the KV cache\. Exact Top\-KKattention computes a score for every cached key,

et,i=qt⊤​kid,i∈\{1,…,Lt\},e\_\{t,i\}=\\frac\{q\_\{t\}^\{\\top\}k\_\{i\}\}\{\\sqrt\{d\}\},\\qquad i\\in\\\{1,\\ldots,L\_\{t\}\\\},\(4\)selects the full\-history supportStS\_\{t\}in Equation[1](https://arxiv.org/html/2607.27692#Sx3.E1), and performs Softmax–value aggregation over the selected entries\. Although this aggregation involves onlyKKentries, obtainingStS\_\{t\}still requires full\-history QK scoring and global Top\-KKselection\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x3.png)Figure 3:Head\-wise Exact Top\-KKsupport coverage on Qwen2\.5\-7B at 16K \(K=512K=512,R=8R=8\): \(a\) layer–head means; \(b–d\) representative decoding traces\.On reuse steps, ReTopK replaces this full\-history selection with a*recall\-before\-rerank*procedure\. It first retrieves historical supports associated with queries similar toqtq\_\{t\}, then constructs a compact candidate set from these supports and a recent window\. ReTopK computes exact current\-query scores only for the candidate keys and selects the final Top\-KKentries within this set\. When reuse is unreliable, it reverts to Exact Top\-KKattention\.

Figure[4](https://arxiv.org/html/2607.27692#Sx3.F4)compares this procedure with Exact Top\-KKattention\. Exact Top\-KKscores allLtL\_\{t\}keys at every step, whereas ReTopK searches only a bounded cache of historical queries and reranksMt≪LtM\_\{t\}\\ll L\_\{t\}candidate keys on the reuse path\. Importantly, ReTopK reuses only historical*index selections*; it does not reuse historical QK scores, attention probabilities, or attention outputs\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x4.png)Figure 4:\(a\) Full\-history Exact Top\-KKselection\. \(b\) ReTopK recall, candidate construction, and reranking\. Dashed boxes and lines denote unselected entries and connections\.
### Recall\-Before\-Rerank Selection

#### Query–support cache\.

For each query head, ReTopK maintains a bounded FIFO cache of capacityCC:

ℬt=\{\(q¯c,S^c\)\}c=1\|ℬt\|,q¯c=qc‖qc‖2,\\mathcal\{B\}\_\{t\}=\\left\\\{\\left\(\\bar\{q\}\_\{c\},\\hat\{S\}\_\{c\}\\right\)\\right\\\}\_\{c=1\}^\{\|\\mathcal\{B\}\_\{t\}\|\},\\qquad\\bar\{q\}\_\{c\}=\\frac\{q\_\{c\}\}\{\\\|q\_\{c\}\\\|\_\{2\}\},\(5\)whereq¯c\\bar\{q\}\_\{c\}is the normalized query representation after positional encoding, andS^c\\hat\{S\}\_\{c\}is the support selected at the corresponding historical step\. The cache is initialized from the finalCCprompt tokens during prefill and updated after each decoding step, whether ReTopK takes the reuse or exact path\. The cache adds bounded metadata to the original KV cache: it stores normalized queries and selected indices, while preserving the complete set of keys and values\.

#### Similarity\-guided recall\.

For the normalized current queryq¯t=qt/‖qt‖2\\bar\{q\}\_\{t\}=q\_\{t\}/\\\|q\_\{t\}\\\|\_\{2\}, ReTopK computes cosine similarities to all cached queries:

ρt,c=q¯t⊤​q¯c,c∈\{1,…,\|ℬt\|\}\.\\rho\_\{t,c\}=\\bar\{q\}\_\{t\}^\{\\top\}\\bar\{q\}\_\{c\},\\qquad c\\in\\\{1,\\ldots,\|\\mathcal\{B\}\_\{t\}\|\\\}\.\(6\)It retrieves the indices of theRRmost similar cache entries,

ℛt=TopRc∈\{1,…,\|ℬt\|\}⁡\(ρt,c,R\),ρtmax=maxc⁡ρt,c\.\\mathcal\{R\}\_\{t\}=\\operatorname\{TopR\}\_\{c\\in\\\{1,\\ldots,\|\\mathcal\{B\}\_\{t\}\|\\\}\}\\left\(\\rho\_\{t,c\},R\\right\),\\;\\rho\_\{t\}^\{\\max\}=\\max\_\{c\}\\rho\_\{t,c\}\.\(7\)Unlike Exact Top\-KK, this recall stage comparesqtq\_\{t\}with onlyCCcached queries rather than allLtL\_\{t\}historical keys\.

#### Candidate construction and reranking\.

The supports associated with the retrieved cache entries are merged with a recent\-token window,

ℒt=\{max⁡\(1,Lt−W\+1\),…,Lt\},\\mathcal\{L\}\_\{t\}=\\left\\\{\\max\(1,L\_\{t\}\-W\+1\),\\ldots,L\_\{t\}\\right\\\},\(8\)whereWWis the window size\. The resulting candidate set is

𝒜t=Unique⁡\(⋃c∈ℛtS^c∪ℒt\),Mt=\|𝒜t\|\.\\mathcal\{A\}\_\{t\}=\\operatorname\{Unique\}\\left\(\\bigcup\_\{c\\in\\mathcal\{R\}\_\{t\}\}\\hat\{S\}\_\{c\}\\;\\cup\\;\\mathcal\{L\}\_\{t\}\\right\),\\qquad M\_\{t\}=\|\\mathcal\{A\}\_\{t\}\|\.\(9\)The recent window ensures that newly appended tokens remain eligible even if they do not appear in any retrieved support\.

ReTopK then computes exact current\-query QK scores only for candidate keys:

e^t,i=qt⊤​kid,i∈𝒜t,\\hat\{e\}\_\{t,i\}=\\frac\{q\_\{t\}^\{\\top\}k\_\{i\}\}\{\\sqrt\{d\}\},\\qquad i\\in\\mathcal\{A\}\_\{t\},\(10\)and selects the reranked support

S^t=TopKi∈𝒜t⁡\(e^t,i,K\)\.\\hat\{S\}\_\{t\}=\\operatorname\{TopK\}\_\{i\\in\\mathcal\{A\}\_\{t\}\}\\left\(\\hat\{e\}\_\{t,i\},K\\right\)\.\(11\)Reranking maps the variable\-size candidate set back to exactlyKKindices, preserving the fixed\-size support stored in each cache entry\. IfSt⊆𝒜tS\_\{t\}\\subseteq\\mathcal\{A\}\_\{t\}, this reranking step exactly recoversStS\_\{t\}\.

Finally, ReTopK performs sparse attention over the reranked support:

o^t=∑i∈S^texp⁡\(e^t,i\)∑j∈S^texp⁡\(e^t,j\)​vi\.\\hat\{o\}\_\{t\}=\\sum\_\{i\\in\\hat\{S\}\_\{t\}\}\\frac\{\\exp\(\\hat\{e\}\_\{t,i\}\)\}\{\\sum\_\{j\\in\\hat\{S\}\_\{t\}\}\\exp\(\\hat\{e\}\_\{t,j\}\)\}v\_\{i\}\.\(12\)Thus, historical supports are used only for candidate generation\. All retained scores, normalized attention weights, and value aggregations are computed using the current query and the original KV cache\.

### Reliable Reuse via Fallback and Refresh

Query similarity provides a useful but imperfect signal for reuse fidelity\. ReTopK therefore takes the Exact Top\-KKpath whenever the maximum cached\-query similarity falls below a thresholdτ\\tau, i\.e\.,ρtmax<τ\\rho\_\{t\}^\{\\max\}<\\tau\. This fallback avoids reliance on historical supports when no sufficiently related query is available\.

Fallback alone does not fully address error accumulation, because supports produced on the reuse path are subsequently inserted into the cache\. To periodically inject exact retrieval decisions, ReTopK performs an Exact Top\-KKrefresh everyTrT\_\{r\}decoding steps; settingTr=0T\_\{r\}=0disables scheduled refresh\. Lettdect\_\{\\mathrm\{dec\}\}denote the decoding\-step offset and𝒫t\\mathcal\{P\}\_\{t\}the selected path\. Then

𝒫t=\{Exact,Tr\>0,tdec∈\{Tr,2​Tr,…\},Exact,ρtmax<τ,Reuse,otherwise\.\\mathcal\{P\}\_\{t\}=\\begin\{cases\}\\textsc\{Exact\},&T\_\{r\}\>0,\\;t\_\{\\mathrm\{dec\}\}\\in\\\{T\_\{r\},2T\_\{r\},\\ldots\\\},\\\\ \\textsc\{Exact\},&\\rho\_\{t\}^\{\\max\}<\\tau,\\\\ \\textsc\{Reuse\},&\\text\{otherwise\}\.\\end\{cases\}\(13\)The two mechanisms are complementary: similarity fallback handles queries without reliable cached\-query matches, whereas periodic refresh limits long\-horizon drift in the online cache\.

After either path, ReTopK caches\(q¯t,S^t\)\(\\bar\{q\}\_\{t\},\\hat\{S\}\_\{t\}\), withS^t=St\\hat\{S\}\_\{t\}=S\_\{t\}on exact steps\.

### Complexity and GPU Implementation

For Exact Top\-KK, full\-history QK scoring requires𝒪​\(Lt​d\)\\mathcal\{O\}\(L\_\{t\}d\)work, followed by global Top\-KKselection overLtL\_\{t\}scores\. In contrast, a ReTopK reuse step requires𝒪​\(C​d\)\\mathcal\{O\}\(Cd\)work for cache matching,𝒪​\(R​K\+W\)\\mathcal\{O\}\(RK\+W\)work to construct the raw candidate list, and𝒪​\(Mt​d\)\\mathcal\{O\}\(M\_\{t\}d\)work for candidate QK scoring, followed by Top\-KKselection overMtM\_\{t\}scores, where

Mt≤R​K\+W≪Lt\.M\_\{t\}\\leq RK\+W\\ll L\_\{t\}\.\(14\)The subsequent sparse Softmax–value aggregation remains𝒪​\(K​dv\)\\mathcal\{O\}\(Kd\_\{v\}\), wheredvd\_\{v\}is the value\-head dimension\. Hence, withCC,RR,KK, andWWfixed, a reuse step replaces both full\-history QK scoring and global selection overLtL\_\{t\}entries with bounded cache lookup and candidate processing overMtM\_\{t\}entries\. Its selector work therefore does not grow with context length\. The additional metadata cost is𝒪​\(C​d\+C​K\)\\mathcal\{O\}\(Cd\+CK\)per head, while the complete KV cache remains unchanged\.

Our GPU implementation maps the reuse path to three GPU stages\. A single fused selector performs cache matching, Top\-RRretrieval, and exact candidate sorting and deduplication\. An indexed QK kernel directly scores the packed candidate keys without materializing a gathered key tensor\. A final fused kernel performs candidate Top\-KKselection, indexed Softmax–value aggregation, and cache update\. Cache state and per\-layer workspaces are preallocated and kept on the GPU\. Exact, fallback, and refresh paths use matched full\-history kernels, so the paths differ only in support discovery\.

MethodPG19 PPL↓\\downarrowSpeedup vs\. Exact Top\-KK↑\\uparrowNIAH: 3\-task Avg\.↑\\uparrowLongBench↑\\uparrow16K32K64K128K16K32K64K128K16K32K64K128K2WikiHotpotMFQAAvg\.Full Attention8\.788\.768\.8211\.622\.08×2\.08\\times1\.18×1\.18\\times0\.95×0\.95\\times0\.87×0\.87\\times99\.597\.398\.593\.255\.159\.849\.854\.9Exact Top\-KK8\.928\.949\.0312\.071\.00×1\.00\\times1\.00×1\.00\\times1\.00×1\.00\\times1\.00×1\.00\\times95\.295\.098\.064\.549\.857\.550\.252\.5StreamingLLM11\.6011\.4912\.1812\.382\.19×\\boldsymbol\{2\.19\\times\}2\.36×\\boldsymbol\{2\.36\\times\}3\.61×\\boldsymbol\{3\.61\\times\}6\.39×\\boldsymbol\{6\.39\\times\}3\.33\.82\.53\.522\.330\.824\.425\.8Quest10\.5517\.0518\.1736\.231\.07×1\.07\\times1\.13×1\.13\\times1\.75×1\.75\\times3\.09×3\.09\\times43\.778\.829\.724\.043\.039\.542\.041\.5SparQ11\.8713\.4313\.5523\.260\.68×0\.68\\times0\.51×0\.51\\times0\.78×0\.78\\times1\.39×1\.39\\times59\.284\.848\.045\.347\.452\.948\.849\.7Loki \(r=64r=64\)61\.0133\.28110\.08199\.320\.57×0\.57\\times0\.60×0\.60\\times0\.93×0\.93\\times1\.37×1\.37\\times4\.70\.00\.00\.040\.439\.546\.942\.3TokenSelect10\.2510\.6011\.1217\.710\.67×0\.67\\times0\.63×0\.63\\times0\.97×0\.97\\times1\.33×1\.33\\times65\.883\.336\.744\.847\.356\.049\.851\.0ReTopK \(τ=0\.85\\tau=0\.85\)8\.989\.039\.1812\.131\.29×1\.29\\times1\.38×1\.38\\times2\.03×2\.03\\times3\.07×3\.07\\times85\.389\.393\.253\.347\.054\.548\.950\.1ReTopK \(τ=0\.90\\tau=0\.90\)8\.958\.989\.1112\.201\.14×1\.14\\times1\.23×1\.23\\times1\.70×1\.70\\times2\.25×2\.25\\times96\.392\.298\.075\.248\.458\.751\.052\.7Table 1:Long\-context performance and attention efficiency \(K=512K=512\)\. Speedup is relative to Exact Top\-KK\. Gray and teal denote reference and ReTopK rows, respectively; boldface and underlining mark the best and second\-best approximate results\.

## Experiments

### Experimental Setup

#### Models and benchmarks\.

We evaluate Qwen2\.5\-7B\(Qwen and others[2024](https://arxiv.org/html/2607.27692#bib.bib20)\)on PG19\(Raeet al\.[2020](https://arxiv.org/html/2607.27692#bib.bib8)\)and Qwen2\.5\-7B\-Instruct\-1M\(Yanget al\.[2025](https://arxiv.org/html/2607.27692#bib.bib21)\)on RULER NIAH\(Hsiehet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib10)\)and LongBench\(Baiet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib9)\)\. PG19 and NIAH are evaluated at 16K, 32K, 64K, and 128K contexts\. For PG19, we report perplexity over a 512\-token teacher\-forced suffix for each of 64 fixed documents\. For NIAH, we report the macro\-average of single\-needle, multi\-key, and multi\-value retrieval over 50 examples per task and context length\. For LongBench, we use the complete test splits of three subsets: 2WikiMQA, HotpotQA, and MultiFieldQA, comprising 550 examples in total\.

#### Baselines\.

We compare ReTopK with Full Attention, Exact Top\-KK, StreamingLLM, Quest, SparQ, Loki, and TokenSelect\(Xiaoet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib2); Tanget al\.[2024](https://arxiv.org/html/2607.27692#bib.bib6); Ribaret al\.[2024](https://arxiv.org/html/2607.27692#bib.bib7); Singhaniaet al\.[2024](https://arxiv.org/html/2607.27692#bib.bib16); Wuet al\.[2025](https://arxiv.org/html/2607.27692#bib.bib14)\)\. Unless otherwise specified, all sparse methods use a sharedK=512K=512active\-token budget\. StreamingLLM allocates this budget to 64 sink and 448 recent tokens, whereas TokenSelect uses 64 initial, 416 selected, and 32 local tokens\. Other baseline configurations are provided in the appendix\.

#### Implementation settings\.

Unless otherwise specified, ReTopK usesC=W=32C=W=32,R=4R=4,τ=0\.85\\tau=0\.85, andTr=128T\_\{r\}=128\. All experiments use BF16\. Attention latency is measured on a single NVIDIA L20 GPU using matched kernels, and speedup is reported relative to Exact Top\-KK\.

### Experimental Results

#### Overall quality and efficiency\.

Table[1](https://arxiv.org/html/2607.27692#Sx3.T1)reports long\-context performance and attention efficiency\. Across all context lengths, the two ReTopK settings rank first and second among approximate methods in PG19 PPL and jointly attain the best approximate results across all reported task metrics, while matching or occasionally exceeding Exact Top\-KK\. The defaultτ=0\.85\\tau=0\.85achieves a3\.07×3\.07\\timesspeedup at 128K with a 0\.50% PPL increase, whereas the quality\-orientedτ=0\.90\\tau=0\.90maintains a1\.141\.14–2\.25×2\.25\\timesspeedup, leads all approximate methods on NIAH at every length, and obtains a LongBench average of 52\.7, exceeding the 52\.5 Exact Top\-KKreference\.

Speedup should be interpreted jointly with task performance\. StreamingLLM is fastest, but its fixed sink\-and\-recent window excludes evidence outside the retained tokens, limiting its NIAH score to 2\.5–3\.8\. TokenSelect reaches 51\.0 on LongBench, yet its selection cache hits on only 34\.7–35\.7% of NIAH steps, restricting its speedup to0\.670\.67–1\.33×1\.33\\times\. Loki uses the sharedK=512K=512budget and rank 64, substantially below the 12\.5–25% context\-proportional budgets used in its paper; its results therefore characterize a strict\-budget operating point\. Paper\-recommended configurations are compared in the appendix\.

Table 2:Cross\-model PG19 results \(64 documents; shared ReTopK settings; reuse weighted over head–token pairs\)\.
#### Cross\-model generalization\.

We apply the configuration selected on Qwen2\.5\-7B to Llama\-3\.1\-8B\(Grattafiori and others[2024](https://arxiv.org/html/2607.27692#bib.bib22)\)and Qwen2\.5\-14B without model\-specific tuning, usingC=W=32C=W=32,R=4R=4,τ=0\.85\\tau=0\.85, andTr=128T\_\{r\}=128, withK=512K=512at 16K–64K andK=1024K=1024at 128K\. Across all model–context pairs, ReTopK achieves 82\.5–89\.6% reuse, PPL changes of−0\.46%\-0\.46\\%to\+2\.76%\+2\.76\\%relative to Exact Top\-KK, and speedups of1\.261\.26–2\.66×2\.66\\times, demonstrating consistent transfer across model families and scales\.

#### Attention\-support fidelity\.

ReTopK need not reproduce every index selected by Exact Top\-KK, provided that the reranked support preserves high\-mass positions and the resulting attention output\. We evaluate this criterion for both a representative head and all layers and query heads\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x5.png)Figure 5:QK supports for Qwen2\.5\-7B layer 18, head 6 after a 1,536\-token prefill \(K=512K=512; 512 decoding steps\)\. White denotes unselected or masked entries\.Figure[5](https://arxiv.org/html/2607.27692#Sx4.F5)compares Full Attention, Exact Top\-KK, StreamingLLM, and ReTopK for the representative head\. ReTopK’s long\-range support geometry aligns more closely with Exact Top\-KKthan StreamingLLM’s fixed sink\-and\-recent pattern\. With 78\.9% Exact Top\-KKsupport recall, ReTopK retains 99\.45% of the attention mass and achieves 99\.95% attention\-distribution cosine, indicating that the mismatched indices receive low normalized attention weights\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x6.png)Figure 6:All\-head ReTopK fidelity on Qwen2\.5\-7B for 4 PG19 documents \(32K prefill, 512 decoding steps,K=512K=512\)\.Figure[6](https://arxiv.org/html/2607.27692#Sx4.F6)extends the analysis to 1,605,632 head–token pairs across all 28 layers and 28 query heads\. Retained mass and head\-output cosine average 92\.4% and 97\.4% over all pairs, and 91\.6% and 97\.2% on reuse\-path pairs despite 56\.9% support recall\. Both metrics increase with cached\-query similarity, supporting the similarity fallback atτ=0\.85\\tau=0\.85; the layer–head maps further show that this behavior is distributed throughout the model\.

#### Hyperparameter sensitivity and component ablation\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x7.png)Figure 7:One\-at\-a\-time ReTopK hyperparameter sensitivity at 64K; orange marks defaults\.Figure[7](https://arxiv.org/html/2607.27692#Sx4.F7)evaluates one\-at\-a\-time sensitivity around the default 64K configuration\. In panels \(a–e\), the defaultK=512K=512lies on the Pareto frontier, with a2\.03×2\.03\\timesspeedup and a 1\.60% PPL increase over Exact Top\-KK\. LargerKKyields gradual PPL improvements at higher reranking cost, reducing speedup to1\.09×1\.09\\timesatK=1024K=1024\. Performance is stable near the defaultCC, while gains from increasingRRdiminish beyondR=4R=4\. Increasingτ\\tautrades speed for quality by triggering more exact fallbacks\. Refresh intervalsTr≥128T\_\{r\}\\geq 128give similar short\-horizon PPL, whereas shorter intervals improve quality through more frequent Exact Top\-KKsteps\.

Panels \(f–h\) decompose these trade\-offs: increasingTrT\_\{r\}reduces the refresh share, while increasingτ\\taushifts decisions from reuse to similarity fallback\. IncreasingW/CW/Cfrom 1 to 16 enlarges the candidate set from approximately 701 to 901 candidates with little PPL variation;W=0W=0is examined in the component ablation\.

Table 3:ReTopK component ablation at 64K \(K=512K=512\)\.Table[3](https://arxiv.org/html/2607.27692#Sx4.T3)isolates the components; Exact and Cand\. denote the exact\-path rate and mean candidate count\. Although recent\-RRretrieval raises the exact\-path rate from 11\.6% to 16\.7%, it has slightly higher PPL and lower speedup than similarity\-guided retrieval\. SmallerRRorCCdegrades PPL or speed, while removing similarity fallback trades PPL for speed\. Eliminating the recent\-token guard causes severe degradation\. Scheduled refresh has little short\-horizon effect, motivating the long\-horizon evaluation\.

#### Long\-horizon stability\.

After a 64K prefill, we evaluate 8K continuations on 8 PG19 documents withK=512K=512andTr∈\{0,32,128,512\}T\_\{r\}\\in\\\{0,32,128,512\\\}\. Panel \(a\) uses shared teacher\-forced tokens; panel \(b\) compares autoregressive ReTopK with an Exact Top\-KKshadow on the same generated prefix\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x8.png)Figure 8:Long\-horizon stability under \(a\) teacher forcing and \(b\) autoregressive generation\.Panel \(a\) shows that quality error remains bounded through 8K:Δ\\DeltaPPL is within 2\.08% for all settings\. More frequent refresh improves the 8K endpoint from 2\.02% without refresh to 1\.73% at the defaultTr=128T\_\{r\}=128and 1\.33% atTr=32T\_\{r\}=32, while the no\-refresh setting remains stable\. Panel \(b\) yields 0\.010–0\.013 mean KL up toTTand 97\.2–97\.6% Top\-1 agreement under autoregressive feedback, with no late\-stage increase in error\.

#### Kernel efficiency analysis\.

Table 4:ReTopK kernel\-fusion stages at 64K and 128K \(K=512K=512; ms/token over 28 layers\)\.Table[4](https://arxiv.org/html/2607.27692#Sx4.T4)reports five cumulative stages under a fixed input and path mixture\. L1 separates retrieval and sparse output; L2 fuses cache search with support union and adds indexed candidate scoring; L3 adds exact sorting and deduplication while fusing selection, sparse output, and cache update; L4 reuses per\-layer workspaces; and L5 fuses candidate construction with output\. Relative to L1, L5 achieves2\.57×2\.57\\timesand2\.29×2\.29\\timesspeedups at 64K and 128K, respectively\.

![Refer to caption](https://arxiv.org/html/2607.27692v1/x9.png)Figure 9:Attention latency composition across context lengths \(K=512K=512\); labels above ReTopK denote speedup over Exact Top\-KK\.Figure[9](https://arxiv.org/html/2607.27692#Sx4.F9)decomposes attention latency across context lengths\. Full Attention remains scan\-dominated at 97–99%, while QK scoring and Top\-KKselection account for an increasing 87–97% of Exact Top\-KKlatency, driven primarily by QK and consistent with Figure[1](https://arxiv.org/html/2607.27692#Sx1.F1)\. ReTopK’s reuse\-path cost remains nearly constant; its growth arises from full\-history QK on fallback and refresh steps\. This growth does not arise from the path mix: reuse rises from 87\.5% to 88\.9%, while low\-similarity fallback falls from 11\.9% to 10\.5%\. Instead, the per\-exact\-path cost grows with context length, increasing ReTopK’s speedup over Exact Top\-KKfrom1\.29×1\.29\\timesto3\.07×3\.07\\times\.

#### End\-to\-end long\-context scaling\.

We measure matched end\-to\-end decoding on Qwen2\.5\-7B\-Instruct\-1M, using the same GPU layout across methods and a fixed ReTopK path mixture across lengths\. Timings exclude prefill, setup, and compilation\.

Table 5:End\-to\-end decoding latency \(ms/token\) across context lengths \(K=512K=512\)\.Table[5](https://arxiv.org/html/2607.27692#Sx4.T5)separates attention gains from shared decoder work\. Non\-attn\. denotes shared latency as a fraction of Exact Top\-KKlatency, and Spd\. is relative to Exact Top\-KK\. As the Non\-attn\. share decreases from 70\.1% at 128K to 22\.9% at 1M and 5\.8% at 5M, ReTopK’s speedup increases from1\.13×1\.13\\timesto2\.30×2\.30\\timesand3\.73×3\.73\\times, respectively\. The end\-to\-end implementation evaluates the fullR​K\+W=2080RK\+W=2080candidate capacity without truncation\.

## Conclusion

We introduced ReTopK, a training\-free recall\-before\-rerank method that constructs candidate sets from cached query–support pairs, reranks them with current\-query scores, and uses similarity fallback and periodic refresh for reliability\. Across long\-context language modeling and retrieval tasks, ReTopK preserves the quality of Exact Top\-KKwhile accelerating dynamic Top\-KKattention by3\.07×3\.07\\timesat 128K with a 0\.50% PPL increase\. These results show that retrieval\-decision reuse reduces full\-history index\-discovery cost without evicting KV entries\.

## References

- IndexCache: accelerating sparse attention via cross\-layer index reuse\.arXiv preprint arXiv:2603\.12201\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3119–3137\.Cited by:[Models and benchmarks\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: dynamic KV cache compression based on pyramidal information funneling\.InSecond Conference on Language Modeling,Cited by:[KV\-cache reduction and structured sparsity\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px2.p1.1)\.
- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[Introduction](https://arxiv.org/html/2607.27692#Sx1.p1.1),[Efficient attention and Exact Top\-KKattention\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px1.p1.3)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.27692#Sx1.p1.1),[Efficient attention and Exact Top\-KKattention\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px1.p1.3)\.
- D\. Deshmukh, S\. Goyal, N\. Kwatra, and R\. Ramjee \(2025\)Kascade: a practical sparse attention method for long\-context LLM inference\.arXiv preprint arXiv:2512\.16391\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. Gao, Z\. Zeng, D\. Du, S\. Cao, P\. Zhou, J\. Qi, J\. Lai, H\. So, T\. Cao, F\. Yang, and M\. Yang \(2025\)SeerAttention: self\-distilled attention gating for efficient long\-context prefilling\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1)\.
- A\. Grattafioriet al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Cross\-model generalization\.](https://arxiv.org/html/2607.27692#Sx4.SSx2.SSS0.Px2.p1.11)\.
- C\. R\. C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. Maheswaran, S\. Zhao, J\. Paik, M\. W\. Mahoney, K\. Keutzer, and A\. Gholami \(2025\)Squeezed attention: accelerating long context length LLM inference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 32631–32652\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,Cited by:[Models and benchmarks\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px1.p1.1)\.
- H\. Jiang, Y\. Li, C\. Zhang, Q\. Wu, X\. Luo, S\. Ahn, Z\. Han, A\. H\. Abdi, D\. Li, C\. Lin, Y\. Yang, and L\. Qiu \(2024\)MInference 1\.0: accelerating pre\-filling for long\-context LLMs via dynamic sparse attention\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[KV\-cache reduction and structured sparsity\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[KV\-cache reduction and structured sparsity\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px2.p1.1)\.
- D\. Liu, M\. Chen, B\. Lu, H\. Jiang, Z\. Han, Q\. Zhang, Q\. Chen, C\. Zhang, B\. Ding, K\. Zhang, C\. Chen, F\. Yang, Y\. Yang, and L\. Qiu \(2025\)RetrievalAttention: accelerating long\-context LLM inference via vector retrieval\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for LLM KV cache compression at test time\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[KV\-cache reduction and structured sparsity\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px2.p1.1)\.
- Qwenet al\.\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[Models and benchmarks\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px1.p1.1)\.
- J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, C\. Hillier, and T\. P\. Lillicrap \(2020\)Compressive transformers for long\-range sequence modelling\.InInternational Conference on Learning Representations,Cited by:[Models and benchmarks\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px1.p1.1)\.
- L\. Ribar, I\. Chelombiev, L\. Hudlass\-Galley, C\. Blake, C\. Luschi, and D\. Orr \(2024\)SparQ attention: bandwidth\-efficient LLM inference\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 42558–42583\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1),[Baselines\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px2.p1.2)\.
- P\. Singhania, S\. Singh, S\. He, S\. Feizi, and A\. Bhatele \(2024\)Loki: low\-rank keys for efficient sparse attention\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Introduction](https://arxiv.org/html/2607.27692#Sx1.p3.1),[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1),[Baselines\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px2.p1.2)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)Quest: query\-aware sparsity for efficient long\-context LLM inference\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 47901–47911\.Cited by:[Introduction](https://arxiv.org/html/2607.27692#Sx1.p3.1),[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1),[Baselines\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px2.p1.2)\.
- T\. Wang, G\. Rattihalli, A\. Dhakal, J\. Li, Z\. Ren, D\. Milojicic, and L\. Shangguan \(2026\)Predict, reuse, and repair: accelerating dynamic sparse attention for long\-context LLM decoding\.arXiv preprint arXiv:2606\.30389\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1)\.
- W\. Wu, Z\. Pan, K\. Fu, C\. Wang, L\. Chen, Y\. Bai, T\. Wang, Z\. Wang, and H\. Xiong \(2025\)TokenSelect: efficient long\-context inference and length extrapolation for LLMs via dynamic token\-level KV cache selection\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 21264–21281\.Cited by:[Query\-aware sparse selection\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px3.p1.1),[Baselines\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px2.p1.2)\.
- G\. Xiao, J\. Tang, J\. Zuo, J\. Guo, S\. Yang, H\. Tang, Y\. Fu, and S\. Han \(2025\)DuoAttention: efficient long\-context LLM inference with retrieval and streaming heads\.InInternational Conference on Learning Representations,Cited by:[KV\-cache reduction and structured sparsity\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px2.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.27692#Sx1.p3.1),[KV\-cache reduction and structured sparsity\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px2.p1.1),[Baselines\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px2.p1.2)\.
- A\. Yang, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Huang, J\. Jiang, J\. Tu, J\. Zhang, J\. Zhou, J\. Lin, K\. Dang, K\. Yang, L\. Yu, M\. Li, M\. Sun, Q\. Zhu, R\. Men, T\. He, W\. Xu, W\. Yin, W\. Yu, X\. Qiu, X\. Ren, X\. Yang, Y\. Li, Z\. Xu, and Z\. Zhang \(2025\)Qwen2\.5\-1M technical report\.arXiv preprint arXiv:2501\.15383\.Cited by:[Models and benchmarks\.](https://arxiv.org/html/2607.27692#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Introduction](https://arxiv.org/html/2607.27692#Sx1.p3.1),[KV\-cache reduction and structured sparsity\.](https://arxiv.org/html/2607.27692#Sx2.SS0.SSS0.Px2.p1.1)\.