S$^4$R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching

# S4R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
Source: [https://arxiv.org/html/2608.00528](https://arxiv.org/html/2608.00528)
Jialong Han,You Wu,Kewei Tu School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging hanjl2022@alumni\.shanghaitech\.edu\.cn,\{wuyou2024,tukw\}@shanghaitech\.edu\.cn

###### Abstract

The growth of context window lengths in Large Language Models \(LLMs\) significantly enhances their long\-context capabilities but incurs prohibitive memory costs due to the Key\-Value \(KV\) cache\. Although low\-rank compression of KV cache is a promising remedy, existing methods face a dilemma: offline approaches depend on external calibration data, whereas online approaches incur substantial compute for full\-prompt decomposition and reconstruction\. In this paper, we proposeS4R, which builds low\-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation\. S4R uses prompt\-aware initialization to build initial key/value bases from a representative prompt subset, trading off calibration\-data dependence against prefilling cost\. Because fully reconstructing the cache at every decoding step is prohibitively expensive and hurts throughput, we further adopt sparse reconstruction to retain only informative positions during decoding\. Extensive experiments on LongBench and RULER with Llama and Qwen model families show that S4R achieves up to5×\\timesKV compression with near full\-cache accuracy, combining the efficiency of fixed compression with the adaptability of prompt\-dependent methods\.

S4R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long\-Context KV Caching

Jialong Han, You Wu, Kewei Tu††thanks:Corresponding author\.School of Information Science and Technology, ShanghaiTech UniversityShanghai Engineering Research Center of Intelligent Vision and Imaginghanjl2022@alumni\.shanghaitech\.edu\.cn,\{wuyou2024,tukw\}@shanghaitech\.edu\.cn

## 1Introduction

The remarkable expansion of context window sizes in Large Language Models \(LLMs\)—now spanning from 128k up to millions of tokensOpenAIet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib34)\); Teamet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib33)\); Yanget al\.\([2025b](https://arxiv.org/html/2608.00528#bib.bib36)\)—has ushered in a new era of capabilities, powering breakthroughs in long\-document summarizationBaiet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib46)\), retrieval\-augmented generationLewiset al\.\([2020](https://arxiv.org/html/2608.00528#bib.bib38)\), and autonomous agentsParket al\.\([2023](https://arxiv.org/html/2608.00528#bib.bib39)\); Wanget al\.\([2024a](https://arxiv.org/html/2608.00528#bib.bib40)\)\. The scaling of context window brings an acute bottleneck: memory consumption\. As context length increases, the Key\-Value \(KV\) cache—used for storing intermediate computations—can rapidly surpass the footprint of the model parameters themselvesKwonet al\.\([2023](https://arxiv.org/html/2608.00528#bib.bib7)\); Popeet al\.\([2023](https://arxiv.org/html/2608.00528#bib.bib8)\); Ainslieet al\.\([2023](https://arxiv.org/html/2608.00528#bib.bib47)\), making memory efficiency the central challenge in LLMs inference\.

To address this challenge, numerous KV cache compression techniques have been explored, among which low\-rank methods—particularly Singular Value Decomposition \(SVD\)—stand out for their effectivenessLIet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib30)\)\. These approaches are motivated by the empirical observation that the KV cache often exhibits strong low\-rank structuresYuet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib6)\), which can be exploited to reduce memory usage\. However, existing low\-rank methods face a trade\-off between efficiency and adaptivity\. Fixed post\-training methods either derive projection bases from calibration activations or factorize KV projection weights before inference, and then reuse the resulting low\-dimensional representations during servingChanget al\.\([2025b](https://arxiv.org/html/2608.00528#bib.bib21)\); Saxenaet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib48)\); Zhanget al\.\([2024a](https://arxiv.org/html/2608.00528#bib.bib17)\); Yanet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib24)\)\. They are efficient at decoding time, but can be sensitive to calibration data, rank allocation, or distribution shifts under aggressive compression ratiosSunet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib27)\); Wanget al\.\([2025b](https://arxiv.org/html/2608.00528#bib.bib3)\); Zhuet al\.\([2026](https://arxiv.org/html/2608.00528#bib.bib20)\)\. Prompt\-dependent methods estimate or adapt low\-rank representations during inference, which improves robustness to the current context but introduces additional computation, especially during prefillingSunet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib27)\); Changet al\.\([2025a](https://arxiv.org/html/2608.00528#bib.bib32)\); Zhuet al\.\([2026](https://arxiv.org/html/2608.00528#bib.bib20)\)\.

Our key insight is that the dominant KV subspace of a long prompt need not be estimated from either an external calibration set or the entire prompt\. Instead, it can be approximated from a carefully selected subset of prompt tokens\. At the same time, KV cache entries should not be compressed or reconstructed uniformly\. Sink tokens serve as stable global pivots in attention and should be preserved in full precisionXiaoet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib9)\), while most non\-sink tokens can be stored in a low\-rank coefficient space and reconstructed only when they are likely to matter for the current decoding step\.

Based on this insight, we proposeS4R, a prompt\-aware low\-rank KV cache compression method with selective sampling and sparse reconstruction\. During prefilling, S4R keeps sink tokens in full precision, samples representative non\-sink tokens from the prompt, normalizes their KV states, and constructs key/value subspaces with truncated SVD\. It then stores non\-sink keys and values as low\-rank coefficients rather than full\-dimensional KV states\. During decoding, S4R estimates token relevance directly in the latent coefficient space, combines a local window with globally important positions, and reconstructs only the selected KV entries for final attention computation\. This design combines the low prefilling overhead of fixed compression with the context adaptivity of prompt\-dependent methods\.

We evaluate S4R on LongBenchBaiet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib49)\)and RULERHsiehet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib41)\)with LlamaGrattafioriet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib35)\)and QwenYanget al\.\([2025b](https://arxiv.org/html/2608.00528#bib.bib36)\); Qwenet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib37)\); Yanget al\.\([2025a](https://arxiv.org/html/2608.00528#bib.bib43)\)model families\. On LongBench, S4R achieves up to5×\\timesKV cache compression and remains close to the full\-cache baseline across model scales\. Across accuracy and efficiency evaluations, S4R consistently matches or outperforms prior compressed\-cache baselines while substantially reducing long\-context inference latency\.

- •We propose S4R, a prompt\-aware low\-rank KV cache compression method that builds compact key/value subspaces from selectively sampled prompt tokens and combines sink\-token preservation, latent\-space relevance estimation, and sparse reconstruction to reduce persistent KV memory while preserving context adaptivity\.
- •We evaluate S4R on LongBench and RULER, showing near full\-cache accuracy under up to5×\\timesKV compression and a favorable accuracy–efficiency trade\-off against strong compressed\-cache baselines\.

## 2Related Work

Low\-rank KV cache compression aims to reduce hidden\-dimensional redundancy in cached keys \(and values\), complementing token eviction and quantization methods surveyed in recent KV cache management literatureLIet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib30)\)\. Prior work provides empirical evidence that KV states or KV\-related projections exhibit exploitable low\-rank structureYuet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib6)\)\.

One group of methods performs fixed post\-training compression before serving\. Palu factorizes key/value projection layers, caches compact intermediate states, and reconstructs keys or fuses value reconstruction during attentionChanget al\.\([2025b](https://arxiv.org/html/2608.00528#bib.bib21)\)\. LoRC similarly applies low\-rank approximation to KV weight matrices, with a progressive layer\-wise strategy motivated by error propagationZhanget al\.\([2024a](https://arxiv.org/html/2608.00528#bib.bib17)\)\. Eigen Attention instead computes low\-rank bases from calibration activations and performs attention in the induced low\-dimensional spaceSaxenaet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib48)\)\. ReCalKV further separates key and value compression, using head\-wise similarity\-aware grouping for keys and offline calibration plus matrix fusion for valuesYanet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib24)\)\. Loki and SALS also determine low\-dimensional key bases from offline calibration before serving and use the induced latent key space for sparse token selection during decodingSinghaniaet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib16)\); Muet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib25)\)\. SALS directly targets KV cache compression in latent space, whereas Loki primarily uses the compressed key space to accelerate sparse attention scoring and does not itself reduce the persistent KV cache footprint\. These approaches are efficient because most decomposition work is done before inference, but their accuracy depends on the fixed compression design, including calibration statistics, rank allocation, and how well the compressed subspace matches the target prompts\.

A second group of methods builds low\-rank representations from the current prompt during inference, reducing dependence on offline calibration at the cost of additional prefilling computation\. ShadowKV performs prompt\-dependent SVD on pre\-RoPE keys, keeps a low\-rank key cache on GPU, offloads values, and reconstructs only selected sparse KV pairs during decodingSunet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib27)\)\. xKV applies cross\-layer SVD during prefilling to share low\-rank structure across grouped layersChanget al\.\([2025a](https://arxiv.org/html/2608.00528#bib.bib32)\)\. S4R follows the prompt\-dependent direction, but avoids full\-prompt decomposition by constructing key/value subspaces from selectively sampled prompt tokens and then combining latent\-space relevance estimation with sparse reconstruction during decoding\. Appendix[C](https://arxiv.org/html/2608.00528#A3)discusses additional related work from other KV cache compression categories and gives a more detailed comparison between S4R and the representative methods discussed above\.

## 3Method

S4R consists of two main stages: theprefillingand thedecodingstages\. We defer background on standard KV cache computation and SVD to Appendix[A](https://arxiv.org/html/2608.00528#A1)\.

### 3\.1Prefilling Stage

The main motivation of the prefilling stage is to efficiently construct suitable low\-rank subspaces for keys and values\. As directly applying SVD to the full long\-context sequence \(e\.g\., 128k tokens\) is computationally prohibitive, this stage starts by selecting a representative subset of tokens from the input prompt to approximate the dominant subspace structure of the full KV states\.

#### 3\.1\.1Sink Token Handling

Due to their role as global pivots in attention, sink tokens typically exhibit consistently high attention weights across queries and layersXiaoet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib9)\)\. As a result, their representations are critical for maintaining stable long\-range dependencies and cannot be reliably approximated by a low\-rank subspace without incurring noticeable information loss\.

To preserve their fidelity, we exclude the firstsssink tokens from the compression pipeline and store their pre\-RoPE key and value representations in full precision\. The remaining tokens are treated as non\-sink tokens and are subject to low\-rank approximation\.

#### 3\.1\.2Subspace Construction

Let the non\-sink tokens in the input prompt be denoted as𝐗∈ℝℬ×𝒯×𝒟\\mathbf\{X\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{T\}\\times\\mathcal\{D\}\}\. We construct a subset of tokens𝐗\[ℐ\]∈ℝℬ×𝒯′×𝒟\\mathbf\{X\}\_\{\[\\mathcal\{I\}\]\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{T\}^\{\\prime\}\\times\\mathcal\{D\}\}by selecting𝒯′≪𝒯\\mathcal\{T\}^\{\\prime\}\\ll\\mathcal\{T\}using a hybrid sampling strategy that combines \(i\) uniform sampling over early positions and \(ii\) a contiguous block of most recent \(right\) tokens\. Here,ℐ∈ℕℬ×𝒯′\\mathcal\{I\}\\in\\mathbb\{N\}^\{\\mathcal\{B\}\\times\\mathcal\{T\}^\{\\prime\}\}denotes the indices of the selected non\-sink tokens\. Correspondingly, the non\-sink tokens are partitioned into two disjoint subsets: the selected tokens \(𝐗\[ℐ\]\\mathbf\{X\}\_\{\[\\mathcal\{I\}\]\}\) and the remaining tokens \(𝐗\[ℐ~\]\\mathbf\{X\}\_\{\[\\widetilde\{\\mathcal\{I\}\}\]\}\), whereℐ~\\widetilde\{\\mathcal\{I\}\}denotes the complement ofℐ\\mathcal\{I\}along the sequence dimension\.

Before applying SVD to the selected tokens, we perform token\-wise normalization to mitigate the influence of varying vector magnitudes on the learned subspace\. In practice, tokens with larger norms may dominate the decomposition and bias the resulting singular vectors toward high\-energy directions\. To alleviate this effect, we normalize each sampled pre\-RoPE key and value representation along the feature dimension:

𝐒\[ℐ\]K/V←normalize​\(𝐒\[ℐ\]K/V,dim=−1\)\.\\mathbf\{S\}^\{K/V\}\_\{\[\\mathcal\{I\}\]\}\\leftarrow\\text\{normalize\}\(\\mathbf\{S\}^\{K/V\}\_\{\[\\mathcal\{I\}\]\},\\text\{dim=\}\-1\)\.\(1\)
This normalization step ensures that the subspace primarily captures directional information rather than being skewed by magnitude differences, leading to a more stable and representative low\-rank approximation\.

Then, we perform truncated SVD for each layer:

𝐔K/V,𝚺K/V,𝐁K/V←SVDr​\(𝐒\[ℐ\]K/V\),\\mathbf\{U\}^\{K/V\},\\mathbf\{\\Sigma\}^\{K/V\},\\mathbf\{B\}^\{K/V\}\\leftarrow\\mathrm\{SVD\}\_\{r\}\(\\mathbf\{S\}^\{K/V\}\_\{\[\\mathcal\{I\}\]\}\),\(2\)where𝐔K/V∈ℝℬ×𝒯′×r\\mathbf\{U\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{T\}^\{\\prime\}\\times r\}are the left singular vectors,𝚺K/V∈ℝℬ×r×r\\mathbf\{\\Sigma\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times r\\times r\}are the singular values, and𝐁K/V∈ℝℬ×𝒟′×r\\mathbf\{B\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{D\}^\{\\prime\}\\times r\}are the top\-rrright singular vectors, which serve as the low\-rank bases\.

We then project the non\-sink key and value representations onto the learned subspaces to obtain their coefficient representations:

𝐂K/V=𝐒K/V​𝐁K/V,\\mathbf\{C\}^\{K/V\}=\\mathbf\{S\}^\{K/V\}\\mathbf\{B\}^\{K/V\},\(3\)where𝐂K/V∈ℝℬ×𝒯×r\\mathbf\{C\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{T\}\\times r\}are the projection coefficients\.

#### 3\.1\.3Initial Local Query Window

To enable efficient relevance estimation without reconstructing the full keys and values during decoding, we also project the local queries into the same low\-rank subspace\.

Let the most recent queries be denoted as𝐐~∈ℝℬ×𝒢×𝒲×𝒟′\\widetilde\{\\mathbf\{Q\}\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times\\mathcal\{W\}\\times\\mathcal\{D\}^\{\\prime\}\}\. The projected query coefficients are then computed as

𝐂Q=𝐐~​𝐁K\\mathbf\{C\}^\{Q\}=\\widetilde\{\\mathbf\{Q\}\}\\mathbf\{B\}^\{K\}\(4\)where𝐂Q∈ℝℬ×𝒢×𝒲×r\\mathbf\{C\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times\\mathcal\{W\}\\times r\}\.

### 3\.2Decoding Stage

At a high level, decoding in S4R keeps the KV cache compressed while reconstructing only a small set of useful tokens at each step\. Each new token is projected into the prefilling\-stage low\-rank subspace to update the coefficient cache\. Recent query coefficients are then used to score cached tokens in latent space, which provides an inexpensive estimate of token relevance\. S4R reconstructs only the selected local and globally important non\-sink tokens, concatenates them with the full\-precision sink tokens, and performs attention over this compact set\.

#### 3\.2\.1Compressed Cache Update

Let𝐬tQ∈ℝℬ×𝒢×1×𝒟′\\mathbf\{s\}\_\{t\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times 1\\times\\mathcal\{D\}^\{\\prime\}\}and𝐬tK/V∈ℝℬ×1×𝒟′\\mathbf\{s\}\_\{t\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times 1\\times\\mathcal\{D\}^\{\\prime\}\}denote the query, key, and value representations at steptt, respectively\. The corresponding coefficient representations are computed as:

𝐜tQ/K\\displaystyle\\mathbf\{c\}\_\{t\}^\{Q/K\}=𝐬tQ/K​𝐁K,\\displaystyle=\\mathbf\{s\}\_\{t\}^\{Q/K\}\\mathbf\{B\}^\{K\},\(5\)𝐜tV\\displaystyle\\mathbf\{c\}\_\{t\}^\{V\}=𝐬tV​𝐁V,\\displaystyle=\\mathbf\{s\}\_\{t\}^\{V\}\\mathbf\{B\}^\{V\},where𝐜tQ∈ℝℬ×𝒢×1×r\\mathbf\{c\}\_\{t\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times 1\\times r\}and𝐜tK/V∈ℝℬ×1×r\\mathbf\{c\}\_\{t\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times 1\\times r\}\.

The key and value coefficients are appended to the cached sequences:

𝐂tK/V=𝐂t−1K/V⊕𝐜tK/V,\\mathbf\{C\}\_\{t\}^\{K/V\}=\\mathbf\{C\}\_\{t\-1\}^\{K/V\}\\oplus\\mathbf\{c\}\_\{t\}^\{K/V\},\(6\)where𝐂tK/V∈ℝℬ×\(𝒯\+t\)×r\\mathbf\{C\}\_\{t\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\(\\mathcal\{T\}\+t\)\\times r\}stores the accumulated coefficients\.

In contrast, query coefficients are maintained as a fixed\-size sliding window of length𝒲\\mathcal\{W\}, forming a first\-in\-first\-out queue𝐂Q∈ℝℬ×𝒢×𝒲×r\\mathbf\{C\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times\\mathcal\{W\}\\times r\}\. At each steptt, the new coefficient𝐜tQ\\mathbf\{c\}\_\{t\}^\{Q\}is appended, while the oldest entry is removed:

𝐂Q←concat\(𝐂Q\[:,:,1:,:\],;𝐜tQ\)\.\\mathbf\{C\}^\{Q\}\\leftarrow\\text\{concat\}\\big\(\\mathbf\{C\}^\{Q\}\[:,:,1:,:\],;\\mathbf\{c\}\_\{t\}^\{Q\}\\big\)\.\(7\)

#### 3\.2\.2Latent\-Space Relevance Estimation

To avoid explicit reconstruction of the full key and value representations, we approximate the attention scores directly in the latent coefficient space\.

Letℒt=𝒯\+t\\mathcal\{L\}\_\{t\}=\\mathcal\{T\}\+tdenote the current non\-sink cache length\. Given the query coefficients𝐂Q∈ℝℬ×𝒢×𝒲×r\\mathbf\{C\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times\\mathcal\{W\}\\times r\}and the cached key coefficients𝐂tK∈ℝℬ×ℒt×r\\mathbf\{C\}\_\{t\}^\{K\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{L\}\_\{t\}\\times r\}, we first compute the latent\-space attention logits:

𝐙=𝐂Q​\(𝐂tK\)⊤r,\\mathbf\{Z\}=\\frac\{\\mathbf\{C\}^\{Q\}\(\\mathbf\{C\}\_\{t\}^\{K\}\)^\{\\top\}\}\{\\sqrt\{r\}\},\(8\)where the inner product is taken along the latent dimensionrr, yielding𝐙∈ℝℬ×𝒢×𝒲×ℒt\\mathbf\{Z\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times\\mathcal\{W\}\\times\\mathcal\{L\}\_\{t\}\}\.

We then apply the causal mask before normalization, so each query only attends to valid historical positions:

𝒜=softmaxℒt​\(Maskcausal​\(𝐙\)\),\\mathcal\{A\}=\\mathrm\{softmax\}\_\{\\mathcal\{L\}\_\{t\}\}\\left\(\\mathrm\{Mask\}\_\{\\text\{causal\}\}\(\\mathbf\{Z\}\)\\right\),\(9\)where the softmax is computed over the cache\-length dimension\.

To obtain a single relevance score for each cached token, we average the attention scores over the query window:

𝒜Q=1𝒲​∑i=1𝒲𝒜\[:,:,i,:\]\.\\mathcal\{A\}^\{Q\}=\\frac\{1\}\{\\mathcal\{W\}\}\\sum\_\{i=1\}^\{\\mathcal\{W\}\}\\mathcal\{A\}\_\{\[:,:,i,:\]\}\.\(10\)Here,𝒜Q∈ℝℬ×𝒢×ℒt\\mathcal\{A\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times\\mathcal\{L\}\_\{t\}\}\.

#### 3\.2\.3Sparse Reconstruction and Attention

Given the relevance scores𝒜Q∈ℝℬ×𝒢×ℒt\\mathcal\{A\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{G\}\\times\\mathcal\{L\}\_\{t\}\}, we first apply a pooling operation along the sequence dimension before selecting global tokens\. This follows SnapKV\-style token clusteringLiet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib28)\), where neighboring tokens are smoothed or grouped to produce more stable importance estimates:

𝒜~=ℱ𝒫​\(𝒜Q;κ\)\\widetilde\{\\mathcal\{A\}\}=\\mathcal\{F\}\_\{\\mathcal\{P\}\}\(\\mathcal\{A\}^\{Q\};\\kappa\)\(11\)whereℱ𝒫\\mathcal\{F\}\_\{\\mathcal\{P\}\}denotes either average pooling or max pooling with kernel sizeκ\\kappa\.

For models with𝒢\>1\\mathcal\{G\}\>1, we further aggregate across the group dimension usingℱ𝒢∈\{mean,max\}\\mathcal\{F\}\_\{\\mathcal\{G\}\}\\in\\\{\\mathrm\{mean\},\\mathrm\{max\}\\\}:

𝒜~=ℱ𝒢​\(𝒜~\),\\widetilde\{\\mathcal\{A\}\}=\\mathcal\{F\}\_\{\\mathcal\{G\}\}\(\\widetilde\{\\mathcal\{A\}\}\),\(12\)where𝒜~∈ℝℬ×ℒt\\widetilde\{\\mathcal\{A\}\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{L\}\_\{t\}\}represents the final relevance score for each cached token\.

We select a subset of historical tokens for reconstruction by combining a local window with globally important tokens\. We always retain the most recent𝒲\\mathcal\{W\}tokens:

ℐlocal=\{ℒt−𝒲,…,ℒt−1\}\.\\mathcal\{I\}\_\{\\text\{local\}\}=\\\{\\mathcal\{L\}\_\{t\}\-\\mathcal\{W\},\\dots,\\mathcal\{L\}\_\{t\}\-1\\\}\.\(13\)From the remaining tokens, we select the top\-kkpositions according to𝒜~\\widetilde\{\\mathcal\{A\}\}:

ℐglobal=TopK​\(𝒜~\[:,:ℒt−𝒲\],k\),\\mathcal\{I\}\_\{\\text\{global\}\}=\\mathrm\{TopK\}\\big\(\\widetilde\{\\mathcal\{A\}\}\_\{\[:,:\\mathcal\{L\}\_\{t\}\-\\mathcal\{W\}\]\},\\,k\\big\),\(14\)wherek=max⁡\(⌊ρ​ℒt⌋−𝒲,0\)k=\\max\(\\lfloor\\rho\\mathcal\{L\}\_\{t\}\\rfloor\-\\mathcal\{W\},\\,0\)is determined by the reconstruction ratioρ\\rho\.

The final index set is given by:

ℐ^=ℐlocal∪ℐglobal\.\\widehat\{\\mathcal\{I\}\}=\\mathcal\{I\}\_\{\\text\{local\}\}\\cup\\mathcal\{I\}\_\{\\text\{global\}\}\.\(15\)
Given the selected indices, we reconstruct the corresponding key and value representations from the low\-rank coefficients:

𝐊^=𝐂tK\(𝐁K\)⊤\[ℐ^\],𝐕^=𝐂tV\(𝐁V\)⊤\[ℐ^\],\\widehat\{\\mathbf\{K\}\}=\\mathbf\{C\}\_\{t\}^\{K\}\{\}\_\{\[\\widehat\{\\mathcal\{I\}\}\]\}\(\\mathbf\{B\}^\{K\}\)^\{\\top\},\\quad\\widehat\{\\mathbf\{V\}\}=\\mathbf\{C\}\_\{t\}^\{V\}\{\}\_\{\[\\widehat\{\\mathcal\{I\}\}\]\}\(\\mathbf\{B\}^\{V\}\)^\{\\top\},\(16\)where𝐊^,𝐕^∈ℝℬ×\|ℐ^\|×𝒟′\\widehat\{\\mathbf\{K\}\},\\widehat\{\\mathbf\{V\}\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\|\\widehat\{\\mathcal\{I\}\}\|\\times\\mathcal\{D\}^\{\\prime\}\}\.

These reconstructed tokens are then concatenated with the sink tokens:

𝐊⋆\\displaystyle\\mathbf\{K\}^\{\\star\}=concat​\(𝐊sink,𝐊^\),\\displaystyle=\\text\{concat\}\(\\mathbf\{K\}\_\{\\text\{sink\}\},\\widehat\{\\mathbf\{K\}\}\),\(17\)𝐕⋆\\displaystyle\\mathbf\{V\}^\{\\star\}=concat​\(𝐕sink,𝐕^\)\.\\displaystyle=\\text\{concat\}\(\\mathbf\{V\}\_\{\\text\{sink\}\},\\widehat\{\\mathbf\{V\}\}\)\.
We then apply RoPE to𝐊⋆\\mathbf\{K\}^\{\\star\}according to their original positions to ensure positional consistency\.

The final attention output is computed as:

𝐎t=softmax​\(RoPE​\(𝐬tQ\)​RoPE​\(𝐊⋆\)⊤dh\)​𝐕⋆\.\\mathbf\{O\}\_\{t\}=\\mathrm\{softmax\}\\left\(\\frac\{\\text\{RoPE\}\(\\mathbf\{s\}\_\{t\}^\{Q\}\)\\text\{RoPE\}\(\\mathbf\{K\}^\{\\star\}\)^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)\\mathbf\{V\}^\{\\star\}\.\(18\)

### 3\.3Memory Analysis

We analyze the per\-layer memory footprint in number of stored scalar elements, omitting the batch dimension for clarity\. At decoding steptt, the full KV cache stores both keys and values for thesssink tokens and theℒt\\mathcal\{L\}\_\{t\}non\-sink tokens:

Mfull​\(t\)=2​\(s\+ℒt\)​𝒟′\.M\_\{\\mathrm\{full\}\}\(t\)=2\(s\+\\mathcal\{L\}\_\{t\}\)\\mathcal\{D\}^\{\\prime\}\.\(19\)
In S4R, sink tokens are kept in full precision, while non\-sink tokens are stored in the low\-rank coefficient cache\. The persistent cache therefore consists of the full sink KV states, the key/value coefficient caches, the query\-coefficient window, and the learned key/value bases:

MS4​R​\(t\)=2​s​𝒟′\+2​ℒt​r\+𝒢​𝒲​r\+2​𝒟′​r\.M\_\{\\mathrm\{S^\{4\}R\}\}\(t\)=2s\\mathcal\{D\}^\{\\prime\}\+2\\mathcal\{L\}\_\{t\}r\+\\mathcal\{G\}\\mathcal\{W\}r\+2\\mathcal\{D\}^\{\\prime\}r\.\(20\)The term𝒢​𝒲​r\\mathcal\{G\}\\mathcal\{W\}rstores the fixed\-size query\-coefficient window, while the last term stores𝐁K\\mathbf\{B\}^\{K\}and𝐁V\\mathbf\{B\}^\{V\}\. The resulting persistent memory ratio is

MS4​R​\(t\)Mfull​\(t\)=s​𝒟′\+ℒt​r\+12​𝒢​𝒲​r\+𝒟′​r\(s\+ℒt\)​𝒟′\.\\frac\{M\_\{\\mathrm\{S^\{4\}R\}\}\(t\)\}\{M\_\{\\mathrm\{full\}\}\(t\)\}=\\frac\{s\\mathcal\{D\}^\{\\prime\}\+\\mathcal\{L\}\_\{t\}r\+\\frac\{1\}\{2\}\\mathcal\{G\}\\mathcal\{W\}r\+\\mathcal\{D\}^\{\\prime\}r\}\{\(s\+\\mathcal\{L\}\_\{t\}\)\\mathcal\{D\}^\{\\prime\}\}\.\(21\)Whenℒt≫s,r\\mathcal\{L\}\_\{t\}\\gg s,r, the ratio approachesr/𝒟′r/\\mathcal\{D\}^\{\\prime\}, showing that the long\-context memory is dominated by the low\-rank coefficient dimension\.

During attention computation, S4R reconstructs only the selected non\-sink tokens\. Let

mt=\|ℐ^\|\\displaystyle m\_\{t\}=\|\\widehat\{\\mathcal\{I\}\}\|=𝒲\+max⁡\(⌊ρ​ℒt⌋−𝒲,0\)\\displaystyle=\\mathcal\{W\}\+\\max\(\\lfloor\\rho\\mathcal\{L\}\_\{t\}\\rfloor\-\\mathcal\{W\},0\)\(22\)=max⁡\(𝒲,⌊ρ​ℒt⌋\)\\displaystyle=\\max\(\\mathcal\{W\},\\lfloor\\rho\\mathcal\{L\}\_\{t\}\\rfloor\)denote the number of reconstructed non\-sink tokens\. The transient working memory is then

Mwork​\(t\)=2​mt​𝒟′,M\_\{\\mathrm\{work\}\}\(t\)=2m\_\{t\}\\mathcal\{D\}^\{\\prime\},\(23\)which corresponds to the reconstructed key/value buffer\. Thus, the peak per\-step memory is approximately

Mpeak​\(t\)≈MS4​R​\(t\)\+Mwork​\(t\)\.M\_\{\\mathrm\{peak\}\}\(t\)\\approx M\_\{\\mathrm\{S^\{4\}R\}\}\(t\)\+M\_\{\\mathrm\{work\}\}\(t\)\.\(24\)Sincemt≪ℒtm\_\{t\}\\ll\\mathcal\{L\}\_\{t\}andr≪𝒟′r\\ll\\mathcal\{D\}^\{\\prime\}, S4R reduces the persistent KV cache substantially while avoiding full\-cache reconstruction during decoding\.

## 4Experiments

In this section, we comprehensively evaluate the effectiveness and memory savings of S4R across multiple benchmarks\. We also conduct speed evaluations to analyze the impact of our method on generation efficiency\. Additionally, we perform ablation studies to analyze the contributions of key components within the S4R method\.

Unless otherwise specified, the rankrris obtained by solving Eq\. \([21](https://arxiv.org/html/2608.00528#S3.E21)\) for the target retention ratioη\\eta, and the reconstruction ratio is set toρ=η\\rho=\\eta\. We uses=4s=4sink tokens following the default setting ofXiaoet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib9)\), a query\-window size of𝒲=32\\mathcal\{W\}=32, and a pooling kernel size ofκ=7\\kappa=7following the default setting ofLiet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib28)\)\. For token relevance aggregation, we use max pooling forℱ𝒫\\mathcal\{F\}\_\{\\mathcal\{P\}\}and mean aggregation forℱ𝒢\\mathcal\{F\}\_\{\\mathcal\{G\}\}\. The number of sampled tokens𝒯′\\mathcal\{T\}^\{\\prime\}is determined by the product of the total prompt length and the target retention ratio, with a recent\-to\-uniform sampling split of7:37\{:\}3\.

### 4\.1LongBench and RULER Results

![Refer to caption](https://arxiv.org/html/2608.00528v1/x1.png)

Figure 1:Performance–speed trade\-off on Qwen3\-4B\. The y\-axis is broken to include the dominated Palu\.Table 1:Performance comparison of different models and methods on LongBench\.LongBenchFigure[1](https://arxiv.org/html/2608.00528#S4.F1)summarizes the main accuracy–efficiency trade\-off on Qwen3\-4B by combining the LongBench average scores in Table[1](https://arxiv.org/html/2608.00528#S4.T1)with the output\-throughput measurements in Table[3](https://arxiv.org/html/2608.00528#S4.T3)\. S4R lies on the favorable part of the trade\-off curve: it achieves a LongBench average score of 47\.73 \(Table[7](https://arxiv.org/html/2608.00528#A4.T7)\) under the latency\-oriented efficiency setting, close to the default LongBench result of 47\.79 in Table[1](https://arxiv.org/html/2608.00528#S4.T1), while reaching 8\.32 output tokens/s in Table[3](https://arxiv.org/html/2608.00528#S4.T3)\. In contrast, xKV has a lower Qwen3\-4B LongBench average score in Table[1](https://arxiv.org/html/2608.00528#S4.T1)and is much slower in Table[3](https://arxiv.org/html/2608.00528#S4.T3), with output throughput dropping to 1\.96 tokens/s and 512\-token latency increasing to 260\.76s\. Palu and ShadowKV are faster, but Table[1](https://arxiv.org/html/2608.00528#S4.T1)shows that Palu collapses under aggressive compression and ShadowKV remains below S4R on Qwen3\-4B accuracy\.

Table[1](https://arxiv.org/html/2608.00528#S4.T1)further expands this comparison across model families\. S4R substantially outperforms Palu across all evaluated models, showing that prompt\-aware subspace construction is important when the KV cache is compressed to 20–30% of its original size\. Compared with ShadowKV and xKV, S4R achieves the best average score on Llama\-3\.2\-1B and Qwen3\-4B, and remains competitive on larger models\. On Llama\-3\.1\-8B and Qwen2\.5\-14B, xKV obtains slightly higher average scores, but this accuracy difference should be interpreted together with the efficiency gap\. Overall, Figure[1](https://arxiv.org/html/2608.00528#S4.F1), Table[1](https://arxiv.org/html/2608.00528#S4.T1), and Table[3](https://arxiv.org/html/2608.00528#S4.T3)show that S4R preserves most of the LongBench accuracy of stronger prompt\-dependent low\-rank baselines while avoiding their large runtime overhead\.

RULERTable[2](https://arxiv.org/html/2608.00528#S4.T2)reports the RULER results under a 2\.5×\\timescompression ratio and a 128k context window\. In addition to low\-rank baselines, we include two representative eviction\-based methods, StreamingLLM \(SLM\) and SnapKV, to compare against token\-retention strategies\. S4R achieves an average score of 93\.61, nearly matching the full KV cache result of 93\.97 and slightly outperforming xKV at 93\.47\. It also improves over Palu, ShadowKV, SnapKV, and SLM by 11\.39, 4\.00, 4\.80, and 45\.22 points on average, respectively\. The gains are especially clear on multi\-key and multi\-query retrieval tasks, where S4R preserves perfect accuracy, suggesting that the sampled low\-rank subspace and sparse reconstruction strategy retain the information needed for long\-context retrieval\. The main remaining gap appears on FWE, where S4R trails full KV and xKV, indicating that some tasks remain sensitive to the exact retained or reconstructed token set\.

Table 2:Comparison of different methods on RULER using Qwen2\.5\-7B\-Instruct\-1M under a 2\.5×\\timescompression ratio and a 128k context window\.Table 3:Model\-level efficiency comparison on Qwen3\-4B using a 120k\-token context sampled from GovReport\. Latency512denotes the average end\-to\-end latency for generating 512 output tokens\. All methods are evaluated without system\-level, kernel\-level, or operator\-level optimizations\. Appendix[B](https://arxiv.org/html/2608.00528#A2)provides the corresponding theoretical efficiency estimates\.- †\\daggerFor ShadowKV, we keep values on GPU rather than following its standard CPU\-offloading implementation, which removes CPU–GPU value\-loading communication overhead and therefore gives ShadowKV a more favorable efficiency estimate\.
- ‡\\ddaggerFor S4R, this efficiency experiment uses a latency\-oriented query window𝒲=4\\mathcal\{W\}=4\. Appendix[D\.1](https://arxiv.org/html/2608.00528#A4.SS1)shows that this has negligible impact on Qwen3\-4B accuracy, with an average score of 47\.73 compared with 47\.79 for𝒲=32\\mathcal\{W\}=32\.

### 4\.2Efficiency Evaluation

We report model\-level generation efficiency using TTFT, average latency for generating 512 output tokens, time per output token \(TPOT\), output throughput, and total throughput\.

Table[3](https://arxiv.org/html/2608.00528#S4.T3)shows that S4R introduces moderate overhead compared with the full KV baseline and fixed/offline\-style methods, mainly because it performs latent\-space scoring and sparse reconstruction during decoding\. Nevertheless, the overhead is small in this setting: S4R reaches a 512\-token latency of 61\.54s, close to Full KV, Palu, and ShadowKV at 54\.89s, 57\.06s, and 56\.92s, respectively\. More importantly, S4R is substantially more efficient than xKV: it reduces TTFT from 79\.90s to 27\.65s and the 512\-token generation latency from 260\.76s to 61\.54s, while improving output throughput from 1\.96 to 8\.32 tokens/s\. This supports the intended trade\-off of S4R: compared with more expensive prompt\-dependent low\-rank approaches, it retains prompt adaptivity while avoiding the cost of decomposing or processing the full prompt during inference\. Compared with Palu and ShadowKV, S4R is slightly slower in this unoptimized implementation, but the LongBench and RULER results show that the additional runtime is exchanged for stronger accuracy under aggressive KV compression\.

### 4\.3Ablation Study

We ablate three components that are central to S4R in the main text\. Unless otherwise specified, ablation tables report grouped LongBench scores: Fewshot, SDQA, and MDQA denote the average score within each task group, while Avg\. is computed over the same 11 tasks as Table[1](https://arxiv.org/html/2608.00528#S4.T1)\. Additional ablations are provided in Appendix[D](https://arxiv.org/html/2608.00528#A4)\.

#### 4\.3\.1Importance of Sink Token

S4R treats the first few tokens as attention sinks and keeps their key/value states in full precision\. Unlike ordinary non\-sink tokens, compressing these positions into a low\-rank coefficient space may introduce errors that are repeatedly exposed to many later queries\. We therefore isolate the effect of sink\-token preservation by comparing the default setting, which stores the firsts=4s=4sink tokens in full precision, with a variant that applies the same low\-rank compression pipeline to all tokens\.

Table 4:Effect of retaining sink tokens in full precision on grouped LongBench performance\. “w/” keeps the firsts=4s=4sink tokens uncompressed, whereas “w/o” applies the low\-rank compression pipeline to all tokens\.Table[4](https://arxiv.org/html/2608.00528#S4.T4)reports the grouped LongBench results\. Retaining sink tokens in full precision improves the average score across all four models\. The gain is largest on smaller models: the average score increases from 24\.60 to 28\.09 on Llama\-3\.2\-1B and from 45\.58 to 47\.79 on Qwen3\-4B\. The improvement remains consistent on larger models, with gains of 0\.71 points on Llama\-3\.1\-8B and 1\.15 points on Qwen2\.5\-14B\. These results indicate that the first few sink tokens provide a stable global context that is difficult to recover once compressed into the same low\-rank coefficient space as ordinary tokens\. The task\-level effects are also informative: Llama\-3\.2\-1B benefits most on Fewshot and LCC, while Qwen3\-4B shows a large improvement on passage retrieval, where PRE increases from 82\.00 to 96\.17\. Since the sink cache only contributes2​s​𝒟′2s\\mathcal\{D\}^\{\\prime\}elements per layer in Eq\. \([21](https://arxiv.org/html/2608.00528#S3.E21)\), its memory overhead is negligible for long contexts\.

#### 4\.3\.2Effect of Token\-wise Normalization

We ablate the token\-wise normalization step before SVD in Table[5](https://arxiv.org/html/2608.00528#S4.T5)\. On Llama\-3\.1\-8B, removing normalization leads to a slightly higher average score, with 53\.32 compared to 53\.03 for the default normalized variant\. However, the effect is model\-dependent: on Qwen2\.5\-14B, removing normalization causes a substantial drop from 53\.19 to 46\.27\. The degradation is especially pronounced on single\-document QA, multi\-document QA, and passage retrieval, indicating that unnormalized high\-magnitude KV states can dominate the SVD basis and produce less transferable low\-rank subspaces\. We therefore keep token\-wise normalization as the default because it improves robustness across model families, even though it is not uniformly optimal on every model\.

Table 5:Effect of token\-wise normalization before SVD\. Norm\. indicates whether sampled KV states are normalized before constructing the subspaces\.
#### 4\.3\.3Pooling and Aggregation

We study how the pooling functionℱ𝒫\\mathcal\{F\}\_\{\\mathcal\{P\}\}over neighboring token scores and the aggregation functionℱ𝒢\\mathcal\{F\}\_\{\\mathcal\{G\}\}affect the final sparse reconstruction set\. Table[6](https://arxiv.org/html/2608.00528#S4.T6)compares max and average pooling along the sequence dimension, combined with mean or max aggregation across group dimensions\. Across all four model settings, the average score varies only mildly across these choices: the largest gap among the four variants is 0\.38 for Llama\-3\.2\-1B, 0\.23 for Qwen3\-4B, 0\.67 for Llama\-3\.1\-8B, and 0\.18 for Qwen2\.5\-14B\. This suggests that S4R is not highly sensitive to the exact smoothing and group\-aggregation rule used for latent\-space relevance scores\.

Table 6:Effect of pooling and aggregation choices for latent\-space relevance estimation\.

## 5Conclusion

We presented S4R, a prompt\-aware low\-rank KV cache compression method that combines selective subspace construction with sparse reconstruction\. Instead of relying on external calibration data or decomposing the full prompt, S4R constructs key/value bases from sampled non\-sink prompt tokens, stores non\-sink KV states as latent coefficients, preserves sink tokens in full precision, and reconstructs only the local window plus globally important positions during decoding\. This design targets the main trade\-off in low\-rank KV compression: fixed methods are efficient but less adaptive, while prompt\-dependent methods are adaptive but can be costly on long prompts\.

Experiments on LongBench and RULER show that S4R maintains strong long\-context accuracy under aggressive KV compression\. On LongBench, it outperforms fixed low\-rank baselines and remains competitive with more expensive prompt\-dependent methods; on RULER, it achieves 93\.61 average score under 2\.5×\\timescompression, nearly matching full KV and slightly surpassing xKV\. Efficiency results further show that S4R substantially reduces the overhead of xKV in a 120k\-token setting, lowering TTFT from 79\.90s to 27\.65s and 512\-token latency from 260\.76s to 61\.54s\. Together, these results indicate that sampled\-token SVD and latent\-space sparse reconstruction provide a practical path toward adaptive KV cache compression for long\-context LLM inference\.

## Limitations

Our implementation lacks system\- or kernel\-level optimization, so reported efficiency reflects algorithmic overhead rather than a fully optimized deployment\. Experiments are limited to Llama and Qwen on LongBench and RULER, which emphasize long\-input understanding with relatively short outputs; behavior under very long generation, other architectures, and broader workloads remains to be validated\. S4R also depends on hyperparameters and a fixed token\-sampling heuristic, and tasks with scattered evidence \(e\.g\., FWE\) can still degrade when latent scoring misses relevant positions\.

## References

- J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebron, and S\. Sanghai \(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.Singapore,pp\. 4895–4901\.External Links:[Link](https://arxiv.org/html/2608.00528v1/anth2023.emnlp-main.298/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.298)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://arxiv.org/html/2608.00528v1/anth2024.acl-long.172/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§E\.1](https://arxiv.org/html/2608.00528#A5.SS1.p1.1),[§1](https://arxiv.org/html/2608.00528#S1.p5.4)\.
- Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2025\)LongBench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.Vienna, Austria,pp\. 3639–3664\.External Links:[Link](https://arxiv.org/html/2608.00528v1/anth2025.acl-long.183/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.183),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: dynamic kv cache compression based on pyramidal information funneling\.External Links:2406\.02069,[Link](https://arxiv.org/abs/2406.02069)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1)\.
- C\. Chang, C\. Lin, Y\. Akhauri, W\. Lin, K\. Wu, L\. Ceze, and M\. S\. Abdelfattah \(2025a\)XKV: cross\-layer svd for kv\-cache compression\.External Links:2503\.18893,[Link](https://arxiv.org/abs/2503.18893)Cited by:[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px1.p2.2),[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p3.2)\.
- C\. Chang, W\. Lin, C\. Lin, C\. Chen, Y\. Hu, P\. Wang, N\. Huang, L\. Ceze, M\. S\. Abdelfattah, and K\. Wu \(2025b\)Palu: KV\-cache compression with low\-rank projection\.Singapore\.External Links:[Link](https://openreview.net/forum?id=LWMS4pk2vK)Cited by:[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p2.1)\.
- Y\. Chen, G\. Wang, J\. Shang, S\. Cui, Z\. Zhang, T\. Liu, S\. Wang, Y\. Sun, D\. Yu, and H\. Wu \(2024\)NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time\.pp\. 7913–7926\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.428),[Link](https://aclanthology.org/2024.acl-long.428/)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1)\.
- Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou \(2025\)Ada\-kv: optimizing kv cache eviction by adaptive budget allocation for efficient llm inference\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,San Diego, CA, USA,pp\. 113152–113188\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/a40ff56daab9f4808b1e18350c8a11ce-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§E\.2](https://arxiv.org/html/2608.00528#A5.SS2.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§E\.1](https://arxiv.org/html/2608.00528#A5.SS1.p2.1),[§1](https://arxiv.org/html/2608.00528#S1.p5.4)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)KVQuant: towards 10 million context length llm inference with kv cache quantization\.Vancouver, Canada,pp\. 1270–1303\.External Links:[Document](https://dx.doi.org/10.52202/079017-0040),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/028fcbcf85435d39a40c4d61b42c99a4-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p3.2)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.Philadelphia, Pennsylvania, United States\.External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[§E\.2](https://arxiv.org/html/2608.00528#A5.SS2.p1.1),[§1](https://arxiv.org/html/2608.00528#S1.p5.4)\.
- S\. Jie, Y\. Tang, K\. Han, Z\. Deng, and J\. Han \(2025\)SpeCache: speculative key\-value caching for efficient generation of LLMs\.Vancouver, Canada,pp\. 27917–27928\.External Links:[Link](https://proceedings.mlr.press/v267/jie25a.html)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p4.2)\.
- V\. Joshi, P\. P\. Brahma, Z\. Liu, and E\. Barsoum \(2025\)TaDA: training\-free recipe for decoding with adaptive KV cache compression and mean\-centering\.Vienna, Austria,pp\. 1435–1443\.External Links:[Link](https://arxiv.org/html/2608.00528v1/anth2025.acl-industry.101/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-industry.101),ISBN 979\-8\-89176\-288\-6Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p3.2)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)GEAR: an efficient error reduction framework for KV cache compression in LLM inference\.InProceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing WorkshopAdvances in Neural Information Processing SystemsProceedings of the 41st International Conference on Machine LearningAdvances in Neural Information Processing SystemsAdvances in Neural Information Processing SystemsAdvances in Neural Information Processing SystemsProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)The Thirteenth International Conference on Learning RepresentationsProceedings of the 41st International Conference on Machine LearningAdvances in Neural Information Processing SystemsAdvances in Neural Information Processing SystemsProceedings of the 42nd International Conference on Machine LearningAdvances in Neural Information Processing SystemsProceedings of the 42nd International Conference on Machine LearningAdvances in Neural Information Processing SystemsProceedings of the 36th Annual ACM Symposium on User Interface Software and TechnologyFirst Conference on Language ModelingThe Twelfth International Conference on Learning RepresentationsProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)Proceedings of the 2023 Conference on Empirical Methods in Natural Language ProcessingFindings of the Association for Computational Linguistics: EMNLP 2024Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)Findings of the Association for Computational Linguistics: ACL 2024Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),M\. Rezagholizadeh, P\. Passban, S\. Samiee, V\. Partovi Nia, Y\. Cheng, Y\. Deng, Q\. Liu, B\. Chen, A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, S\. Levine, R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, F\. Berkenkamp, A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, C\. Zhang, D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, C\. Zhang, R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, F\. Berkenkamp, D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, S\. Levine, A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, J\. Zhu, A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, C\. Zhang, A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, J\. Zhu, H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, H\. Lin, W\. Che, J\. Nabende, E\. Shutova, M\. T\. Pilehvar, H\. Bouamor, J\. Pino, K\. Bali, Y\. Al\-Onaizan, M\. Bansal, Y\. Chen, L\. Ku, A\. Martins, V\. Srikumar, L\. Ku, A\. Martins, V\. Srikumar, G\. Rehm, and Y\. Li \(Eds\.\),Proceedings of Machine Learning ResearchProceedings of Machine Learning ResearchProceedings of Machine Learning ResearchProceedings of Machine Learning ResearchProceedings of Machine Learning ResearchUIST ’23, Vol\.2623623537383723538362673726733,Vancouver, Canada,pp\. 305–321\.External Links:[Link](https://proceedings.mlr.press/v262/kang24a.html)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p3.2)\.
- J\. Kim, J\. Kim, S\. Kwon, J\. W\. Lee, S\. Yun, and H\. O\. Song \(2025\)KVzip: query\-agnostic kv cache compression with context reconstruction\.San Diego, CA, USA,pp\. 167563–167591\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/f4eaa4b8f2d08edb3f0af990d56134ea-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p4.2)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- J\. Lee, S\. Hwang, A\. Qiao, D\. Campos, Z\. Yao, and Y\. He \(2025\)TALE: token\-adaptive low\-rank KVCache approximation with reconstruction elimination\.Transactions of the Association for Computational Linguistics13,pp\. 1298–1318\.External Links:[Link](https://arxiv.org/html/2608.00528v1/anth2025.tacl-1.59/),[Document](https://dx.doi.org/10.1162/tacl.a.39)Cited by:[§D\.2](https://arxiv.org/html/2608.00528#A4.SS2.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Virtual,pp\. 9459–9474\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- H\. LI, Y\. Li, A\. Tian, T\. Tang, Z\. Xu, X\. Chen, N\. HU, W\. Dong, L\. Qing, and L\. Chen \(2025\)A survey on large language model acceleration based on KV cache management\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=z3JZzu9EA3)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: llm knows what you are looking for before generation\.Vancouver, Canada,pp\. 22947–22970\.External Links:[Document](https://dx.doi.org/10.52202/079017-0722),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/28ab418242603e0f7323e54185d19bde-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1),[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px2.p1.1),[§3\.2\.3](https://arxiv.org/html/2608.00528#S3.SS2.SSS3.p1.1),[§4](https://arxiv.org/html/2608.00528#S4.p2.10)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.New Orleans, LA, USA,pp\. 52342–52364\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a452a7c6c463e4ae8fbdc614c6e983e6-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024\)KIVI: a tuning\-free asymmetric 2bit quantization for KV cache\.Vienna, Austria,pp\. 32332–32344\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bz.html)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p3.2)\.
- J\. Mu, H\. Huang, J\. Zhang, M\. Yu, T\. Wang, and Y\. Li \(2025\)SALS: sparse attention in latent space for kv cache compression\.San Diego, CA, USA,pp\. 384–403\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/00a0ebcad584c59dbc439c2af8793638-Paper-Conference.pdf)Cited by:[§D\.5](https://arxiv.org/html/2608.00528#A4.SS5.p1.2),[§2](https://arxiv.org/html/2608.00528#S2.p2.1)\.
- OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, J\. Belgum, I\. Bello, J\. Berdine, G\. Bernadett\-Shapiro, C\. Berner, L\. Bogdonoff, O\. Boiko, M\. Boyd, A\. Brakman, G\. Brockman, T\. Brooks, M\. Brundage, K\. Button, T\. Cai, R\. Campbell, A\. Cann, B\. Carey, C\. Carlson, R\. Carmichael, B\. Chan, C\. Chang, F\. Chantzis, D\. Chen, S\. Chen, R\. Chen, J\. Chen, M\. Chen, B\. Chess, C\. Cho, C\. Chu, H\. W\. Chung, D\. Cummings, J\. Currier, Y\. Dai, C\. Decareaux, T\. Degry, N\. Deutsch, D\. Deville, A\. Dhar, D\. Dohan, S\. Dowling, S\. Dunning, A\. Ecoffet, A\. Eleti, T\. Eloundou, D\. Farhi, L\. Fedus, N\. Felix, S\. P\. Fishman, J\. Forte, I\. Fulford, L\. Gao, E\. Georges, C\. Gibson, V\. Goel, T\. Gogineni, G\. Goh, R\. Gontijo\-Lopes, J\. Gordon, M\. Grafstein, S\. Gray, R\. Greene, J\. Gross, S\. S\. Gu, Y\. Guo, C\. Hallacy, J\. Han, J\. Harris, Y\. He, M\. Heaton, J\. Heidecke, C\. Hesse, A\. Hickey, W\. Hickey, P\. Hoeschele, B\. Houghton, K\. Hsu, S\. Hu, X\. Hu, J\. Huizinga, S\. Jain, S\. Jain, J\. Jang, A\. Jiang, R\. Jiang, H\. Jin, D\. Jin, S\. Jomoto, B\. Jonn, H\. Jun, T\. Kaftan, Ł\. Kaiser, A\. Kamali, I\. Kanitscheider, N\. S\. Keskar, T\. Khan, L\. Kilpatrick, J\. W\. Kim, C\. Kim, Y\. Kim, J\. H\. Kirchner, J\. Kiros, M\. Knight, D\. Kokotajlo, Ł\. Kondraciuk, A\. Kondrich, A\. Konstantinidis, K\. Kosic, G\. Krueger, V\. Kuo, M\. Lampe, I\. Lan, T\. Lee, J\. Leike, J\. Leung, D\. Levy, C\. M\. Li, R\. Lim, M\. Lin, S\. Lin, M\. Litwin, T\. Lopez, R\. Lowe, P\. Lue, A\. Makanju, K\. Malfacini, S\. Manning, T\. Markov, Y\. Markovski, B\. Martin, K\. Mayer, A\. Mayne, B\. McGrew, S\. M\. McKinney, C\. McLeavey, P\. McMillan, J\. McNeil, D\. Medina, A\. Mehta, J\. Menick, L\. Metz, A\. Mishchenko, P\. Mishkin, V\. Monaco, E\. Morikawa, D\. Mossing, T\. Mu, M\. Murati, O\. Murk, D\. Mély, A\. Nair, R\. Nakano, R\. Nayak, A\. Neelakantan, R\. Ngo, H\. Noh, L\. Ouyang, C\. O’Keefe, J\. Pachocki, A\. Paino, J\. Palermo, A\. Pantuliano, G\. Parascandolo, J\. Parish, E\. Parparita, A\. Passos, M\. Pavlov, A\. Peng, A\. Perelman, F\. de Avila Belbute Peres, M\. Petrov, H\. P\. de Oliveira Pinto, Michael, Pokorny, M\. Pokrass, V\. H\. Pong, T\. Powell, A\. Power, B\. Power, E\. Proehl, R\. Puri, A\. Radford, J\. Rae, A\. Ramesh, C\. Raymond, F\. Real, K\. Rimbach, C\. Ross, B\. Rotsted, H\. Roussez, N\. Ryder, M\. Saltarelli, T\. Sanders, S\. Santurkar, G\. Sastry, H\. Schmidt, D\. Schnurr, J\. Schulman, D\. Selsam, K\. Sheppard, T\. Sherbakov, J\. Shieh, S\. Shoker, P\. Shyam, S\. Sidor, E\. Sigler, M\. Simens, J\. Sitkin, K\. Slama, I\. Sohl, B\. Sokolowsky, Y\. Song, N\. Staudacher, F\. P\. Such, N\. Summers, I\. Sutskever, J\. Tang, N\. Tezak, M\. B\. Thompson, P\. Tillet, A\. Tootoonchian, E\. Tseng, P\. Tuggle, N\. Turley, J\. Tworek, J\. F\. C\. Uribe, A\. Vallone, A\. Vijayvergiya, C\. Voss, C\. Wainwright, J\. J\. Wang, A\. Wang, B\. Wang, J\. Ward, J\. Wei, C\. Weinmann, A\. Welihinda, P\. Welinder, J\. Weng, L\. Weng, M\. Wiethoff, D\. Willner, C\. Winter, S\. Wolrich, H\. Wong, L\. Workman, S\. Wu, J\. Wu, M\. Wu, K\. Xiao, T\. Xu, S\. Yoo, K\. Yu, Q\. Yuan, W\. Zaremba, R\. Zellers, C\. Zhang, M\. Zhang, S\. Zhao, T\. Zheng, J\. Zhuang, W\. Zhuk, and B\. Zoph \(2024\)GPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- J\. S\. Park, J\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.New York, NY, USA\.External Links:ISBN 9798400701320,[Link](https://doi.org/10.1145/3586183.3606763),[Document](https://dx.doi.org/10.1145/3586183.3606763)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- B\. Peng, J\. Quesnelle, H\. Fan, and E\. Shippole \(2024\)YaRN: efficient context window extension of large language models\.Vienna, Austria\.External Links:[Link](https://openreview.net/forum?id=wHBfxhZu1u)Cited by:[§E\.1](https://arxiv.org/html/2608.00528#A5.SS1.p2.1)\.
- J\. Peng, Y\. Wu, H\. Wu, J\. Han, X\. Xie, K\. Tu, and J\. Lai \(2026\)GRKV: global regression for training\-free kv cache compression in long\-context llms\.arXiv preprint arXiv:2605\.31105\.Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p2.1)\.
- R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. Dean \(2023\)Efficiently scaling transformer inference\.InProceedings of Machine Learning and Systems,D\. Song, M\. Carbin, and T\. Chen \(Eds\.\),Vol\.5,Miami Beach, Florida, USA,pp\. 606–624\.External Links:[Link](https://proceedings.mlsys.org/paper_files/paper/2023/file/c4be71ab8d24cdfb45e3d06dbfca2780-Paper-mlsys2023.pdf)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§E\.1](https://arxiv.org/html/2608.00528#A5.SS1.p2.1),[§1](https://arxiv.org/html/2608.00528#S1.p5.4)\.
- I\. Rehg \(2024\)KV\-compress: paged kv\-cache compression with variable compression rates per attention head\.External Links:2410\.00161,[Link](https://arxiv.org/abs/2410.00161)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p4.2)\.
- U\. Saxena, G\. Saha, S\. Choudhary, and K\. Roy \(2024\)Eigen attention: attention in low\-rank space for KV cache compression\.Miami, Florida, USA,pp\. 15332–15344\.External Links:[Link](https://arxiv.org/html/2608.00528v1/anth2024.findings-emnlp.899/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.899)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p2.1)\.
- P\. Singhania, S\. Singh, S\. He, S\. Feizi, and A\. Bhatele \(2024\)Loki: low\-rank keys for efficient sparse attention\.Vancouver, Canada,pp\. 16692–16723\.External Links:[Document](https://dx.doi.org/10.52202/079017-0532),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/1e027da6bec9ceb2ec37951ceeccae93-Paper-Conference.pdf)Cited by:[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px2.p2.3),[§2](https://arxiv.org/html/2608.00528#S2.p2.1)\.
- H\. Sun, L\. Chang, W\. Bao, S\. Zheng, N\. Zheng, X\. Liu, H\. Dong, Y\. Chi, and B\. Chen \(2025\)ShadowKV: KV cache in shadows for high\-throughput long\-context LLM inference\.Vancouver, Canada,pp\. 57355–57373\.External Links:[Link](https://proceedings.mlr.press/v267/sun25b.html)Cited by:[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px1.p2.2),[§D\.2](https://arxiv.org/html/2608.00528#A4.SS2.p1.1),[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p3.2)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)QUEST: query\-aware sparsity for efficient long\-context LLM inference\.Vienna, Austria,pp\. 47901–47911\.External Links:[Link](https://proceedings.mlr.press/v235/tang24l.html)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1),[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px2.p2.3)\.
- G\. Team, P\. Georgiev, V\. I\. Lei, R\. Burnell, L\. Bai, A\. Gulati, G\. Tanzer, D\. Vincent, Z\. Pan, S\. Wang, S\. Mariooryad, Y\. Ding, X\. Geng, F\. Alcober, R\. Frostig, M\. Omernick, L\. Walker, C\. Paduraru, C\. Sorokin, A\. Tacchetti, C\. Gaffney, S\. Daruki, O\. Sercinoglu, Z\. Gleicher, J\. Love, P\. Voigtlaender, R\. Jain, G\. Surita, K\. Mohamed, R\. Blevins, J\. Ahn, T\. Zhu, K\. Kawintiranon, O\. Firat, Y\. Gu, Y\. Zhang, M\. Rahtz, M\. Faruqui, N\. Clay, J\. Gilmer, J\. Co\-Reyes, I\. Penchev, R\. Zhu, N\. Morioka, K\. Hui, K\. Haridasan, V\. Campos, M\. Mahdieh, M\. Guo, S\. Hassan, K\. Kilgour, A\. Vezer, H\. Cheng, R\. de Liedekerke, S\. Goyal, P\. Barham, D\. Strouse, S\. Noury, J\. Adler, M\. Sundararajan, S\. Vikram, D\. Lepikhin, M\. Paganini, X\. Garcia, F\. Yang, D\. Valter, M\. Trebacz, K\. Vodrahalli, C\. Asawaroengchai, R\. Ring, N\. Kalb, L\. B\. Soares, S\. Brahma, D\. Steiner, T\. Yu, F\. Mentzer, A\. He, L\. Gonzalez, B\. Xu, R\. L\. Kaufman, L\. E\. Shafey, J\. Oh, T\. Hennigan, G\. van den Driessche, S\. Odoom, M\. Lucic, B\. Roelofs, S\. Lall, A\. Marathe, B\. Chan, S\. Ontanon, L\. He, D\. Teplyashin, J\. Lai, P\. Crone, B\. Damoc, L\. Ho, S\. Riedel, K\. Lenc, C\. Yeh, A\. Chowdhery, Y\. Xu, M\. Kazemi, E\. Amid, A\. Petrushkina, K\. Swersky, A\. Khodaei, G\. Chen, C\. Larkin, M\. Pinto, G\. Yan, A\. P\. Badia, P\. Patil, S\. Hansen, D\. Orr, S\. M\. R\. Arnold, J\. Grimstad, A\. Dai, S\. Douglas, R\. Sinha, V\. Yadav, X\. Chen, E\. Gribovskaya, J\. Austin, J\. Zhao, K\. Patel, P\. Komarek, S\. Austin, S\. Borgeaud, L\. Friso, A\. Goyal, B\. Caine, K\. Cao, D\. Chung, M\. Lamm, G\. Barth\-Maron, T\. Kagohara, K\. Olszewska, M\. Chen, K\. Shivakumar, R\. Agarwal, H\. Godhia, R\. Rajwar, J\. Snaider, X\. Dotiwalla, Y\. Liu, A\. Barua, V\. Ungureanu, Y\. Zhang, B\. Batsaikhan, M\. Wirth, J\. Qin, I\. Danihelka, T\. Doshi, M\. Chadwick, J\. Chen, S\. Jain, Q\. Le, A\. Kar, M\. Gurumurthy, C\. Li, R\. Sang, F\. Liu, L\. Lamprou, R\. Munoz, N\. Lintz, H\. Mehta, H\. Howard, M\. Reynolds, L\. Aroyo, Q\. Wang, L\. Blanco, A\. Cassirer, J\. Griffith, D\. Das, S\. Lee, J\. Sygnowski, Z\. Fisher, J\. Besley, R\. Powell, Z\. Ahmed, D\. Paulus, D\. Reitter, Z\. Borsos, R\. Joshi, A\. Pope, S\. Hand, V\. Selo, V\. Jain, N\. Sethi, M\. Goel, T\. Makino, R\. May, Z\. Yang, J\. Schalkwyk, C\. Butterfield, A\. Hauth, A\. Goldin, W\. Hawkins, E\. Senter, S\. Brin, O\. Woodman, M\. Ritter, E\. Noland, M\. Giang, V\. Bolina, L\. Lee, T\. Blyth, I\. Mackinnon, M\. Reid, O\. Sarvana, D\. Silver, A\. Chen, L\. Wang, L\. Maggiore, O\. Chang, N\. Attaluri, G\. Thornton, C\. Chiu, O\. Bunyan, N\. Levine, T\. Chung, E\. Eltyshev, X\. Si, T\. Lillicrap, D\. Brady, V\. Aggarwal, B\. Wu, Y\. Xu, R\. McIlroy, K\. Badola, P\. Sandhu, E\. Moreira, W\. Stokowiec, R\. Hemsley, D\. Li, A\. Tudor, P\. Shyam, E\. Rahimtoroghi, S\. Haykal, P\. Sprechmann, X\. Zhou, D\. Mincu, Y\. Li, R\. Addanki, K\. Krishna, X\. Wu, A\. Frechette, M\. Eyal, A\. Dafoe, D\. Lacey, J\. Whang, T\. Avrahami, Y\. Zhang, E\. Taropa, H\. Lin, D\. Toyama, E\. Rutherford, M\. Sano, H\. Choe, A\. Tomala, C\. Safranek\-Shrader, N\. Kassner, M\. Pajarskas, M\. Harvey, S\. Sechrist, M\. Fortunato, C\. Lyu, G\. Elsayed, C\. Kuang, J\. Lottes, E\. Chu, C\. Jia, C\. Chen, P\. Humphreys, K\. Baumli, C\. Tao, R\. Samuel, C\. N\. dos Santos, A\. Andreassen, N\. Rakićević, D\. Grewe, A\. Kumar, S\. Winkler, J\. Caton, A\. Brock, S\. Dalmia, H\. Sheahan, I\. Barr, Y\. Miao, P\. Natsev, J\. Devlin, F\. Behbahani, F\. Prost, Y\. Sun, A\. Myaskovsky, T\. S\. Pillai, D\. Hurt, A\. Lazaridou, X\. Xiong, C\. Zheng, F\. Pardo, X\. Li, D\. Horgan, J\. Stanton, M\. Ambar, F\. Xia, A\. Lince, M\. Wang, B\. Mustafa, A\. Webson, H\. Lee, R\. Anil, M\. Wicke, T\. Dozat, A\. Sinha, E\. Piqueras, E\. Dabir, S\. Upadhyay, A\. Boral, L\. A\. Hendricks, C\. Fry, J\. Djolonga, Y\. Su, J\. Walker, J\. Labanowski, R\. Huang, V\. Misra, J\. Chen, R\. Skerry\-Ryan, A\. Singh, S\. Rijhwani, D\. Yu, A\. Castro\-Ros, B\. Changpinyo, R\. Datta, S\. Bagri, A\. M\. Hrafnkelsson, M\. Maggioni, D\. Zheng, Y\. Sulsky, S\. Hou, T\. L\. Paine, A\. Yang, J\. Riesa, D\. Rogozinska, D\. Marcus, D\. E\. Badawy, Q\. Zhang, L\. Wang, H\. Miller, J\. Greer, L\. L\. Sjos, A\. Nova, H\. Zen, R\. Chaabouni, M\. Rosca, J\. Jiang, C\. Chen, R\. Liu, T\. Sainath, M\. Krikun, A\. Polozov, J\. Lespiau, J\. Newlan, Z\. Cankara, S\. Kwak, Y\. Xu, P\. Chen, A\. Coenen, C\. Meyer, K\. Tsihlas, A\. Ma, J\. Gottweis, J\. Xing, C\. Gu, J\. Miao, C\. Frank, Z\. Cankara, S\. Ganapathy, I\. Dasgupta, S\. Hughes\-Fitt, H\. Chen, D\. Reid, K\. Rong, H\. Fan, J\. van Amersfoort, V\. Zhuang, A\. Cohen, S\. S\. Gu, A\. Mohananey, A\. Ilic, T\. Tobin, J\. Wieting, A\. Bortsova, P\. Thacker, E\. Wang, E\. Caveness, J\. Chiu, E\. Sezener, A\. Kaskasoli, S\. Baker, K\. Millican, M\. Elhawaty, K\. Aisopos, C\. Lebsack, N\. Byrd, H\. Dai, W\. Jia, M\. Wiethoff, E\. Davoodi, A\. Weston, L\. Yagati, A\. Ahuja, I\. Gao, G\. Pundak, S\. Zhang, M\. Azzam, K\. C\. Sim, S\. Caelles, J\. Keeling, A\. Sharma, A\. Swing, Y\. Li, C\. Liu, C\. G\. Bostock, Y\. Bansal, Z\. Nado, A\. Anand, J\. Lipschultz, A\. Karmarkar, L\. Proleev, A\. Ittycheriah, S\. H\. Yeganeh, G\. Polovets, A\. Faust, J\. Sun, A\. Rrustemi, P\. Li, R\. Shivanna, J\. Liu, C\. Welty, F\. Lebron, A\. Baddepudi, S\. Krause, E\. Parisotto, R\. Soricut, Z\. Xu, D\. Bloxwich, M\. Johnson, B\. Neyshabur, J\. Mao\-Jones, R\. Wang, V\. Ramasesh, Z\. Abbas, A\. Guez, C\. Segal, D\. D\. Nguyen, J\. Svensson, L\. Hou, S\. York, K\. Milan, S\. Bridgers, W\. Gworek, M\. Tagliasacchi, J\. Lee\-Thorp, M\. Chang, A\. Guseynov, A\. J\. Hartman, M\. Kwong, R\. Zhao, S\. Kashem, E\. Cole, A\. Miech, R\. Tanburn, M\. Phuong, F\. Pavetic, S\. Cevey, R\. Comanescu, R\. Ives, S\. Yang, C\. Du, B\. Li, Z\. Zhang, M\. Iinuma, C\. H\. Hu, A\. Roy, S\. Bijwadia, Z\. Zhu, D\. Martins, R\. Saputro, A\. Gergely, S\. Zheng, D\. Jia, I\. Antonoglou, A\. Sadovsky, S\. Gu, Y\. Bi, A\. Andreev, S\. Samangooei, M\. Khan, T\. Kocisky, A\. Filos, C\. Kumar, C\. Bishop, A\. Yu, S\. Hodkinson, S\. Mittal, P\. Shah, A\. Moufarek, Y\. Cheng, A\. Bloniarz, J\. Lee, P\. Pejman, P\. Michel, S\. Spencer, V\. Feinberg, X\. Xiong, N\. Savinov, C\. Smith, S\. Shakeri, D\. Tran, M\. Chesus, B\. Bohnet, G\. Tucker, T\. von Glehn, C\. Muir, Y\. Mao, H\. Kazawa, A\. Slone, K\. Soparkar, D\. Shrivastava, J\. Cobon\-Kerr, M\. Sharman, J\. Pavagadhi, C\. Araya, K\. Misiunas, N\. Ghelani, M\. Laskin, D\. Barker, Q\. Li, A\. Briukhov, N\. Houlsby, M\. Glaese, B\. Lakshminarayanan, N\. Schucher, Y\. Tang, E\. Collins, H\. Lim, F\. Feng, A\. Recasens, G\. Lai, A\. Magni, N\. D\. Cao, A\. Siddhant, Z\. Ashwood, J\. Orbay, M\. Dehghani, J\. Brennan, Y\. He, K\. Xu, Y\. Gao, C\. Saroufim, J\. Molloy, X\. Wu, S\. Arnold, S\. Chang, J\. Schrittwieser, E\. Buchatskaya, S\. Radpour, M\. Polacek, S\. Giordano, A\. Bapna, S\. Tokumine, V\. Hellendoorn, T\. Sottiaux, S\. Cogan, A\. Severyn, M\. Saleh, S\. Thakoor, L\. Shefey, S\. Qiao, M\. Gaba, S\. Chang, C\. Swanson, B\. Zhang, B\. Lee, P\. K\. Rubenstein, G\. Song, T\. Kwiatkowski, A\. Koop, A\. Kannan, D\. Kao, P\. Schuh, A\. Stjerngren, G\. Ghiasi, G\. Gibson, L\. Vilnis, Y\. Yuan, F\. T\. Ferreira, A\. Kamath, T\. Klimenko, K\. Franko, K\. Xiao, I\. Bhattacharya, M\. Patel, R\. Wang, A\. Morris, R\. Strudel, V\. Sharma, P\. Choy, S\. H\. Hashemi, J\. Landon, M\. Finkelstein, P\. Jhakra, J\. Frye, M\. Barnes, M\. Mauger, D\. Daun, K\. Baatarsukh, M\. Tung, W\. Farhan, H\. Michalewski, F\. Viola, F\. de Chaumont Quitry, C\. L\. Lan, T\. Hudson, Q\. Wang, F\. Fischer, I\. Zheng, E\. White, A\. Dragan, J\. Alayrac, E\. Ni, A\. Pritzel, A\. Iwanicki, M\. Isard, A\. Bulanova, L\. Zilka, E\. Dyer, D\. Sachan, S\. Srinivasan, H\. Muckenhirn, H\. Cai, A\. Mandhane, M\. Tariq, J\. W\. Rae, G\. Wang, K\. Ayoub, N\. FitzGerald, Y\. Zhao, W\. Han, C\. Alberti, D\. Garrette, K\. Krishnakumar, M\. Gimenez, A\. Levskaya, D\. Sohn, J\. Matak, I\. Iturrate, M\. B\. Chang, J\. Xiang, Y\. Cao, N\. Ranka, G\. Brown, A\. Hutter, V\. Mirrokni, N\. Chen, K\. Yao, Z\. Egyed, F\. Galilee, T\. Liechty, P\. Kallakuri, E\. Palmer, S\. Ghemawat, J\. Liu, D\. Tao, C\. Thornton, T\. Green, M\. Jasarevic, S\. Lin, V\. Cotruta, Y\. Tan, N\. Fiedel, H\. Yu, E\. Chi, A\. Neitz, J\. Heitkaemper, A\. Sinha, D\. Zhou, Y\. Sun, C\. Kaed, B\. Hulse, S\. Mishra, M\. Georgaki, S\. Kudugunta, C\. Farabet, I\. Shafran, D\. Vlasic, A\. Tsitsulin, R\. Ananthanarayanan, A\. Carin, G\. Su, P\. Sun, S\. V, G\. Carvajal, J\. Broder, I\. Comsa, A\. Repina, W\. Wong, W\. W\. Chen, P\. Hawkins, E\. Filonov, L\. Loher, C\. Hirnschall, W\. Wang, J\. Ye, A\. Burns, H\. Cate, D\. G\. Wright, F\. Piccinini, L\. Zhang, C\. Lin, I\. Gog, Y\. Kulizhskaya, A\. Sreevatsa, S\. Song, L\. C\. Cobo, A\. Iyer, C\. Tekur, G\. Garrido, Z\. Xiao, R\. Kemp, H\. S\. Zheng, H\. Li, A\. Agarwal, C\. Ngani, K\. Goshvadi, R\. Santamaria\-Fernandez, W\. Fica, X\. Chen, C\. Gorgolewski, S\. Sun, R\. Garg, X\. Ye, S\. M\. A\. Eslami, N\. Hua, J\. Simon, P\. Joshi, Y\. Kim, I\. Tenney, S\. Potluri, L\. N\. Thiet, Q\. Yuan, F\. Luisier, A\. Chronopoulou, S\. Scellato, P\. Srinivasan, M\. Chen, V\. Koverkathu, V\. Dalibard, Y\. Xu, B\. Saeta, K\. Anderson, T\. Sellam, N\. Fernando, F\. Huot, J\. Jung, M\. Varadarajan, M\. Quinn, A\. Raul, M\. Le, R\. Habalov, J\. Clark, K\. Jalan, K\. Bullard, A\. Singhal, T\. Luong, B\. Wang, S\. Rajayogam, J\. Eisenschlos, J\. Jia, D\. Finchelstein, A\. Yakubovich, D\. Balle, M\. Fink, S\. Agarwal, J\. Li, D\. Dvijotham, S\. Pal, K\. Kang, J\. Konzelmann, J\. Beattie, O\. Dousse, D\. Wu, R\. Crocker, C\. Elkind, S\. R\. Jonnalagadda, J\. Lee, D\. Holtmann\-Rice, K\. Kallarackal, R\. Liu, D\. Vnukov, N\. Vats, L\. Invernizzi, M\. Jafari, H\. Zhou, L\. Taylor, J\. Prendki, M\. Wu, T\. Eccles, T\. Liu, K\. Kopparapu, F\. Beaufays, C\. Angermueller, A\. Marzoca, S\. Sarcar, H\. Dib, J\. Stanway, F\. Perbet, N\. Trdin, R\. Sterneck, A\. Khorlin, D\. Li, X\. Wu, S\. Goenka, D\. Madras, S\. Goldshtein, W\. Gierke, T\. Zhou, Y\. Liu, Y\. Liang, A\. White, Y\. Li, S\. Singh, S\. Bahargam, M\. Epstein, S\. Basu, L\. Lao, A\. Ozturel, C\. Crous, A\. Zhai, H\. Lu, Z\. Tung, N\. Gaur, A\. Walton, L\. Dixon, M\. Zhang, A\. Globerson, G\. Uy, A\. Bolt, O\. Wiles, M\. Nasr, I\. Shumailov, M\. Selvi, F\. Piccinno, R\. Aguilar, S\. McCarthy, M\. Khalman, M\. Shukla, V\. Galic, J\. Carpenter, K\. Villela, H\. Zhang, H\. Richardson, J\. Martens, M\. Bosnjak, S\. R\. Belle, J\. Seibert, M\. Alnahlawi, B\. McWilliams, S\. Singh, A\. Louis, W\. Ding, D\. Popovici, L\. Simicich, L\. Knight, P\. Mehta, N\. Gupta, C\. Shi, S\. Fatehi, J\. Mitrovic, A\. Grills, J\. Pagadora, T\. Munkhdalai, D\. Petrova, D\. Eisenbud, Z\. Zhang, D\. Yates, B\. Mittal, N\. Tripuraneni, Y\. Assael, T\. Brovelli, P\. Jain, M\. Velimirovic, C\. Akbulut, J\. Mu, W\. Macherey, R\. Kumar, J\. Xu, H\. Qureshi, G\. Comanici, J\. Wiesner, Z\. Gong, A\. Ruddock, M\. Bauer, N\. Felt, A\. GP, A\. Arnab, D\. Zelle, J\. Rothfuss, B\. Rosgen, A\. Shenoy, B\. Seybold, X\. Li, J\. Mudigonda, G\. Erdogan, J\. Xia, J\. Simsa, A\. Michi, Y\. Yao, C\. Yew, S\. Kan, I\. Caswell, C\. Radebaugh, A\. Elisseeff, P\. Valenzuela, K\. McKinney, K\. Paterson, A\. Cui, E\. Latorre\-Chimoto, S\. Kim, W\. Zeng, K\. Durden, P\. Ponnapalli, T\. Sosea, C\. A\. Choquette\-Choo, J\. Manyika, B\. Robenek, H\. Vashisht, S\. Pereira, H\. Lam, M\. Velic, D\. Owusu\-Afriyie, K\. Lee, T\. Bolukbasi, A\. Parrish, S\. Lu, J\. Park, B\. Venkatraman, A\. Talbert, L\. Rosique, Y\. Cheng, A\. Sozanschi, A\. Paszke, P\. Kumar, J\. Austin, L\. Li, K\. Salama, B\. Perz, W\. Kim, N\. Dukkipati, A\. Baryshnikov, C\. Kaplanis, X\. Sheng, Y\. Chervonyi, C\. Unlu, D\. de Las Casas, H\. Askham, K\. Tunyasuvunakool, F\. Gimeno, S\. Poder, C\. Kwak, M\. Miecnikowski, V\. Mirrokni, A\. Dimitriev, A\. Parisi, D\. Liu, T\. Tsai, T\. Shevlane, C\. Kouridi, D\. Garmon, A\. Goedeckemeyer, A\. R\. Brown, A\. Vijayakumar, A\. Elqursh, S\. Jazayeri, J\. Huang, S\. M\. Carthy, J\. Hoover, L\. Kim, S\. Kumar, W\. Chen, C\. Biles, G\. Bingham, E\. Rosen, L\. Wang, Q\. Tan, D\. Engel, F\. Pongetti, D\. de Cesare, D\. Hwang, L\. Yu, J\. Pullman, S\. Narayanan, K\. Levin, S\. Gopal, M\. Li, A\. Aharoni, T\. Trinh, J\. Lo, N\. Casagrande, R\. Vij, L\. Matthey, B\. Ramadhana, A\. Matthews, C\. Carey, M\. Johnson, K\. Goranova, R\. Shah, S\. Ashraf, K\. Dasgupta, R\. Larsen, Y\. Wang, M\. R\. Vuyyuru, C\. Jiang, J\. Ijazi, K\. Osawa, C\. Smith, R\. S\. Boppana, T\. Bilal, Y\. Koizumi, Y\. Xu, Y\. Altun, N\. Shabat, B\. Bariach, A\. Korchemniy, K\. Choo, O\. Ronneberger, C\. Iwuanyanwu, S\. Zhao, D\. Soergel, C\. Hsieh, I\. Cai, S\. Iqbal, M\. Sundermeyer, Z\. Chen, E\. Bursztein, C\. Malaviya, F\. Biadsy, P\. Shroff, I\. Dhillon, T\. Latkar, C\. Dyer, H\. Forbes, M\. Nicosia, V\. Nikolaev, S\. Greene, M\. Georgiev, P\. Wang, N\. Martin, H\. Sedghi, J\. Zhang, P\. Banzal, D\. Fritz, V\. Rao, X\. Wang, J\. Zhang, V\. Patraucean, D\. Du, I\. Mordatch, I\. Jurin, L\. Liu, A\. Dubey, A\. Mohan, J\. Nowakowski, V\. Ion, N\. Wei, R\. Tojo, M\. A\. Raad, D\. A\. Hudson, V\. Keshava, S\. Agrawal, K\. Ramirez, Z\. Wu, H\. Nguyen, J\. Liu, M\. Sewak, B\. Petrini, D\. Choi, I\. Philips, Z\. Wang, I\. Bica, A\. Garg, J\. Wilkiewicz, P\. Agrawal, X\. Li, D\. Guo, E\. Xue, N\. Shaik, A\. Leach, S\. M\. Khan, J\. Wiesinger, S\. Jerome, A\. Chakladar, A\. W\. Wang, T\. Ornduff, F\. Abu, A\. Ghaffarkhah, M\. Wainwright, M\. Cortes, F\. Liu, J\. Maynez, A\. Terzis, P\. Samangouei, R\. Mansour, T\. Kępa, F\. Aubet, A\. Algymr, D\. Banica, A\. Weisz, A\. Orban, A\. Senges, E\. Andrejczuk, M\. Geller, N\. D\. Santo, V\. Anklin, M\. A\. Merey, M\. Baeuml, T\. Strohman, J\. Bai, S\. Petrov, Y\. Wu, D\. Hassabis, K\. Kavukcuoglu, J\. Dean, and O\. Vinyals \(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.External Links:2403\.05530,[Link](https://arxiv.org/abs/2403.05530)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. Wen \(2024a\)A survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\)\.External Links:ISSN 2095\-2236,[Link](http://dx.doi.org/10.1007/s11704-024-40231-1),[Document](https://dx.doi.org/10.1007/s11704-024-40231-1)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p1.1)\.
- T\. Wang, M\. Huang, F\. Li, L\. Chen, J\. Zhang, and J\. Ren \(2025a\)DynaKV: enabling accurate and efficient long\-sequence llm decoding on smartphones\.External Links:2511\.07427,[Link](https://arxiv.org/abs/2511.07427)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p2.1)\.
- Y\. Wang, H\. Qiao, L\. Li, Q\. Zhu, and W\. Che \(2025b\)CommonKV: compressing kv cache with cross\-layer parameter sharing\.External Links:2508\.16134,[Link](https://arxiv.org/abs/2508.16134)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p2.1)\.
- Z\. Wang, B\. Jin, Z\. Yu, and M\. Zhang \(2024b\)Model tells you where to merge: adaptive kv cache merging for llms on long\-context tasks\.External Links:2407\.08454,[Link](https://arxiv.org/abs/2407.08454)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p2.1)\.
- G\. Xiao, J\. Tang, J\. Zuo, J\. Guo, S\. Yang, H\. Tang, Y\. Fu, and S\. Han \(2025\)DuoAttention: Efficient Long\-Context LLM Inference with Retrieval and Streaming Heads\.InInternational Conference on Learning Representations,Singapore\.External Links:[Link](https://openreview.net/forum?id=cFu7ze7xUm)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations,Vienna, Austria\.External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1),[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.00528#S1.p3.1),[§3\.1\.1](https://arxiv.org/html/2608.00528#S3.SS1.SSS1.p1.1),[§4](https://arxiv.org/html/2608.00528#S4.p2.10)\.
- X\. Yan, Z\. Li, T\. Zhang, H\. Qin, L\. Kong, Y\. Zhang, and X\. Yang \(2025\)ReCalKV: low\-rank kv cache compression via head reordering and offline calibration\.External Links:2505\.24357,[Link](https://arxiv.org/abs/2505.24357)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025a\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§E\.1](https://arxiv.org/html/2608.00528#A5.SS1.p2.1),[§1](https://arxiv.org/html/2608.00528#S1.p5.4)\.
- A\. Yang, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Huang, J\. Jiang, J\. Tu, J\. Zhang, J\. Zhou, J\. Lin, K\. Dang, K\. Yang, L\. Yu, M\. Li, M\. Sun, Q\. Zhu, R\. Men, T\. He, W\. Xu, W\. Yin, W\. Yu, X\. Qiu, X\. Ren, X\. Yang, Y\. Li, Z\. Xu, and Z\. Zhang \(2025b\)Qwen2\.5\-1m technical report\.External Links:2501\.15383,[Link](https://arxiv.org/abs/2501.15383)Cited by:[§E\.2](https://arxiv.org/html/2608.00528#A5.SS2.p1.1),[§1](https://arxiv.org/html/2608.00528#S1.p1.1),[§1](https://arxiv.org/html/2608.00528#S1.p5.4)\.
- D\. Yang, X\. Han, Y\. Gao, Y\. Hu, S\. Zhang, and H\. Zhao \(2024\)PyramidInfer: pyramid KV cache compression for high\-throughput LLM inference\.Bangkok, Thailand,pp\. 3258–3270\.External Links:[Link](https://arxiv.org/html/2608.00528v1/anth2024.findings-acl.195/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.195)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1)\.
- H\. Yu, Z\. Yang, S\. Li, Y\. Li, and J\. Wu \(2024\)Effectively compress kv heads for llm\.External Links:2406\.07056,[Link](https://arxiv.org/abs/2406.07056)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p1.1)\.
- R\. Zhang, K\. Wang, L\. Liu, S\. Wang, H\. Cheng, C\. Zhang, and Y\. Shen \(2024a\)LoRC: low\-rank compression for llms kv cache with a progressive compression strategy\.External Links:2410\.03111,[Link](https://arxiv.org/abs/2410.03111)Cited by:[§C\.2](https://arxiv.org/html/2608.00528#A3.SS2.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2608.00528#S1.p2.1),[§2](https://arxiv.org/html/2608.00528#S2.p2.1)\.
- Y\. Zhang, Y\. Du, G\. Luo, Y\. Zhong, Z\. Zhang, S\. Liu, and R\. Ji \(2024b\)CaM: cache merging for memory\-efficient LLMs inference\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,Vienna, Austria,pp\. 58840–58850\.External Links:[Link](https://proceedings.mlr.press/v235/zhang24n.html)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p2.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. "\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.New Orleans, LA, USA,pp\. 34661–34710\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f8-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2608.00528#A3.SS1.p1.1)\.
- Y\. Zhu, D\. H\. Yang, M\. M\. Amiri, K\. Murugesan, T\. Pedapati, and P\. Chen \(2026\)OjaKV: context\-aware online low\-rank kv cache compression\.External Links:2509\.21623,[Link](https://arxiv.org/abs/2509.21623)Cited by:[§1](https://arxiv.org/html/2608.00528#S1.p2.1)\.

## Appendix APreliminaries

### A\.1KV Cache Computation and Update

The inference process of typical LLMs consists of two main stages:prefillinganddecoding\. During theprefillingstage, the input prompt is passed through all layers of the model to generate the KV cache for each transformer layer\. This cache stores the key and value tensors, which are reused in subsequentdecodingsteps, significantly reducing redundant computation\. Additionally, during thedecodingstage, each newly generated token is also projected into key and value tensors, which are appended to the existing KV cache\. In this way, the KV cache is progressively updated as the sequence grows, facilitating efficient autoregressive generation\.

##### Prefilling

Let𝐗∈ℝℬ×𝒯×𝒟\\mathbf\{X\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{T\}\\times\\mathcal\{D\}\}denote the input tensor, whereℬ\\mathcal\{B\}is the batch size,𝒯\\mathcal\{T\}is the prompt length, and𝒟\\mathcal\{D\}is the hidden dimension\. The corresponding key and value tensors,𝐒K\\mathbf\{S\}^\{K\}and𝐒V\\mathbf\{S\}^\{V\}, are computed as follows:

𝐒K/V=𝐗𝐖K/V,\\mathbf\{S\}^\{K/V\}=\\mathbf\{X\}\\mathbf\{W\}^\{K/V\},\(25\)where𝐖K/V∈ℝ𝒟×𝒟′\\mathbf\{W\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{D\}\\times\\mathcal\{D\}^\{\\prime\}\}are the key and value projection matrices, respectively\. Here,𝒟′=hk​v⋅dh\\mathcal\{D\}^\{\\prime\}=h\_\{kv\}\\cdot d\_\{h\}, withhk​vh\_\{kv\}representing the number of KV heads anddhd\_\{h\}the head dimension\. The computed𝐒K/V∈ℝℬ×𝒯×𝒟′\\mathbf\{S\}^\{K/V\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times\\mathcal\{T\}\\times\\mathcal\{D\}^\{\\prime\}\}are stored as the initial KV cache to enable efficient retrieval duringdecoding\.

Similarly, the query tensor is obtained as

𝐐=𝐗𝐖Q,\\mathbf\{Q\}=\\mathbf\{X\}\\mathbf\{W\}^\{Q\},\(26\)where𝐖Q∈ℝ𝒟×𝒢​𝒟′\\mathbf\{W\}^\{Q\}\\in\\mathbb\{R\}^\{\\mathcal\{D\}\\times\\mathcal\{G\}\\mathcal\{D\}^\{\\prime\}\}\. The parameter𝒢\\mathcal\{G\}denotes the query\-to\-KV head ratio\. When𝒢\>1\\mathcal\{G\}\>1, the model corresponds to grouped\-query attention \(GQA\); when𝒢=1\\mathcal\{G\}=1, it reduces to multi\-head attention \(MHA\)\.

##### Decoding

At eachdecodingtimesteptt, the newly generated input token𝐱t∈ℝℬ×1×𝒟\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{\\mathcal\{B\}\\times 1\\times\\mathcal\{D\}\}is projected to obtain the corresponding key and value vectors𝐬tK/V\\mathbf\{s\}^\{K/V\}\_\{t\}:

𝐬tK/V=𝐱t​𝐖K/V\.\\mathbf\{s\}^\{K/V\}\_\{t\}=\\mathbf\{x\}\_\{t\}\\mathbf\{W\}^\{K/V\}\.\(27\)The KV cache is then updated by appending the new entries:

𝐒tK/V=𝐒t−1K/V⊕𝐬tK/V,\\mathbf\{S\}^\{K/V\}\_\{t\}=\\mathbf\{S\}^\{K/V\}\_\{t\-1\}\\oplus\\mathbf\{s\}^\{K/V\}\_\{t\},\(28\)where⊕\\oplusdenotes concatenation along𝒯\\mathcal\{T\}\.

### A\.2Singular Value Decomposition

SVD is a fundamental tool in matrix analysis that enables us to decompose any matrix into a set of orthogonal basis vectors and singular values, facilitating an understanding of its structure and enabling efficient computations such as dimensionality reduction\. Given a matrix𝐀∈ℝm×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{m\\times n\}, the SVD factorizes it as follows:

SVD​\(𝐀\)=𝐔​𝚺​𝐙⊤,\\text\{SVD\}\(\\mathbf\{A\}\)=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{Z\}^\{\\top\},\(29\)where𝐔∈ℝm×m\\mathbf\{U\}\\in\\mathbb\{R\}^\{m\\times m\}and𝐙∈ℝn×n\\mathbf\{Z\}\\in\\mathbb\{R\}^\{n\\times n\}are orthogonal matrices whose columns are, respectively, the left and right singular vectors of𝐀\\mathbf\{A\}\. The matrix𝚺∈ℝm×n\\mathbf\{\\Sigma\}\\in\\mathbb\{R\}^\{m\\times n\}is diagonal \(typically rectangular\), with non\-negative singular values arranged in descending order\.

For practical purposes such as dimensionality reduction and compression, one often uses the truncated rank\-rrSVD, which retains only therrlargest singular values and their corresponding singular vectors:

SVDr​\(𝐀\)=𝐔r​𝚺r​𝐙r⊤,\\text\{SVD\}\_\{r\}\(\\mathbf\{A\}\)=\\mathbf\{U\}\_\{r\}\\mathbf\{\\Sigma\}\_\{r\}\\mathbf\{Z\}\_\{r\}^\{\\top\},\(30\)where𝐔r∈ℝm×r\\mathbf\{U\}\_\{r\}\\in\\mathbb\{R\}^\{m\\times r\}and𝐙r∈ℝn×r\\mathbf\{Z\}\_\{r\}\\in\\mathbb\{R\}^\{n\\times r\}contain the firstrrleft and right singular vectors, respectively, and𝚺r∈ℝr×r\\mathbf\{\\Sigma\}\_\{r\}\\in\\mathbb\{R\}^\{r\\times r\}is a diagonal matrix of the toprrsingular values\.

## Appendix BDecoding\-Time Complexity Comparison

This section provides a detailed decoding\-time complexity comparison between S4R and a full\-reconstruction low\-rank baseline\. We focus on the per\-layer cost at a single decoding step and omit the batch dimension for clarity\. Letℒt\\mathcal\{L\}\_\{t\}denote the current non\-sink cache length,ssthe number of sink tokens,𝒟′\\mathcal\{D\}^\{\\prime\}the KV hidden dimension,rrthe low\-rank dimension,𝒢\\mathcal\{G\}the query\-to\-KV head ratio,𝒲\\mathcal\{W\}the query\-window size, andρ\\rhothe sparse reconstruction ratio\. The number of non\-sink tokens reconstructed by S4R is

mt=\|ℐ^\|=max⁡\(𝒲,⌊ρ​ℒt⌋\)\.m\_\{t\}=\|\\widehat\{\\mathcal\{I\}\}\|=\\max\(\\mathcal\{W\},\\lfloor\\rho\\mathcal\{L\}\_\{t\}\\rfloor\)\.\(31\)For long contexts whereρ​ℒt≥𝒲\\rho\\mathcal\{L\}\_\{t\}\\geq\\mathcal\{W\}, we havemt≈ρ​ℒtm\_\{t\}\\approx\\rho\\mathcal\{L\}\_\{t\}\.

### B\.1Full\-Reconstruction Low\-Rank Decoding

A natural alternative to sparse reconstruction is to store the non\-sink KV cache in coefficient form but reconstruct the entire cache at every decoding step before applying attention\. In this full\-reconstruction baseline, all cached non\-sink keys and values are recovered as

𝐊t=𝐂tK​\(𝐁K\)⊤,𝐕t=𝐂tV​\(𝐁V\)⊤\.\\mathbf\{K\}\_\{t\}=\\mathbf\{C\}\_\{t\}^\{K\}\(\\mathbf\{B\}^\{K\}\)^\{\\top\},\\qquad\\mathbf\{V\}\_\{t\}=\\mathbf\{C\}\_\{t\}^\{V\}\(\\mathbf\{B\}^\{V\}\)^\{\\top\}\.\(32\)The reconstruction cost for keys and values is therefore

O​\(2​ℒt​r​𝒟′\),O\(2\\mathcal\{L\}\_\{t\}r\\mathcal\{D\}^\{\\prime\}\),\(33\)which we write asO​\(ℒt​r​𝒟′\)O\(\\mathcal\{L\}\_\{t\}r\\mathcal\{D\}^\{\\prime\}\)after dropping constant factors\. The final attention is then computed over the full non\-sink cache, together with the sink tokens\. Ignoring the small sink\-token term for long contexts, the attention score computation and value aggregation have cost

O​\(𝒢​ℒt​𝒟′\)\.O\(\\mathcal\{G\}\\mathcal\{L\}\_\{t\}\\mathcal\{D\}^\{\\prime\}\)\.\(34\)Thus, the dominant per\-step decoding complexity of full reconstruction is

Cfull​\-​rec\\displaystyle C\_\{\\mathrm\{full\\text\{\-\}rec\}\}=O​\(ℒt​r​𝒟′\+𝒢​ℒt​𝒟′\)\\displaystyle=O\\left\(\\mathcal\{L\}\_\{t\}r\\mathcal\{D\}^\{\\prime\}\+\\mathcal\{G\}\\mathcal\{L\}\_\{t\}\\mathcal\{D\}^\{\\prime\}\\right\)\(35\)=O​\(ℒt​𝒟′​\(r\+𝒢\)\)\.\\displaystyle=O\\left\(\\mathcal\{L\}\_\{t\}\\mathcal\{D\}^\{\\prime\}\(r\+\\mathcal\{G\}\)\\right\)\.This baseline reduces persistent KV memory because it stores low\-rank coefficients, but its decoding computation still scales with the full cache length in both reconstruction and attention\.

### B\.2S4R Sparse\-Reconstruction Decoding

S4R avoids reconstructing the full cache\. At each decoding step, the new query, key, and value states are first projected into the learned low\-rank subspaces\. The cost of this cache update is

O​\(\(𝒢\+2\)​𝒟′​r\),O\(\(\\mathcal\{G\}\+2\)\\mathcal\{D\}^\{\\prime\}r\),\(36\)where the𝒢\\mathcal\{G\}term comes from grouped query projection and the remaining two terms come from key and value projection\. This term does not scale withℒt\\mathcal\{L\}\_\{t\}and is therefore lower order for long contexts\.

S4R then estimates token relevance in the latent space by multiplying the query\-coefficient window with the cached key coefficients:

𝐙=𝐂Q​\(𝐂tK\)⊤r,𝐙∈ℝ𝒢×𝒲×ℒt\.\\mathbf\{Z\}=\\frac\{\\mathbf\{C\}^\{Q\}\(\\mathbf\{C\}\_\{t\}^\{K\}\)^\{\\top\}\}\{\\sqrt\{r\}\},\\qquad\\mathbf\{Z\}\\in\\mathbb\{R\}^\{\\mathcal\{G\}\\times\\mathcal\{W\}\\times\\mathcal\{L\}\_\{t\}\}\.\(37\)This latent scoring step costs

O​\(𝒢​𝒲​ℒt​r\)\.O\(\\mathcal\{G\}\\mathcal\{W\}\\mathcal\{L\}\_\{t\}r\)\.\(38\)The subsequent softmax, pooling, group aggregation, and TopK selection add lower\-order terms such asO​\(𝒢​𝒲​ℒt\)O\(\\mathcal\{G\}\\mathcal\{W\}\\mathcal\{L\}\_\{t\}\)andO​\(ℒt​log⁡k\)O\(\\mathcal\{L\}\_\{t\}\\log k\), or approximatelyO​\(ℒt\)O\(\\mathcal\{L\}\_\{t\}\)with efficient selection\.

After token selection, S4R reconstructs onlymtm\_\{t\}selected non\-sink tokens:

𝐊^=𝐂tK\(𝐁K\)⊤\[ℐ^\],𝐕^=𝐂tV\(𝐁V\)⊤\[ℐ^\]\.\\widehat\{\\mathbf\{K\}\}=\\mathbf\{C\}\_\{t\}^\{K\}\{\}\_\{\[\\widehat\{\\mathcal\{I\}\}\]\}\(\\mathbf\{B\}^\{K\}\)^\{\\top\},\\qquad\\widehat\{\\mathbf\{V\}\}=\\mathbf\{C\}\_\{t\}^\{V\}\{\}\_\{\[\\widehat\{\\mathcal\{I\}\}\]\}\(\\mathbf\{B\}^\{V\}\)^\{\\top\}\.\(39\)The sparse reconstruction cost is

O​\(2​mt​r​𝒟′\),O\(2m\_\{t\}r\\mathcal\{D\}^\{\\prime\}\),\(40\)which isO​\(mt​r​𝒟′\)O\(m\_\{t\}r\\mathcal\{D\}^\{\\prime\}\)after omitting constant factors\. The final exact attention is performed only over the reconstructed non\-sink tokens and the full\-precision sink tokens, with cost

O​\(𝒢​\(s\+mt\)​𝒟′\)\.O\(\\mathcal\{G\}\(s\+m\_\{t\}\)\\mathcal\{D\}^\{\\prime\}\)\.\(41\)Ignoring lower\-order terms and the small sink\-token contribution, the dominant S4R decoding complexity is therefore

CS4​R=O​\(𝒢​𝒲​ℒt​r\+mt​r​𝒟′\+𝒢​mt​𝒟′\)\.C\_\{\\mathrm\{S^\{4\}R\}\}=O\\left\(\\mathcal\{G\}\\mathcal\{W\}\\mathcal\{L\}\_\{t\}r\+m\_\{t\}r\\mathcal\{D\}^\{\\prime\}\+\\mathcal\{G\}m\_\{t\}\\mathcal\{D\}^\{\\prime\}\\right\)\.\(42\)Whenmt≈ρ​ℒtm\_\{t\}\\approx\\rho\\mathcal\{L\}\_\{t\}, this becomes

CS4​R=O​\(𝒢​𝒲​ℒt​r\+ρ​ℒt​r​𝒟′\+ρ​𝒢​ℒt​𝒟′\)\.C\_\{\\mathrm\{S^\{4\}R\}\}=O\\left\(\\mathcal\{G\}\\mathcal\{W\}\\mathcal\{L\}\_\{t\}r\+\\rho\\mathcal\{L\}\_\{t\}r\\mathcal\{D\}^\{\\prime\}\+\\rho\\mathcal\{G\}\\mathcal\{L\}\_\{t\}\\mathcal\{D\}^\{\\prime\}\\right\)\.\(43\)Equivalently,

CS4​R=O​\(ℒt​\[𝒢​𝒲​r\+ρ​𝒟′​\(r\+𝒢\)\]\)\.C\_\{\\mathrm\{S^\{4\}R\}\}=O\\left\(\\mathcal\{L\}\_\{t\}\\left\[\\mathcal\{G\}\\mathcal\{W\}r\+\\rho\\mathcal\{D\}^\{\\prime\}\(r\+\\mathcal\{G\}\)\\right\]\\right\)\.\(44\)

### B\.3Asymptotic Comparison

Equations \([35](https://arxiv.org/html/2608.00528#A2.E35)\) and \([44](https://arxiv.org/html/2608.00528#A2.E44)\) show the main difference between the two approaches\. Full reconstruction pays the reconstruction and attention cost over allℒt\\mathcal\{L\}\_\{t\}cached non\-sink tokens; S4R reduces both reconstruction and exact attention fromℒt\\mathcal\{L\}\_\{t\}tokens to approximatelyρ​ℒt\\rho\\mathcal\{L\}\_\{t\}tokens, but introduces latent\-space relevance scoring\. Therefore, S4R is computationally favorable over full reconstruction when

𝒢​𝒲​r\+ρ​𝒟′​\(r\+𝒢\)<𝒟′​\(r\+𝒢\)\.\\mathcal\{G\}\\mathcal\{W\}r\+\\rho\\mathcal\{D\}^\{\\prime\}\(r\+\\mathcal\{G\}\)<\\mathcal\{D\}^\{\\prime\}\(r\+\\mathcal\{G\}\)\.\(45\)Equivalently,

𝒢​𝒲​r<\(1−ρ\)​𝒟′​\(r\+𝒢\)\.\\mathcal\{G\}\\mathcal\{W\}r<\(1\-\\rho\)\\mathcal\{D\}^\{\\prime\}\(r\+\\mathcal\{G\}\)\.\(46\)This condition is usually satisfied when the query window𝒲\\mathcal\{W\}and rankrrare much smaller than the original KV dimension𝒟′\\mathcal\{D\}^\{\\prime\}, and when the reconstruction ratioρ\\rhois substantially below one\.

### B\.4Concrete Example

We instantiate the above comparison using the latency\-oriented efficiency setting in Table[3](https://arxiv.org/html/2608.00528#S4.T3), where𝒲=4\\mathcal\{W\}=4\. Considerr=204r=204,ρ=0\.2\\rho=0\.2, and𝒟′=1024\\mathcal\{D\}^\{\\prime\}=1024\. After removing the common factorℒt\\mathcal\{L\}\_\{t\}, the full\-reconstruction cost coefficient is

r​𝒟′\+𝒢​𝒟′\\displaystyle r\\mathcal\{D\}^\{\\prime\}\+\\mathcal\{G\}\\mathcal\{D\}^\{\\prime\}=204×1024\+1024​𝒢\\displaystyle=04\\times 024\+024\\mathcal\{G\}\(47\)=208,896\+1024​𝒢\.\\displaystyle=08\{,\}96\+024\\mathcal\{G\}\.For S4R, the latent\-scoring coefficient is

𝒢​𝒲​r=4×204​𝒢=816​𝒢,\\mathcal\{G\}\\mathcal\{W\}r=4\\times 204\\mathcal\{G\}=816\\mathcal\{G\},\(48\)while the sparse reconstruction coefficient is

ρ​r​𝒟′=0\.2×204×1024=41,779\.2,\\rho r\\mathcal\{D\}^\{\\prime\}=0\.2\\times 204\\times 1024=41\{,\}779\.2,\(49\)and the sparse attention coefficient is

ρ​𝒢​𝒟′=0\.2×1024​𝒢=204\.8​𝒢\.\\rho\\mathcal\{G\}\\mathcal\{D\}^\{\\prime\}=0\.2\\times 1024\\mathcal\{G\}=204\.8\\mathcal\{G\}\.\(50\)Thus,

CS4​R/ℒt≈41,779\+1021​𝒢,C\_\{\\mathrm\{S^\{4\}R\}\}/\\mathcal\{L\}\_\{t\}\\approx 41\{,\}779\+1021\\mathcal\{G\},\(51\)whereas

Cfull​\-​rec/ℒt≈208,896\+1024​𝒢\.C\_\{\\mathrm\{full\\text\{\-\}rec\}\}/\\mathcal\{L\}\_\{t\}\\approx 208\{,\}896\+1024\\mathcal\{G\}\.\(52\)For𝒢=1\\mathcal\{G\}=1, this gives approximately

CS4​RCfull​\-​rec≈42,800209,920≈0\.204\.\\frac\{C\_\{\\mathrm\{S^\{4\}R\}\}\}\{C\_\{\\mathrm\{full\\text\{\-\}rec\}\}\}\\approx\\frac\{42\{,\}800\}\{209\{,\}920\}\\approx 0\.204\.\(53\)For𝒢=4\\mathcal\{G\}=4, the ratio is

CS4​RCfull​\-​rec≈45,862212,922≈0\.215\.\\frac\{C\_\{\\mathrm\{S^\{4\}R\}\}\}\{C\_\{\\mathrm\{full\\text\{\-\}rec\}\}\}\\approx\\frac\{45\{,\}862\}\{212\{,\}922\}\\approx 0\.215\.\(54\)These estimates indicate that, under this setting, S4R reduces the dominant reconstruction\-and\-attention arithmetic by roughly4\.64\.6–5×5\\timesrelative to full reconstruction\. The latent scoring term is small compared with the saved reconstruction cost because

𝒲​r=4×204=816≪ρ​r​𝒟′=41,779\.2\.\\mathcal\{W\}r=4\\times 204=816\\ll\\rho r\\mathcal\{D\}^\{\\prime\}=41\{,\}779\.2\.\(55\)Consequently, the leading effect is that sparse reconstruction replaces the full reconstruction costO​\(ℒt​r​𝒟′\)O\(\\mathcal\{L\}\_\{t\}r\\mathcal\{D\}^\{\\prime\}\)withO​\(ρ​ℒt​r​𝒟′\)O\(\\rho\\mathcal\{L\}\_\{t\}r\\mathcal\{D\}^\{\\prime\}\), yielding an approximate1/ρ1/\\rhoreconstruction\-compute reduction when the latent\-scoring overhead is small\.

## Appendix CMore Related Work

We provide additional discussion of related work beyond low\-rank KV cache compression, together with a fine\-grained comparison between S4R and representative methods from each category\.

### C\.1Beyond Low\-Rank Compression

Beyond low\-rank compression, KV cache management has been studied from several complementary perspectives\. A major line of work reduces the number of cached or attended tokens through eviction or sparse selection\. StreamingLLM identifies attention sinks and combines them with a sliding window to support stable streaming inferenceXiaoet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib9)\)\. H2O keeps heavy\-hitter tokens based on their accumulated attention importanceZhanget al\.\([2023](https://arxiv.org/html/2608.00528#bib.bib11)\), while Scissorhands exploits the persistence of token importance during generationLiuet al\.\([2023](https://arxiv.org/html/2608.00528#bib.bib26)\)\. NACL proposes a general eviction framework that combines local and global importance signalsChenet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib19)\), and Ada\-KV further improves eviction by adaptively allocating cache budgets across headsFenget al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib1)\)\. SnapKV clusters recent prompt attention to select useful historical tokens before generationLiet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib28)\)\. PyramidInfer and PyramidKV exploit the observation that different layers require different cache budgets, retaining more tokens in layers that carry more crucial context informationYanget al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib50)\); Caiet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib22)\)\. Quest performs query\-aware sparse KV selection for long\-context inferenceTanget al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib23)\), and DuoAttention separates retrieval heads from streaming heads to avoid treating all attention heads uniformlyXiaoet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib4)\)\. These methods primarily reduce the sequence dimension of the KV cache or the set of tokens used in attention\. In contrast, S4R stores most non\-sink tokens in a low\-rank coefficient space and uses sparse reconstruction only at decoding time, reducing the hidden dimension of the persistent cache while still allowing globally important positions to be recovered when needed\.

Another family of methods compresses the cache by merging or clustering similar KV states\. CaM merges cache entries to reduce memory while preserving representative informationZhanget al\.\([2024b](https://arxiv.org/html/2608.00528#bib.bib2)\), KVMerger adaptively identifies suitable KV states for merging in long\-context tasksWanget al\.\([2024b](https://arxiv.org/html/2608.00528#bib.bib18)\), and Chelsea performs online KV cache clustering based on sequence\-wise key similarityWanget al\.\([2025a](https://arxiv.org/html/2608.00528#bib.bib5)\)\. GRKV formulates KV\-cache merging as a global regression problemPenget al\.\([2026](https://arxiv.org/html/2608.00528#bib.bib45)\)\. These approaches preserve information by replacing multiple tokens with merged representatives\. S4R instead keeps token identities in the coefficient cache and reconstructs selected positions from learned key/value bases, which avoids irreversible token merging during prefilling\.

Quantization methods reduce the numerical precision of cached keys and values\. KIVI proposes asymmetric low\-bit KV quantization tailored to different key and value distributionsLiuet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib12)\), while KVQuant targets very long\-context inference with KV cache quantizationHooperet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib14)\)\. GEAR combines quantization with low\-rank error compensation for near\-lossless compressionKanget al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib10)\), and TaDA adapts quantization precision across layers with mean\-centeringJoshiet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib51)\)\. Quantization is largely orthogonal to S4R: it compresses the bit width of stored entries, whereas S4R changes the representation by storing low\-rank coefficients and bases\. Combining coefficient\-space compression with quantization is a promising direction for further reducing memory\.

System\-oriented methods optimize where and how the KV cache is stored or reused\. KV\-Compress evicts contiguous KV blocks within a PagedAttention\-style memory managerRehg \([2024](https://arxiv.org/html/2608.00528#bib.bib13)\), KVzip constructs query\-agnostic compressed caches that can be reused across queriesKimet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib15)\), and SpeCache offloads the full KV cache to CPU memory while using a compact GPU\-side copy to guide fetchingJieet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib29)\)\. These methods focus on memory layout, cache reuse, or CPU–GPU movement, whereas S4R focuses on the algorithmic representation of the cached keys and values\. They are therefore complementary to S4R and could potentially be combined with prompt\-aware low\-rank coefficient storage\.

### C\.2Detailed Comparison between S4R and Representative Methods

##### Low\-rank weight or cache compression\.

Palu and LoRC are fixed post\-training methods that compress KV\-related projection weights before servingChanget al\.\([2025b](https://arxiv.org/html/2608.00528#bib.bib21)\); Zhanget al\.\([2024a](https://arxiv.org/html/2608.00528#bib.bib17)\)\. Palu factorizes the key/value projection matrices and caches compact intermediate states, while LoRC applies low\-rank approximation with a progressive layer\-wise compression strategy to reduce error propagation\. Both methods have low prompt\-time decomposition overhead because their compression is largely determined before inference, but the resulting low\-rank structure is not prompt\-specific\. S4R differs by constructing key and value bases from the current prompt, using only a sampled subset of non\-sink tokens to avoid full\-prompt SVD\. Thus, S4R trades a small amount of prompt\-time adaptation for better alignment with the current context\.

ShadowKV and xKV are closer to S4R because they adapt the low\-rank representation to the current prompt rather than relying on a fixed calibration\-derived basisSunet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib27)\); Changet al\.\([2025a](https://arxiv.org/html/2608.00528#bib.bib32)\)\. Their main difference lies in where the prompt\-dependent cost is paid\. ShadowKV performs SVD on pre\-RoPE keys, keeps a low\-rank key cache on GPU, and uses sparse reconstruction during decoding while relying on value offloading and chunk\-level organization\. It partitions keys into chunks and maintains local chunks and outlier chunks in full precision so that selected sparse KV pairs can be reconstructed accurately\. These full\-precision chunks are important for ShadowKV’s sparse reconstruction quality, but under a fixed compression ratio they consume part of the cache budget and therefore reduce the effective rank available to the low\-rank key representation\. xKV instead performs cross\-layer SVD during prefilling and shares prompt\-specific low\-rank structure across grouped layers, which can preserve accuracy but incurs substantial prefill latency on long prompts\. S4R is designed as a lower\-overhead prompt\-dependent alternative: it performs SVD only on selectively sampled non\-sink tokens, constructs both key and value bases, stores non\-sink values as low\-rank coefficients rather than relying primarily on value offloading, and explicitly preserves only sink tokens in full precision\. This is why it has lower TTFT and generation latency than xKV in our efficiency study while retaining the sparse\-reconstruction benefits shared with ShadowKV\.

##### Eviction and sparse selection\.

StreamingLLM and SnapKV reduce computation by retaining a subset of tokens rather than changing the hidden\-dimensional representation of every cached tokenXiaoet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib9)\); Liet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib28)\)\. StreamingLLM keeps attention sinks and a recent window, making it effective for streaming generation but limited when important evidence appears outside the retained window\. SnapKV uses prompt attention patterns to cluster and select important historical tokens before generation\. S4R also uses sink tokens and local windows, but it does not permanently discard all non\-selected tokens from the persistent representation; non\-sink tokens remain available in the coefficient cache and can be reconstructed if selected by latent\-space relevance\.

Quest and Loki are sparse\-attention methods that avoid attending to all tokensTanget al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib23)\); Singhaniaet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib16)\)\. Quest builds query\-aware sparse KV selection to accelerate long\-context inference, while Loki computes approximate attention scores in a reduced key space to select top\-kktokens\. Their main target is reducing attention computation through sparse token access\. In contrast, S4R targets persistent KV memory compression and decoding\-time reconstruction together: it stores compressed key/value coefficients, scores tokens in latent space, and reconstructs only the local window plus globally important positions for exact attention over the selected set\. This makes S4R closer to a joint memory\-compression and sparse\-reconstruction method than a pure sparse\-attention method\.

##### Summary\.

Overall, S4R differs from fixed low\-rank methods such as Palu and LoRC by being prompt\-aware, from xKV by avoiding full\-prompt cross\-layer SVD, from ShadowKV by using sampled key/value bases and coefficient\-space value storage, from eviction methods such as StreamingLLM and SnapKV by retaining compressed representations of non\-selected tokens, and from sparse\-attention methods such as Quest and Loki by explicitly reducing the persistent KV representation rather than only selecting tokens for attention\.

## Appendix DMore Ablation Studies

### D\.1Effect of Query Window

S4R uses a sliding query window of length𝒲\\mathcal\{W\}for two related purposes during decoding\. First, the projected queries in this window are used to estimate token relevance in the latent coefficient space, where attention scores are averaged across recent queries before selecting globally important positions\. Second, the same window determines the local tokens that are always reconstructed and attended to in full precision\. Thus,𝒲\\mathcal\{W\}controls how much short\-range decoding context is used for stable token scoring and how much recent context is guaranteed to be preserved in the sparse reconstruction set\.

This ablation varies𝒲\\mathcal\{W\}while keeping the rank, sampling strategy, reconstruction ratio, and pooling configuration fixed\. When𝒲=1\\mathcal\{W\}=1, token selection is driven only by the current query, which provides the lowest query\-coefficient storage and latent scoring cost but may produce noisier global\-token estimates\. Larger windows aggregate relevance over multiple recent queries and preserve more local tokens, which can improve robustness for tasks where the answer depends on a short span of recent decoding context\. However, increasing𝒲\\mathcal\{W\}also increases the query\-coefficient buffer size and the latent attention cost, and it allocates more of the reconstruction budget to local tokens, leaving fewer slots for globally selected positions when the total reconstruction ratio is fixed\.

Table[7](https://arxiv.org/html/2608.00528#A4.T7)compares𝒲∈\{1,4,8,16,32,64\}\\mathcal\{W\}\\in\\\{1,4,8,16,32,64\\\}across LongBench task groups\. The results show a clear preference for moderate query windows\. Using only the current query is consistently weaker: increasing𝒲\\mathcal\{W\}from11to the best\-performing setting improves the average score from 25\.90 to 28\.68 on Llama\-3\.2\-1B, from 47\.31 to 47\.85 on Qwen3\-4B, from 51\.92 to 53\.50 on Llama\-3\.1\-8B, and from 52\.82 to 53\.67 on Qwen2\.5\-14B\. These gains indicate that aggregating relevance over several recent queries produces more stable global\-token estimates than single\-query selection\.

The benefit, however, saturates quickly\. The strongest averages are obtained by𝒲=16\\mathcal\{W\}=16for Llama\-3\.2\-1B, Qwen3\-4B, and Qwen2\.5\-14B, and by𝒲=4\\mathcal\{W\}=4for Llama\-3\.1\-8B\. The default𝒲=32\\mathcal\{W\}=32remains close to the optimum, trailing the best setting by 0\.59, 0\.06, 0\.47, and 0\.48 points on the four models, respectively, while maintaining a larger guaranteed local reconstruction window\. Increasing the window further to𝒲=64\\mathcal\{W\}=64lowers the average score for all models compared with𝒲=32\\mathcal\{W\}=32\. This suggests that overly large query windows dilute current\-step relevance and reserve more reconstruction budget for local tokens, thereby reducing the coverage of globally important positions\. We therefore keep𝒲=32\\mathcal\{W\}=32as the default in the main experiments not because it is always the best setting in this ablation, but because it provides near\-optimal accuracy and matches the default window size used by SnapKV\.

Table 7:Effect of the query\-window size𝒲∈\{1,4,8,16,32,64\}\\mathcal\{W\}\\in\\\{1,4,8,16,32,64\\\}on grouped LongBench performance\. All other S4R hyperparameters are fixed, and shaded rows denote the default setting used in the main experiments\.
### D\.2Effect of Different Rank Allocation

Prior work observes that pre\-RoPE keys exhibit stronger low\-rank characteristics than valuesLeeet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib52)\); Sunet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib27)\)\. This asymmetry suggests that keys may tolerate a smaller latent rank, while values may require a larger rank to preserve the information aggregated into the final attention output\. Motivated by this observation, we restrict this ablation to settings where the key rank is no larger than the value rank, i\.e\.,rK≤rVr^\{K\}\\leq r^\{V\}\. The goal is to test whether shifting part of the coefficient budget from keys to values improves reconstruction quality without substantially degrading latent\-space token scoring\.

To study this trade\-off, we compare several key\-to\-value rank ratios while keeping the total coefficient budget fixed\. Concretely, for a target budget corresponding to the default rankrr, we choose\(rK,rV\)\(r^\{K\},r^\{V\}\)such thatrK\+rV=2​rr^\{K\}\+r^\{V\}=2rand vary the allocation ratiorK:rVr^\{K\}:r^\{V\}\.

Table[8](https://arxiv.org/html/2608.00528#A4.T8)reports the results across LongBench task groups\. The results show that asymmetric rank allocation can be beneficial, but the effect depends on model scale\. On Llama\-3\.2\-1B, the average scores are nearly tied, with1:1\.51\{:\}1\.5slightly outperforming1:11\{:\}1by 0\.08 points and1:21\{:\}2by 0\.03 points\. On Qwen3\-4B, allocating more rank to values improves the average score from 47\.79 to 48\.08 for1:1\.51\{:\}1\.5and 48\.00 for1:21\{:\}2\. The same trend is clearer on Qwen2\.5\-14B, where1:1\.51\{:\}1\.5improves the average score from 53\.19 to 54\.09 and1:21\{:\}2reaches 53\.81\. In contrast, Llama\-3\.1\-8B favors the balanced allocation, with1:11\{:\}1achieving 53\.03 compared with 52\.89 and 52\.65 for1:1\.51\{:\}1\.5and1:21\{:\}2\. Overall, these results support the motivation that values can require more rank than pre\-RoPE keys, but an overly value\-heavy allocation is not uniformly optimal\. We therefore keep the balanced allocation as the default in the main experiments for simplicity and robustness, while noting that a mildly value\-biased allocation such as1:1\.51\{:\}1\.5can further improve some model families\.

Table 8:Effect of key–value rank allocation under a fixed total coefficient budget\. The ratiorK:rVr^\{K\}:r^\{V\}specifies the key and value ranks, and shaded rows denote the balanced default allocation\.
### D\.3Effect of Different Recent\-to\-Uniform Sampling Ratios

Table 9:Effect of the recent\-to\-uniform sampling ratio used to construct prompt\-aware key/value bases\. The total number of sampled tokens is fixed, and shaded rows denote the default7:37\{:\}3split\.S4R constructs the prompt\-aware key/value bases from a sampled subset of non\-sink tokens\. The sampling policy combines a contiguous block of recent tokens with uniformly sampled earlier tokens\. These two sources are complementary: recent tokens tend to be more relevant to the immediate decoding state and better match the local distribution near generation, while uniformly sampled tokens provide broader coverage of the full prompt and reduce the risk of missing globally important context\. The default configuration uses a recent\-to\-uniform split of7:37:3\.

Table[9](https://arxiv.org/html/2608.00528#A4.T9)compares several ratios while keeping the total number of sampled tokens fixed\. The results show that S4R is relatively robust to moderate changes in the split, but combining recent and uniform samples is generally preferable to using only recent tokens\. For Llama\-3\.2\-1B,6:46:4obtains the best average score of 28\.44, while the default7:37:3remains close at 28\.09\. For Qwen3\-4B,5:55:5and7:37:3are nearly tied, with average scores of 47\.80 and 47\.79, respectively; however,7:37:3performs better on passage retrieval than5:55:5\. For Llama\-3\.1\-8B, the default7:37:3achieves the best average score of 53\.03, outperforming the other ratios by 0\.21–0\.55 points\. For Qwen2\.5\-14B, more uniform coverage is beneficial:5:55:5and4:64:6both reach 53\.75, compared with 53\.19 for7:37:3\. In contrast, the recent\-only10:010:0setting is not consistently competitive, suggesting that relying only on the most recent sampled tokens can miss useful global prompt structure\. Overall, the results indicate that no single split is universally optimal across all models, but hybrid recent\-plus\-uniform sampling is important\. We keep7:37:3as the default because it is consistently strong, especially on Llama models and passage retrieval, while maintaining a simple fixed sampling rule\.

### D\.4Effect of Reconstruction Ratio

The reconstruction ratioρ\\rhocontrols how many non\-sink positions are reconstructed and attended to in full precision at each decoding step\. In the main experiments, we setρ\\rhoto match the target compression ratioη\\eta, so the sparse reconstruction budget scales with the memory budget used by each model setting\. This ablation tests whether S4R can use a smaller reconstruction set during decoding while keeping the compressed coefficient cache and other hyperparameters unchanged\. Reducingρ\\rhodirectly lowers the number of reconstructed key/value vectors and the size of the exact attention set, but it may also remove globally relevant positions from the final sparse attention computation\.

Table[10](https://arxiv.org/html/2608.00528#A4.T10)compares the default reconstruction ratio with a smaller setting ofρ=0\.15\\rho=0\.15\. The results show that S4R is reasonably robust to reconstructing fewer tokens, but the default setting remains consistently stronger\. On Llama\-3\.2\-1B, reducingρ\\rhofrom0\.20\.2to0\.150\.15only decreases the average score from 28\.09 to 27\.90, and even improves MDQA, LCC, and PRE slightly\. On Qwen3\-4B, the average score drops by 0\.33 points, mainly due to lower Fewshot and PRE scores\. The larger Llama\-3\.1\-8B and Qwen2\.5\-14B settings are more sensitive: their average scores decrease by 0\.95 and 1\.07 points, respectively\. This suggests that a smaller reconstruction set can preserve most of the performance when further decoding\-time savings are desired, but the defaultρ=η\\rho=\\etaprovides a safer accuracy–efficiency trade\-off, especially for stronger models whose outputs rely more on recovering a broader set of relevant context positions\.

Table 10:Effect of the reconstruction ratioρ\\rhoduring sparse decoding\. The coefficient cache and other hyperparameters are unchanged, and shaded rows denote the defaultρ\\rhofor each model setting\.
### D\.5Effect of Layer\-wise Full Reconstruction

SALS applies full reconstruction to selected layers based on its layer\-wise analysis, specifically the first two layers and the last layer, to reduce the approximation error in layers that are more sensitive to compressionMuet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib25)\)\. Motivated by this design, we test whether a similar layer\-wise full\-reconstruction strategy is also beneficial for S4R\. In this ablation, “w/” denotes applying full reconstruction to these selected layers, while “w/o” denotes the default S4R configuration that uses sparse reconstruction consistently across layers\. All other hyperparameters are kept unchanged\.

Table[11](https://arxiv.org/html/2608.00528#A4.T11)shows that layer\-wise full reconstruction does not consistently improve S4R\. On Llama\-3\.2\-1B, it improves the average score from 28\.09 to 28\.45, mainly due to gains on Fewshot, MDQA, LCC, and PRE\. However, the same strategy reduces the average score on the other three models: from 47\.79 to 47\.45 on Qwen3\-4B, from 53\.03 to 52\.33 on Llama\-3\.1\-8B, and from 53\.19 to 52\.94 on Qwen2\.5\-14B\. The drops are especially visible in MDQA and PRE for Qwen3\-4B and in Fewshot and MDQA for Llama\-3\.1\-8B\.

These results suggest that the layer\-wise full\-reconstruction heuristic used by SALS is not directly transferable to S4R\. S4R already preserves sink tokens and reconstructs a query\-dependent sparse set at every layer; forcing full reconstruction on fixed layers can change the balance between exact local/global token recovery and the prompt\-adaptive sparse reconstruction mechanism\. It also introduces additional decoding\-time reconstruction cost\. Since the accuracy gains are limited to the smallest model and are not robust across model families, we do not use layer\-wise full reconstruction as the default\.

Table 11:Effect of applying layer\-wise full reconstruction to selected layers\. “w/” applies full reconstruction to the first two layers and the last layer, whereas “w/o” is the default sparse\-reconstruction setting; shaded rows denote the default\.DatasetSourceAvg lenDecoding lenMetricLanguage\#dataSingle\-Document QANarrativeQALiterature, Film18,409128F1English200QasperScience3,619128F1English200MultiFieldQA\-enMulti\-field4,55964F1English150Multi\-Document QAHotpotQAWikipedia9,15132F1English2002WikiMultihopQAWikipedia4,88732F1English200MuSiQueWikipedia11,21432F1English200Few\-shot LearningTRECWeb question5,17764Accuracy \(CLS\)English200TriviaQAWikipedia, Web8,20932F1English200SAMSumDialogue6,258128Rouge\-LEnglish200Synthetic TaskPassageRetrieval\-enWikipedia9,28932Accuracy \(EM\)English200Code CompletionLCCGithub1,23564Edit SimPython/C\#/Java500Table 12:Summary of LongBench dataset statistics\. ‘Avg len’ denotes average input length \(number of words for English/code datasets\)\. ‘Decoding len’ is the number of tokens in the decoding sequence\. ‘Accuracy \(CLS\)’ indicates classification accuracy, and ‘Accuracy \(EM\)’ stands for exact match accuracy\.Table 13:Seven task configurations in RULER\.

## Appendix EImplementation Details

### E\.1LongBench

In this section, we provide the detailed statistics of the LongBench benchmarkBaiet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib49)\)used in our evaluation\. As shown in Table[12](https://arxiv.org/html/2608.00528#A4.T12), the selected datasets cover five major categories: single\-document QA, multi\-document QA, few\-shot learning, synthetic tasks, and code completion\. These tasks represent typical long\-context scenarios with average input lengths ranging from approximately 1\.2k to over 18k words\. We also specify the decoding lengths and the primary evaluation metrics \(e\.g\., F1 score, Rouge\-L, and Exact Match\) for each task to provide a comprehensive reference for our experimental setup in Section[4\.1](https://arxiv.org/html/2608.00528#S4.SS1)\.

We evaluate our method on the LongBench benchmark using Llama\-3\.2\-1B\-Instruct, Llama\-3\.1\-8B\-InstructGrattafioriet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib35)\), Qwen3\-4B\-Instruct\-2507Yanget al\.\([2025a](https://arxiv.org/html/2608.00528#bib.bib43)\)and Qwen2\.5\-14B\-InstructQwenet al\.\([2025](https://arxiv.org/html/2608.00528#bib.bib37)\)\. Llama\-3\.2\-1B\-Instruct is a 1\-billion\-parameter model with a 128k\-token context window, while Llama\-3\.1\-8B\-Instruct is an 8\-billion\-parameter model with a 128k\-token context window\. Qwen3\-4B\-Instruct\-2507 is a 4\-billion\-parameter model with a 32k\-token context window natively and 128k tokens with YaRNPenget al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib44)\), while Qwen2\.5\-14B\-Instruct is a 14\-billion\-parameter model with a 32k\-token context window\. For LongBench samples exceeding 32k tokens \(e\.g\., in NarrativeQA\), we truncate the input to the rightmost 32k tokens to fit within the Qwen2\.5\-14B\-Instruct’s constraints\.

### E\.2RULER

RULERHsiehet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib41)\)is built for maximal flexibility, supporting a vast range of sequence lengths and task complexities\. Each task allows for a combinatorial number of configuration options, as summarized in Table[13](https://arxiv.org/html/2608.00528#A4.T13)\. In our primary experiments, we assess Qwen2\.5\-7B\-Instruct\-1MYanget al\.\([2025b](https://arxiv.org/html/2608.00528#bib.bib36)\)on seven representative tasks spanning RULER’s two primary categories: retrieval and multi\-hop tracing\. For each task, we prepare samples of 128k tokens and generate 20 samples to assess performance\. Unless otherwise specified, the decoding length is set to 32 tokens\.

Both LongBench and RULER are evaluated using LM\-Evaluation\-HarnessGaoet al\.\([2024](https://arxiv.org/html/2608.00528#bib.bib42)\)\.