Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

# Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
Source: [https://arxiv.org/html/2607.01237](https://arxiv.org/html/2607.01237)
Han Shen kingsutherszsz48@gmail\.com&Yuyang Wu11footnotemark:1 Carnegie Mellon University Pittsburgh, PA, USA yuyangwu@andrew\.cmu\.edu

###### Abstract

Reasoning language models often generate long chain\-of\-thought \(CoT\), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput\. To address these issues, KV cache compression has emerged as a promising technique for reducing memory overhead by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding\. Nevertheless, we identify two key limitations in existing KV cache compression methods: 1\) their threshold\-triggered compression policy may provide limited throughput improvement or even reduce throughput, and may fully eliminate KV pairs from certain blocks of the sequence, potentially worsening information loss\. 2\) they typically retain either isolated KV pairs or fixed\-size chunks with rigid boundaries, failing to preserve important flexible\-sized chunks at arbitrary token positions\.

To overcome these limitations, we propose Kara, a sliding\-window KV cache compression method that performs decoding\-time compression by operating only on the recently generated context\. Kara leverages bidirectional attention to score and select informative KV pairs in the window\. To enable flexible preservation of important semantic information, we design a Token2Chunk module to expand a subset of selected KV pairs into chunks\. Furthermore, we adapt Kara to PagedAttention and develop KvLLM, an inference framework built upon vLLM, which reduces KV cache memory usage and effectively improves output throughput\. Extensive experiments demonstrate consistent performance improvements of proposed Kara and KvLLM\.

## 1Introduction

Chain\-of\-thought reasoningGuoet al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib7)\); Weiet al\.\([2022](https://arxiv.org/html/2607.01237#bib.bib2)\); Yanget al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib3)\)has endowed LLMs with powerful capabilities for solving complex tasks\. However, reasoning models often generate long traces, which lead to a substantial growth of KV cache during decoding and impose significant memory overheadLiet al\.\([2024a](https://arxiv.org/html/2607.01237#bib.bib4)\); Yuanet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib5)\)\. For example, caching 128K tokens in Qwen3\-14B with FP16 precision requires approximately 20 GB memory\. This overhead becomes especially pronounced in large\-batch serving settings, where the rapidly growing KV cache increases decoding latency and causes requests to wait for available memory resources, thereby reducing overall output throughput\.

To address these issues, KV cache compressionCaiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib6)\); Fenget al\.\([2026b](https://arxiv.org/html/2607.01237#bib.bib9)\); Liet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\), an inference\-time approach, has garnered increasing attention\. By selectively retaining important KV pairs, KV cache compression can reduce memory usage while keeping performance degradation within an acceptable range \(*e\.g\.,*within a 1% performance drop\)\. Existing worksLiet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\); Yuanet al\.\([2024a](https://arxiv.org/html/2607.01237#bib.bib32)\)have shown strong performance on multiple tasks while retaining only a small fraction of KV cache \(*e\.g\.,*30%\\%\)\. Early compression methodsXiaoet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib10)\); Zhanget al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib11)\)relied on heuristic techniques such as attention sinks to preserve KV pairs at fixed positions, ignoring the varying importance of different tokens and resulting in suboptimal performance\. More recently, research has shifted toward score\-based KV cache compressionCaiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib6)\); Liet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\); Łańcuckiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib12)\); Kimet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib13)\), which assigns importance scores to multi\-head KV pairs to determine cache retention\. These methods typically maintain a query cache of recent query states and use them to score historical KV pairs via attention with their key states\. The score\-based compression methods often achieve state\-of\-the\-art performance, as they leverage the internal information of LLMs to identify critical KV pairsDevotoet al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib14)\)\.

![Refer to caption](https://arxiv.org/html/2607.01237v1/x1.png)Figure 1:\(a\) Average throughput under different batch sizes, where batch size denotes the predefined maximum decoding sequences\. We observe that vLLM with SnapKV achieves lower throughput than vanilla vLLM as the batch size increases\. \(b\) The actual number of decoding sequences varies with decoding steps\. The decoding step denotes the number of global decoding iterations and predefined maximum decoding sequences is 256\. We observe that vLLM with SnapKV consistently maintains a higher actual decoding sequences than vanilla vLLM, as the saved memory is used to decode more sequences\.
![Refer to caption](https://arxiv.org/html/2607.01237v1/x2.png)Figure 2:Remaining KV pairs under different compression counts\. We track the remaining KV pairs of tokens from different regions in certain attention head across repeated compression steps\. We observe that the semantic information of certain regions is completely eliminated as compression proceeds\.

Despite their success, we identify two key limitations in existing KV cache compression methods\.

- •These methods mainly rely on a threshold\-triggered compression policy, which can degrade both reasoning performance and inference efficiency\.Under the threshold\-triggered compression policy, the KV cache is compressed back to a smaller target length once it reaches a predefined threshold, after which this grow\-and\-compress pattern repeats\. While effective in low\-frequency or personal deployment settings, this policy can cause a concurrency–throughput inversion effect\. Specifically, when the number of decoding sequences becomes comparable to the gap between the threshold and the post\-compression cache length, compression may be triggered frequently, and the associated computational overhead can decrease output throughput even as the number of concurrently decoded sequences increases\. In addition, since compression is applied over the entire KV cache, previously compressed regions of the sequence may be compressed again, which can further worsen information loss and poison reasoning quality\. To empirically validate these challenges, we conduct a simple experiment using the vLLM111We use nano\-vLLM for our experiments\. It retains the core mechanisms of vLLM, including PagedAttention, continuous batching, tensor parallelism, and recomputation, and is convenient for further developmentframeworkKwonet al\.\([2023](https://arxiv.org/html/2607.01237#bib.bib18)\)with a representative decoding\-time compression method \(SnapKVLiet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\)\) on the MATH\-500 datasetLinet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib19)\)with distilled Llama\-8B modelsGrattafioriet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib20)\)\. As shown in Figures[2](https://arxiv.org/html/2607.01237#S1.F2)and[2](https://arxiv.org/html/2607.01237#S1.F2), the output throughput of vLLM with SnapKV decreases even as the number of concurrently decoded sequences increases, and the corresponding KV pairs of earlier tokens are almost completely removed as compression is applied repeatedly\. Detailed experimental settings are provided in Appendix[A](https://arxiv.org/html/2607.01237#A1)\.
- •These methods retain either isolated KV pairs or fixed\-size chunks with rigid boundaries, failing to preserve important flexible\-sized chunks at arbitrary token positions\.Mainstream KV cache compression methods select isolated KV pairs for retention based on scores, which can lead to significant loss of semantic information\. To address this issue, several methods further propose retaining contiguous KV pairs to form chunks\. For example, ChunkKVLiuet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib15)\)partitions the historical KV cache into chunks of the same length with fixed boundaries\. However, such retention paradigms are often rigid and fail to capture flexible\-sized KV information distributed at arbitrary positions across the sequence\.

To overcome these limitations, we propose Kara, a sliding\-window KV cache compression method that performs decoding\-time compression only on the recently generated context\. Through empirical analysis, we find that the accumulated bidirectional attention received by each KV pair serves as an effective indicator of its importance and informativeness\. Building on this observation, Kara uses the accumulated bidirectional attention score as the importance measure to identify discrete candidate KV pairs within the window\. To preserve flexible\-sized contiguous KV pairs at arbitrary positions, we introduce a simple yet effective Token2Chunk module\. Given the candidate discrete KV pairs, Token2Chunk treats every two consecutive candidate indices as the endpoints of a chunk, and filters chunks according to their width and the importance scores of the endpoint KV pairs\. Finally, we preserve the candidate discrete KV pairs as well as all KV pairs inside the selected chunks\. In summary, Kara produces a compressed KV cache that combines both isolated KV pairs and chunk\-level KV pairs within the sliding window\. Furthermore, we adapt Kara to PagedAttentionKwonet al\.\([2023](https://arxiv.org/html/2607.01237#bib.bib18)\)and develop KvLLM222KvLLM is built upon the nano‑vLLM framework\., a vLLM\-based inference framework equipped with a periodic compression policy\. To avoid frequent compression triggering during decoding, the periodic compression policy compresses the trailing blocks of the selected sequences every fixed number of decoding steps\. Extensive experiments demonstrate that the proposed Kara and KvLLM framework can reduce KV cache memory usage while improving throughput and concurrency, and further preserving reasoning quality\.

In summary, our main contributions are as follows:

- •We propose Kara, a KV cache compression method that uses sliding\-window bidirectional attention to identify candidate discrete KV pairs and employs a Token2Chunk module to generate chunks with contiguous KV pairs\.
- •We adapt the proposed method to PagedAttention and design KvLLM inference framework with a periodic compression policy, which can effectively improve throughput, concurrency, and reasoning quality compared with existing compression methods\.
- •We conduct extensive experiments, including reasoning and NIAH \(Needle\-in\-a\-Haystack\) evaluations, to demonstrate the superiority and effectiveness of the proposed method and inference framework\.

## 2Preliminary

#### Notation\.

We consider an autoregressive Transformer\-based LLMSiaet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib17)\); Achiamet al\.\([2023](https://arxiv.org/html/2607.01237#bib.bib16)\)that generates a token sequence denoted by𝐗=\(x1,x2,…,xS\)\\mathbf\{X\}=\(x\_\{1\},x\_\{2\},\\dots,x\_\{S\}\), whereSSis the sequence length\. The model consists ofLLTransformer layers, each withHHattention heads and head dimensiondd\. For thehh\-th attention head, we denote the projected query, key, and value states as𝐐l,h,𝐊l,h,𝐕l,h∈ℝS×d\\mathbf\{Q\}^\{l,h\},\\mathbf\{K\}^\{l,h\},\\mathbf\{V\}^\{l,h\}\\in\\mathbb\{R\}^\{S\\times d\}, respectively\. The scaled dot\-product attention probabilities are defined as:

𝐏l,h=softmax​\(𝐐l,h​\(𝐊l,h\)⊤d\),\\mathbf\{P\}^\{l,h\}=\\text\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}^\{l,h\}\(\\mathbf\{K\}^\{l,h\}\)^\{\\top\}\}\{\\sqrt\{d\}\}\\right\),\(1\)where𝐏i,jl,h\\mathbf\{P\}^\{l,h\}\_\{i,j\}denotes the attention probability from theii\-th query token to thejj\-th key token in layerlland headhh\. For simplicity, we use𝐐\\mathbf\{Q\}and𝐊\\mathbf\{K\}to denote the query and key states of a specific layerlland headhh\.

#### LLM Inference\.

During autoregressive decoding, the model maintains a key cache𝒦\\mathcal\{K\}and a value cache𝒱\\mathcal\{V\}to store the key and value states corresponding to historical tokens\. Compared with computing attention over the entire prefix, KV caching substantially reduces the quadratic computation cost and improves inference efficiencyLIet al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib21)\)\.

In practice, the decoding efficiency of LLM inference is commonly measured by TPOT \(Time Per Output Token\), which denotes the average time required to generate one output token during decoding\. Beyond TPOT, throughput is another important metric that measures the number of output tokens generated per unit time, and is commonly used to reflect the concurrency and latency\. In general, more decoding sequences enables higher throughput In long\-context reasoning scenarios, the inference overhead may be dominated by memory access to the KV cache, leaving the inference memory\-bound\. The detailed formulas for TPOT and throughput are presented in Appendix[B](https://arxiv.org/html/2607.01237#A2)\.

To improve the serving efficiency of LLM inference, frameworksKwonet al\.\([2023](https://arxiv.org/html/2607.01237#bib.bib18)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib22)\)such as vLLM have been widely developed, they aim to maximize memory utilization and sustain efficient decoding under high\-concurrency and long\-context generation\. The core idea of vLLM is PagedAttention, which views the KV cache as a group of fixed\-size blocks\. Each block stores a contiguous segment of KV pairs \(*e\.g\.,*256 tokens\), and memory allocation and release are managed at the block level during inference\. However, in memory\-constrained settings, the available memory can be rapidly consumed as decoding proceeds, which may force the inference framework to reduce the number of running requests and thereby limit overall output throughput \(shown in Figure[2](https://arxiv.org/html/2607.01237#S1.F2)\(b\)\)\.

#### KV Cache Compression\.

KV cache compressionCaiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib6)\); Parket al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib26)\); Ramachandranet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib24)\); Fenget al\.\([2026a](https://arxiv.org/html/2607.01237#bib.bib23)\); Xiaoet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib10)\)reduces the memory footprint of LLM inference by evicting unimportant ones\. Most compression methodsCaiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib6)\); Liet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\); Qinet al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib33)\)estimate an importance score𝐀il,h\\mathbf\{A\}\_\{i\}^\{l,h\}for the KV pair of token positioniiin attention headhhand layerll, and then select the most important KV pairs under a cache budget when the cache length reaches a predefined threshold\. A representative score\-based compression paradigm can be formulated as:

ℐl,h=TopK​\(\{𝐀il,h\}i=1Ct,N\)\\mathcal\{I\}^\{l,h\}=\\textnormal\{TopK\}\(\\\{\\mathbf\{A\}\_\{i\}^\{l,h\}\\\}\_\{i=1\}^\{C\_\{t\}\},\\,N\)\(2\)\(𝒦l,h,𝒱l,h\)←\(𝒦l,h,𝒱l,h\)​\[ℐl,h\],when​Ct≥Cmax,\(\\mathcal\{K\}^\{l,h\},\\mathcal\{V\}^\{l,h\}\)\\leftarrow\(\\mathcal\{K\}^\{l,h\},\\mathcal\{V\}^\{l,h\}\)\[\\mathcal\{I\}^\{l,h\}\],\\quad\\text\{when \}C\_\{t\}\\geq C\_\{\\max\},\(3\)whereℐl,h\\mathcal\{I\}^\{l,h\}is the index of selected KV pairs of headhhand layerll,NNis the predefined KV budget,CtC\_\{t\}denote the number of KV pairs currently stored in the cache \(*i\.e\.,*the cache length\) at decoding steptt,CmaxC\_\{\\max\}denote the predefined threshold\. Once compressed, the cache length is constrained within the interval\[N,Cmax\]\[N,\\,C\_\{\\max\}\]as it grows fromNNback toCmaxC\_\{\\max\}before the next compression\. In the following, we omit the notationllandhhas all operations performed in the layer and attention head are same in our method\. Although the compression paradigm in Equation[2](https://arxiv.org/html/2607.01237#S2.E2)and[3](https://arxiv.org/html/2607.01237#S2.E3)is widely adopted, it suffers from two key limitations:

- •Threshold\-triggered compression may degrade both reasoning performance and inference efficiency\.Since the paradigm operates compression over the entire KV cache, it may fully remove all KV pairs within a long contiguous span of context \(*e\.g\.,*xi∼xi\+256x\_\{i\}\\sim x\_\{i\+256\}in Figure[2](https://arxiv.org/html/2607.01237#S1.F2)\), potentially exacerbating information loss\. Furthermore, when the decoding batch size increases toward the gap betweenCmaxC\_\{\\max\}andNN\(*i\.e\.,*Cmax−NC\_\{\\max\}\-N\), compression can be triggered frequently across concurrently decoded sequences, making the overall compression overhead more pronounced and potentially reducing throughput \(shown in Figure[2](https://arxiv.org/html/2607.01237#S1.F2)\)\.
- •Retention granularity is often either isolated KV pairs or rigid fixed\-size chunks\.Mainstream score\-based methods select isolated KV pairs for retention, which can lead to the loss of semantic information\. Although some methods retain contiguous KV pairs to form chunks, such chunks are often extracted with rigid boundaries \(*e\.g\.,*fixed spans like𝐊1∼𝐊256\\mathbf\{K\}\_\{1\}\\sim\\mathbf\{K\}\_\{256\}\), which may fail to capture flexible\-sized cache information distributed at arbitrary positions across the sequence\.

Given the flaws of existing methods, we argue for using sliding\-window bidirectional\-attention scores to guide KV\-pair retention within the window, together with a retention strategy that incorporates both token\-level and chunk\-level KV pairs\. We further adapt our method to PagedAttention and develop the KvLLM framework with a periodic compression policy, which aims to improve throughput under high concurrency while maintaining reasoning quality\.

## 3Method

![Refer to caption](https://arxiv.org/html/2607.01237v1/x3.png)Figure 3:\(a\) Distribution comparison between causal attention and bidirectional attention of a specific token\. \(b\) Average causal attention weight versus bidirectional attention percentile\. We first compute the bidirectional attention weights for all token pairs\(xi,xj\)\(x\_\{i\},x\_\{j\}\)withj\>ij\>i, usingxix\_\{i\}as the query andxjx\_\{j\}as the key\. We then sort these weights in ascending order and group the pairs by percentiles\. Finally, for each percentile group, we compute the causal attention weight for the same pairs in the reverse direction \(xjx\_\{j\}as the query andxix\_\{i\}as the key\), average the causal attention weights within the group, and plot the resulting averages against the bidirectional\-attention percentiles\.
![Refer to caption](https://arxiv.org/html/2607.01237v1/x4.png)Figure 4:Comparison between accumulated causal attention and bidirectional attention distributions\. We compute causal attention and bidirectional attention probability matrices in a given sequence and accumulate the attention received by each token, respectively\.

### 3\.1Sliding\-Window Bidirectional Attention for KV Importance Estimation

During decoding, we maintain a sliding window that dynamically covers the recently generated tokens, and we apply KV cache compression only to KV pairs within the window\. Once compression is performed, the window slides forward to cover newly generated tokens that is uncompressed, and the same procedure is applied repeatedly\. Inspired byCaiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib6)\)andLiet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\), we additionally reserve a buffer at the end of the window to protect the latest generated tokens from being compressed, which helps preserve reasoning quality\.

To obtain important discrete KV pairs, we first extract the query states of all tokens inside the window and compute bidirectional attention to the key states of tokens inside the window except those in the buffer\. We then accumulate the attention weights received by each KV pair to estimate its importance score\. Finally, we select the TopK important KV pairs inside the window as retention candidates\. In summary, the sliding\-window bidirectional attention scoring can be formulated as:

𝐀j=∑i∈𝒲exp⁡\(⟨𝐐i,𝐊j⟩/d\)∑j′∈𝒲∖𝒰exp⁡\(⟨𝐐i,𝐊j′⟩/d\),j∈𝒲∖𝒰,\\mathbf\{A\}\_\{j\}=\\sum\_\{i\\in\\mathcal\{W\}\}\\frac\{\\exp\(\\langle\\mathbf\{Q\}\_\{i\},\\mathbf\{K\}\_\{j\}\\rangle/\\sqrt\{d\}\)\}\{\\sum\\limits\_\{j^\{\\prime\}\\in\\mathcal\{W\}\\setminus\\mathcal\{U\}\}\\exp\(\\langle\\mathbf\{Q\}\_\{i\},\\mathbf\{K\}\_\{j^\{\\prime\}\}\\rangle/\\sqrt\{d\}\)\},\\qquad j\\in\\mathcal\{W\}\\setminus\\mathcal\{U\},\(4\)ℐ=TopK​\(\{𝐀j\}j∈𝒲∖𝒰,⌈r​\(\|𝒲\|−\|𝒰\|\)⌉\)\.\\mathcal\{I\}=\\textnormal\{TopK\}\(\\\{\\mathbf\{A\}\_\{j\}\\\}\_\{j\\in\\mathcal\{W\}\\setminus\\mathcal\{U\}\},\\left\\lceil r\(\|\\mathcal\{W\}\|\-\|\\mathcal\{U\}\|\)\\right\\rceil\)\.\(5\)where𝐀j\\mathbf\{A\}\_\{j\}is the estimated importance score of the KV pair from tokenxjx\_\{j\},𝒲\\mathcal\{W\}denotes the token indices of the current sliding window, and𝒰⊂𝒲\\mathcal\{U\}\\subset\\mathcal\{W\}denotes the buffer at the end of the window \(\|𝒰\|<\|𝒲\|\|\\mathcal\{U\}\|<\|\\mathcal\{W\}\|\),𝒲∖𝒰\\mathcal\{W\}\\setminus\\mathcal\{U\}is the compressible region in the window with length\|𝒲\|−\|𝒰\|\|\\mathcal\{W\}\|\-\|\\mathcal\{U\}\|\.r∈\(0,1\]r\\in\(0,1\]is the retention ratio, andℐ\\mathcal\{I\}denotes the index set of candidate discrete KV pairs to be retained within the current window\.

Since tokens in𝒰\\mathcal\{U\}are excluded from compression, Equation \([4](https://arxiv.org/html/2607.01237#S3.E4)\) serves as an approximation to bidirectional attention over all tokens within the window\. The importance score𝐀∈ℝ\|ℐ\|\\mathbf\{A\}\\in\\mathbb\{R\}^\{\|\\mathcal\{I\}\|\}can also be viewed as combining bidirectional attention scores over the compressible region with causal\-attention scores induced by the query states in the buffer\. After compression, the window moves forward by\|𝒲\|−\|𝒰\|\|\\mathcal\{W\}\|\-\|\\mathcal\{U\}\|steps once there are\|𝒲\|−\|𝒰\|\|\\mathcal\{W\}\|\-\|\\mathcal\{U\}\|newly generated uncompressed tokens beyond the current window\.

#### Empirical Analysis\.

We conduct simple experiments to empirically demonstrate the superiority of bidirectional attention in importance estimation\. Given a sequence, we first compute both the causal attention and bidirectional attention probability matrices over all token pairs, and visualize the cumulative attention received by each token’s key state from all query tokens\. As shown in Figure[4](https://arxiv.org/html/2607.01237#S3.F4), most accumulated casual attention scores concentrate in the prefix region of the window, as earlier positions are accessible to more queries than later ones\. Then, we visualize the specific token’s bidirectional attention probability distribution over all tokens in the sequence, as well as its causal attention distribution over its preceding tokens\. Finally, we examine the correlation between the bidirectional attention probability from former tokenxix\_\{i\}to later tokenxjx\_\{j\}\(j\>ij\>i\) and the causal attention probability from later tokenxjx\_\{j\}to former tokenxix\_\{i\}\. As shown in Figures[4](https://arxiv.org/html/2607.01237#S3.F4), we observe that 1\) tokenxix\_\{i\}’s causal attention distribution largely matches its bidirectional attention distribution on preceding positions, and 2\) when former tokenxix\_\{i\}assigns a higher bidirectional attention weight to future tokenxjx\_\{j\}, tokenxjx\_\{j\}also tends to assign a higher causal attention weight to former tokenxix\_\{i\}\.

The above analyses suggest that bidirectional attention can identify important preceding context information similar to causal attention, and highlight future tokens that are likely to attend strongly to the current token\. Building on this observation, we use bidirectional attention to identify KV pairs that are important for the current window context\. It also selects KV pairs whose corresponding query states are highly active during causal\-attention computation, and these KV pairs often carry rich contextual information\.

#### Implementation Details\.

We can obtain the query states required for KV scoring via recomputation while reusing the existing KV cache\. Specifically, at each compression, we run an additional forward pass on the tokens in the current window and compute only the query projections, the attention computation and the FFN \(feed\-forward network\)\. The recomputation avoids maintaining an explicit query cache during decoding\. The sliding\-window mechanism can also be applied to compress the prompt KV cache before decoding\. Specifically, during the prefill stage, we can partition the prompt into multiple windows of length\|𝒲\|\|\\mathcal\{W\}\|and perform window\-wise compression on the KV pairs corresponding to positions inside each window in parallel\. After compression, the window moves forward by\|𝒲\|−\|𝒰\|\|\\mathcal\{W\}\|\-\|\\mathcal\{U\}\|steps once there are\|𝒲\|−\|𝒰\|\|\\mathcal\{W\}\|\-\|\\mathcal\{U\}\|uncompressed tokens beyond the current window during decoding\.

### 3\.2Token2Chunk Module

Building on the discrete KV indicesℐ\\mathcal\{I\}obtained from sliding\-window bidirectional attention, we introduce a lightweight Token2Chunk module to preserve flexible\-sized contiguous KV pairs at arbitrary positions within the window\. Specifically, Token2Chunk takes as input a discrete retained indicesℐ\\mathcal\{I\}and the corresponding importance scores𝐀∈ℝ\|ℐ\|\\mathbf\{A\}\\in\\mathbb\{R\}^\{\|\\mathcal\{I\}\|\}\. We first define the chunk budget that controls maximum additional KV pairs can be introduced by this module, together with a maximum chunk length\. We then sort the indices inℐ\\mathcal\{I\}in ascending order and divide the compressible region into chunks by treating every two consecutive indices as the endpoints of a candidate chunk\. Each candidate chunk indicates a contiguous span, whose interior positions correspond to KV pairs that can be additionally retained as a chunk\. Next, we filter candidate chunks by the maximum chunk length, and score each remaining chunk using both its length and the importance scores of the two endpoint KV pairs\. Finally, we preserve all KV pairs inside the selected chunks, and union them withℐ\\mathcal\{I\}to obtain the final retained indices\. For thekk\-th chunk \(1≤k≤\|ℐ\|−11\\leq k\\leq\|\\mathcal\{I\}\|\-1\), Token2Chunk can be formulated as:

𝐑k=\(𝐀ℐ​\[k\]\+𝐀ℐ​\[k\+1\]\)×\(ℐ​\[k\+1\]−ℐ​\[k\]−1\),\\mathbf\{R\}\_\{k\}=\(\\mathbf\{A\}\_\{\\mathcal\{I\}\[k\]\}\+\\mathbf\{A\}\_\{\\mathcal\{I\}\[k\+1\]\}\)\\times\(\\mathcal\{I\}\[k\+1\]\-\\mathcal\{I\}\[k\]\-1\),\(6\)𝒫=TopK​\(\{𝐑k\}k=1\|ℐ\|−1,⌊αγ⌋\),s\.t\.​ℐ​\[k\+1\]−ℐ​\[k\]≤γ\\mathcal\{P\}=\\mathrm\{TopK\}\(\\\{\\,\\mathbf\{R\}\_\{k\}\\,\\\}\_\{k=1\}^\{\|\\mathcal\{I\}\|\-1\},\\;\\left\\lfloor\\frac\{\\alpha\}\{\\gamma\}\\right\\rfloor\),\\quad\\text\{s\.t\. \}\\ \\mathcal\{I\}\[k\+1\]\-\\mathcal\{I\}\[k\]\\leq\\gamma\(7\)ℐ^=ℐ∪⋃k∈𝒫\{j\|ℐ​\[k\]<j<ℐ​\[k\+1\]\}\.\\hat\{\\mathcal\{I\}\}=\\mathcal\{I\}\\ \\cup\\ \\bigcup\_\{k\\in\\mathcal\{P\}\}\\left\\\{\\,j\\ \\middle\|\\ \\mathcal\{I\}\[k\]<j<\\mathcal\{I\}\[k\+1\]\\,\\right\\\}\.\(8\)where𝐑k\\mathbf\{R\}\_\{k\}is thekk\-th chunk’s importance score,ℐ\\mathcal\{I\}is the discrete retained indices obtained from sliding\-window bidirectional attention \(sorted in ascending order\), andℐ​\[k\]\\mathcal\{I\}\[k\]denotes thekk\-th smallest element inℐ\\mathcal\{I\}\.Aℐ​\[k\]A\_\{\\mathcal\{I\}\[k\]\}is the importance score of the KV pair at indexℐ​\[k\]\\mathcal\{I\}\[k\]\.γ\\gammais the predefined maximum chunk length, andα\\alphais the predefined chunk budget controlling the number of additional KV pairs introduced by Token2Chunk\.𝒫\\mathcal\{P\}denotes selected chunk indices returned byTopK​\(⋅\)\\mathrm\{TopK\}\(\\cdot\), andℐ^\\hat\{\\mathcal\{I\}\}denotes the final retained KV pair indices after adding all KV pairs inside the selected chunks\. We then useℐ^\\hat\{\\mathcal\{I\}\}to compress the KV cache of the current sliding window:

\(𝒦,𝒱\)←\(𝒦,𝒱\)​\[ℐ^\],\(\\mathcal\{K\},\\mathcal\{V\}\)\\leftarrow\(\\mathcal\{K\},\\mathcal\{V\}\)\[\\hat\{\\mathcal\{I\}\}\],\(9\)where\(𝒦,𝒱\)\(\\mathcal\{K\},\\mathcal\{V\}\)denotes the KV cache within the current sliding window\.

Equation \([6](https://arxiv.org/html/2607.01237#S3.E6)\) assigns each candidate chunk a score that associated with both its length and the importance of its two endpoints\. Intuitively, a longer chunk preserves more contextual information, while important endpoints indicate that the chunk is likely to be important for the window context\. Equation \([7](https://arxiv.org/html/2607.01237#S3.E7)\) selects the TopK important chunks under the maximum\-length constraint\.

#### Implementation Details\.

During decoding, Token2Chunk is applied to the compressible region immediately after the bidirectional\-attention scoring stage\. Token2Chunk is a lightweight module, and its additional complexity isO​\(n\)O\(n\), wheren=\|ℐ\|n=\|\\mathcal\{I\}\|\. Equation \([6](https://arxiv.org/html/2607.01237#S3.E6)\) and Equation \([7](https://arxiv.org/html/2607.01237#S3.E7)\) ensure that the number of additional KV pairs is bounded by the chunk budgetα\\alpha\. If the selected chunks introduce fewer thanα\\alphaadditional KV pairs, we fill the chunk budget by selecting discrete KV pairs from the unselected positions in𝒲∖𝒰\\mathcal\{W\}\\setminus\\mathcal\{U\}according to score𝐀i\\mathbf\{A\}\_\{i\}\.

### 3\.3KvLLM: Periodic Compression with PagedAttention

We further adapt Kara to PagedAttention and developKvLLM, an inference framework that supports applying KV cache compression during serving to improve decoding efficiency\. To mitigate the concurrency\-throughput inversion effect observed in Figure[2](https://arxiv.org/html/2607.01237#S1.F2), KvLLM adopts a periodic compression policy that is aligned with Kara’s sliding\-window design\. Concretely, KvLLM maintains a global decoding\-step counter from the beginning of inference and applies a fixed periodic compression schedule\. At each scheduled compression, KvLLM selects a subset of running sequences and treats the trailing PagedAttention blocks of each selected sequence as a compression window, on which Kara is applied to compress the KV cache\. The compression period is set to be longer than the window span, so that each invocation operates on newly generated, previously uncompressed tokens\. By using periodic compression, KvLLM avoids frequent compression triggering across concurrent requests\. In practice, this policy can further reduce the KV\-cache memory footprint while improving overall output throughput\.

![Refer to caption](https://arxiv.org/html/2607.01237v1/x5.png)Figure 5:Performance of different KV cache compression methods across varying retention levels\. The dash line represents the accuracy of the vanilla LLM model without compression\.

## 4Experiments

#### Benchmarks\.

We evaluate on three mathematical reasoning benchmarks: MATH\-500Linet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib19)\), AIME24Zhang and Math\-AI \([2024](https://arxiv.org/html/2607.01237#bib.bib27)\), and AMC23\. We use decoding andreport accuracy of zero\-shot pass@1, computed as the fraction of problems answered correctly\. We set the maximum generation length to 16,384 tokens for MATH\-500 and AMC 2023, and 32,768 tokens for AIME 2024\. We also include a long\-context evaluation on Needle\-in\-a\-Haystack \(NIAH\) benchmark\.

#### Baselines\.

We compare Kara against representative KV cache compression baselines, including SnapKVLiet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\), ChunkKVLiuet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib15)\), StreamingLLMXiaoet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib10)\), PyramidKVCaiet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib30)\), and AdaKVFenget al\.\([2026b](https://arxiv.org/html/2607.01237#bib.bib9)\)\. For each baseline, we use its best\-performing configuration as recommended by the original paper\.

#### Configurations\.

We conduct experiments on DeepSeek\-R1\-Distill\-Llama\-8B \(R1\-Llama\-8B\)Guoet al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib7)\), Qwen3\-14B, and Qwen3\-4BYanget al\.\([2025](https://arxiv.org/html/2607.01237#bib.bib3)\), using BF16 precision\. For Kara, we sweep the window length\|𝒲\|∈\{256,384,512\}\|\\mathcal\{W\}\|\\in\\\{256,384,512\\\}and set the buffer length\|𝒰\|∈\{32,64\}\|\\mathcal\{U\}\|\\in\\\{32,64\\\}\. For Token2Chunk, we use a fixed maximum chunk sizeγ\\gammaof88and set the additional chunk budgetα\\alphato1616or3232\. We evaluate retention ratiosr∈\{20,30,40,50,60\}%r\\in\\\{20,30,40,50,60\\\}\\%, and apply the same configuration to all layers and attention heads\. Since Kara does not strictly enforce a fixed compressed cache length, different sequences may result in various cache sizes\. We ensure a fair comparison by first running Kara to obtain the compressed KV cache length for each sequence and setting the per\-sequence KV budget of each baseline to match Kara under the corresponding setting\. Note that we evaluate Kara’s reasoning pass@1 without the periodic compression policy for a fair accuracy comparison, since this policy is primarily designed to improve throughput in practical application, and we only activate it in throughput experiments\.

![Refer to caption](https://arxiv.org/html/2607.01237v1/x6.png)Figure 6:Needle\-In\-A\-Haystack \(NIAH\) performance\. The x\-axis denotes the input context length and the y\-axis denotes the needle insertion depth\. Each cell reports the retrieval score for the corresponding \(length, depth\) setting, and we also report the mean accuracy averaged over all cells\.
### 4\.1Main Results

Table 1:Ablation study of Kara with Qwen3\-4B at a 40% KV retention ratio\.DatasetsMATH\-500AMC23w/o Bi\-Attention85\.2072\.50w/o Token2Chunk88\.2077\.50Kara90\.0082\.50#### Reasoning Performance\.

Figure[5](https://arxiv.org/html/2607.01237#S3.F5)reports the reasoning performance of Kara compared with KV cache compression methods under various retention ratios\. For a fair comparison, we first run Kara to obtain the realized compressed KV cache length for each sequence, and then set the per\-sequence KV budget of each baseline to match Kara under the corresponding setting\. We observe that: 1\) Kara preserves accuracy with no or only minor degradation relative to the vanilla LLM without compression\. For example, on MATH\-500 with Qwen3\-14B, Kara maintains nearly unchanged accuracy at a 30% retention ratio\. 2\) Kara generally outperforms other compression methods across retention ratios\. For example, on AMC23 with Qwen3\-4B, Kara remains better than the baselines at a 20% retention ratio\. 3\) Kara outperforms AdaKV and PyramidKV in most cases, which allocate different memory budgets across layers and heads\. In contrast, we use a unified configuration and retention ratio for all layers and heads, which is architecturally more compatible with tensor parallelism and PagedAttention in inference frameworks\. Overall, the results demonstrate the effectiveness of Kara for decoding\-time KV cache compression\.

#### NIAH Performance\.

To evaluate Kara’s ability to preserve context information under KV cache compression, we conduct Needle\-in\-a\-Haystack \(NIAH\) experiment\. In this experiment, a short needle statement is inserted into a long context at varying depths, and the model is queried to retrieve the needle\. Figure[6](https://arxiv.org/html/2607.01237#S4.F6)compares Kara with ChunkKV and AdaKV across different context lengths and insertion depths\. We observe that Kara achieves higher retrieval accuracy than the baselines across most depths and context lengths\. In particular, Kara reduces the accuracy drop that some methods exhibit at mid\-context depths, indicating a less information loss\. This behavior is consistent with Kara’s joint retention of discrete KV pairs and flexible\-sized contiguous chunks within the window, which helps preserve relevant context that may be distributed across positions\. Overall, the NIAH results support Kara’s effectiveness in retaining contextual information under compression\.

### 4\.2Ablation Studies

#### Effects of Bidirectional Attention Scoring\.

To evaluate the effectiveness of the sliding\-window bidirectional attention scoring in Equation \([4](https://arxiv.org/html/2607.01237#S3.E4)\), we remove bidirectional attention and scores KV pairs in the compressible region using only the query states from the protected buffer𝒰\\mathcal\{U\}\(denoted as w/o Bi\-Attention\)\. As shown in Table[1](https://arxiv.org/html/2607.01237#S4.T1), remove bidirectional attention consistently reduces reasoning accuracy\. These results indicate that utilizing all query states in the window with bidirectional attention provides a stronger importance signal than buffer\-only scoring, leading to better preservation of informative context under the same KV retention ratio\.

#### Effects of Token2Chunk\.

To assess the effectiveness of the Token2Chunk module in Equation \([6](https://arxiv.org/html/2607.01237#S3.E6)\), we remove Token2Chunk and retain KV pairs using only sliding\-window bidirectional\-attention scoring, selecting a target number of discrete KV pairs under the same retention ratio \(w/o Token2Chunk\)\. As shown in Table[1](https://arxiv.org/html/2607.01237#S4.T1), removing Token2Chunk leads to lower performance under compression\. These results show that expanding a subset of selected discrete KV pairs into flexible\-sized contiguous chunks at arbitrary positions helps preserve additional semantic context and improves compression performance\.

## 5Related Work

KV cache compressionLiet al\.\([2024a](https://arxiv.org/html/2607.01237#bib.bib4)\); Yuanet al\.\([2024a](https://arxiv.org/html/2607.01237#bib.bib32)\)aims to reduce inference\-time memory usage while keeping performance degradation within an acceptable range, typically by retaining important KV pairs and evicting unimportant ones\. Early compression methods, such as StreamingLLMXiaoet al\.\([2024](https://arxiv.org/html/2607.01237#bib.bib10)\), preserve KV pairs at fixed prefix positions, ignoring the varying importance of different tokens\. Mainstream KV cache compression methodsCaiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib6)\); Liet al\.\([2024b](https://arxiv.org/html/2607.01237#bib.bib8)\); Ramachandranet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib24)\); Joneset al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib28)\); Maoet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib29)\)leverage LLM\-internal query signals to score historical KV pairs and typically achieve state\-of\-the\-art compression performance\. Nevertheless, we observe two limitations of score\-based KV cache compression\. First, most methods rely on a threshold\-triggered policy for decoding\-time compression, where compression is executed once the cache length reaches a predefined threshold\. This policy may degrade throughput even when compression increases the number of concurrently running sequences\. Moreover, most methods perform eviction over the entire KV cache, which may fully remove KV pairs from a contiguous span of context and exacerbate information loss\. Some works mitigate issues induced by this policy by adopting delayed compression\. For example, DMSŁańcuckiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib12)\)scores KV pairs within a recent sliding window and executes eviction after tokens leave the window\. TRIM\-KVBuiet al\.\([2026](https://arxiv.org/html/2607.01237#bib.bib35)\)further introduces a time\-decay mechanism on KV importance to avoid immediate eviction after leaving the window\. However, these methods may still require frequent eviction operations under concurrency and may still fail to avoid eliminating KV pairs from long contiguous context spans\. Second, existing methods often fail to preserve flexible\-size context information distributed at arbitrary positions across the sequence\. Most methods retain either isolated KV pairs or fixed\-size chunks with rigid boundaries\. Different from the above works, we score KV pairs using sliding\-window bidirectional attention and introduce Token2Chunk to expand selected discrete KV pairs into flexible\-sized chunks at arbitrary positions\. Our sliding\-window bidirectional attention scoring also shares a similar intuition with DeepSeek\-V4’s heavily compressed attentionDeepSeek\-AI \([2026](https://arxiv.org/html/2607.01237#bib.bib36)\), which directly merges the KV pairs of contiguous tokens within a long span into a single entry\. We also develop the KvLLM inference framework equipped with a periodic compression policy to reduce KV cache overhead and improve throughput\.

## 6Conclusion

We propose Kara, a decoding\-time sliding\-window KV cache compression method that scores and retains KV pairs in the recent context using bidirectional attention and augments discrete retention with a Token2Chunk module for flexible\-size chunks\. We further integrate it into PagedAttention and develop KvLLM framework with periodic trailing\-block compression to reduce KV\-cache memory usage and improve throughput\. In the future, we plan to combine KV offloading and KV retrieval to enable lossless KV cache compression, maintaining decoding performance with substantially smaller cache footprints\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px1.p1.7)\.
- \[2\]N\. Bui, S\. Sharma, S\. Lamba, S\. Mishra, and R\. Ying\(2026\)Cache what lasts: token retention for memory\-bounded KV cache in LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qCaq3jGb0S)Cited by:[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[3\]Z\. Cai, W\. Xiao, H\. Sun, C\. Luo, Y\. Zhang, K\. Wan, Y\. Li, Y\. Zhou, L\. Chang, J\. Gu, Z\. Dong, A\. Anandkumar, A\. Asi, and J\. Hu\(2026\)R\-KV: redundancy\-aware KV cache compression for reasoning models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=2jwAjomEDB)Cited by:[Appendix A](https://arxiv.org/html/2607.01237#A1.p1.1),[§1](https://arxiv.org/html/2607.01237#S1.p2.1),[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px3.p1.4),[§3\.1](https://arxiv.org/html/2607.01237#S3.SS1.p1.1),[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[4\]Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu,et al\.\(2024\)Pyramidkv: dynamic kv cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px2.p1.1)\.
- \[5\]DeepSeek\-AI\(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[6\]A\. Devoto, M\. Jeblick, and S\. Jégou\(2025\)Expected attention: kv cache compression by estimating attention from future queries distribution\.arXiv preprint arXiv:2510\.00636\.External Links:[Link](https://arxiv.org/abs/2510.00636)Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p2.1)\.
- \[7\]Y\. Feng, H\. Guo, J\. Lv, S\. K\. Zhou, and X\. Xie\(2026\)DefensiveKV: taming the fragility of KV cache eviction in LLM inference\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nJgS06sX3O)Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px3.p1.4)\.
- \[8\]Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou\(2026\)Ada\-KV: optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=tcisuhGsQZ)Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p2.1),[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px2.p1.1)\.
- \[9\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[1st item](https://arxiv.org/html/2607.01237#S1.I1.i1.p1.1)\.
- \[10\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p1.1),[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px3.p1.8)\.
- \[11\]D\. Jones, J\. Park, M\. J\. Morse, M\. Lee, M\. H\. Langston, and C\. Lott\(2026\)QuoKA: query\-oriented KV selection for efficient LLM prefill\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YS4N1YxXSM)Cited by:[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[12\]J\. Kim, J\. Kim, S\. Kwon, J\. W\. Lee, S\. Yun, and H\. O\. Song\(2026\)KVzip: query\-agnostic KV cache compression with context reconstruction\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=JFygzwx8SJ)Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p2.1)\.
- \[13\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[1st item](https://arxiv.org/html/2607.01237#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2607.01237#S1.p4.1),[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px2.p3.1)\.
- \[14\]A\. Łańcucki, K\. Staniszewski, P\. Nawrot, and E\. Ponti\(2026\)Inference\-time hyper\-scaling with KV cache compression\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=8ZiElzQxf1)Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p2.1),[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[15\]H\. Li, Y\. Li, A\. Tian, T\. Tang, Z\. Xu, X\. Chen, N\. Hu, W\. Dong, Q\. Li, and L\. Chen\(2024\)A survey on large language model acceleration based on kv cache management\.arXiv preprint arXiv:2412\.19442\.Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p1.1),[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[16\]H\. LI, Y\. Li, A\. Tian, T\. Tang, Z\. Xu, X\. Chen, N\. HU, W\. Dong, L\. Qing, and L\. Chen\(2025\)A survey on large language model acceleration based on KV cache management\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=z3JZzu9EA3)Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px2.p1.2)\.
- \[17\]Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen\(2024\)SnapKV: LLM knows what you are looking for before generation\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=poE54GOq2l)Cited by:[1st item](https://arxiv.org/html/2607.01237#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2607.01237#S1.p2.1),[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px3.p1.4),[§3\.1](https://arxiv.org/html/2607.01237#S3.SS1.p1.1),[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[18\]Z\. Lin, Z\. Gou, T\. Liang, R\. Luo, H\. Liu, and Y\. Yang\(2024\)Criticbench: benchmarking llms for critique\-correct reasoning\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 1552–1587\.Cited by:[1st item](https://arxiv.org/html/2607.01237#S1.I1.i1.p1.1),[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px1.p1.1)\.
- \[19\]X\. Liu, Z\. Tang, P\. Dong, Z\. Li, Liuyue, B\. Li, X\. Hu, and X\. Chu\(2026\)ChunkKV: semantic\-preserving KV cache compression for efficient long\-context LLM inference\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=20JDhbJqn3)Cited by:[2nd item](https://arxiv.org/html/2607.01237#S1.I1.i2.p1.1),[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px2.p1.1)\.
- \[20\]Y\. Mao, Q\. Wang, M\. Ester, and K\. Li\(2026\)IceCache: memory\-efficient KV\-cache management for long\-sequence LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yHxSKM9kdr)Cited by:[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[21\]J\. Park, D\. Jones, M\. J\. Morse, R\. Goel, M\. Lee, and C\. Lott\(2026\)KeyDiff: key similarity\-based KV cache eviction for long\-context LLM inference in resource\-constrained environments\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=uBaFH7aQnC)Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px3.p1.4)\.
- \[22\]Z\. Qin, Y\. Cao, M\. Lin, W\. Hu, S\. Fan, K\. Cheng, W\. Lin, and J\. Li\(2025\)CAKE: cascading and adaptive KV cache eviction with layer preferences\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=EQgEMAD4kv)Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px3.p1.4)\.
- \[23\]A\. Ramachandran, M\. Neseem, C\. Sakr, R\. Venkatesan, B\. Khailany, and T\. Krishna\(2026\)ThinKV: thought\-adaptive KV cache compression for efficient reasoning models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=M3CeHnZKNC)Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px3.p1.4),[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[24\]S\. Sia, D\. Mueller, and K\. Duh\(2024\)Where does in\-context learning \\\\ happen in large language models?\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=LLuSjg59an)Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px1.p1.7)\.
- \[25\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p1.1)\.
- \[26\]G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis\(2024\)Efficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p2.1),[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px3.p1.4),[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[27\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p1.1),[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px3.p1.8)\.
- \[28\]J\. Yuan, H\. Liu, S\. Zhong, Y\. Chuang, S\. Li, G\. Wang, D\. Le, H\. Jin, V\. Chaudhary, Z\. Xu, Z\. Liu, and X\. Hu\(2024\-11\)KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4623–4648\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.266/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.266)Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p2.1),[§5](https://arxiv.org/html/2607.01237#S5.p1.1)\.
- \[29\]Z\. Yuan, Y\. Shang, Y\. Zhou, Z\. Dong, Z\. Zhou, C\. Xue, B\. Wu, Z\. Li, Q\. Gu, Y\. J\. Lee,et al\.\(2024\)Llm inference unveiled: survey and roofline model insights\.arXiv preprint arXiv:2402\.16363\.Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p1.1)\.
- \[30\]Q\. Zhang, A\. Cheng, M\. Lu, R\. Zhang, Z\. Zhuo, J\. Cao, S\. Guo, Q\. She, and S\. Zhang\(2025\)Beyond text\-visual attention: exploiting visual cues for effective token pruning in vlms\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 20857–20867\.Cited by:[§1](https://arxiv.org/html/2607.01237#S1.p2.1)\.
- \[31\]Y\. Zhang and T\. Math\-AI\(2024\)American invitational mathematics examination \(aime\) 2024\.Cited by:[§4](https://arxiv.org/html/2607.01237#S4.SS0.SSS0.Px1.p1.1)\.
- \[32\]L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng\(2024\)SGLang: efficient execution of structured language model programs\.NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§2](https://arxiv.org/html/2607.01237#S2.SS0.SSS0.Px2.p3.1)\.

## Appendix ADetailed Experimental Settings for Figure[2](https://arxiv.org/html/2607.01237#S1.F2)and[2](https://arxiv.org/html/2607.01237#S1.F2)

This section provides detailed settings for the simple experiment shown in Figure[2](https://arxiv.org/html/2607.01237#S1.F2)and[2](https://arxiv.org/html/2607.01237#S1.F2)\. We run all experiments on the MATH\-500 dataset using DeepSeek\-R1\-Distill\-LLaMA\-8B\. In Figure[2](https://arxiv.org/html/2607.01237#S1.F2), we vary the maximum number of decoding sequences \(*i\.e\.,*the maximum concurrency\) from 16 to 256\. On MATH\-500, the model’s average generation length is approximately 3,000 tokens\[[3](https://arxiv.org/html/2607.01237#bib.bib6)\]\. For vLLM, We use tensor parallelism with size 4, the maximum number of tokens per batch is set to 16,384 and the PagedAttention block size is set to 256 tokens and the maximum generation length per sequence is set to 8,192 token\. These experiments are conducted on RTX 5090 GPU\. For Figure[2](https://arxiv.org/html/2607.01237#S1.F2), we visualize results from a certain layer and attention head\.

## Appendix BFormula of TPOT and Throughput

Considering a simplified decoding setting with decoding batch sizeBB333BBvaries dynamically during decoding\., and BF16 KV cache, and an FFN intermediate size set to four times the hidden size, then TPOT can be approximated as

TPOT≈max⁡\(L​\(12​B​D2\+2​B​S​D\)ρ,4​L​B​S​Dβ\),\\text\{TPOT\}\\approx\\max\\left\(\\frac\{L\(12BD^\{2\}\+2BSD\)\}\{\\rho\},\\frac\{4LBSD\}\{\\beta\}\\right\),\(10\)whereρ\\rhodenotes the effective GPU compute throughput,D=H​dD=Hdis the hidden dimension andβ\\betadenotes the effective HBM bandwidth, and modern GPUs generally have a high compute\-to\-bandwidth ratio\. The first term in Equation \([10](https://arxiv.org/html/2607.01237#A2.E10)\) corresponds to the computation cost of one decoding forward step, while the second term captures the memory access overhead of KV cache\. The output throughput can be formulated as:

Throughput≈BTPOT,\\text\{Throughput\}\\approx\\frac\{B\}\{\\text\{TPOT\}\},\(11\)From Equation[10](https://arxiv.org/html/2607.01237#A2.E10)and[11](https://arxiv.org/html/2607.01237#A2.E11), we can observe that in long\-context reasoning scenarios, the inference overhead may be dominated by memory access to the KV cache, leaving the inference memory\-bound, and higherBBgenerally enables higher throughput\.

## Appendix CDetail Settings of Experiments

Kara experiments are running on 8 H200 GPUs and 8 H100 GPUs\.