Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

# Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving [Experiment, Analysis & Benchmark]
Source: [https://arxiv.org/html/2607.05399](https://arxiv.org/html/2607.05399)
###### Abstract\.

Large language model serving is increasingly limited by KV\-cache growth under long\-context workloads, yet existing KV\-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks\. This paper presents a workload\-aware benchmark of representative KV\-cache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench\-style multi\-document QA, single\-document QA, few\-shot learning, and summarization workloads using Llama\-3\.1\-8B\-Instruct and Mistral\-7B\-Instruct\-v0\.3\. The benchmark measures task quality, mean output throughput, mean time\-to\-first\-token, and realized compression ratio across context\-length buckets\. The results show that the compression ratio alone is a poor predictor of end\-to\-end performance\. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long\-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio\. These findings motivate workload\-aware selection of KV\-cache mechanisms rather than one\-size\-fits\-all compression and provide deployment guidance for long\-context serving systems\.

PVLDB Reference Format: PVLDB, 14\(1\): XXX\-XXX, 2020\. [doi:XX\.XX/XXX\.XX](https://doi.org/XX.XX/XXX.XX)††This work is licensed under the Creative Commons BY\-NC\-ND 4\.0 International License\. Visit[https://creativecommons\.org/licenses/by\-nc\-nd/4\.0/](https://creativecommons.org/licenses/by-nc-nd/4.0/)to view a copy of this license\. For any use beyond those covered by this license, obtain permission by emailing[info@vldb\.org](https://arxiv.org/html/2607.05399v1/mailto:[email protected])\. Copyright is held by the owner/author\(s\)\. Publication rights licensed to the VLDB Endowment\. Proceedings of the VLDB Endowment, Vol\. 14, No\. 1 ISSN 2150\-8097\. [doi:XX\.XX/XXX\.XX](https://doi.org/XX.XX/XXX.XX)

## 1\.Introduction

The adoption of large language models \(LLMs\) has grown rapidly across a wide range of data\-intensive applications, including document summarization, multi\-turn dialogue, and code analysis\(Duanet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib56); Barbon Junioret al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib57)\)\. As these applications increasingly rely on long\-context inputs, efficient LLM serving has emerged as a critical systems challenge\. In particular, the Key–Value \(KV\) cache used in LLMs grows linearly with input length, leading to substantial memory and bandwidth overhead during inference\(Liuet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib55)\)\. This has motivated a surge of recent work on KV cache optimization techniques, such as quantization, pruning, and merging, to enable scalable long\-context serving\.

The data management community has recently begun to address these challenges by rethinking LLM serving stacks and system\-level optimizations\. Previous works have explored enhancing LLM serving by improving efficiency in inference pipelines, memory\-aware scheduling, and hardware\-conscious optimizations\(Pan and Li,[2025](https://arxiv.org/html/2607.05399#bib.bib58); Liet al\.,[2025b](https://arxiv.org/html/2607.05399#bib.bib59); Yuanet al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib61); Liet al\.,[2024a](https://arxiv.org/html/2607.05399#bib.bib62)\)\. A growing research area addresses KV cache compression techniques to reduce memory footprint and improve decoding efficiency\(Wanget al\.,[2025b](https://arxiv.org/html/2607.05399#bib.bib60)\)\. However, the proposed methods have only been evaluated in isolation or compared with few other techniques\. The corresponding publications make a direct comparison difficult, as they use different models, datasets, compression budgets, and system configurations\. Thus, it remains unclear how these methods compare among themselves, and which methods are most suitable for different workloads\.

This lack of a unified and workload\-aware evaluation is a key gap in the literature\. In particular, existing studies often focus on either model quality or system efficiency in isolation, without jointly analyzing their trade\-offs\(Liuet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib55); Yuanet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib54)\)\. Furthermore, there is limited understanding of how KV cache optimizations affect system metrics such as memory and bandwidth across various task types, input lengths, and model architectures\. This slows the development of robust KV cache compression techniques and system\-level optimization strategies\.

In this paper, we present a comprehensive benchmark for KV cache optimization methods that jointly evaluates task quality and system performance under long\-context workloads\. We focus on representative inference\-time techniques spanning three major paradigms: quantization, pruning, and merging\. We employ widely used instruction\-tuned models, Llama\-3\.1\-8B\-Instruct\(Heet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib16)\)and Mistral\-7B\-Instruct\-v0\.3\(Jianget al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib53)\), and cover both task\-level accuracy and system\-level efficiency\. Our benchmark is built on the LongBench suite\(Baiet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib51)\), from which we select six datasets to evaluate task quality across four categories: multi\-document QA, single\-document QA, few\-shot learning, and summarization\. To assess system performance, we use three representative long\-context datasets: NarrativeQA, GovReport, and Qasper\. We measure key metrics, including time\-to\-first\-token \(TTFT\), output throughput, and prefill KV cache memory footprint\. This dual evaluation of quality and system performance enables us to capture the trade\-offs between accuracy preservation and inference efficiency across different KV cache compression strategies\.

Our main contributions are as follows:

- •We present a workload\-aware benchmark that systematically evaluates KV cache compression methods across both task quality and system performance dimensions\. This provides a holistic perspective on the methods and their associated trade\-offs\.
- •We conduct a unified evaluation of representative methods from quantization \(KIVI, TurboQuant\), pruning and eviction \(SnapKV\) and merging \(CaM\) with consistent models, datasets, and experimental settings\. This provides a fair comparison under realistic conditions\.
- •We study the tradeoffs between accuracy, throughput, latency, and memory footprint, and demonstrate that the compression ratio alone is not enough to evaluate the end\-to\-end performance of a KV compression approach\. This insight has an immediate practical use and also guides future research on new KV cache optimizations\.

The paper is organized as follows\. Section[2](https://arxiv.org/html/2607.05399#S2)introduces the necessary background of LLM serving and KV cache optimizations, including a detailed introduction of the techniques we use in our benchmark\. In Section[3](https://arxiv.org/html/2607.05399#S3), we provide a detailed account of the experimental setup, including workloads, setup and metrics\. Based on that, we discuss the results in Section[4](https://arxiv.org/html/2607.05399#S4)\. From the results, we derive practical insights and lessons learned in Section[5](https://arxiv.org/html/2607.05399#S5)\. Finally, we discuss related work in Section[6](https://arxiv.org/html/2607.05399#S6)before concluding the paper in Section[7](https://arxiv.org/html/2607.05399#S7)\.

## 2\.Background

![Refer to caption](https://arxiv.org/html/2607.05399v1/x1.jpg)Figure 1\.Prefill and decode during LLM inference\. Token T0 is detokenized right after the prefill stage\. After each token is generated in the decode step, the KV cache is updated\. Detokenization happens after each decode step\. Each token \(T0, T1, T2, T3,…\.,Tn\) is detokenized and output sequentially\.### 2\.1\.LLM Inference Serving

Large language model \(LLM\) inference serving refers to the system\-level process of executing queries on pretrained models to generate outputs in real time\. Fig\.[1](https://arxiv.org/html/2607.05399#S2.F1)depicts a typical serving pipeline, where an input prompt is first tokenized, i\.e\., converted from raw text into a sequence of discrete token IDs that can be processed by the model\. The tokenized input is then processed during the prefill phase\. In this phase, the entire input prompt is encoded, and the intermediate keys and values \(KV\) are computed and stored\. This is followed by thedecoding phase, where tokens are generated autoregressively, one step at a time\(Wanget al\.,[2025b](https://arxiv.org/html/2607.05399#bib.bib60)\)\. Each generated token is subsequently detokenized, i\.e\., mapped back from token IDs to human\-readable text, enabling streaming output to the user\.

#### Prefill Phase\.

LetX∈ℝb×lprompt×dX\\in\\mathbb\{R\}^\{b\\times l\_\{\\text\{prompt\}\}\\times d\}denote the input tensor, wherebbis the batch size,lpromptl\_\{\\text\{prompt\}\}is the prompt length, andddis the model hidden size\. For simplicity, we omit the layer index\. The key and value tensors are computed as

XK=X​WK,XV=X​WVX\_\{K\}=XW\_\{K\},\\quad X\_\{V\}=XW\_\{V\}whereWK,WV∈ℝd×dW\_\{K\},W\_\{V\}\\in\\mathbb\{R\}^\{d\\times d\}are the projection matrices for keys and values, respectively\. Once computed,XKX\_\{K\}andXVX\_\{V\}are stored in the KV cache to facilitate efficient decoding\(Liuet al\.,[2024c](https://arxiv.org/html/2607.05399#bib.bib17)\)\.

#### Decoding Phase

Lett∈ℝb×1×dt\\in\\mathbb\{R\}^\{b\\times 1\\times d\}denote the current input token embedding\. The corresponding key and value outputs are computed astK=t​WKt\_\{K\}=tW\_\{K\}andtV=t​WVt\_\{V\}=tW\_\{V\}, respectively\. First, the KV cache is updated by appending the new entries:

XK←Concat​\(XK,tK\),XV←Concat​\(XV,tV\)\.X\_\{K\}\\leftarrow\\mathrm\{Concat\}\(X\_\{K\},t\_\{K\}\),\\quad X\_\{V\}\\leftarrow\\mathrm\{Concat\}\(X\_\{V\},t\_\{V\}\)\.
Next, the attention output is computed as:

tQ\\displaystyle t\_\{Q\}=t​WQ,\\displaystyle=tW\_\{Q\},A\\displaystyle A=Softmax​\(tQ​XK⊤\),\\displaystyle=\\mathrm\{Softmax\}\(t\_\{Q\}X\_\{K\}^\{\\top\}\),tO\\displaystyle t\_\{O\}=A​XV,\\displaystyle=AX\_\{V\},whereWQW\_\{Q\}denotes the query projection matrix\. For simplicity, we omit the attention output projection layer and other components of the full inference pipeline\(Liuet al\.,[2024c](https://arxiv.org/html/2607.05399#bib.bib17)\)\.

To avoid recomputing attention over all previous tokens at each decoding step, modern LLM systems maintain aKV cache\. For a sequence of lengthNN, each Transformer layer stores key and value tensors corresponding to all past tokens\. During decoding, the query vector of the current token attends to the cached keys and values, reducing the computational complexity fromO​\(N2\)O\(N^\{2\}\)toO​\(N\)O\(N\)per step\(Liuet al\.,[2024b](https://arxiv.org/html/2607.05399#bib.bib63)\)\. This reuse of previously computed representations is critical for achieving low\-latency inference\.

However, the KV cache can introduce a major system bottleneck\. Its memory footprint grows linearly with both sequence length and the number of layers, often dominating GPU memory usage in long\-context scenarios\(Liet al\.,[2025b](https://arxiv.org/html/2607.05399#bib.bib59)\)\. For large models and long inputs, KV cache storage and access become the primary limiting factors for throughput and scalability\. This has motivated many works on KV cache optimization techniques, including quantization, pruning, and merging, which aim to reduce memory usage while preserving model accuracy\.

### 2\.2\.Taxonomy of KV Cache Optimizations

To compress the KV cache size, there have been proposed four principal methods\.

#### Quantization

The first principal method maintain all entries in the KV cache, but reduce the bit\-length \(precision\) of the encoding of the stored keys and values\. A common theme that runs among many KV quantization algorithm is mixed precision, where important tokens are kept at higher precision while heavily quantizing others\. For example, ZipCache\(Heet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib16)\)and QAQ\(Chenget al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib20)\)use attention\-derived properties to identify important tokens and store them at higher precision\. In ZipCache, a channel\-wise token quantization is proposed to reduce parameter overhead, and a normalized attention score guides which tokens to preserve\. Similarly, KIVI\(Heet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib16)\)finds that keys have a few large\-magnitude channels while values do not\. It thus applies per\-channel quantization for keys and per\-token quantization for values, enabling 2\-bit KV storage with almost no accuracy loss\. KVQuant\(Hooperet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib19)\)also exploits KV structure\. It quantizes keys before rotary embeddings and isolates outliers per vector by keeping them in higher precision\.

Another class uses vector transforms or codebooks to ease quantization\. For example, CommVQ\(Liet al\.,[2025a](https://arxiv.org/html/2607.05399#bib.bib22)\)and PQCache\(Chenget al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib20)\)apply vector quantization\. They split each KV vector into subvectors or additive code components and store compact indices instead of full float values\. CommVQ\(Liet al\.,[2025a](https://arxiv.org/html/2607.05399#bib.bib22)\)even designs its codebooks to commute with rotary embeddings, enabling as low as 1\-bit precision with minimal loss\. PolarQuant\(Wuet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib23)\)and TurboQuant\(Zandiehet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib24)\)apply random rotations or coordinate changes: PolarQuant\(Wuet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib23)\)transforms KV vectors to polar coordinates after a random rotation, yielding tightly distributed angles that can be quantized without extra scaling parameters\. TurboQuant\(Zandiehet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib24)\)shows that a random rotation makes each coordinate follow a concentrated Beta distribution, so simple optimal scalar quantizers can nearly achieve the theoretical best distortion\.

Quantization approaches trade memory for extra computation or design complexity\. Mixed\-precision schemes require computing token importance using attention scores or norms, and per\-channel methods need special grouping of data\. Transform\-based methods incur the cost of rotating or projecting vectors at decode time, and product quantization uses a cookbook for encoding/decoding\. However, the extra computational cost provides orders\-of\-magnitude memory savings with only small drops in accuracy\. Aggressive quantization increases compression but risks output quality\. Methods that preserve outliers or adapt precision mitigate this at the cost of extra overhead\(Hooperet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib19); Zandiehet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib24)\)\.

#### Pruning

The second principal method approaches the problem by compressing the KV cache along the token or structural dimension, either by discarding less important tokens or by sparsifying their representations\. The core idea is to rank tokens by some importance signal and keep only a subset\. Similar to quantization, many pruning methods use attention\-derived scores to decide which tokens to retain\. For example, SAGE\-KV\(Wanget al\.,[2025a](https://arxiv.org/html/2607.05399#bib.bib67)\)computes attention after prefill and retains only the top\-k tokens for inference\. This exploits the “attention sparsity” observation, usually only a few tokens have high influence\. Similar approaches such as H2O\(Zhanget al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib26)\)and SnapKV\(Liet al\.,[2024b](https://arxiv.org/html/2607.05399#bib.bib25)\), aggregate past attention to pick important tokens\. Another simple yet effective approach is recency\. Sliding\-window schemes such as StreamingLLM\(Xiaoet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib27)\)keep only the most recent tokens plus a few early attention sinks to stabilize performance\. These methods incur minimal overhead as no scoring of tokens is required, but can mistakenly drop older relevant context\. Other signals such as L2 norm has been explored to guide eviction decisions\(Devotoet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib28)\)\.

Beyond token level pruning, structured pruning removes fixed groups of tokens at once\. For example, PagedEviction operates on block “pages” of the KV cache\(Chitty\-Venkataet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib30)\)\. It evicts full blocks aligned to the memory layout, resulting in less fragmentation and overhead, but coarser granularity\. The trade\-off here is that aggressive pruning saves memory, but has the risk of losing useful context\. Attention\-based pruning is precise but adds computation; static policies are simpler but less adaptable\. Empirically, hybrid\-strategies often works best\. In all cases, the main cost is potential accuracy loss\. Evicted tokens cannot be recovered, so if importance is misjudged, model outputs degrade\.

#### Merging

The third principal method aim to compress the KV cache by combining redundant or highly similar tokens into shared representations, rather than discarding them outright\. The core idea is to cluster redundant tokens and represent them with a shared state\. For example, Adaptive KV\(Geet al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib7)\)clusters KV tokens by similarity and merges each cluster into one representative\. Bolya et al\.\(Bolyaet al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib38)\)merges based on feature similarity on\-the\-fly during inference\. Other work performs multi\-level merging of tokens in a hierarchical manner, progressively compressing context while preserving high\-level semantic structure\(Songet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib37)\)\.

Merging incurs extra computation cost for calculating pairwise similarity or clustering similar tokens but can retain much of the original information compared to pruning\. It avoids outright deletion of content by combining tokens that have less importance with the retained tokens, preserving their relevance in a compressed form\(Geet al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib7)\)\. Few methods proposed by\(Munkhdalaiet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib39)\)and\(Kimet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib40)\)progressively push past tokens into a fixed\-size recurrent memory via a small updater network, so the KV cache acts like an RNN summary\. This makes the KV cache quite complex and the merged states may lead to loss of fine grained details among the tokens\. However, compared to pruning, merging yield smoother accuracy, since information is not fully lost\.

#### Cross\-Layer/Head Sharing and Low\-Rank Methods

Quantization, pruning, and merging work within the token dimension of the KV cache\. Another orthogonal approach compresses the KV cache across layers or attention heads, i\.e\., along the model’s feature dimensions\. They typically require model\-level changes or retraining\. A common theme is sharing KV projections\. For example, Multi\-Query Attention \(MQA\)\(Shazeer,[2019](https://arxiv.org/html/2607.05399#bib.bib48)\)uses one shared key/value projection per layer instead of per head, whereas Grouped\-Query Attention \(GQA\)\(Ainslieet al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib47)\)uses a few projections shared among groups of heads\. MLKV\(Zuhriet al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib43)\)extends this idea to multiple layers by sharing the same KV projections across layers, achieving up to 6× more compression beyond MQA\. Similarly, YOCO\(Sunet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib44)\)introduces a two\-stage transformer, which is a combination of a self\-decoder and a cross\-decoder, so that KV is computed globally once and reused by all layers\. Other schemes such as\(Liuet al\.,[2024a](https://arxiv.org/html/2607.05399#bib.bib45)\)and KVSharer\(Yanget al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib46)\)selectively merge KV states across layers based on similarity\. These methods drastically cut KV size and bandwidth by design, often with some pre\-training or architecture redesign\. The downside is complexity\. They require changes to the model architecture or training\.

Another direction targets the feature dimension using low\-rank approximations\. Methods such as Palu\(Changet al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib49)\)factorize attention into low\-rank components and cache compressed representations, reconstructing full KV states on demand\. LoRC\(Zhanget al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib50)\)applies layer\-wise low\-rank projections to compress KV weights without retraining\. By exploiting redundancy in hidden dimensions, these methods shrink KV storage, though they introduce approximation error and some extra compute for compression and reconstruction\.

Overall, these methods structurally reduce KV size and are complementary to token\-level techniques such as pruning or quantization\. Their main drawback is integration complexity and potential accuracy loss, while their key advantage is a consistent reduction in memory usage and bandwidth, independent of input length\.

### 2\.3\.Benchmarked Techniques

Selection rationale for this study:In our benchmarking study, we focus on drop\-in replaceable KV cache compression techniques that do not require deep changes to the model architecture or even retraining\. From the practical perspective of data management techniques, it is desirable that a data system architect or administrator can easily integrate the techniques into existing infrastructure\. Following this rationale, we selected representative techniques from the three categories of quantization \(KIVI, Turboquant\), pruning \(SnapKV\), and merging \(CaM\)\. We discuss the details of these techniques in the following\.

#### KIVI

The mathematical core of KIVI\(Liuet al\.,[2024c](https://arxiv.org/html/2607.05399#bib.bib17)\)is built upon the observation that the key cache \(KK\) and value cache \(VV\) exhibit different outlier structures\. In the key cache, outliers are concentrated in specific channels, meaning certain dimensions across all tokens have much higher magnitudes\. Conversely, the value cache contains outliers that are token\-specific, where certain entire tokens have higher magnitudes across all channels\. KIVI uses an asymmetric quantization scheme to map these high\-precision floating\-point values into a discreteBB\-bit integer space\.

For any given tensorXX, the quantization process is defined by finding a scaling factorsXs\_\{X\}and a zero\-pointzXz\_\{X\}\. The quantization functionQ​\(X\)Q\(X\)maps the input to the nearest integer in the range\[0,2B−1\]\[0,2^\{B\}\-1\]using the following formulas:

zX=min⁡\(X\),\\displaystyle z\_\{X\}=\\min\(X\),sX=max⁡\(X\)−min⁡\(X\)2B−1,\\displaystyle s\_\{X\}=\\frac\{\\max\(X\)\-\\min\(X\)\}\{2^\{B\}\-1\},Q\(X\)=⌊X−zXsX⌉\\displaystyle Q\(X\)=\\left\\lfloor\\frac\{X\-z\_\{X\}\}\{s\_\{X\}\}\\right\\rceil
KIVI adopts different quantization axes for keys and values to align with their distinct outlier patterns\. It appliesper\-channel quantizationto keys \(column\-wise\) so that large values concentrated in certain dimensions do not distort the scaling for the entire tensor\. For values, it appliesper\-token quantization\(row\-wise\) so that tokens with unusually large magnitudes are handled locally, preventing them from degrading the precision of other tokens\.

During the inference phase, the B\-bit quantization integers must be converted back to the original precision for the model to perform the attention operation\. This process can be expressed as

X′=Q​\(X\)⋅sX\+zXX^\{\\prime\}=Q\(X\)\\cdot s\_\{X\}\+z\_\{X\}
where⌊⋅⌉\\lfloor\\cdot\\rceildenotes the rounding operator to the nearest integer\.

To further stabilize performance, KIVI employs a streaming strategy involving a residual cache\. New tokens are kept in full precision because recent tokens often contribute most significantly to the attention output, and their distributions have not yet stabilized\. Once the residual cache reaches a pre\-defined threshold, the oldest tokens in the residual are quantized using the per\-channel/per\-token logic and appended to the compressed 2\-bit KV cache\. This hybrid precision approach ensures that the “local” context remains highly accurate while the “distant” context is efficiently compressed\.

#### TurboQuant

TurboQuant\(Zandiehet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib24)\)approaches transforms KV cache vectors into a distribution that is amendable to efficient quantization while preserving attention\-relevant inner products\. Given an input vectorx∈ℝdx\\in\\mathbb\{R\}^\{d\}, TurboQuant first applies a randomized orthogonal transformation using a matrixQ∈ℝd×dQ\\in\\mathbb\{R\}^\{d\\times d\}, producingy=Q​xy=Qx\. This transformation preserves the Euclidean norm while redistributing the vector’s energy uniformly across dimensions\. In high\-dimensional settings, the coordinatesyiy\_\{i\}approximate a Gaussian distribution, which enables the use of simple, data\-independent scalar quantizers\.

For a given vectoryy, TurboQuant performs coordinate\-wisebb\-bit quantization\. LetQs​\(⋅\)Q\_\{s\}\(\\cdot\)denote the scalar quantizer; the quantized vector is defined as

y^=\[Qs​\(y1\),Qs​\(y2\),…,Qs​\(yd\)\]⊤,\\hat\{y\}=\[Q\_\{s\}\(y\_\{1\}\),Q\_\{s\}\(y\_\{2\}\),\\dots,Q\_\{s\}\(y\_\{d\}\)\]^\{\\top\},with total distortion

MSE​\(y,y^\)=∑i=1d𝔼​\[\(yi−y^i\)2\]\.\\mathrm\{MSE\}\(y,\\hat\{y\}\)=\\sum\_\{i=1\}^\{d\}\\mathbb\{E\}\[\(y\_\{i\}\-\\hat\{y\}\_\{i\}\)^\{2\}\]\.By leveraging the Gaussian\-like distribution induced by the rotation, this approach achieves near\-optimal rate–distortion performance using simple scalar quantization\.

To mitigate the contraction bias introduced by MSE\-optimal quantization, TurboQuant incorporates a residual correction mechanism\. After quantization, the residual vectorr=y−y^r=y\-\\hat\{y\}is computed and encoded using a 1\-bit Quantized Johnson–Lindenstrauss \(QJL\) transform, storing only the sign information:

s=sign​\(r\)\.s=\\mathrm\{sign\}\(r\)\.During inference, the quantized vectory^\\hat\{y\}and residual signsssare combined to produce an unbiased estimate of inner products, ensuring that attention scores remain accurate despite aggressive compression\.

This design enables TurboQuant to compress KV cache vectors to very low bit rates while maintaining high fidelity in both reconstruction error and attention computation\.

#### SnapKV

SnapKV\(Liet al\.,[2024b](https://arxiv.org/html/2607.05399#bib.bib25)\)leverages the intrinsic sparsity and consistency of attention patterns in LLMs to compress the KV cache\. The method identifies salient historical tokens by observing the attention distribution at the end of a prompt and evicting non\-essential entries to maintain a constant\-sized cache\. This process is governed by a heuristic selection mechanism that utilizes pooled attention scores to determine which KV pairs are critical for future token generation\.

For a prompt of lengthNN, SnapKV computes attention weights using an observation window of queriesQobsQ\_\{\\text\{obs\}\}attending over prefix keysKprefixK\_\{\\text\{prefix\}\}:

A=Softmax​\(Qobs​Kprefix⊤d\)\.A=\\mathrm\{Softmax\}\\left\(\\frac\{Q\_\{\\text\{obs\}\}K\_\{\\text\{prefix\}\}^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)\.
To estimate token importance, the attention weights are aggregated across the observation window and smoothed using a 1D average pooling operation:

S=AvgPool​\(A,kernel=s\)\.S=\\mathrm\{AvgPool\}\(A,\\text\{kernel\}=s\)\.The most salient tokens are then selected via a Top\-kkoperation under a fixed cache budgetCC:

I=Top​\-​k​\(S,k=C−Lrecent\)\.I=\\mathrm\{Top\}\\text\{\-\}k\(S,k=C\-L\_\{\\text\{recent\}\}\)\.
The final compressed KV cache consists of the union of selected salient tokens and a recent token window:

K​Vcompressed=\{K​Vi∣i∈I\}∪\{K​Vj∣j\>N−Lrecent\}\.KV\_\{\\text\{compressed\}\}=\\\{KV\_\{i\}\\mid i\\in I\\\}\\cup\\\{KV\_\{j\}\\mid j\>N\-L\_\{\\text\{recent\}\}\\\}\.
By keeping bothCCandLrecentL\_\{\\text\{recent\}\}fixed, SnapKV preserves long\-range dependencies through salient token retention while maintaining local coherence via recent tokens, enabling efficient long\-context inference without retraining\.

#### CaM

Cache Merging \(CaM\) departs from conventional KV cache pruning methods by avoiding hard eviction and instead redistributing the contribution of removed tokens into retained ones\. The key idea is to preserve the attention output by merging values of evicted tokens into nearby tokens, thereby reducing the bias introduced by removing low\-probability but non\-negligible contributions\.

In a standard attention layer, the output for a query is given by

O=∑i=1nαi​Vi,O=\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}V\_\{i\},whereαi\\alpha\_\{i\}are attention weights andViV\_\{i\}are value vectors\. When a tokenkkis removed, CaM merges its contribution into a retained tokenjjby updating

Vj′=Vj\+αkαj​Vk,V\_\{j\}^\{\\prime\}=V\_\{j\}\+\\frac\{\\alpha\_\{k\}\}\{\\alpha\_\{j\}\}V\_\{k\},which preserves the attention output since

αj​Vj′=αj​Vj\+αk​Vk\.\\alpha\_\{j\}V\_\{j\}^\{\\prime\}=\\alpha\_\{j\}V\_\{j\}\+\\alpha\_\{k\}V\_\{k\}\.This formulation shows that, given exact attention ratios, merging can be lossless\.

Since future attention weights are unknown during inference, CaM approximates this process usingeven merging, where an evicted tokenViV\_\{i\}is distributed across a local window ofmmretained tokens:

Vj′=Vj\+1m​Vi\.V\_\{j\}^\{\\prime\}=V\_\{j\}\+\\frac\{1\}\{m\}V\_\{i\}\.This approximation assumes locally similar attention magnitudes and reduces variance compared to single\-point merging\.

To further improve robustness, CaM employs an adaptive merging strategy based on cumulative attention scoresA¯i\\bar\{A\}\_\{i\}, which provide a stable estimate of token importance\. The decision to merge a token is modeled as

Mi∼Bernoulli​\(clamp​\(A¯i1m​∑j∈windowA¯j,0,1\)\)\.M\_\{i\}\\sim\\mathrm\{Bernoulli\}\\left\(\\mathrm\{clamp\}\\left\(\\frac\{\\bar\{A\}\_\{i\}\}\{\\frac\{1\}\{m\}\\sum\_\{j\\in\\text\{window\}\}\\bar\{A\}\_\{j\}\},0,1\\right\)\\right\)\.Tokens are merged when their relative importance is sufficiently high; otherwise, they are evicted\.

Overall, CaM compresses the KV cache by converting discrete token removal into a continuous redistribution of value representations\. This design preserves attention outputs more faithfully than standard pruning while maintaining a bounded memory footprint\.

## 3\.Experimental Setup

Table 1\.Datasets, token ranges, and workload types\.#### Workloads

We evaluate KV cache compression methods using the LongBench benchmark\(Baiet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib51)\), which consists of many long\-context tasks\. Table[1](https://arxiv.org/html/2607.05399#S3.T1)shows the datasets we use along with the minimum and maximum context length test examples it consist of and the workload type\. We consider four varying categories: \(1\) multi\-document QA, which needs to extract and combine information from several documents to obtain the answer, \(2\) single\-document QA, which tests the long context understanding ability with longer documents, \(3\) few\-shot learning, which is a practical setting requiring long\-context understanding over provided examples, and \(4\) summarization, which requires a global understanding of the whole context in this work\(Yuanet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib54)\)\. For task quality evaluation, we use six representative datasets: HotpotQA and 2WikiMQA for multi\-document QA, Qasper and MultiFieldQA\_en for single\-document QA, TriviaQA for few\-shot learning, and MultiNews for summarization\. These datasets span a wide range of context lengths, reasoning complexity, and dependency patterns, from multi\-hop retrieval across documents to long\-form generation tasks\. For system efficiency evaluation, we focus on NarrativeQA, Qasper, and GovReport, as they contain substantially longer contexts, making them better suited to capturing realistic KV cache behavior and more representative system\-level performance under long\-context workloads\. This enables us to systematically study the impact of KV cache compression on task quality and system efficiency for various long context tasks\.

#### Compared Methods

We select a small but representative set of KV cache compression methods to capture the core design trade\-offs relevant for long\-context serving systems\. Rather than exhaustively evaluating all prior work, we focus on widely adopted, training\-free methods that can be directly applied at inference time and reflect distinct system\-level behaviors\.

For quantization, we evaluate KIVI\(Liuet al\.,[2024c](https://arxiv.org/html/2607.05399#bib.bib17)\)and TurboQuant\(Zandiehet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib24)\), which represent two complementary techniques\. KIVI is a data\-dependent approach that explicitly models the outlier structure in keys and values via asymmetric quantization, achieving consistent accuracy despite aggressive compression\. On the other hand, TurboQuant is a data\-oblivious, rotation\-based approach with strong theoretical guarantees, allowing efficient low\-bit compression that behaves distinctly under varying workloads\. For pruning and eviction, we include SnapKV\(Liet al\.,[2024b](https://arxiv.org/html/2607.05399#bib.bib25)\), a state\-of\-the\-art training\-free approach that leverages consistent attention patterns to retain salient tokens under a fixed cache budget\. This makes it representative of practical token\-level sparsification strategies used in long\-context inference\. For merging\-based compression, we evaluate CaM\(Zhanget al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib35)\), which departs from hard eviction by redistributing evicted token contributions to preserve attention outputs\. This provides a fundamentally different trade\-off, prioritizing output integrity over strict sparsity\.

These KV cache compression methods can be easily deployed in real systems\. It enables us to systematically analyze how different KV cache optimization strategies impact both task quality and end\-to\-end system performance under diverse workloads\.

#### LLMs

We conduct experiments on two widely used instruction\-tuned models: Llama\-3\.1\-8B\-Instruct\(Grattafiori and et al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib52)\)and Mistral\-7B\-Instruct\-v0\.3\(Jianget al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib53)\)\. Both models use Grouped\-Query Attention \(GQA\)\. The differentiator is that Mistral\-7B specifically utilizes Sliding Window Attention \(SWA\) and a rolling buffer cache for extreme efficiency, whereas Llama\-3\.1\-8B focuses on maximizing reasoning performance, utilizing a much larger 128k context window compared to 32k in Mistral\-7B\.

These architectural differences lead to different attention behaviors and KV cache usage patterns, providing a diversity of models for evaluating compression strategies\. Using both models allows us to determine whether the observed performance trends are consistent across LLM families or sensitive to model\-specific characteristics\.

#### Setup

All methods were evaluated using the same models and datasets\. Models were loaded in FP16 with FlashAttention2\(Dao,[2024](https://arxiv.org/html/2607.05399#bib.bib64)\)enabled, and prompts were formatted using the corresponding chat template for each instruction\-tuned model\. We used the LongBench prompt templates and dataset\-specific generation budgets\(Baiet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib51)\)\.

For KIVI\(Liuet al\.,[2024c](https://arxiv.org/html/2607.05399#bib.bib17)\), we use the official KIVI\-style asymmetric KV\-cache quantization implementation integrated into the Llama and Mistral attention modules\. We evaluate both 2\-bit and 4\-bit KV\-cache variants\. In both cases, keys and values are quantized to the same bit width, with group size for group\-wise quantization and residual length for preserving the latest tokens in full precision as 32\. The KIVI’s FP16 model where both keys and values are stored in full precision without any quantization is used as the full\-cache baseline\.

For TurboQuant\(Zandiehet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib24)\), we used the Transformers 4\.45\-compatible TurboQuant cache implementation by Omar Hory\(Hory,[2026](https://arxiv.org/html/2607.05399#bib.bib66)\)\. We evaluated 3\-bit and 4\-bit KV\-cache quantization\. The 3\-bit configuration used outlier\-aware quantization: for a head dimension of 128, 32 outlier channels were stored at 4 bits, giving an effective precision of 3\.25 bits per value\. The 4\-bit setting used uniform 4\-bit quantization without outlier channels\. The reported runs used the unpacked cache representation, so the reported memory usage may be slightly higher than with additional bit\-packing optimizations\.

For SnapKV, we used the SnapKVPress implementation from KVPress\(Devotoet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib65)\)\. The cache was compressed just after prefill using SnapKV’s attention\-based token selection\(Liet al\.,[2024b](https://arxiv.org/html/2607.05399#bib.bib25)\)\. We set the compression ratio to 0\.75\. In SnapKV, window size denotes the number of most recent query tokens used to estimate the importance of earlier KV entries, while kernel size denotes the width of the average\-pooling filter used to smooth the resulting attention\-based importance scores\. We set the window size to 32 and the kernel size to 7\.

For CaM\(Zhanget al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib35)\), we evaluated the CAMPress wrapper with SnapKV as the base compressor provided by KVPress\(Devotoet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib65)\)\. We used CAMPress with a compression interval of 32, which means that CaM applies one merge\-and\-prune compression pass after every 32 decoding steps\. After each compression pass, the KV cache is reduced to a pre\-defined number of retained tokens per layer, known as the target cache size\. We set it to 1024\. The hidden\-state buffer size is set to 64, which provides longer recent decoding context available for scoring and compressing\. A merge budget of 32 helps redistribute the token value across up to 32 subsequent tokens instead of being simply dropped\. The underlying SnapKV base press uses a window size of 16 and a kernel size of 7\. We use a smaller SnapKV window inside CaM because CaM performs periodic decoding\-time compression using recent buffered hidden states and cumulative attention, so a shorter observation window emphasizes the most recent local decoding context and keeps the base scorer aligned with interval\-based compression\. Thus, CaM performs periodic cache merging during generation rather than only applying a one\-shot prefill compression\.

Multi\-Doc QASingle\-Doc QAFew\-Shot LearningSummarizationModelsHotpotQA2WikiMQAQasperMultifieldQA\_enTrivia\_QAMulti\-newsLlama\-3\.1\-8b\-InstructAll KV55\.7244\.2745\.7653\.9291\.6527\.15KIVI254\.4242\.9444\.0754\.9892\.5626\.99KIVI455\.9946\.6345\.4254\.7491\.6626\.95TurboQuant353\.1941\.5746\.6051\.4684\.8416\.39\*TurboQuant456\.1045\.0746\.0353\.3190\.1316\.16\*SnapKV\-0\.7559\.249\.4146\.3756\.0692\.0423\.04CaM59\.851\.3347\.3256\.4291\.7117\.56Mistral\-7b\-Instruct\-v0\.3All KV51\.5436\.1638\.2750\.1088\.5926\.42KIVI248\.4638\.3439\.8954\.1287\.8227\.04KIVI448\.9839\.1141\.1053\.5688\.5427\.68Turboquant349\.4534\.3137\.1650\.2188\.0515\.86\*Turboquant449\.4333\.9440\.3950\.6588\.0916\.01\*SnapKV\-0\.7548\.4338\.4636\.4451\.2386\.3124\.25CaM48\.5737\.5940\.5552\.485\.3716\.74Table 2\.Accuracy results on LongBench tasks for Llama\-3\.1\-8B\-Instruct and Mistral\-7B\-Instruct\-v0\.3 under different KV compression methods\. Bold and italic values denote the best and second\-best performance per column, highlighting the trade\-offs between compression efficiency and task\-specific accuracy\.Note: \*Summarization task using TurboQuant was only ran once as the run takes 3\-4 days to complete\.We conduct our experiments on a NVIDIA A100 GPU with 40GB of memory\. As a result, we limit our evaluation to models in the 7\-8B parameter range, which represents a practical balance between model capability and feasible long\-context inference under KV cache compression\. The main trends we observe are expected to generalize to larger models and distributed settings\. KV cache size and memory bandwidth costs grow with model size and context length, so KV compression carries the same importance for larger models\(Liuet al\.,[2024c](https://arxiv.org/html/2607.05399#bib.bib17)\)\. The workload\-dependent patterns we observe are driven by task structure and are expected to persist\. However, exact memory usage, throughput, and latency will depend on system\-level optimizations, so quantitative gains may differ in larger deployments\.

#### Metrics

We evaluate KV cache compression methods along two dimensions:task qualityandsystem efficiency\.

We report both per\-task performance and category\-level averages to capture method behavior across heterogeneous workloads\. In doing so, we rely on accuracy metrics provided by LongBench for each dataset\(Baiet al\.,[2024](https://arxiv.org/html/2607.05399#bib.bib51)\)\. For the question answering tasks HotpotQA, 2WikiMQA, Qasper, and MultiFieldQA\_en, we use the F1 score, which measures the overlap between predicted and ground\-truth answers\. For TriviaQA, we use the exact match \(EM\), which checks whether the model’s predicted answer matches the ground\-truth answer exactly, after applying basic normalization\. For summarization datasets such as MultiNews, we report ROUGE scores, which evaluate the quality of generated summaries based on n\-gram overlap with reference summaries\. These metrics reflect the model’s ability to preserve semantic correctness and coherence under KV cache compression\. We report both per\-dataset performance and category\-level averages\.

We focus our system\-level evaluation on three representative datasets: NarrativeQA, GovReport, and Qasper, as they have longer context lengths\. We measure efficiency metrics such as time\-to\-first\-token \(TTFT\), output throughput, and prefill KV memory\.

Time\-to\-First\-Tokenmeasures the time elapsed from when a user submits a prompt to when the model generates its very first output token\. It captures the total latency of the prefill stage, including KV cache construction and initial model overhead\. It is heavily influenced by input prompt length, KV cache compression strategies, and includes the computation time required to generate the very first output token\. Lower TTFT indicates faster perceived responsiveness in real\-time applications such as chatbots, voice AI, and interactive streaming because it reduces the initial wait time before the user sees the first piece of output\.

Output Throughputis measured as the average number of tokens generated per second during inference\. This metric reflects the efficiency of the decoding stage and is directly impacted by the KV cache compression used\. Higher throughput indicates better system performance\.

Prefill KV memorymeasures how much VRAM is consumed to hold the context during that initial KV processing stage, which is a major factor in overall peak memory\. This metric captures the effectiveness of each compression method in reducing memory usage\. Lower KV cache memory enables longer context lengths and improved scalability under given hardware constraints\.

## 4\.Results

Table 3\.Time to first token of different KV caching methods across 3 datasets \(NarrativeQA, GovReport, Qasper\) and varying context length ranges for Llama\-3\.1\-8B\-Instruct and Mistral\-7B\-Instruct\-v0\.3\.
Note:We do not present results for Gov\_Report using TurboQuant because it takes up to 3 days for one single run\.Our results are organized into two sections:Task QualityandSystem Efficiency\.

### 4\.1\.Task Quality

We first discuss general observations and trends across all tasks before discussing the detailed results for each workload in separate paragraphs\. Table[2](https://arxiv.org/html/2607.05399#S3.T2)shows all accuracy results\.

#### General Observations

We observe clear differences between KV\-cache compression paradigms across all workloads and models\. Overall, moderate quantization methods, especially KIVI4, achieve the most stable accuracy across workloads, while pruning using SnapKV and merging using CaM often yield higher peak accuracy across the QA workload, but the accuracy is variable across other tasks\. This can be attributed to the fact that, despite aggressive compression, KIVI handles key/value outliers in an efficient manner\. Aggressive quantization, such as in TurboQuant, leads to a drop in quality, especially for generation\-heavy workloads such as summarization\. This indicates that, although rotation\-based quantization is theoretically efficient, it introduces reconstruction errors that negatively affect tasks requiring global context understanding\.

#### Multi\-Document QA

For Multi\-Document QA, CaM and SnapKV achieve good performance across both models\. On Llama\-3\.1\-8B, both CaM and SnapKV even outperform the All\-KV baseline\. This can be explained by the nature of the Multi\-Doc QA task, which requires the retrieval and aggregation of sparse but highly relevant information across documents\. Pruning preserves important tokens based on attention scores, ensuring that important cross\-document context is retained, and merging redistributes information instead of discarding it, which reduces information loss compared to pruning\. Heavy quantization with KIVI2 and TurboQuant3 shows slight performance degradation\. This is likely because precision loss in attention scores affects multi\-hop reasoning, where small numerical differences can propagate across reasoning chains\.

#### Single\-Document QA

In Single\-Document QA tasks, results are more balanced, especially with the Llama model\. Similar to Multi\-Document QA, CaM achieves the best performance across all the compression methods with Llama, while KIVI4 performs consistently well across both models\.

The good performance of CaM indicates that merging benefits tasks that require holistic document understanding, as it preserves broader context\. KIVI4 achieves the best Qasper accuracy score and near\-best MultifieldQA scores for Mistral\-7b\-Instruct\-v0\.3\. The relatively weaker performance of SnapKV on Mistral suggests that aggressive token selection may remove context needed for detailed comprehension, especially in long single documents\.

#### Few\-Shot Learning

In few\-shot learning, quantization methods perform surprisingly well, with KIVI2 achieving the highest score\. This suggests that few\-shot learning tasks are less sensitive to fine\-grained KV precision, as they rely more on pattern recognition from examples rather than on exact long\-range dependencies\. Few\-shot learning workload is robust to compression as long as recent tokens, which here are few\-shot examples, are preserved\. KIVI explicitly ensures this via its residual cache mechanism\.

In contrast, aggressive TurboQuant compression using the Llama\-3\.1\-8b\-Instruct model shows a noticeable degradation in quality of almost 7%\. This indicates that aggressive low\-bit quantization may still harm the pattern extraction required for few\-shot learning tasks\.

SnapKV and CaM remain competitive but do not outperform KIVI\. This can likely be attributed to the fact that token selection and eviction or merging provide less benefit when most few\-shot examples are already highly relevant\.

#### Summarization

Summarization workload exhibits the largest largest performance drop across compression methods, especially for TurboQuant and CaM\.

Quantization using KIVI remains closest to baseline All\-KV quality performance, mainly because KIVI retains all tokens, only reducing precision rather than changing the token structure\. TurboQuant underperforms in summarization because its uniform quantization introduces small errors across all tokens, which accumulate and disrupt the global context coherence required for high\-quality summaries\. SnapKV shows a moderate degradation in quality of approximately 15% and 8% using Llama\-3\.1\-8b\-Instruct and Mistral\-7b\-Instruct\-v0\.3 respectively\. This reflects that pruning might remove information that may still be important for coherent summaries\. Whereas merging, especially using CaM, introduces approximation errors when combining tokens, which results in a quality degradation of approximately 35\-36% using both models\.

Thus, summarization is most sensitive to structural modifications of the KV cache, confirming that compression methods that preserve full context are preferable\.

### 4\.2\.System Efficiency

Compression RateContextMin\_tokensMax\_TokensKIVI2KIVI4TurboQuant3TurboQuant4SnapKV\-0\.75CaMLlama\-3\.1\-8b\-InstructNarrativeQA4\-8K796479645\.253\.174\.273\.764\.001\.008K\+8962652715\.323\.194\.273\.764\.001\.00GovReport0\-4K202039195\.163\.15\-\-4\.003\.094\-8K413879795\.243\.17\-\-4\.006\.048K\+8118513935\.293\.18\-\-4\.0013\.82Qasper0\-4K184739345\.163\.154\.273\.764\.001\.264\-8K404478745\.233\.174\.273\.764\.001\.178K\+8027211115\.283\.184\.273\.764\.001\.22Mistral\-7b\-Instruct\-v0\.3NarrativeQA8K\+9570814965\.313\.195\.024\.434\.001\.00GovReport0\-4K220439885\.163\.15\-\-4\.003\.214\-8K414079895\.323\.17\-\-4\.005\.868K\+8071583345\.303\.18\-\-4\.0014\.20Qasper0\-4K209139865\.173\.154\.273\.764\.001\.094\-8K400278415\.243\.174\.273\.764\.001\.078K\+8024241215\.283\.184\.273\.764\.001\.13Table 4\.Compression rates of different KV caching methods across 3 datasets \(NarrativeQA, GovReport, Qasper\) and varying context length ranges for Llama\-3\.1\-8B\-Instruct and Mistral\-7B\-Instruct\-v0\.3\. We report compression ratios alongside minimum and maximum token lengths for each method, illustrating how compression efficiency varies with sequence length and task\.
Note:We do not present results for Gov\_Report using TurboQuant because it takes upto 3 days for one single run\.#### Time to first token

Table[3](https://arxiv.org/html/2607.05399#S4.T3)shows TTFT over different context length buckets for all KV cache compression methods\. In terms of TTFT, we observe only moderate differences across KV compression techniques, which are largely consistent across both models, as illustrated in Fig\.[2](https://arxiv.org/html/2607.05399#S4.F2)\. Overall, most methods remain close to the All\-KV baseline, confirming that KV cache optimizations have limited impact on prefill latency\. KIVI and SnapKV show slightly increased TTFT compared to All\-KV, reflecting the additional overhead introduced by quantization and attention\-based token selection during the prefill phase\. CaM performs similarly, with only minor deviations, as its merging operations are applied periodically and do not heavily impact the initial prefill stage\.

In contrast, TurboQuant exhibits the highest TTFT overhead among all methods, particularly for longer contexts such as NarrativeQA and Qasper\. This is expected, as TurboQuant applies computationally expensive transformations, including random rotations and residual corrections, which increase the cost of KV cache construction before the first token is generated\. Additionally, we observe that TTFT increases with context length across all methods, highlighting that prompt processing remains the dominant factor in prefill latency\.

Overall, these results indicate that KV cache compression techniques introduce only limited additional latency for the first token, with the exception of TurboQuant, whose more complex quantization pipeline leads to noticeable overhead\. This suggests that most compression methods can be safely applied in latency\-sensitive applications without significantly affecting perceived responsiveness, aligning with the observation that TTFT is primarily dominated by prompt encoding rather than KV cache operations\.

#### Throughput

In terms of throughput, we see large differences between the compression techniques that are consistent across both models as can be seen in Fig\.[3](https://arxiv.org/html/2607.05399#S4.F3)\. SnapKV achieves the best throughput, on par with All\-KV on Llama and even surpassing All\-KV on Mistral\. KIVI and CaM are in the middle, suffering of only modest throughput degradation\. The most overhead is introduced in TurboQuant, leading to severe throughput drop\.

These results are expected, as SnapKV comes with very low overhead and effectively reduces the*number*of tokens in the KV cache instead of just compressing their representation\. CaM also reduces the number of tokens, but induces higher overheads as it redistributes the contributions of evicted tokens\. The quantization techniques KIVI and TurboQuant do not reduce the number of tokens in the KV cache, but only compress them\. Here, the compression techniques in TurboQuant are more aggressive, leading to higher overheads that have significant impact on throughput\.

![Refer to caption](https://arxiv.org/html/2607.05399v1/figures/speedup_over_allkv.png)Figure 2\.Relative time for TTFT \(normalized to All KV\) of KV compression methods across models and tasks\.
Note:We do not present results for Gov\_Report using TurboQuant because it takes upto 3 days for one single run\.
#### Compression rate on KV memory

Table[4](https://arxiv.org/html/2607.05399#S4.T4)shows the compression rates of the various KV cache optimizations, broken down across workloads and context\-lengths within each workload\. This allows for a detailed look into the*effectiveness*of the techniques in achieving their primary goal: reducing the size of the KV cache in GPU memory\. We make two major observations, one on stability of compression rates and one on the maximally achievable compression\.

Stability:KIVI and SnapKV achieve the most stable and predictable compression rates\. There are no major fluctuations across the different workloads and context lengths\. TurboQuant is also relatively stable, but shows an outlier with the NarrativeQA workload where the compression rate is slightly higher This can be attributed to its rotation\-based quantization scheme, which depends on the statistical distribution of KV vectors; longer and more diverse contexts, such as in NarrativeQA, tend to produce more uniformly distributed representations, enabling slightly more effective compression\(Donget al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib68)\)\. For CaM, we see a completely different picture: Compression lies between extreme compression of 14\.2 times \(GovReport on 8K\+ context length\) andnocompression at all \(compression rate of 1\.0 for NarrativeQA\)\. This variability stems from CaM’s adaptive, attention\-driven merging strategy, which dynamically decides whether tokens should be merged or retained based on their estimated importance\. In workloads like GovReport, where redundancy is higher and many tokens can be safely merged, CaM achieves very high compression\. However, in tasks such as NarrativeQA, where a larger portion of the context remains relevant for downstream generation, fewer tokens qualify for merging, resulting in minimal compression\. From a data management perspective, such unpredcitability of compression effectiveness may make it challenge to operate CaM in real\-world deployments\.

![Refer to caption](https://arxiv.org/html/2607.05399v1/figures/accuracy_avg_throughput_dual_axis_llama_mistral.png)Figure 3\.Average accuracy and throughput across different KV compression methodsEffectiveness:Among the quantization\-based compression techniques, KIVI2 shows the highest effectiveness with an average compression rate between 5\.16 and 5\.32\. The compression rates of TurboQuant3 and Turboquant4 are between those of KIVI2 and KIVI4\. SnapKV is also exactly in the middle between both KIVI settings\. This shows that, by using the quantization width as a parameter, it is possible to carefully tune the approaches to achieve a desired compression rate\. This is good news for practical data management, providing an effective tuning knob that can be adapted to given hardware constraints and workload characteristics\. If extreme compression is needed under certain long\-context workloads, CaM can be a good choice, as it achieved the highest compression rates under the long\-context GovReports tasks \(up to 14\.2\)\.

## 5\.Lessons Learned

We present lessons learned and relate them to conventional data management wisdom, comparing our lessons to insights from recent data management papers on related topics where appropriate\.

KV cache compression can preserve high accuracy\.All\-KV was not the consistent winner in terms of accuracy across the various workloads in our benchmark\. Instead, under most workloads, one of the compression techniques achieved even better results\. This shows that it is not important to retain*all*information in the KV cache, but to identify the*task\-relevant*information\. Summarizing across all workloads, we show that the decision to use a KV cache compression technique doesnotnecessarily mean trading accuracy for memory efficiency\. This is different from conventional data management wisdom about lossy compression, where information loss is considered an undesirable but necessary cost to achieve a high compression rate\(Jianget al\.,[2018](https://arxiv.org/html/2607.05399#bib.bib1); Behmeet al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib11)\)\.

Compression overhead can be amortized by reducing KV cache size\.In terms of throughput, we observe that some compression techniques indeed cause a certain throughput degradation, but not all\. By reducing the number of tokens in the KV cache, SnapKV yielded even higher throughput than the All\-KV baseline\. Again, this is different from conventional data management wisdom, where compression and decompression overheads are considered an additional cost that one has to pay to achieve a reduction in data size\. One could compare our results to Isenko et al\.\(Isenkoet al\.,[2022](https://arxiv.org/html/2607.05399#bib.bib12)\), who showed that data compression in machine learning preprocessing pipelines can in some cases be amortized by overcoming communication bottlenecks\.

Stable compression rates are preferable in real\-world deployments\.While CaM achieves extremely high compression rates on some long\-context workloads, it fails to be effective on shorter context workloads\. This discrepancy has consequences for practicability\. It may be extremely hard to predict under which circumstances an LLM may be used, especially if it accepts prompts from users\. Indeed, it is one of the strengths of LLMs that they are general\-purpose across a variety of different tasks\. In such settings, from a data systems management perspective, having more predictable performance is key\. Hence, we recommend using quantization or pruning methods except for scenarios with repeated and uniform tasks, such as using LLM\-based summarization for similar kinds of documents\.

Latency in terms of time\-to\-first\-token is merely affected by KV cache optimizations\.While there are overheads involved in decompressing KV cache \(especially under quantization\), these do not significantly affect the perceived latency for the user\. This is because KV\-cache overheads are insignificant when compared to decoding and detokenization phases of LLM inference\. As a result, KV cache optimizations can safely be applied in latency\-sensitive settings such as user\-facing interactions\.

KV cache optimization trends are similar across LLM models\.We find that the relative performance of the various KV cache compression methods is consistent for both Llama\-3\.1\-8B\-Instruct and Mistral\-7B\-Instruct\-v0\.3\. In particular, KIVI provides consistent accuracy, SnapKV consistently improves throughput, and TurboQuant incurs additional overheads across both models\. While absolute performance scores vary due to architectural differences such as sliding\-window attention in Mistral, the qualitative trends are consistent\. From a data management perspective, this implies that system\-level insights from KV cache optimization are transferrable across model families, enabling generalizable optimization strategies without the need for model\-specific tuning\.

Accuracy stability is different in different workloads\.We observe that the impact of KV cache compression on accuracy strongly depends on the task type\. Tasks such as few\-shot learning are relatively robust to compression, as they rely primarily on recent tokens and local patterns, whereas summarization is highly sensitive to any form of KV modification due to its reliance on global context\. Multi\-document QA lies in between, benefiting from selective pruning or merging that preserves important information while removing redundancy\. From a data management perspective, this implies that workload characteristics such as context redundancy, dependency structure, and sensitivity to global information determine the effectiveness of compression\. Consequently, KV cache optimization should be treated as a workload\-dependent decision rather than a uniform system configuration\.

Quantization provides robustness across unknown workloads, but not all methods behave equally\.We observe that KIVI is largely insensitive to workload type, maintaining stable accuracy across QA, few\-shot, and summarization tasks\. This robustness is due to its asymmetric, data\-aware design that takes into account explicitly different outlier structures in keys and values and preserves recent tokens in higher precision with a residual cache\. Thus, KIVI preserves both local and global context information even under aggressive compression, making it a reliable default choice when workload characteristics are unknown\.

In contrast, TurboQuant is not as robust, despite also being a quantization\-based method\. Its data\-oblivious design relying on random rotations and uniform scalar quantization, leads to small but systematic reconstruction errors on all tokens\. While these may be acceptable for local or pattern\-based tasks, they accumulate in workloads requiring an understanding of global context, such as summarization, resulting in significant quality degradation\. Moreover, TurboQuant introduces higher computational overhead due to its transformation and residual correction steps, which adversely affect system performance\. From a data management perspective, this implies that not only the compression paradigm, but also the underlying design principles \(data\-aware vs\. data\-oblivious\) dictate robustness and suitability for deployment\.

## 6\.Related Work

Recent work has increasingly framed KV cache optimization as a system\-level problem in LLM inference, closely aligned with data management concerns such as memory efficiency, scheduling, and resource utilization\. Surveys such as Towards Efficient Large Language Model Serving\(Jianget al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib69)\)emphasize that KV cache optimization spans multiple system dimensions, including execution scheduling, memory placement, and representation design, highlighting that KV cache is a central bottleneck in modern LLM serving systems \. Similarly, Li et al\.\(LIet al\.,[2025](https://arxiv.org/html/2607.05399#bib.bib70)\)categorize KV cache management techniques into token\-level, model\-level, and system\-level optimizations, reflecting the need for holistic approaches that jointly consider computation and memory trade\-offs \. Xu et al\.\(Liuet al\.,[2026](https://arxiv.org/html/2607.05399#bib.bib55)\)further show that no single optimization strategy dominates across workloads, and that adaptive, workload\-dependent KV cache management is essential for scalable inference \.

From a data management perspective, prior work on ML pipelines has demonstrated that performance bottlenecks are often driven by memory movement and data representation rather than pure computation, and that compression can improve system efficiency despite introducing approximation errors\(Jianget al\.,[2018](https://arxiv.org/html/2607.05399#bib.bib1); Isenkoet al\.,[2022](https://arxiv.org/html/2607.05399#bib.bib12); Behmeet al\.,[2023](https://arxiv.org/html/2607.05399#bib.bib11)\)\. However, these studies typically assume a fixed trade\-off between compression and accuracy, and evaluate optimizations at isolated stages of the pipeline\. Earlier works such as CLA\(Elgoharyet al\.,[2016](https://arxiv.org/html/2607.05399#bib.bib2)\)considered lossless compression only, which is hard to achieve in KV cache as there is little redundancy in KV cache tensors that could be exploited\.

In contrast, our work positions KV cache compression as a core data management problem within LLM inference pipelines, and provides a unified evaluation across accuracy, latency, throughput, and memory under realistic long\-context workloads\. Unlike prior survey and systems work, which primarily categorize techniques or analyze them in isolation, we empirically show that the effectiveness of KV cache optimization is highly workload\-dependent and that compression does not necessarily imply a loss in task quality\. This extends existing data management insights by demonstrating that workload\-aware KV cache strategies can simultaneously improve both system efficiency and model quality, challenging traditional assumptions about lossy compression in data systems\.

## 7\.Conclusions

LLM serving pipelines receive growing attention in the data management community, as they involve difficult data management and systems challenges\. In this paper, we investigate an important component of LLM serving: the KV cache which stores previously computed attention keys and values so a transformer can reuse them during autoregressive decoding, avoiding recomputation and dramatically speeding up token generation\. KV cache optimizations like quantization, pruning, and merging reduce the memory footprint and bandwidth of stored attention keys and values with the goal of enabling faster and more scalable decoding with minimal impact on output quality\.

Our benchmarking study reveals various trade\-offs in using the existing methods, and provides surprising but practical insights that in parts defy common data management wisdom about compression\. Counter\-intuitive findings such as the fact that lossy compression could*improve*accuracy in a data system challenge the way we may think about data management for LLMs\. At the same time, we found that not all compression techniques are easily deployable in general\-purpose LLM serving stacks, such as CaM that could yield by far the best compression rate, or no compression at all, depending on the data set and task\. This shows significant challenges when setting up and optimizing an LLM inference system\. The data management community with its decade\-long experience in query optimization may play a big role in workload\-aware optimization of future LLM serving stacks\. Our benchmarking study may be a starting point of such explorations\.

## Acknowledgments

We thank the Kuenneth Research Group at the University of Bayreuth for providing access to an NVIDIA A100 40GB GPU, which enabled us to run our benchmarks\.

## References

- J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebron, and S\. Sanghai \(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://openreview.net/forum?id=hmOwOZWzYE)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p4.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px5.p2.1)\.
- S\. Barbon Junior, P\. Ceravolo, S\. Groppe, M\. Jarrar, S\. Maghool, F\. Sèdes, S\. Sahri, and M\. Van Keulen \(2024\)Are large language models the new interface for data pipelines?\.InProceedings of the International Workshop on Big Data in Emergent Distributed Environments,BiDEDE ’24,New York, NY, USA\.External Links:ISBN 9798400706790,[Link](https://doi.org/10.1145/3663741.3664785),[Document](https://dx.doi.org/10.1145/3663741.3664785)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p1.1)\.
- L\. Behme, S\. Thirumuruganathan, A\. R\. Mahdiraji, J\. Quiané\-Ruiz, and V\. Markl \(2023\)The art of losing to win: using lossy image compression to improve data loading in deep learning pipelines\.In2023 IEEE 39th International Conference on Data Engineering \(ICDE\),Vol\.,pp\. 936–949\.External Links:[Document](https://dx.doi.org/10.1109/ICDE55515.2023.00077)Cited by:[§5](https://arxiv.org/html/2607.05399#S5.p2.1),[§6](https://arxiv.org/html/2607.05399#S6.p2.1)\.
- D\. Bolya, C\. Fu, X\. Dai, P\. Zhang, C\. Feichtenhofer, and J\. Hoffman \(2023\)Token merging: your ViT but faster\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px3.p1.1)\.
- C\. Chang, W\. Lin, C\. Lin, C\. Chen, Y\. Hu, P\. Wang, N\. Huang, L\. Ceze, M\. S\. Abdelfattah, and K\. Wu \(2025\)Palu: KV\-cache compression with low\-rank projection\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=LWMS4pk2vK)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p2.1)\.
- W\. Cheng, S\. Dong, J\. Qin, and W\. Wang \(2025\)QAQ: quality adaptive quantization for llm kv cache\.In2025 IEEE/CVF International Conference on Computer Vision Workshops \(ICCVW\),Vol\.,pp\. 2563–2571\.External Links:[Document](https://dx.doi.org/10.1109/ICCVW69036.2025.00267)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p2.1)\.
- K\. T\. Chitty\-Venkata, J\. Ye, S\. Raskar, A\. Kougkas, X\. Sun, M\. Emani, V\. Vishwanath, and B\. Nicolae \(2026\)PagedEviction: structured block\-wise KV cache pruning for efficient large language model inference\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 3207–3218\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.168/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.168),ISBN 979\-8\-89176\-386\-9Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px2.p2.1)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=mZn2Xyh9Ec)Cited by:[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p1.1)\.
- A\. Devoto, M\. Jeblick, and S\. Jégou \(2026\)Expected attention: KV cache compression by estimating attention from future queries distribution\.External Links:[Link](https://openreview.net/forum?id=VmojW15eRc)Cited by:[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p4.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p5.1)\.
- A\. Devoto, Y\. Zhao, S\. Scardapane, and P\. Minervini \(2024\)A simple and effectiveL​\_​2L\\\_2norm\-based strategy for KV cache compression\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 18476–18499\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1027/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1027)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px2.p1.1)\.
- Z\. Dong, J\. Li, J\. Jiang, M\. Xu, W\. X\. Zhao, B\. Wang, and W\. Chen \(2025\)LongReD: mitigating short\-text degradation of long\-context large language models via restoration distillation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 10687–10707\.External Links:[Link](https://aclanthology.org/2025.acl-long.524/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.524),ISBN 979\-8\-89176\-251\-0Cited by:[§4\.2](https://arxiv.org/html/2607.05399#S4.SS2.SSS0.Px3.p2.1)\.
- H\. Duan, J\. Wei, C\. Wang, H\. Liu, Y\. Fang, S\. Zhang, D\. Lin, and K\. Chen \(2024\)BotChat: evaluating LLMs’ capabilities of having multi\-turn dialogues\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 3184–3200\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.201/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.201)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p1.1)\.
- A\. Elgohary, M\. Boehm, P\. J\. Haas, F\. R\. Reiss, and B\. Reinwald \(2016\)Compressed linear algebra for large\-scale machine learning\.Proc\. VLDB Endow\.9\(12\),pp\. 960–971\.External Links:ISSN 2150\-8097,[Link](https://doi.org/10.14778/2994509.2994515),[Document](https://dx.doi.org/10.14778/2994509.2994515)Cited by:[§6](https://arxiv.org/html/2607.05399#S6.p2.1)\.
- S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. Gao \(2023\)Model tells you what to discard: adaptive kv cache compression for llms\.arXiv preprint arXiv:2310\.01801\.Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px3.p1.1),[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px3.p2.1)\.
- A\. Grattafiori and A\. D\. et al\. \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px3.p1.1)\.
- Y\. He, L\. Zhang, W\. Wu, J\. Liu, H\. Zhou, and B\. Zhuang \(2024\)ZipCache: accurate and efficient kv cache quantization with salient token identification\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p1.1)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)KVQuant: towards 10 million context length llm inference with kv cache quantization\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p3.1)\.
- O\. Hory \(2026\)TurboQuant: open\-source implementation of google’s turboquant\.Note:[https://github\.com/OmarHory/turboquant](https://github.com/OmarHory/turboquant)Accessed: 2026\-04\-29Cited by:[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p3.1)\.
- A\. Isenko, R\. Mayer, J\. Jedele, and H\. Jacobsen \(2022\)Where is my training bottleneck? hidden trade\-offs in deep learning preprocessing pipelines\.InProceedings of the 2022 International Conference on Management of Data,SIGMOD ’22,New York, NY, USA,pp\. 1825–1839\.External Links:ISBN 9781450392495,[Link](https://doi.org/10.1145/3514221.3517848),[Document](https://dx.doi.org/10.1145/3514221.3517848)Cited by:[§5](https://arxiv.org/html/2607.05399#S5.p3.1),[§6](https://arxiv.org/html/2607.05399#S6.p2.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p4.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px3.p1.1)\.
- J\. Jiang, P\. Yang, R\. Zhang, and F\. Liu \(2026\)Towards efficient large language model serving: a survey on system\-aware kv cache optimization\.TechRxiv\.External Links:[Document](https://dx.doi.org/10.36227/techrxiv.176046306.66521015/v3)Cited by:[§6](https://arxiv.org/html/2607.05399#S6.p1.1)\.
- J\. Jiang, F\. Fu, T\. Yang, and B\. Cui \(2018\)SketchML: accelerating distributed machine learning with data sketches\.InProceedings of the 2018 International Conference on Management of Data,SIGMOD ’18,New York, NY, USA,pp\. 1269–1284\.External Links:ISBN 9781450347037,[Link](https://doi.org/10.1145/3183713.3196894),[Document](https://dx.doi.org/10.1145/3183713.3196894)Cited by:[§5](https://arxiv.org/html/2607.05399#S5.p2.1),[§6](https://arxiv.org/html/2607.05399#S6.p2.1)\.
- J\. Kim, J\. Yeom, S\. Yun, and H\. O\. Song \(2024\)Compressed context memory for online language model interaction\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=64kSvC4iPg)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px3.p2.1)\.
- G\. Li, X\. Zhou, and X\. Zhao \(2024a\)LLM for data management\.Proc\. VLDB Endow\.17\(12\),pp\. 4213–4216\.External Links:ISSN 2150\-8097,[Link](https://doi.org/10.14778/3685800.3685838),[Document](https://dx.doi.org/10.14778/3685800.3685838)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p2.1)\.
- H\. LI, Y\. Li, A\. Tian, T\. Tang, Z\. Xu, X\. Chen, N\. HU, W\. Dong, L\. Qing, and L\. Chen \(2025\)A survey on large language model acceleration based on KV cache management\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=z3JZzu9EA3)Cited by:[§6](https://arxiv.org/html/2607.05399#S6.p1.1)\.
- J\. Li, Y\. Zhang, M\. Y\. Hassan, T\. Chafekar, T\. Cai, Z\. Ren, P\. Guo, B\. Karimzadeh, C\. J\. Reed, C\. Wang, and C\. Gan \(2025a\)CommVQ: commutative vector quantization for kv cache compression\.InProceedings of the 42nd International Conference on Machine Learning,ICML’25\.Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p2.1)\.
- Y\. Li, R\. Gu, C\. Huan, Z\. Wang, R\. Yao, C\. Tian, and G\. Chen \(2025b\)HotPrefix: hotness\-aware kv cache scheduling for efficient prefix sharing in llm inference systems\.Proc\. ACM Manag\. Data3\(4\)\.External Links:[Link](https://doi.org/10.1145/3749168),[Document](https://dx.doi.org/10.1145/3749168)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.05399#S2.SS1.SSS0.Px2.p4.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024b\)SnapKV: llm knows what you are looking for before generation\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px2.p1.1),[§2\.3](https://arxiv.org/html/2607.05399#S2.SS3.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px2.p2.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p4.1)\.
- A\. Liu, J\. Liu, Z\. Pan, Y\. He, G\. Haffari, and B\. Zhuang \(2024a\)MiniCache: kv cache compression in depth dimension for large language models\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p1.1)\.
- Y\. Liu, J\. Fu, S\. Liu, Y\. Zou, S\. Zhang, and J\. Zhou \(2026\)KV cache compression for inference efficiency in llms: a review\.InProceedings of the 4th International Conference on Artificial Intelligence and Intelligent Information Processing,AIIIP ’25,New York, NY, USA,pp\. 207–212\.External Links:ISBN 9798400719356,[Link](https://doi.org/10.1145/3778534.3778567),[Document](https://dx.doi.org/10.1145/3778534.3778567)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p1.1),[§1](https://arxiv.org/html/2607.05399#S1.p3.1),[§6](https://arxiv.org/html/2607.05399#S6.p1.1)\.
- Y\. Liu, H\. Li, Y\. Cheng, S\. Ray, Y\. Huang, Q\. Zhang, K\. Du, J\. Yao, S\. Lu, G\. Ananthanarayanan,et al\.\(2024b\)Cachegen: kv cache compression and streaming for fast large language model serving\.InProceedings of the ACM SIGCOMM 2024 Conference,pp\. 38–56\.Cited by:[§2\.1](https://arxiv.org/html/2607.05399#S2.SS1.SSS0.Px2.p3.3)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. \(\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024c\)KIVI: a tuning\-free asymmetric 2bit quantization for kv cache\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§2\.1](https://arxiv.org/html/2607.05399#S2.SS1.SSS0.Px1.p1.7),[§2\.1](https://arxiv.org/html/2607.05399#S2.SS1.SSS0.Px2.p2.1),[§2\.3](https://arxiv.org/html/2607.05399#S2.SS3.SSS0.Px1.p1.3),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px2.p2.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p2.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p6.1)\.
- T\. Munkhdalai, M\. Faruqui, and S\. Gopal \(2024\)Leave no context behind: efficient infinite context transformers with infini\-attention\.External Links:2404\.07143,[Link](https://arxiv.org/abs/2404.07143)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px3.p2.1)\.
- J\. Pan and G\. Li \(2025\)Database perspective on llm inference systems\.Proc\. VLDB Endow\.18\(12\),pp\. 5504–5507\.External Links:ISSN 2150\-8097,[Link](https://doi.org/10.14778/3750601.3750703),[Document](https://dx.doi.org/10.14778/3750601.3750703)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p2.1)\.
- N\. Shazeer \(2019\)Fast transformer decoding: one write\-head is all you need\.External Links:1911\.02150,[Link](https://arxiv.org/abs/1911.02150)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p1.1)\.
- W\. Song, S\. Oh, S\. Mo, J\. Kim, S\. Yun, J\. Ha, and J\. Shin \(2024\)Hierarchical context merging: better long context understanding for pre\-trained llms\.ICLR\.Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px3.p1.1)\.
- Y\. Sun, L\. Dong, Y\. Zhu, S\. Huang, W\. Wang, S\. Ma, Q\. Zhang, J\. Wang, and F\. Wei \(2024\)You only cache once: decoder\-decoder architectures for language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=25Ioxw576r)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p1.1)\.
- G\. Wang, S\. Upasani, C\. Wu, D\. Gandhi, J\. L\. Li, C\. Hu, B\. Li, and U\. Thakker \(2025a\)LLMs know what to drop: self\-attention guided KV cache eviction for efficient long\-context inference\.InSparsity in LLMs \(SLLM\): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference,External Links:[Link](https://openreview.net/forum?id=qg9dlCcNzr)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px2.p1.1)\.
- J\. Wang, J\. Han, X\. Wei, S\. Shen, D\. Zhang, C\. Fang, R\. Chen, W\. Yu, and H\. Chen \(2025b\)KVCache cache in the wild: characterizing and optimizing kvcache cache at a large cloud provider\.InProceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference,USENIX ATC ’25,USA\.External Links:ISBN 978\-1\-939133\-48\-9Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.05399#S2.SS1.p1.1)\.
- S\. Wu, A\. Lv, xiao feng, Y\. zhang, X\. Zhang, G\. Yin, W\. Lin, and R\. Yan \(2026\)PolarQuant: leveraging polar transformation for key cache quantization and decoding acceleration\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=JCTTLKEBza)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p2.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px2.p1.1)\.
- Y\. Yang, zouying cao, Q\. Chen, L\. Qin, D\. Yang, Z\. Chen, and hai zhao \(2024\)KVSharer: efficient inference via layer\-wise dissimilar KV cache sharing\.External Links:[Link](https://openreview.net/forum?id=2Akf4BBCKo)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p1.1)\.
- H\. Yuan, X\. Ai, Q\. Wang, P\. Li, J\. Yu, C\. Chen, X\. Yang, Y\. Zhang, Z\. Fu, Y\. Wen, and G\. Yu \(2025\)DepCache: a kv cache management framework for graphrag with dependency attention\.Proc\. ACM Manag\. Data3\(6\)\.External Links:[Link](https://doi.org/10.1145/3769778),[Document](https://dx.doi.org/10.1145/3769778)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p2.1)\.
- J\. Yuan, H\. Liu, S\. Zhong, Y\. Chuang, S\. Li, G\. Wang, D\. Le, H\. Jin, V\. Chaudhary, Z\. Xu, Z\. Liu, and X\. Hu \(2024\)KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4623–4648\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.266/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.266)Cited by:[§1](https://arxiv.org/html/2607.05399#S1.p3.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px1.p1.1)\.
- A\. Zandieh, M\. Daliri, M\. Hadian, and V\. Mirrokni \(2026\)TurboQuant: online vector quantization with near\-optimal distortion rate\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tO3ASKZlok)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p2.1),[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px1.p3.1),[§2\.3](https://arxiv.org/html/2607.05399#S2.SS3.SSS0.Px2.p1.4),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px2.p2.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p3.1)\.
- R\. Zhang, K\. Wang, L\. Liu, S\. Wang, H\. Cheng, C\. Zhang, and yelong shen \(2025\)LoRC: low\-rank compression for LLMs KV cache with a progressive compression strategy\.External Links:[Link](https://openreview.net/forum?id=NI8AUSAc4i)Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p2.1)\.
- Y\. Zhang, Y\. Du, G\. Luo, Y\. Zhong, Z\. Zhang, S\. Liu, and R\. Ji \(2024\)CaM: cache merging for memory\-efficient LLMs inference\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 58840–58850\.External Links:[Link](https://proceedings.mlr.press/v235/zhang24n.html)Cited by:[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px2.p2.1),[§3](https://arxiv.org/html/2607.05399#S3.SS0.SSS0.Px4.p5.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px2.p1.1)\.
- Z\. M\. K\. Zuhri, M\. F\. Adilazuarda, A\. Purwarianti, and A\. F\. Aji \(2025\)MLKV: multi\-layer key\-value heads for memory efficient transformer decoding\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 5531–5540\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.305/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.305),ISBN 979\-8\-89176\-195\-7Cited by:[§2\.2](https://arxiv.org/html/2607.05399#S2.SS2.SSS0.Px4.p1.1)\.