Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

# Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
Source: [https://arxiv.org/html/2607.08057](https://arxiv.org/html/2607.08057)
Jiantong Jiang1,Peiyu Yang1,Rui Zhang2,Feng Liu1 1School of Computing and Information Systems, The University of Melbourne, 2School of Computer Science and Technology, Huazhong University of Science and Technology \(www\.ruizhang\.info\) \{jiantong\.jiang, peiyu\.yang\}@unimelb\.edu\.au rayteam@yeah\.net, fengliu\.ml@gmail\.com [![[Uncaptioned image]](https://arxiv.org/html/2607.08057v1/figs/github-mark.png)https://github\.com/jjiantong/Awesome\-KV\-Cache\-Optimization](https://github.com/jjiantong/Awesome-KV-Cache-Optimization)

###### Abstract

Despite the rapid advancements of large language models \(LLMs\), LLM serving systems remain memory\-intensive and costly\. The key\-value \(KV\) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low\-latency, high\-throughput LLM inference serving\. In this survey, we focus onsystem\-awareKVinfrastructure forserving LLMs \(abbreviated assKis\)\. We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling \(temporal\), placement and migration \(spatial\), and representation and retention \(structural\)\. Furthermore, we analyzecross\-behavior co\-design affinity and behavior\-objective links, highlighting future opportunities\. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure\.

Towards Efficient Large Language Model Serving: A Survey on System\-Aware KV Cache Optimization

Jiantong Jiang1, Peiyu Yang1††thanks:Peiyu Yang is the corresponding author\., Rui Zhang2, Feng Liu11School of Computing and Information Systems, The University of Melbourne,2School of Computer Science and Technology,Huazhong University of Science and Technology \(www\.ruizhang\.info\)\{jiantong\.jiang, peiyu\.yang\}@unimelb\.edu\.aurayteam@yeah\.net, fengliu\.ml@gmail\.com[![[Uncaptioned image]](https://arxiv.org/html/2607.08057v1/figs/github-mark.png)https://github\.com/jjiantong/Awesome\-KV\-Cache\-Optimization](https://github.com/jjiantong/Awesome-KV-Cache-Optimization)

## 1Introduction

Large language models \(LLMs\) have showcased exceptional abilities across diverse applicationsZhaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib2)\), with notable examples like GPTRadfordet al\.\([2018](https://arxiv.org/html/2607.08057#bib.bib16),[2019](https://arxiv.org/html/2607.08057#bib.bib17)\); Brownet al\.\([2020](https://arxiv.org/html/2607.08057#bib.bib18)\); OpenAI \([2023](https://arxiv.org/html/2607.08057#bib.bib19)\), LLaMATouvronet al\.\([2023a](https://arxiv.org/html/2607.08057#bib.bib20),[b](https://arxiv.org/html/2607.08057#bib.bib21)\), and OPTZhanget al\.\([2022](https://arxiv.org/html/2607.08057#bib.bib22)\)\. These models excel at large\-scale high\-quality language understanding and generation, and their extensions also support plenty of multi\-modal generation tasksZhanget al\.\([2025d](https://arxiv.org/html/2607.08057#bib.bib193)\); Zhang and Bian \([2025](https://arxiv.org/html/2607.08057#bib.bib194)\)\. These abilities are largely powered by the Transformer architectureVaswaniet al\.\([2017](https://arxiv.org/html/2607.08057#bib.bib15)\), which efficiently captures long\-range dependencies via self\-attention\.

Despite their success, serving LLMs efficiently remains non\-trivialLiet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib5)\)\. Transformer\-based LLMs generate tokens autoregressively, with each token conditioned on all previous ones\. To avoid redundant compute, serving systems adopt akey\-value \(KV\) cachePopeet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib1)\)to store intermediate KV tensors of the generated tokens\. Yet, as prompt and output length grow, the KV cache can reach millions of tokensDinget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib25)\), creating memory bottlenecks and highlighting the critical role of KV cache optimization\. Thus, a growing body of KV\-centric techniques has emerged, yielding memory savings and efficiency gains in throughput and latencyLiet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib13)\)\.

![Refer to caption](https://arxiv.org/html/2607.08057v1/x1.png)Figure 1:Positioning of the survey scope \(“sKis”\)\.To this end, we argue that it deserves a deep investigation of system\-aware, serving\-time, KV\-centric optimization methods, as shown in Fig\.[1](https://arxiv.org/html/2607.08057#S1.F1), which we call this scopesKis\. We adopt a system\-oriented taxonomy to offer a comprehensive understanding of sKis, categorizing methods along three fundamental axes of system behaviors, as shown in Fig\.[2](https://arxiv.org/html/2607.08057#S1.F2): \(i\)execution and schedulingfocuses on thetemporalcontrol of when KV data is accessed, computed, or scheduled \(cf\. §[3](https://arxiv.org/html/2607.08057#S3)\); \(ii\)placement and migrationcaptures thespatialdecisions of where KV data is placed or moved across memory tiers or devices \(cf\. §[4](https://arxiv.org/html/2607.08057#S4)\); and \(iii\)representation and retentionconcerns thestructuraltreatment of how KV data is compressed or managed \(cf\. §[5](https://arxiv.org/html/2607.08057#S5)\)\. We further analyze cross\-behaviorco\-design patternsand behavior–objective effects to reveal overlooked regions and open challenges \(cf\. §[6](https://arxiv.org/html/2607.08057#S6)\)\.

![Refer to caption](https://arxiv.org/html/2607.08057v1/x2.png)Figure 2:Taxonomy of the survey that covers temporal, spatial, and structural dimensions\.Table 1:Comparison of our work with surveys related to efficient LLM inference or serving\.SurveyKV\-centricServingonlyNoretrainOrganizingprincipleMiaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib7)\)✓Algorithm\-systemYuanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib4)\)✓Optimization layerLiet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib5)\)✓✓System componentZhouet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib3)\)✓Optimization layerZhenet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib11)\)✓✓Serving scaleShiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib12)\)✓Lifecycle stageLiet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib13)\)✓✓Optimization layerLiuet al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib14)\)✓✓Compression typesThis survey \(sKis\)✓✓System behavior

While prior surveys span efficient LLM inference and servingZhouet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib3)\); Yuanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib4)\); Miaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib7)\); Liet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib5)\); Zhenet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib11)\), they are general surveys where the KV cache is discussed only as a minor component\. KV\-specific surveys are closest to our topicShiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib12)\); Liet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib13)\); Liuet al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib14)\), but they typically organize by lifecycle stages or optimization layers\.Instead, this survey focuses exclusively on sKis and distinguishes itself by offering a novel behavior\-oriented perspective and a deeper understanding\.We compare related surveys in Tab\.[1](https://arxiv.org/html/2607.08057#S1.T1)and provide further details in App\.[C](https://arxiv.org/html/2607.08057#A3)\.

To the best of our knowledge, we are the first to frame KV cache optimization as a temporal\-spatial\-structural behavior space, enabling principled analysis and actionable future directions\.Because this design space is decoupled from model and kernel details, it also offers a stable lens for situating new techniques in this rapidly evolving area\.

## 2Foundations, Scope and Taxonomy

![Refer to caption](https://arxiv.org/html/2607.08057v1/x3.png)Figure 3:Prefill and decode phases of LLM inference\.##### LLM Inference and KV Cache\.

LLMs generate tokens autoregressively, as shown in Fig\.[3](https://arxiv.org/html/2607.08057#S2.F3)\(see preliminaries on Transformer\-based LLMs in App\.[A](https://arxiv.org/html/2607.08057#A1)\)\. At each step, the model consumes the input and previously generated tokens to generate the next token\. This process has two phases:prefillprocesses the initial input and generates the first output token, anddecodegenerates tokens autoregressively\. Due to the quadratic cost of self\-attention, repeatedly computing attention across tokens is expensive\. To this end,key\-value \(KV\) cacheis used to store the intermediate KV tensors computed previously, allowing the model to efficiently reuse them without recomputing attention over the entire sequence\.

##### Scope\.

This survey investigates recent advances in the sKis scope, as shown in Fig\.[1](https://arxiv.org/html/2607.08057#S1.F1)\.

sKis denotessystem\-awareKVinfrastructure forserving LLMs\. A method belongs to sKis if it: \(i\) operates during serving \(inference\), \(ii\) centers on KV caches as the primary optimization target, and \(iii\) aims to improve system metrics without retraining the base LLM’s weights or modifying its Transformer architecture\.

We acknowledge related but out\-of\-scope directions that redesign the attention or KV mechanism\(DeepSeek\-AI,[2024](https://arxiv.org/html/2607.08057#bib.bib146); Yuanet al\.,[2025](https://arxiv.org/html/2607.08057#bib.bib147); Sunet al\.,[2026](https://arxiv.org/html/2607.08057#bib.bib149)\), or modify the backbone model structure\(Ashkbooset al\.,[2024](https://arxiv.org/html/2607.08057#bib.bib151); Liet al\.,[2024d](https://arxiv.org/html/2607.08057#bib.bib153); Luet al\.,[2026](https://arxiv.org/html/2607.08057#bib.bib152)\)\. Although these directions can reshape the KV footprint and improve efficiency, they do so by altering the model itself or requiring model adaptation\. In contrast, this survey focuses on serving\-time KV cache optimizations for off\-the\-shelf models, which remains a common deployment setting in the community\.

##### Taxonomy\.

We organize literature on sKis by low\-level system behaviors, as shown in Fig\.[2](https://arxiv.org/html/2607.08057#S1.F2)\. Further details are provided in App\.[B](https://arxiv.org/html/2607.08057#A2)\. However, similar to how a modern OS includes components for scheduling, memory, and I/O, LLM serving systems often involve techniques spanning various system dimensions\. Thus, a single paper may naturally touch on several categories\. For clarity and focus, we mention 1\-2 primary categories per work based on its main contributions\. Minor associations are not elaborated, and we refer to App\.[D](https://arxiv.org/html/2607.08057#A4)for details\. We summarize the organization and methods in Fig\.[4](https://arxiv.org/html/2607.08057#S2.F4)and the key takeaways in App\.[E](https://arxiv.org/html/2607.08057#A5)\.

\{forest\}

Figure 4:Taxonomy of sKis and associated methods\. Each method is annotated with its primary contributions for conciseness\. Minor category associations are omitted here and listed in App\.[D](https://arxiv.org/html/2607.08057#A4), Tab\.[9](https://arxiv.org/html/2607.08057#A4.T9)\.

## 3KV Execution and Scheduling

This section captures the temporal behaviors of KV cache usage, including how cache entries are scheduled and executed efficiently at runtime\.

### 3\.1KV\-centric Scheduling

While scheduling is a long\-studied system problem, KV\-centric scheduling \(KVS\) methods explicitly integrate KV characteristics into runtime decisions\.

At therequest level, some methods adopt KV usage\-aware scheduling to balance resource load and reduce contentionHuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib34)\); Duanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib52)\); Xionget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib49)\); Shahoutet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib51)\); Wuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib71)\)\. For example, TetriInferHuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib34)\)prioritizes requests using predicted KV usage to mitigate prefill\-decode interference\. Another line is reuse\-aware, prioritizing high\-reuse requests to maximize KV cache hit rateZhenget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib47)\)or using KV reuse potential as a key signal in decisionsSrivatsaet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib72)\); Qinet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib69)\)\.

At finer granularity,token\-levelmethods decide which KV entries participate in attention based on estimated contributionsTanget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib60)\); Ribaret al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib75)\); Singhaniaet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib63)\); Behnamet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib137)\); Xuet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib78)\); Wuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib53)\), for example via periodic refresh that alternates full\-context and subset attentionXuet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib78)\)\. At thekernel level, methods like FlashInferYeet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib46)\)schedule attention workloads across CUDA thread blocks based on query and KV lengths\.

Table 2:Summary of OVLP methods\.“Comp” denotes compute, “I/O” denotes KV data movement \(host–device transfer or on\-device memory movement\), and “comm” denotes collective communication\.ModeMethodOverlapped operations \(with transfer path\)GranularityComp–CompFastDecodeHe and Zhai \([2024](https://arxiv.org/html/2607.08057#bib.bib30)\)CPU R\-part comp↔\\leftrightarrowGPU S\-part compToken\-wiseNeoJianget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib67)\)CPU attention comp↔\\leftrightarrowGPU linear opsSub\-batch\-wiseComp–I/OCCompPark and Egger \([2024](https://arxiv.org/html/2607.08057#bib.bib38)\)CPU MHSA comp↔\\leftrightarrowFFN data transfer \(CPU→\\rightarrowGPU\)Split pointCachedAttentionGaoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib50)\)GPU comp↔\\leftrightarrowKV load/store \(CPU↔\\leftrightarrowGPU\)Layer\-wiseAsyncKVDonget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib66)\)GPU attention comp↔\\leftrightarrowGPU KV prefetch \(HBM→\\rightarrowL2\)KV block\-wiseKVPRJianget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib65)\)GPU KV re\-comp↔\\leftrightarrowKV transfer \(CPU↔\\leftrightarrowGPU\)Split pointCXL\-SpecKVLiu and Yu \([2026](https://arxiv.org/html/2607.08057#bib.bib83)\)GPU comp↔\\leftrightarrowGPU KV prefetch \(CXL memory→\\rightarrowL2\)KV page\-wiseI/O–CommPRESERVEYüzügüleret al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib64)\)GPU KV prefetch \(HBM→\\rightarrowL2\)↔\\leftrightarrowGPU collective commOperator\-wise

### 3\.2Pipelining and Overlapping

Pipelining and overlapping \(OVLP\) methods hide KV\-related latency by overlapping compute, I/O, and communication\. Though often embedded in broader systems, Tab\.[2](https://arxiv.org/html/2607.08057#S3.T2)highlights methods where OVLP forms the core technical contribution\.We summarize them by mode and list the corresponding overlapped operations and granularity\.OVLP is key to reducing idle time and improving efficiency\.

### 3\.3Hardware\-aware Execution

This section focuses on hardware\-aware execution \(HAE\) methods that adapt KV\-related operations to the underlying heterogeneous hardware\.

#### 3\.3\.1Disaggregated Inference

Disaggregated inference decouples inference compute onto distinct hardware resources to reduce contention and improve utilization\. Infinite\-LLMLinet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib35)\)adopts this idea at the operator level by splitting attention across distributed instances\. Several systems instead apply prefill\-decode \(PD\) disaggregation, assigning compute\-bound prefill and memory\-bound decode to different compute poolsHuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib34)\); Patelet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib31)\); Zhonget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib33)\); Stratiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib70)\)\. MooncakeQinet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib69)\)further couples PD disaggregation with a KV\-centric scheduler and distributed cache pool, while MuxServeDuanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib52)\)colocates PD jobs within each GPU through SM partitioning\.

#### 3\.3\.2Compute Offloading

Compute offloading relocates partial compute to auxiliary devices to reduce GPU bottlenecks, utilizing hardware heterogeneity and workload features\.

A practical instantiation isCPU offloading, which leverages host CPUs for memory\-intensive computeHe and Zhai \([2024](https://arxiv.org/html/2607.08057#bib.bib30)\); Park and Egger \([2024](https://arxiv.org/html/2607.08057#bib.bib38)\); Chenet al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib62)\); Jianget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib67)\)\. They often follow a compute\-near\-cache principle for better locality\. For example, FastDecodeHe and Zhai \([2024](https://arxiv.org/html/2607.08057#bib.bib30)\)and NeoJianget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib67)\)offload both attention and KV caches, using cost\-aware hardware selection and a load\-aware scheduler, respectively\.

Beyond CPUs, several methods offload compute toalternative devices, such as computational storage drive \(CSD\)Panet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib68)\)and processing\-in\-memory \(PIM\)Heet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib73)\); Parket al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib74)\)\. These methods expand the compute offloading space to broader device heterogeneity\.

Takeaways & Limitations – Temporal Behavior•KVS and OVLP directly target KV reuse and stall hiding\. Lightweight cost models or predictors often enhance them\. However, they are typically evaluated on controlled workloads, with limited analysis of robustness under bursty traffic or multi\-tenant settings\.•HAE improves throughput and hardware utilization by decoupling compute and specializing kernels, but its reliance on low\-level primitives can make portability non\-trivial for practitioners in some cases\.More analysis is provided in Apps\.[E\.1](https://arxiv.org/html/2607.08057#A5.SS1)and[E\.2](https://arxiv.org/html/2607.08057#A5.SS2)\.

## 4KV Placement and Migration

This section focuses on the spatial behaviors of how KV caches are placed and migrated across memory hierarchies and between compute devices\. Figure[5](https://arxiv.org/html/2607.08057#S4.F5)visualizes the architecture and transfer paths\.

### 4\.1Memory Hierarchy KV Orchestration

To scale under memory limits, we survey memory hierarchy KV orchestration \(MHO\) methods that distribute KV caches across memory hierarchies\.

##### Cross\-device Memory Hierarchy\.

A broad range of methods migrate KV entries across faster but limited GPU HBM, and larger but slower alternatives like CPU DRAM or SSD\. Most of these works areimportance\-aware, designing importance scoring policies that maintain only critical KV entries on the GPUZhaoet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib39)\); Leeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib58)\); Xiaoet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib59)\); Chenet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib184),[2025b](https://arxiv.org/html/2607.08057#bib.bib80)\); Yuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib55)\); Haoet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib81)\); Liuet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib61)\); Zhanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib100)\); Sunet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib109)\); Jieet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib182)\); Longet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib180)\); Liuet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib57)\)\. For instance, ArkValeChenet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib184)\), OmniKVHaoet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib81)\), ClusterKVLiuet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib61)\), PQCacheZhanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib100)\), and SpeCacheJieet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib182)\)offload the full KV cache to the CPU and keep only a lightweight proxy signal on the GPU\. They then estimate importance via the proxy signal to guide the next prefetch\.

Another line of cross\-device methods optimizes KV placement and migration from asystem costview\. FlexGenShenget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib27)\)places KV caches across GPU, CPU, and disk via a cost model that maximizes throughput under bandwidth and latency constraints\. Recent works further extend this hierarchy to compute express link \(CXL\)\-based shared or disaggregated memory pools, enabling low\-latency KV access and management\(Liu and Yu,[2026](https://arxiv.org/html/2607.08057#bib.bib83); Yanget al\.,[2025](https://arxiv.org/html/2607.08057#bib.bib82); Yoonet al\.,[2025](https://arxiv.org/html/2607.08057#bib.bib84)\)\. At runtime, many systems make online decisions about KV offloading or reloading based on system\-level signals, such as queueing state, memory pressure, compute and I/O costs, and future reuse signalsWuet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib32)\); Gaoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib50)\); Stratiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib70)\); Xionget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib49)\); Shenet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib48)\); Jinet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib76)\); Chenget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib183)\); Panet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib181)\)\.

Table 3:Summary of KV cache quantization \(q\.\) methods\. “Avg\. bits” shows the average bitwidth per KV elementbased on the reported main results\. This metric indicates memory savings and is comparable across methods\.MethodGranularityPrec\.modeImportantregionOutlierhandlingAvg\.bitsKeysValuesSmoothQuantXiaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib101)\)Channel\-wiseFixed–Smoothing via scaling8FlexGenShenget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib27)\)Group\-wiseFixed––4WKVQuantYueet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib86)\)2D \(channel & token\)MixedCurrent tokenDynamic token\-wise q\.~4MiKVYanget al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib87)\)Token\-wiseMixedExisting policyOutlier balancing~4QAQDonget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib88)\)Token\-wiseMixedAttention\-awareSparse matrix \(FP16\)1\.8\-2\.7AtomZhaoet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib93)\)Group\-wiseMixedOutlier channelsSelective high\-bits4\.25KIVILiuet al\.\([2024d](https://arxiv.org/html/2607.08057#bib.bib91)\)Channel\-wiseToken\-wiseMixedRecent tokensChannel\-wise confining~2CacheGenLiuet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib29)\)Layer\-wiseMixedShallow layers–1\.9\-2\.9DecoQuantLiuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib98)\)Decomposed\-tensor\-wiseMixedSmall tensorsTensor decomposition4GEARKanget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib89)\)Channel\-wiseToken\-wiseFixed–Sparse matrix \(FP16\)4\.4/5\.0SKVQDuanmuet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib94)\)Group\-wiseMixedInit\. & recent tokensClipped dynamic q\.2\.25KVQuantHooperet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib92)\)Channel\-wiseToken\-wiseMixedFirst tokenSparse matrix \(FP16\)4\.3CQZhanget al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib90)\)Token\-wise channel\-groupFixed––1\.3ZipCacheHeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib102)\)Channel\-wiseChan\.\-sep\. token\-wiseMixedNorm\. attentionChannel\-wise norm\.3\.2QJLZandiehet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib85)\)Token\-wiseFixed–Selective high\-bits3/5VQ\-LLMLiuet al\.\([2025d](https://arxiv.org/html/2607.08057#bib.bib96)\)Group\-wise \(configurable\)Fixed––2/4SQuatWanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib95)\)Block\-wiseToken\-wiseFixed––3\.1QoQLinet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib97)\)Channel\-wiseFixed–Smooth attention4CommVQLiet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib103)\)Token\-wise vectorFixed––2OTTSuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib104)\)Channel\-wiseToken\-wiseMixedOutlier & recent tokensFull precision2\.5NSNQuantSonet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib99)\)Token\-wise vectorFixed–Token\-wise norm\.1\.2/2\.2ChanMixLiao and Wen \([2026](https://arxiv.org/html/2607.08057#bib.bib105)\)Channel\-wiseToken\-wiseMixedRetrieval channelsChan\.\-wise bit reallocation3\.1

##### Intra\-GPU Memory Hierarchy\.

Another line of MHO methods migrates KV entries between on\-chip L1/L2 caches and off\-chip HBM\.Donget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib66)\)asynchronously prefetched upcoming KV blocks from HBM into L2 so that subsequent attention steps mostly hit in L2\. Similarly, PRESERVEYüzügüleret al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib64)\)fetches KV caches and inserts such operations selectively via graph\-level optimization to avoid cache pollution\.

![Refer to caption](https://arxiv.org/html/2607.08057v1/x4.png)Figure 5:Illustration of KV cache placement and migration across memory hierarchies and compute devices\.

### 4\.2Compute Device KV Orchestration

Unlike hierarchical memory, compute device KV orchestration \(CDO\) places and moves KV across compute\-capable devices to enable distributed or heterogeneous serving\. A common line performs intra\-cluster orchestration, typically coupled with PD disaggregationWuet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib32)\); Patelet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib31)\); Zhonget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib33)\); Linet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib35)\); Chenget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib183)\); Yoonet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib84)\)\. For instance, DistServeZhonget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib33)\)proposes placement schemes for prefill and decode across high and low node\-affinity GPU clusters, and uses a pull\-based scheme where decode GPUs fetch KV as needed from prefill GPUs\. TraCT\(Yoonet al\.,[2025](https://arxiv.org/html/2607.08057#bib.bib84)\)exploits CXL shared memory as a substrate for KV transfer across disaggregated PD workers\.

Beyond tightly coupled clusters, CacheGenLiuet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib29)\)targets remote KV transfer in network setups\. It reduces network delay by encoding KV tensors into bitstreams and adaptively streaming them based on runtime bandwidth\. Finally, CDO also extends to heterogeneous accelerators, offloading attention to devices such as PIMs and CSDsParket al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib74)\); Panet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib68)\)\.

Takeaways & Limitations – Spatial BehaviorMHO and CDO directly target interconnect bottlenecks through tiered KV management, and most systems overlap KV transfers with compute to hide latency, which is a central factor in their effectiveness\. However, bandwidth is typically handled without explicit modeling contention among concurrent KV transfers, making tail behavior hard to analyze\. Another gap is the explicit joint optimization of offload and prefetch under shared memory and interconnect budgets\. More takeaways are provided in App\.[E\.3](https://arxiv.org/html/2607.08057#A5.SS3)\.

## 5KV Representation and Retention

This section focuses on structural system behaviors of KV cache representation and retention\.

### 5\.1KV Cache Compression

KV cache compression \(KVCC\) is a central research thrust as it directly reduces memory usage\.

#### 5\.1\.1KV Cache Quantization

Quantization compresses floating\-point tensors into lower\-precision formats\. Early works enable 8\- and 4\-bit KVXiaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib101)\); Shenget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib27)\)\. Later schemes adopt mixed precision, assigning high precision to critical KV entries\. We compare methodsalong core algorithmic axes and effective bitwidthsin Tab\.[3](https://arxiv.org/html/2607.08057#S4.T3)and present key insights here\.

One recurring pattern isasymmetric KV quantization\(cf\. “granularity” in Tab\.[3](https://arxiv.org/html/2607.08057#S4.T3)\), as keys and values exhibit distinct outlier patterns and quantization sensitivities\. For example, a common practice is to quantize keys per\-channel and values per\-token\. A second insight is thatoutliersplay a crucial role in quantization, so many methods store them in higher bitwidths or design dedicated outlier handling techniques\(cf\. “outlier handling” in Tab\.[3](https://arxiv.org/html/2607.08057#S4.T3)\)\. These issues become more pronounced at very low bitwidths, where outliers and heterogeneous sensitivities are harder to preserve under such tight budgets\. Thus, the compression\-quality trade\-off is an important practical consideration in this regime\.

Recent advances usevector quantization \(VQ\)to compress groups with codebooks and capture inter\-element correlation\. CQZhanget al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib90)\)couples channels and learns centroids for 1\-bit KV\. VQ\-LLMLiuet al\.\([2025d](https://arxiv.org/html/2607.08057#bib.bib96)\)and CommVQLiet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib103)\)reduce overhead via fused VQ kernels and RoPE\-commutative codebooks\.Sonet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib99)\)further improved calibration robustness\.

It is worth noting that lower bitwidth does not always translate into end\-to\-end system gains\. Realized system gains also depend on the runtime cost of \(de\)quantization, extra kernel boundaries, and whether these costs can be fused or overlapped\. In practice, memory savings are more likely to yield latency or throughput gains when workloads are memory\- or I/O\-bound and quantization is effectively integrated into the system stack\.

Table 4:Summary of low\-rank approximation methods\.TargetMethodGranularityRankCached KVtensorsxKVChanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib114)\)LGFReCalKVYanet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib113)\)K: HG; V: LBShadowKVSunet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib109)\)L \(K\-only\)FWKW^\{K\},WVW^\{V\}LoRCZhanget al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib108)\)LRPaluChanget al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib111)\)HGSQKVEigenAttentionSaxenaet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib115)\)LB

- •Granularity:L = layer\-wise; LG = layer\-group\-wise; HG = head\-group\-wise\.
- •Rank policy:Ffixed;Ssearched;Bbudget\-driven;Rrule\-based\.

#### 5\.1\.2KV Cache Low\-rank Approximation

Low\-rank methods constrain KV\-related tensors to a low\-dimensional subspace,as summarized in Tab\.[4](https://arxiv.org/html/2607.08057#S5.T4)by target: \(i\) cached KV, \(ii\) KV projection weights\(WK,WV\)\(W^\{K\},W^\{V\}\), or \(iii\) QKV attention subspace\.For instance, xKVChanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib114)\)applies layer\-group singular value decomposition to cached KV, while PaluChanget al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib111)\)factorizes\(WK,WV\)\(W^\{K\},W^\{V\}\)with searched rank allocation\. KV tensor methods are most plug\-and\-play but add projection cost, whereas weight and QKV ones increase kernel coupling and engineering overhead\. Some low\-rank methods learn extra parameters, requiring training and thus out of scope under sKis\.

Table 5:Summary of structural compression methods\.FamilyMethodUnitSignalMergingKVMergerWanget al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib142)\)TokenASCaMZhanget al\.\([2024d](https://arxiv.org/html/2607.08057#bib.bib143)\)TokenAKVSharerYanget al\.\([2024d](https://arxiv.org/html/2607.08057#bib.bib144)\)LayerSMiniCacheLiuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib145)\)LayerSD2OWanet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib124)\)TokenSPruningCHAIAgarwalet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib138)\)HeadSThinKXuet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib139)\)ChannelQClusterAttnZhanget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib77)\)TokenA

- •Signal:Aattention score;Ssimilarity/dissimilarity;Qquery norm\.

Table 6:Summary of representative KV cache eviction methods in chronological order\.MethodModeEviction policyBudget policyH2OZhanget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib117)\)DynamicR\+ H2\(via accumulated attention\)UniformScissorhandsLiuet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib121)\)DynamicR\+ Attention scoresUniformRoCoRen and Zhu \([2024](https://arxiv.org/html/2607.08057#bib.bib125)\)DynamicMean & std\. dev\. of attention scoresUniformFastGenGeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib118)\)StaticHybrid \(special/punctuation/locality/H2\)UniformStreamingLLMXiaoet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib122)\)DynamicRSUniformKeyformerAdnanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib119)\)DynamicR\+ Key \(via Gumbel\-softmax scores\)UniformPyramidKVCaiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib126)\)StaticObservation window\-based identificationPreset \(L, pyramid\)NACLChenet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib129)\)DynamicAttention w\.r\.t\. proxy tokens & randomnessUniformPyramidInferYanget al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib128)\)DynamicR\+ PvC \(via ensemble attention\)Preset \(L, pyramid\)BUZZZhaoet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib133)\)DynamicRS\+ Segmented local H2UniformTOVAOrenet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib130)\)DynamicDrop lowest attention score token at each stepUniformVATPGuoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib134)\)DynamicS\+ Attention & valueL1L\_\{1\}\-normUniformL2KVDevotoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib127)\)DynamicKeyL2L\_\{2\}\-normUniformSnapKVLiet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib120)\)StaticObservation window\-based identificationUniformCAKEQinet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib116)\)DynamicR\+ Mean & var\. of attention scoresAdaptive \(L, layer preference\)D2OWanet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib124)\)DynamicRS\+ H2& recall via merging \(§[5\.1\.3](https://arxiv.org/html/2607.08057#S5.SS1.SSS3)\)Adaptive \(L, attention density\)SepLLMChenet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib56)\)DynamicRS\+ Separator tokensUniformLaCacheShiet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib79)\)DynamicLadder pattern basedPreset \(L, ladder\)KVComposeAkulovet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib135)\)DynamicAggregated attention & form composite tokenAdaptive \(L, composite importance\)DiffKVZhanget al\.\([2025e](https://arxiv.org/html/2607.08057#bib.bib41)\)DynamicR\+ Relative significance of attention scoresAdaptive \(H, sparsity pattern\)EvolKVYu and Chai \([2025](https://arxiv.org/html/2607.08057#bib.bib136)\)DynamicPlug\-in \(adopt existing eviction methods\)Adaptive \(L, evolutionary search\)DynamicKVZhouet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib131)\)StaticR\+ Attention w\.r\.t\. instruction tokensAdaptive \(L, task\-aware\)Ada\-KVFenget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib132)\)DynamicPlug\-in \(adopt existing eviction methods\)Adaptive \(H, attention sparsity\)

- •Eviction policy:Rrecent tokens;Sattention sink tokensXiaoet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib122)\), which means initial tokens\.Budget policy:L = layer\-wise, H = head\-wise\.

#### 5\.1\.3KV Cache Structural Compression

Unlike value\-level methods, structural compression reduces KV memory by modifying cache organization\(e\.g\., layer, head, channel, token\)\. We compare existing methods in Tab\.[5](https://arxiv.org/html/2607.08057#S5.T5), including \(i\)pruning, which drops a subset of structural units, and \(ii\)merging, which fuses units into shared forms\. The decisions are often guided by attention or similarity measures \(cf\. “signal” in Tab\.[5](https://arxiv.org/html/2607.08057#S5.T5)\), with clustering sometimes used to form groupsWanget al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib142)\); Agarwalet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib138)\); Zhanget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib77)\)\.

### 5\.2KV Cache Retention Management

Going beyond representations, this section focuses on mechanisms that efficiently manage the retention of the KV cache \(KVRM\) during serving\.

#### 5\.2\.1KV Cache Allocation and Reuse

Structure\-awaremethods redesign KV cache layouts for flexible allocation and reuse\. One line targets virtualized allocationKwonet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib26)\); Xuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib40)\); Shenet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib48)\); Prabhuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib43)\)\. A famous example is PagedAttentionKwonet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib26)\), which uses fixed\-size pages with logical\-to\-physical mapping to reduce fragmentation and support memory reuse\. Another line builds structured indices for prompt sharingGimet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib54)\); Yeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib44)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib47)\), exemplified by radix treeZhenget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib47)\)\. A third line standardizes KV layouts for kernels;Yeet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib46)\)introduced a block\-sparse and composable format\.

Orthogonally,semantics\-guidedmethods further reduce materialization by computing KV only for critical tokensFuet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib42)\)and extend reuse to disaggregated LLM serving via an elastic MemPool systemHuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib45)\)\.

Table 7:Behavior×\\timesobjective matrix of sKis methods\. Side bars encode research density \(rows/columns\)\. Cells mark relevance levels \(●= direct,○= indirect\) and high\-prevalence flags \(⋆\\star:≥70%\\geq 70\\%of papers report gains\)\.BehaviorsMeanlatencyTaillatencyThrough\-putGPUmemoryInter\-connect I/OEnergy/powerQualityimpact↓\\downarrowRowdensityKV\-centric scheduling●⋆\\star●●○○○○Pipelining and overlapping●⋆\\star○●⋆\\star○○○Hardware\-aware execution●⋆\\star●●⋆\\star●●○Memory hierarchy KV orchestration○○○○●⋆\\star●●Compute device KV orchestration○○●⋆\\star○●●●KV cache compression○○○●⋆\\star●●●KV cache retention management○○○●⋆\\star●●●Column density
- •Behavior dimension: temporal,spatial,structural\.Row density0–1011–2021–3031\+Column density0–2021–4041–6061\+

#### 5\.2\.2KV Cache Eviction

KV cache eviction reduces memory by discarding less critical token KV states under a budget\.We compare algorithmic details of existing methods in Tab\.[6](https://arxiv.org/html/2607.08057#S5.T6)and highlight three key insights\.

First, methods differ in when eviction is applied \(cf\. “mode” in Tab\.[6](https://arxiv.org/html/2607.08057#S5.T6)\)\.Static methods evict once during or after prefill and keep the retained set fixed in decoding, while dynamic ones update online during decoding to track importance shifts\.Second, eviction policies often retain a recent window or attention sink tokens, and select extra tokens by lightweight signals such as attention\-derived scores, heuristics, or robust variants that mitigate bias in local attention statistics \(cf\. “eviction policy” in Tab\.[6](https://arxiv.org/html/2607.08057#S5.T6)\)\. Third, recent works move beyond uniform budgets and instead assign budgets across layers and even heads via preset and adaptive allocation \(cf\. “budget policy” in Tab\.[6](https://arxiv.org/html/2607.08057#S5.T6)\)\.Some methods treat budget policy as a plug\-in to existing eviction rules\.

Takeaways & Limitations – Structural Behavior•KVCC delivers the most direct memory relief, but such savings may not translate into system gains without system co\-design, due to tail \(e\.g\., outlier\) behavior, compression overhead, and kernel or runtime constraints\. Practical deployment of KVCC remains a challenge\.•KVRM improves effective capacity by deciding which KV states exist at runtime\. The key challenge is fast and stable utility estimation\. In practice, policies are often workload\-sensitive, and robustness under complex serving environments remains under\-studied\.More analysis is provided in Apps\.[E\.4](https://arxiv.org/html/2607.08057#A5.SS4)and[E\.5](https://arxiv.org/html/2607.08057#A5.SS5)\.

## 6Observations and Open Challenges

In this section, we identify key observations from two complementary lenses: \(i\) a behavior×\\timesobjective matrix and \(ii\) a behavior\-behavior co\-design affinity network, which naturally motivate open challenges of future sKis research\.

### 6\.1Key Observations

Table[7](https://arxiv.org/html/2607.08057#S5.T7)\(behavior×\\timesobjective matrix\) marks each behavior’s impact on serving objectives as direct \(●\) or indirect \(○\); stars \(⋆\\star\) on direct cells statistically flag≥70%\\geq 70\\%of papers reporting such gains\. Side bars show research density\. Serving objectives cover latency, throughput, GPU memory, interconnect I/O, and energy\. We also include quality impact↓\\downarrowto capture degradation as a trade\-off\. Figure[6](https://arxiv.org/html/2607.08057#S6.F6)\(behavior\-behaviorco\-design affinitynetwork\) visualizes cross\-behavior co\-occurrence in the literature, with edge thickness proportional to normalized weights\. Low\-score edges are omitted, and computation details are provided in App\.[F](https://arxiv.org/html/2607.08057#A6)\)\. This affinity reflects observed co\-design patterns rather than validated performance gains\. We conclude key observations \(O1\-O7\) as follows\.

O1\. Structural works are most studied and dominate memory savings,while others yield savings indirectly \(e\.g\., via migration or reuse\), indicating a community bias toward memory efficiency\.

O2\. Temporal behaviors act most directly on latency and throughput,since KVS, OVLP, and HAE map cleanly to reductions in scheduling stalls, pipeline bubbles, and device under\-utilization\. However, tail latency reporting is sparse\.

O3\. Spatial methods primarily target interconnect I/O, often paired with OVLP\.Their core focus is KV transfer, and by overlapping it with compute, they effectively hide transfer latency\.

![Refer to caption](https://arxiv.org/html/2607.08057v1/x5.png)Figure 6:Behavior\-behaviorco\-design affinity network\.O4\. Energy is under\-explored,although many methods reduce memory or compute intensity that should translate to energy benefits\.

O5\. Quality loss is universal\.Temporal methods risk inconsistent request handling; spatial methods risk missed KV data; and structural methods directly reduce KV precision\. The practical question is to ensure such degradation is controllable\.

O6\. HAE–CDO is the strongestco\-design pattern\.Compute layouts that exploit device heterogeneity often co\-design with KV colocating or transfer, yielding joint gains in utilization and I/O\.

O7\. KVCC remains isolateddespite its popularity, which suggests a missed opportunity for co\-design\.

### 6\.2Open Challenges

The above key observations \(O1\-O7\) reveal both progress and gaps of current sKis research, which naturally motivate the next set of system\-level open challenges \(C1\-C6\)\.

C1\. SLO\-driven tail control←\\leftarrowO2\.Service\-level objectives \(SLOs\) are critical to LLM serving, with tail latency dominating user experienceDean and Barroso \([2013](https://arxiv.org/html/2607.08057#bib.bib158)\); Wanget al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib159)\), yet most systems omit tail metrics\. Under long contexts and bursts, KV generation, migration, and compression may trigger SLO violations, which often arise from cross\-behavior interference rather than a single operation\. The real challenge is to attribute these SLO violations to concrete KV behaviors and paths, motivating future sKis studies on standardized preemption and degradation semantics to make tail outcomes controllable\.

C2\. Energy\-aware sKis←\\leftarrowO4\.With surging data center demand, AI infrastructure should be energy\-awareHe \([2025](https://arxiv.org/html/2607.08057#bib.bib154)\), but energy is still rarely reported or optimized in sKis methods\. Future research could integrate power profiling into runtime decisions, establish serving\-time energy models, and jointly optimize energy\-latency\-quality under power constraints\. Beyond aggregate power, energy should also be measured as per\-request or per\-token energy under serving traces and budgets\. Another possible direction is to study energy\-friendly KV granularities and layouts\.

C3\. Trustworthy and efficient sKis←\\leftarrowO5\.LLM serving must ensure not only quality but also trustworthinessHanet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib155)\); Yanget al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib156)\), yet trust risks are rarely considered, leaving a gap between efficiency gains and trust failures\. For example, structural methods can harm robustness in ways standard metrics miss, as policies may evict or compress low\-salience but crucial context, causing severe errors under workload shifts despite stable mean accuracy\. Notably, sKis techniques can be dual\-use:Jianget al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib157)\)turned KV eviction into a defense against jailbreak attacks, suggesting that sKis techniques may become trust mechanisms\. Trust concerns also span reliability, privacy, and safety across diverse sKis behaviors, as discussed in App\.[G\.1](https://arxiv.org/html/2607.08057#A7.SS1), and arise in various downstream settingsDonget al\.\([2026](https://arxiv.org/html/2607.08057#bib.bib162)\); Chenet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib163)\); Jiang and Ferraro \([2026](https://arxiv.org/html/2607.08057#bib.bib164)\)\. Beyond average quality, future work should expose worst\-case failures and quality lower bounds under stress, making trustworthiness behavior\-attributed and SLO\-aware\.

C4\. Generalizable HAE–CDO←\\leftarrowO6\.While HAE and CDO form the strongest co\-design pattern, policies are often tailored to specific fabric or single\-tenant settings\. More broadly, although current sKis practice is largely GPU\-centric, the temporal–spatial–structural lens itself is hardware\-agnostic\. What changes across hardware platforms is the concrete hierarchy, interconnect, and kernel or runtime support\. Future directions include making such pattern portable across heterogeneous topologies \(e\.g\., NVLink, NVSwitch, PCIe, CXL\) and adaptive to multi\-tenant settings\.

C5\. Co\-optimization and intermediate semantics←\\leftarrowO7\.Most sKis optimize behaviors in isolation, despite their interactions under bandwidth and latency constraints\. Future studies could explore co\-optimization under shared budgets\. For instance, to co\-decide eviction, offload, and prefetch given predicted reuse, success probability, and I/O contention\. In resource\-constrained edge settings, this also suggests coupling KVCC with OVLP: KVCC reduces memory footprint, while OVLP helps prevent \(de\)compression overhead from negating system gains\. Another promising direction is to exploit fine\-grained intermediate semantics for behaviors and view co\-optimization as state transitions over them\. We give concrete examples in App\.[G\.2](https://arxiv.org/html/2607.08057#A7.SS2)illustrating how intermediate semantics enable co\-optimizing eviction, compression, and migration\.

C6\. Unified benchmarks\.A fundamental challenge in sKis lies in establishing standardized benchmarks\. We review LLM inference benchmarking practices in App\.[G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1)\. We find inconsistent metric definitions and measures across tools, preventing reliable apples\-to\-apples comparisons across papers\. We therefore advocate unified sKis benchmarksand offer a concise set of benchmark design principles, covering metrics \(e\.g\., trust metrics and KV\-centric resource metrics\), representative stress workloads, and reporting standards, detailed in App\.[G\.3\.2](https://arxiv.org/html/2607.08057#A7.SS3.SSS2)\.

## 7Conclusion

This survey presents a systematic overview of sKis, offering a system behavior\-oriented taxonomy covering temporal, spatial, and structural dimensions\. By cross\-analyzing behavior\-objective impacts and behavior\-behaviorco\-design patterns, we reveal overlooked regions and open challenges\. We hope this survey inspires continued exploration toward efficient and trustworthy LLM serving\.

## Limitations

This paper offers a comprehensive review and summary of current methods in the area of system\-aware KV cache optimization\. However, given the extensive body of related work and the rapidly evolving nature of this research area, we may have overlooked some equally valuable contributions\. We tried to include all relevant studies and references wherever feasible\.

Additionally, this survey conducts no new experiments\. Our claims synthesize results reported in public papers and open\-source implementations, primarily under mainstream platforms and common configurations, which may constrain the generality of our conclusions\.We avoid aggregating raw speedup or memory numbers across papers, because the reported gains are tightly coupled with model, hardware, workload, or baseline choices\.

## Acknowledgments

Feng Liu is supported by the ARC through grants DE240101089, LP240100101, DP230101540, and by the NSF&CSIRO Responsible AI program under grant 2303037\. ChatGPT was used only for limited wording and grammar suggestions\.

## References

- M\. Adnan, A\. Arunkumar, G\. Jain, P\. J\. Nair, I\. Soloveychik, and P\. Kamath \(2024\)Keyformer: KV cache reduction through key tokens selection for efficient generative inference\.Proceedings of Machine Learning and Systems6,pp\. 114–127\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.19.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.8.8.8.2)\.
- S\. Agarwal, B\. Acun, B\. Hosmer, M\. Elhoushi, Y\. Lee, S\. Venkataraman, D\. Papailiopoulos, and C\. Wu \(2024\)CHAI: clustered head attention for efficient LLM inference\.InInternational Conference on Machine Learning,pp\. 291–312\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.33.1),[§5\.1\.3](https://arxiv.org/html/2607.08057#S5.SS1.SSS3.p1.1.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.7.7.7.7.7.7.7.7.7.3)\.
- D\. Akulov, M\. Sana, A\. De Domenico, T\. S\. Salem, N\. Piovesan, and F\. Ayed \(2025\)KVCompose: efficient structured KV cache compression with composite tokens\.arXiv preprint arXiv:2509\.05165\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.48.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.32.1)\.
- R\. Y\. Aminabadi, S\. Rajbhandari, A\. A\. Awan, C\. Li, D\. Li, E\. Zheng, O\. Ruwase, S\. Smith, M\. Zhang, J\. Rasley, and Y\. He \(2022\)DeepSpeed\-Inference: enabling efficient inference of transformer models at unprecedented scale\.InInternational Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–15\.Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.p1.1)\.
- S\. Ashkboos, M\. L\. Croci, M\. G\. d\. Nascimento, T\. Hoefler, and J\. Hensman \(2024\)SliceGPT: compress large language models by deleting rows and columns\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.08057#S2.SS0.SSS0.Px2.p3.1)\.
- G\. Bai, Z\. Chai, C\. Ling, S\. Wang, J\. Lu, N\. Zhang, T\. Shi, Z\. Yu, M\. Zhu, Y\. Zhang, C\. Yang, Y\. Cheng, and L\. Zhao \(2024\)Beyond efficiency: a systematic survey of resource\-efficient large language models\.arXiv preprint arXiv:2401\.00625\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.7.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p1.1.3)\.
- P\. Behnam, Y\. Fu, R\. Zhao, P\. Tsai, Z\. Yu, and A\. Tumanov \(2025\)RocketKV: accelerating long\-context LLM inference via two\-stage KV cache compression\.InInternational Conference on Machine Learning,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.41.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p3.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. J\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- H\. Cai, B\. Shen, L\. Jin, L\. Hu, and X\. Fan \(2025\)Does tone change the answer? evaluating prompt politeness effects on modern LLMs: GPT, Gemini, LLaMA\.arXiv preprint arXiv:2512\.12812\.External Links:2512\.12812,[Document](https://dx.doi.org/10.48550/arXiv.2512.12812),[Link](https://arxiv.org/abs/2512.12812)Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p2.1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, B\. Chang, J\. Hu, and W\. Xiao \(2024\)PyramidKV: dynamic KV cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.22.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.27.1)\.
- C\. Chang, C\. Lin, Y\. Akhauri, W\. Lin, K\. Wu, L\. Ceze, and M\. S\. Abdelfattah \(2025a\)xKV: cross\-layer SVD for KV\-cache compression\.arXiv preprint arXiv:2503\.18893\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.18.1),[§5\.1\.2](https://arxiv.org/html/2607.08057#S5.SS1.SSS2.p1.2),[Table 4](https://arxiv.org/html/2607.08057#S5.T4.1.1.1.1.1.1.1.1.1.3)\.
- C\. Chang, W\. Lin, C\. Lin, C\. Chen, Y\. Hu, P\. Wang, N\. Huang, L\. Ceze, M\. S\. Abdelfattah, and K\. Wu \(2025b\)Palu: KV\-cache compression with low\-rank projection\.InInternational Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.24.1),[§5\.1\.2](https://arxiv.org/html/2607.08057#S5.SS1.SSS2.p1.2),[Table 4](https://arxiv.org/html/2607.08057#S5.T4.7.7.7.7.7.7.7.7.7.2)\.
- G\. Chen, H\. Shi, J\. Li, Y\. Gao, X\. Ren, Y\. Chen, X\. Jiang, Z\. Li, W\. Liu, and C\. Huang \(2025a\)SepLLM: accelerate large language models by compressing one segment into one separator\.InInternational Conference on Machine Learning,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.37.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.22.22.22.3)\.
- J\. Chen, H\. Lin, X\. Han, and L\. Sun \(2024a\)Benchmarking large language models in retrieval\-augmented generation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 17754–17762\.Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p4.1)\.
- R\. Chen, Z\. Wang, B\. Cao, T\. Wu, S\. Zheng, X\. Li, X\. Wei, S\. Yan, M\. Li, and Y\. Liang \(2024b\)ArkVale: efficient generative LLM inference with recallable key\-value eviction\.Advances in Neural Information Processing Systems37,pp\. 113134–113155\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.12.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- W\. Chen, S\. He, H\. Qu, R\. Zhang, S\. Yang, P\. Chen, Y\. Zheng, B\. Huai, and G\. Chen \(2025b\)IMPRESS: an importance\-informed multi\-tier prefix KV storage system for large language model inference\.InUSENIX Conference on File and Storage Technologies,pp\. 187–201\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.15.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- Y\. Chen, G\. Wang, J\. Shang, S\. Cui, Z\. Zhang, T\. Liu, S\. Wang, Y\. Sun, D\. Yu, and H\. Wu \(2024c\)NACL: a general and effective KV cache eviction framework for LLM at inference time\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 7913–7926\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.41.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.28.1)\.
- Z\. Chen, R\. Sadhukhan, Z\. Ye, Y\. Zhou, J\. Zhang, N\. Nolte, Y\. Tian, M\. Douze, L\. Bottou, Z\. Jia, and B\. Chen \(2025c\)MagicPIG: LSH sampling for efficient LLM generation\.InInternational Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.27.1),[§3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2.p2.1)\.
- Y\. Cheng, Y\. Liu, J\. Yao, Y\. An, X\. Chen, S\. Feng, Y\. Huang, S\. Shen, K\. Du, and J\. Jiang \(2025\)LMCache: an efficient KV cache layer for enterprise\-scale LLM inference\.arXiv preprint arXiv:2510\.09665\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.49.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p1.1)\.
- K\. T\. Chitty\-Venkata, S\. Raskar, B\. Kale, F\. Ferdaus, A\. Tanikanti, K\. Raffenetti, V\. Taylor, M\. Emani, and V\. Vishwanath \(2024\)LLM\-Inference\-Bench: inference benchmarking of large language models on AI accelerators\.InWorkshops of the International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1362–1379\.Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.Px2.p1.1)\.
- J\. Dean and L\. A\. Barroso \(2013\)The tail at scale\.Communications of the ACM56\(2\),pp\. 74–80\.Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p2.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-V2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§2](https://arxiv.org/html/2607.08057#S2.SS0.SSS0.Px2.p3.1)\.
- A\. Devoto, Y\. Zhao, S\. Scardapane, and P\. Minervini \(2024\)A simple and effectiveL​\_​2L\\\_2norm\-based strategy for KV cache compression\.InConference on Empirical Methods in Natural Language Processing,pp\. 18476–18499\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.58.1),[3rd item](https://arxiv.org/html/2607.08057#A5.I5.i3.p1.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.15.15.15.2)\.
- Y\. Ding, L\. L\. Zhang, C\. Zhang, Y\. Xu, N\. Shang, J\. Xu, F\. Yang, and M\. Yang \(2024\)LongRoPE: extending LLM context window beyond 2 million tokens\.arXiv preprint arXiv:2402\.13753\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p2.1)\.
- R\. Dong, Z\. Ye, J\. Qi, L\. Feng, F\. Liu, G\. Niu, and M\. Sugiyama \(2026\)CARPRT: class\-aware zero\-shot prompt reweighting for vision\-language model\.InInternational Conference on Learning Representations,Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p4.1)\.
- S\. Dong, W\. Cheng, J\. Qin, and W\. Wang \(2024\)QAQ: quality adaptive quantization for LLM KV cache\.arXiv preprint arXiv:2403\.04643\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.14.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.7.1)\.
- Y\. Dong, Y\. Miao, W\. Li, X\. Zheng, C\. Wang, and F\. Lyu \(2025\)Accelerating LLM inference throughput via asynchronous KV cache prefetching\.arXiv preprint arXiv:2504\.06319\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.23.1),[4th item](https://arxiv.org/html/2607.08057#A5.I1.i4.p1.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.8.8.8.3),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px2.p1.1)\.
- J\. Duan, R\. Lu, H\. Duanmu, X\. Li, X\. Zhang, D\. Lin, I\. Stoica, and H\. Zhang \(2024\)MuxServe: flexible spatial\-temporal multiplexing for multiple LLM serving\.InInternational Conference on Machine Learning,pp\. 11905–11917\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.35.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1),[§3\.3\.1](https://arxiv.org/html/2607.08057#S3.SS3.SSS1.p1.1)\.
- H\. Duanmu, Z\. Yuan, X\. Li, J\. Duan, X\. Zhang, and D\. Lin \(2024\)SKVQ: sliding\-window key and value cache quantization for large language models\.arXiv preprint arXiv:2405\.06219\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.48.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.13.1)\.
- Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou \(2025\)Ada\-KV: optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference\.Advances in Neural Information Processing Systems\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.55.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.34.1)\.
- Q\. Fu, M\. Cho, T\. Merth, S\. Mehta, M\. Rastegari, and M\. Najibi \(2024\)LazyLLM: dynamic token pruning for efficient long context LLM inference\.arXiv preprint arXiv:2407\.14057\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.29.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p2.1)\.
- B\. Gao, Z\. He, P\. Sharma, Q\. Kang, D\. Jevdjic, J\. Deng, X\. Yang, Z\. Yu, and P\. Zuo \(2024\)Cost\-efficient large language model serving for multi\-turn conversations with CachedAttention\.InUSENIX Annual Technical Conference,pp\. 111–126\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.28.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.6.6.6.3),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1)\.
- S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. Gao \(2024\)Model tells you what to discard: adaptive KV cache compression for LLMs\.InInternational Conference on Learning Representations,Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.16.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.5.5.5.2)\.
- I\. Gim, G\. Chen, S\. Lee, N\. Sarda, A\. Khandelwal, and L\. Zhong \(2024\)Prompt Cache: modular attention reuse for low\-latency inference\.Proceedings of Machine Learning and Systems6,pp\. 325–338\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.21.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- R\. Gray \(1984\)Vector quantization\.IEEE Assp Magazine1\(2\),pp\. 4–29\.Cited by:[2nd item](https://arxiv.org/html/2607.08057#A5.I4.i2.p1.1)\.
- X\. Gu, T\. Pang, C\. Du, Q\. Liu, F\. Zhang, C\. Du, Y\. Wang, and M\. Lin \(2025\)When attention sink emerges in language models: an empirical view\.InInternational Conference on Learning Representations,Cited by:[2nd item](https://arxiv.org/html/2607.08057#A5.I5.i2.p1.1)\.
- Z\. Guo, H\. Kamigaito, and T\. Watanabe \(2024\)Attention score is not all you need for token importance indicator in KV cache reduction: value also matters\.InConference on Empirical Methods in Natural Language Processing,pp\. 21158–21166\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.57.1),[3rd item](https://arxiv.org/html/2607.08057#A5.I5.i3.p1.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.14.14.14.3)\.
- B\. Han, J\. Yao, T\. Liu, B\. Li, S\. Koyejo, and F\. Liu \(2025\)Trustworthy machine learning: from data to models\.Foundations and Trends® in Privacy and Security7\(2\-3\),pp\. 74–246\.Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p4.1)\.
- J\. Hao, Q\. Huang, Y\. Wang, M\. Zhang, and J\. Yu \(2026\)DeltaKV: residual\-based kv cache compression via long\-range similarity\.arXiv preprint arXiv:2602\.08005\.Cited by:[3rd item](https://arxiv.org/html/2607.08057#A5.I4.i3.p1.1)\.
- J\. Hao, Y\. Zhu, T\. Wang, J\. Yu, X\. Xin, B\. Zheng, Z\. Ren, and S\. Guo \(2025\)OmniKV: dynamic context selection for efficient long\-context LLMs\.InInternational Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.28.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- J\. He and J\. Zhai \(2024\)FastDecode: high\-throughput GPU\-efficient LLM serving using heterogeneous pipelines\.arXiv preprint arXiv:2403\.11421\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.13.1),[§3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2.p2.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.1.1.1.3)\.
- Q\. He \(2025\)A unified metric architecture for AI infrastructure: a cross\-layer taxonomy integrating performance, efficiency, and cost\.arXiv preprint arXiv:2511\.21772\.Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p3.1)\.
- Y\. He, L\. Zhang, W\. Wu, J\. Liu, H\. Zhou, and B\. Zhuang \(2024\)ZipCache: accurate and efficient KV cache quantization with salient token identification\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 68287–68307\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.6.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.16.1)\.
- Y\. He, H\. Mao, C\. Giannoula, M\. Sadrosadati, J\. Gómez\-Luna, H\. Li, X\. Li, Y\. Wang, and O\. Mutlu \(2025\)PAPI: exploiting dynamic parallelism in large language model decoding with a processing\-in\-memory\-enabled computing system\.InACM International Conference on Architectural Support for Programming Languages and Operating Systems,pp\. 766–782\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.21.1),[§3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2.p3.1)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)KVQuant: towards 10 million context length LLM inference with KV cache quantization\.Advances in Neural Information Processing Systems37,pp\. 1270–1303\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.4.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.14.1)\.
- C\. Hu, H\. Huang, J\. Hu, J\. Xu, X\. Chen, T\. Xie, C\. Wang, S\. Wang, Y\. Bao, N\. Sun, and Y\. Shan \(2024a\)MemServe: context caching for disaggregated LLM serving with elastic memory pool\.arXiv preprint arXiv:2406\.17565\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.13.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p2.1)\.
- C\. Hu, H\. Huang, L\. Xu, X\. Chen, J\. Xu, S\. Chen, H\. Feng, C\. Wang, S\. Wang, Y\. Bao, N\. Sun, and Y\. Shan \(2024b\)Inference without interference: disaggregate LLM inference for mixed downstream workloads\.arXiv preprint arXiv:2401\.11181\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.9.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I1.i2.p1.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1),[§3\.3\.1](https://arxiv.org/html/2607.08057#S3.SS3.SSS1.p1.1)\.
- C\. Jiang, L\. Gao, H\. E\. Zarch, and M\. Annavaram \(2025a\)KVPR: efficient LLM inference with i/o\-aware KV cache partial recomputation\.InAnnual Meeting of the Association for Computational Linguistics,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.45.1),[4th item](https://arxiv.org/html/2607.08057#A5.I1.i4.p1.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.10.10.10.3)\.
- J\. Jiang, Z\. Wen, A\. Mansoor, and A\. Mian \(2024\)Fast inference for probabilistic graphical models\.InUSENIX Annual Technical Conference,pp\. 95–110\.Cited by:[Appendix B](https://arxiv.org/html/2607.08057#A2.p1.1)\.
- J\. Jiang, Z\. Wen, and A\. Mian \(2022\)Fast parallel Bayesian network structure learning\.InIEEE International Parallel and Distributed Processing Symposium,pp\. 617–627\.Cited by:[Appendix B](https://arxiv.org/html/2607.08057#A2.p1.1)\.
- T\. Jiang, Z\. Wang, J\. Liang, C\. Li, Y\. Wang, and T\. Wang \(2025b\)RobustKV: defending large language models against jailbreak attacks via kv eviction\.InInternational Conference on Learning Representations,Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p4.1)\.
- X\. Jiang, Y\. Zhou, S\. Cao, I\. Stoica, and M\. Yu \(2025c\)Neo: saving GPU memory crisis with CPU offloading for online LLM inference\.Proceedings of Machine Learning and Systems\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.31.1),[§3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2.p2.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.2.2.2.2)\.
- Y\. Jiang and F\. Ferraro \(2026\)SCRIBE: structured mid\-level supervision for tool\-using language models\.arXiv preprint arXiv:2601\.03555\.External Links:2601\.03555,[Document](https://dx.doi.org/10.5281/zenodo.19647050),[Link](https://doi.org/10.5281/zenodo.19647050)Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p4.1)\.
- S\. Jie, Y\. Tang, K\. Han, Z\. Deng, and J\. Han \(2025\)SpeCache: speculative key\-value caching for efficient generation of LLMs\.InInternational Conference on Machine Learning,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.40.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- C\. Jin, Z\. Zhang, X\. Jiang, F\. Liu, X\. Liu, X\. Liu, and X\. Jin \(2025\)RAGCache: efficient knowledge caching for retrieval\-augmented generation\.ACM Transactions on Computer Systems\.Note:Just AcceptedExternal Links:ISSN 0734\-2071,[Link](https://doi.org/10.1145/3768628),[Document](https://dx.doi.org/10.1145/3768628)Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.47.1),[1st item](https://arxiv.org/html/2607.08057#A5.I3.i1.p1.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1)\.
- L\. Jurkschat, P\. Gattogi, S\. Vahdati, and J\. Lehmann \(2025\)BALI\-a benchmark for accelerated language model inference\.IEEE Access\.Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.Px2.p1.1)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)GEAR: an efficient KV cache compression recipe for near\-lossless generative inference of LLM\.arXiv preprint arXiv:2403\.05527\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.46.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.12.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 611–626\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.6.1),[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.p1.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- W\. Lee, J\. Lee, J\. Seo, and J\. Sim \(2024\)InfiniGen: efficient generative inference of large language models with dynamic KV cache management\.InUSENIX Symposium on Operating Systems Design and Implementation,pp\. 155–172\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.27.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- B\. Li, Y\. Jiang, V\. Gadepally, and D\. Tiwari \(2024a\)LLM inference serving: survey of recent advances and opportunities\.arXiv preprint arXiv:2407\.12391\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.4.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p1.1.3),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.4.1),[§1](https://arxiv.org/html/2607.08057#S1.p2.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.
- H\. Li, J\. Zhao, J\. Bazin, P\. Kim, K\. Joo, Z\. Zhao, and Y\. Liu \(2023\)Hong Kong world: leveraging structural regularity for line\-based SLAM\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(11\),pp\. 13035–13053\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- H\. Li, Y\. Li, A\. Tian, T\. Tang, Z\. Xu, X\. Chen, N\. Hu, W\. Dong, Q\. Li, and L\. Chen \(2024b\)A survey on large language model acceleration based on KV cache management\.arXiv preprint arXiv:2412\.19442\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.10.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p2.1),[§G\.3](https://arxiv.org/html/2607.08057#A7.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.8.1),[§1](https://arxiv.org/html/2607.08057#S1.p2.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.
- J\. Li, Y\. Zhang, M\. Y\. Hassan, T\. Chafekar, T\. Cai, Z\. Ren, P\. Guo, F\. Karimzadeh, C\. Wang, and C\. Gan \(2025a\)CommVQ: commutative vector quantization for KV cache compression\.InInternational Conference on Machine Learning,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.38.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I4.i2.p1.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.21.1),[§5\.1\.1](https://arxiv.org/html/2607.08057#S5.SS1.SSS1.p3.1)\.
- X\. Li, Y\. Lu, J\. Cao, Y\. Ma, Z\. Li, and Y\. Zhou \(2025b\)CATCH: a modular cross\-domain adaptive template with hook\.InInternational Symposium on Visual Computing,pp\. 41–52\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024c\)SnapKV: LLM knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.7.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.30.1)\.
- Y\. Li, J\. Dong, C\. Yang, S\. Wen, P\. Koniusz, T\. Huang, Y\. Tian, and Y\. Ong \(2025c\)MMT\-ARD: multimodal multi\-teacher adversarial distillation for robust vision\-language models\.arXiv preprint arXiv:2511\.17448\.Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p1.1)\.
- Y\. Li, K\. Li, X\. Yin, Z\. Yang, Z\. Dong, Z\. Yao, H\. Xu, Y\. Tian, and Y\. Lu \(2026\)SepPrune: structured pruning for efficient deep speech separation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 31861–31869\.Cited by:[Appendix B](https://arxiv.org/html/2607.08057#A2.p1.1)\.
- Y\. Li, Y\. Lu, J\. Dong, Z\. Dong, C\. Yang, X\. Yin, Y\. Chen, J\. Gou, Y\. Tian, and T\. Huang \(2024d\)SGLP: a similarity guided fast layer partition pruning for compressing large deep models\.arXiv preprint arXiv:2410\.14720\.Cited by:[§2](https://arxiv.org/html/2607.08057#S2.SS0.SSS0.Px2.p3.1)\.
- C\. Liao and Z\. Wen \(2026\)Channel\-aware mixed\-precision quantization for efficient long\-context inference\.InInternational Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.60.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.24.1)\.
- B\. Lin, T\. Peng, C\. Zhang, M\. Sun, L\. Li, H\. Zhao, W\. Xiao, Q\. Xu, X\. Qiu, S\. Li, Z\. Ji, Y\. Li, and W\. Lin \(2024\)Infinite\-LLM: efficient LLM service for long context with distattention and distributed KVCache\.arXiv preprint arXiv:2401\.02669\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.26.1),[§3\.3\.1](https://arxiv.org/html/2607.08057#S3.SS3.SSS1.p1.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p1.1)\.
- Y\. Lin, H\. Tang, S\. Yang, Z\. Zhang, G\. Xiao, C\. Gan, and S\. Han \(2025\)QServe: W4A8KV4 quantization and system co\-design for efficient LLM serving\.Proceedings of Machine Learning and Systems\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.29.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.20.1)\.
- A\. Liu, J\. Liu, Z\. Pan, Y\. He, G\. Haffari, and B\. Zhuang \(2024a\)MiniCache: KV cache compression in depth dimension for large language models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 139997–140031\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.8.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.5.5.5.5.5.5.5.5.5.2)\.
- D\. Liu, M\. Chen, B\. Lu, H\. Jiang, Z\. Han, Q\. Zhang, Q\. Chen, C\. Zhang, B\. Ding, K\. Zhang, C\. Chen, F\. Yang, Y\. Yang, and L\. Qiu \(2025a\)RetrievalAttention: accelerating long\-context LLM inference via vector retrieval\.Advances in Neural Information Processing Systems\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.54.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Liu and Y\. Yu \(2026\)CXL\-SpecKV: a disaggregated FPGA speculative KV\-Cache for datacenter llm serving\.InACM/SIGDA International Symposium on Field Programmable Gate Arrays,pp\. 56–66\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.59.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.12.12.12.4),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1)\.
- G\. Liu, C\. Li, J\. Zhao, C\. Zhang, and M\. Guo \(2025b\)ClusterKV: manipulating LLM KV cache in semantic space for recallable compression\.InACM/IEEE Design Automation Conference,pp\. 1–7\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.34.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- P\. Liu, Z\. Gao, X\. Zhao, Y\. Ma, T\. Wang, and J\. Wen \(2024b\)Unlocking data\-free low\-bit quantization with matrix decomposition for KV cache compression\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 2430–2440\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.40.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.11.1)\.
- Y\. Liu, J\. Fu, S\. Liu, Y\. Zou, Y\. Fu, J\. Zhou, and S\. Zhang \(2025c\)KV cache compression for inference efficiency in LLMs: a review\.arXiv preprint arXiv:2508\.06297\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.11.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p2.1),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.9.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.
- Y\. Liu, H\. Li, Y\. Cheng, S\. Ray, Y\. Huang, Q\. Zhang, K\. Du, J\. Yao, S\. Lu, G\. Ananthanarayanan, M\. Maire, H\. Hoffmann, A\. Holtzman, and J\. Jiang \(2024c\)CacheGen: KV cache compression and streaming for fast large language model serving\.InProceedings of the ACM SIGCOMM 2024 Conference,pp\. 38–56\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.39.1),[3rd item](https://arxiv.org/html/2607.08057#A5.I3.i3.p1.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p2.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.10.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for LLM KV cache compression at test time\.Advances in Neural Information Processing Systems36,pp\. 52342–52364\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.8.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.4.4.4.2)\.
- Z\. Liu, X\. Luo, J\. Guo, W\. Ni, Y\. Zhou, Y\. Guan, C\. Guo, W\. Cui, Y\. Feng, M\. Guo, Y\. Zhu, M\. Zhang, J\. Leng, and C\. Jin \(2025d\)VQ\-LLM: high\-performance code generation for vector quantization augmented LLM inference\.InIEEE International Symposium on High Performance Computer Architecture,pp\. 1496–1509\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.17.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I4.i2.p1.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.18.1),[§5\.1\.1](https://arxiv.org/html/2607.08057#S5.SS1.SSS1.p3.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024d\)KIVI: a tuning\-free asymmetric 2bit quantization for KV cache\.InInternational Conference on Machine Learning,pp\. 32332–32344\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.32.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.9.1)\.
- L\. Long, R\. Yang, Y\. Huang, D\. Hui, A\. Zhou, and J\. Yang \(2025\)SlimInfer: accelerating long\-context LLM inference via dynamic token pruning\.arXiv preprint arXiv:2508\.06447\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.46.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- Y\. Lu, H\. Cheng, Y\. Fang, Z\. Wang, J\. Wei, D\. Xu, Q\. Xuan, X\. Yang, and Z\. Zhu \(2026\)Reassessing layer pruning in LLMs: new insights and methods\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.08057#S2.SS0.SSS0.Px2.p3.1)\.
- X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, H\. Jin, T\. Chen, and Z\. Jia \(2023\)Towards efficient generative large language model serving: a survey from algorithms to systems\.arXiv preprint arXiv:2312\.15234\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.2.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p1.1),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.2.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.
- NVIDIA \(2023\)TensorRT\-LLM\.Note:[https://github\.com/NVIDIA/TensorRT\-LLM](https://github.com/NVIDIA/TensorRT-LLM)Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.p1.1)\.
- NVIDIA \(2025a\)GenAI\-Perf\.NVIDIA Docs\.External Links:[Link](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html)Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.Px1.p1.1)\.
- NVIDIA \(2025b\)NVIDIA NIM LLMs benchmarking\.NVIDIA Docs\.External Links:[Link](https://docs.nvidia.com/nim/benchmarking/llm/latest/overview.html#introduction-to-llm-inference-benchmarking)Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.Px1.p1.1)\.
- OpenAI \(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- M\. Oren, M\. Hassid, N\. Yarden, Y\. Adi, and R\. Schwartz \(2024\)Transformers are multi\-state RNNs\.InConference on Empirical Methods in Natural Language Processing,pp\. 18724–18741\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.56.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.29.1)\.
- X\. Pan, E\. Li, Q\. Li, S\. Liang, Y\. Shan, K\. Zhou, Y\. Luo, X\. Wang, and J\. Zhang \(2024\)InstInfer: in\-storage attention offloading for cost\-effective long\-context LLM inference\.arXiv preprint arXiv:2409\.04992\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.44.1),[§3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2.p3.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p2.1)\.
- Z\. Pan, A\. Patel, Z\. Hu, Y\. Shen, Y\. Guan, W\. Li, L\. Qin, Y\. Wang, and Y\. Ding \(2025\)KVFlow: efficient prefix caching for accelerating LLM\-based multi\-agent workflows\.Advances in Neural Information Processing Systems\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.57.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1)\.
- D\. Park and B\. Egger \(2024\)Improving throughput\-oriented LLM inference with CPU computations\.InInternational Conference on Parallel Architectures and Compilation Techniques,pp\. 233–245\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.50.1),[§3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2.p2.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.4.4.4.4)\.
- J\. Park, J\. Choi, K\. Kyung, M\. J\. Kim, Y\. Kwon, N\. S\. Kim, and J\. H\. Ahn \(2024\)AttAcc\! unleashing the power of PIM for batched transformer\-based generative model inference\.InACM International Conference on Architectural Support for Programming Languages and Operating Systems,pp\. 103–119\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.15.1),[§3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2.p3.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p2.1)\.
- P\. Patel, E\. Choukse, C\. Zhang, A\. Shah, Í\. Goiri, S\. Maleki, and R\. Bianchini \(2024\)Splitwise: efficient generative LLM inference using phase splitting\.InACM/IEEE Annual International Symposium on Computer Architecture,pp\. 118–132\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.23.1),[Appendix D](https://arxiv.org/html/2607.08057#A4.p2.1),[§3\.3\.1](https://arxiv.org/html/2607.08057#S3.SS3.SSS1.p1.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p1.1)\.
- R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. Dean \(2023\)Efficiently scaling transformer inference\.Proceedings of Machine Learning and Systems5,pp\. 606–624\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p2.1)\.
- R\. Prabhu, A\. Nayak, J\. Mohan, R\. Ramjee, and A\. Panwar \(2025\)vAttention: dynamic memory management for serving LLMs without PagedAttention\.InACM International Conference on Architectural Support for Programming Languages and Operating Systems,pp\. 1133–1150\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.20.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- R\. Qin, Z\. Li, W\. He, M\. Zhang, Y\. Wu, W\. Zheng, and X\. Xu \(2024\)Mooncake: a KVCache\-centric disaggregated architecture for LLM serving\.arXiv preprint arXiv:2407\.00079\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.14.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I1.i2.p1.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1),[§3\.3\.1](https://arxiv.org/html/2607.08057#S3.SS3.SSS1.p1.1)\.
- Z\. Qin, Y\. Cao, M\. Lin, W\. Hu, S\. Fan, K\. Cheng, W\. Lin, and J\. Li \(2025\)CAKE: cascading and adaptive KV cache eviction with layer preferences\.InInternational Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.25.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.16.16.16.2)\.
- H\. Qiu, W\. Mao, A\. Patke, S\. Cui, S\. Jha, C\. Wang, H\. Franke, Z\. Kalbarczyk, T\. Başar, and R\. K\. Iyer \(2024\)Power\-aware deep learning model serving withμ\\mu\-serve\.InUSENIX Annual Technical Conference,pp\. 75–93\.Cited by:[Appendix B](https://arxiv.org/html/2607.08057#A2.p1.1)\.
- A\. Radford, K\. Narasimhan, T\. Salimans, and I\. Sutskever \(2018\)Improving language understanding by generative pre\-training\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- S\. Rajbhandari, J\. Rasley, O\. Ruwase, and Y\. He \(2020\)ZeRO: memory optimizations toward training trillion parameter models\.InInternational Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–16\.Cited by:[Appendix B](https://arxiv.org/html/2607.08057#A2.p1.1)\.
- Ray \(2024\)LLMPerf\.Note:[https://github\.com/ray\-project/llmperf](https://github.com/ray-project/llmperf)Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.Px1.p1.1)\.
- V\. J\. Reddi, C\. Cheng, D\. Kanter, P\. Mattson, G\. Schmuelling, C\. Wu, B\. Anderson, M\. Breughe, M\. Charlebois, W\. Chou, R\. Chukka, C\. Coleman, S\. Davis, P\. Deng, G\. Diamos, J\. Duke, D\. Fick, J\. S\. Gardner, I\. Hubara, S\. Idgunji, T\. B\. Jablin, J\. Jiao, T\. St\. John, P\. Kanwar, D\. Lee, J\. Liao, A\. Lokhmotov, F\. Massa, P\. Meng, P\. Micikevicius, C\. Osborne, G\. Pekhimenko, A\. T\. R\. Rajan, D\. Sequeira, A\. Sirasao, F\. Sun, H\. Tang, M\. Thomson, F\. Wei, E\. Wu, L\. Xu, K\. Yamada, B\. Yu, G\. Yuan, A\. Zhong, P\. Zhang, and Y\. Zhou \(2020\)MLPerf inference benchmark\.InACM/IEEE Annual International Symposium on Computer Architecture,pp\. 446–459\.Cited by:[§G\.3\.1](https://arxiv.org/html/2607.08057#A7.SS3.SSS1.Px2.p1.1)\.
- S\. Ren and K\. Q\. Zhu \(2024\)On the efficacy of eviction policy for key\-value constrained generative language model inference\.arXiv preprint arXiv:2402\.06262\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.10.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.26.1)\.
- L\. Ribar, I\. Chelombiev, L\. Hudlass\-Galley, C\. Blake, C\. Luschi, and D\. Orr \(2024\)SparQ Attention: bandwidth\-efficient LLM inference\.InInternational Conference on Machine Learning,pp\. 42558–42583\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.37.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p3.1)\.
- U\. Saxena, G\. Saha, S\. Choudhary, and K\. Roy \(2024\)Eigen Attention: attention in low\-rank space for KV cache compression\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 15332–15344\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.55.1),[Table 4](https://arxiv.org/html/2607.08057#S5.T4.8.8.8.8.8.8.8.8.8.3)\.
- R\. Shahout, C\. Liang, S\. Xin, Q\. Lao, Y\. Cui, M\. Yu, and M\. Mitzenmacher \(2024\)Fast inference for augmented large language models\.arXiv preprint arXiv:2410\.18248\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.52.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I1.i2.p1.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1)\.
- A\. Shen, Z\. Li, and M\. Gao \(2024\)FastSwitch: optimizing context switching efficiency in fairness\-aware large language model serving\.arXiv preprint arXiv:2411\.18424\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.59.1),[3rd item](https://arxiv.org/html/2607.08057#A5.I3.i3.p1.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- Q\. Shen and J\. Zhang \(2026\)MFTFormer: meteorological\-frequency\-temporal transformer with block\-aligned fusion for traffic flow prediction\.Research Square\.Note:Preprint, doi:10\.21203/rs\.3\.rs\-8770196/v1Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p1.1)\.
- Y\. Sheng, L\. Zheng, B\. Yuan, Z\. Li, M\. Ryabinin, B\. Chen, P\. Liang, C\. Ré, I\. Stoica, and C\. Zhang \(2023\)FlexGen: high\-throughput generative inference of large language models with a single gpu\.InInternational Conference on Machine Learning,pp\. 31094–31116\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.5.1),[1st item](https://arxiv.org/html/2607.08057#A5.I3.i1.p1.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.4.1),[§5\.1\.1](https://arxiv.org/html/2607.08057#S5.SS1.SSS1.p1.1)\.
- D\. Shi, Y\. Fu, X\. Yuan, Z\. Yu, H\. You, S\. Li, X\. Dong, J\. Kautz, P\. Molchanov, and Y\. Lin \(2025\)LaCache: ladder\-shaped KV caching for efficient long\-context modeling of large language models\.InInternational Conference on Machine Learning,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.39.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.31.1)\.
- L\. Shi, H\. Zhang, Y\. Yao, Z\. Li, and H\. Zhao \(2024\)Keep the cost down: a review on methods to optimize LLM’s KV\-cache consumption\.arXiv preprint arXiv:2407\.18003\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.9.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p2.1),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.7.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.
- P\. Singhania, S\. Singh, S\. He, S\. Feizi, and A\. Bhatele \(2024\)Loki: low\-rank keys for efficient sparse attention\.Advances in Neural Information Processing Systems37,pp\. 16692–16723\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.11.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p3.1)\.
- D\. Son, E\. Choi, and S\. Yoo \(2025\)NSNQuant: a double normalization approach for calibration\-free low\-bit vector quantization of KV cache\.Advances in Neural Information Processing Systems\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.56.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I4.i2.p1.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.23.1),[§5\.1\.1](https://arxiv.org/html/2607.08057#S5.SS1.SSS1.p3.1)\.
- V\. Srivatsa, Z\. He, R\. Abhyankar, D\. Li, and Y\. Zhang \(2024\)Preble: efficient distributed prompt scheduling for llm serving\.InInternational Conference on Learning Representations,Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.18.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1)\.
- F\. Strati, S\. McAllister, A\. Phanishayee, J\. Tarnawski, and A\. Klimovic \(2024\)Déjàvu: KV\-cache streaming for fast, fault\-tolerant generative LLM serving\.InInternational Conference on Machine Learning,pp\. 46745–46771\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.38.1),[§3\.3\.1](https://arxiv.org/html/2607.08057#S3.SS3.SSS1.p1.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1)\.
- Y\. Su, Y\. Zhou, Q\. Qiu, J\. Li, Q\. Xia, P\. Li, X\. Duan, Z\. Wang, and M\. Zhang \(2025\)Accurate KV cache quantization with outlier tokens tracing\.InAnnual Meeting of the Association for Computational Linguistics,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.44.1),[1st item](https://arxiv.org/html/2607.08057#A5.I4.i1.p1.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.22.1)\.
- H\. Sun, L\. Chang, W\. Bao, S\. Zheng, N\. Zheng, X\. Liu, H\. Dong, Y\. Chi, and B\. Chen \(2025\)ShadowKV: KV cache in shadows for high\-throughput long\-context LLM inference\.InInternational Conference on Machine Learning,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.36.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2607.08057#S5.T4.3.3.3.3.3.3.3.3.3.2)\.
- J\. Sun, S\. Wang, Z\. Zhang, Z\. Liu, Y\. Xu, P\. Sun, B\. Zhao, B\. He, F\. Wu, and Z\. Wang \(2026\)Bat: efficient generative recommender serving with bipartite attention\.InACM International Conference on Architectural Support for Programming Languages and Operating Systems,pp\. 223–238\.Cited by:[§2](https://arxiv.org/html/2607.08057#S2.SS0.SSS0.Px2.p3.1)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017\)Axiomatic attribution for deep networks\.InInternational conference on machine learning,pp\. 3319–3328\.Cited by:[3rd item](https://arxiv.org/html/2607.08057#A5.I5.i3.p1.1)\.
- I\. O\. Sunmola, Z\. Zhao, S\. Schmidgall, Y\. Wang, P\. M\. Scheikl, and A\. Krieger \(2025\)Surgical gaussian surfels: highly accurate real\-time surgical scene rendering\.arXiv preprint arXiv:2503\.04079\.Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p1.1)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)QUEST: query\-aware sparsity for efficient long\-context LLM inference\.InInternational Conference on Machine Learning,pp\. 47901–47911\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.36.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p3.1)\.
- H\. Tian, F\. Liu, K\. C\. Cheung, Z\. Fang, S\. See, T\. Liu, and B\. Han \(2026\)Cross\-domain few\-shot classification via invariant\-content feature reconstruction\.International Journal of Computer Vision134\(2\),pp\. 54\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- H\. Tian, F\. Liu, T\. Liu, B\. Du, Y\. Cheung, and B\. Han \(2024\)MOKD: cross\-domain finetuning for few\-shot classification via maximizing optimized kernel dependence\.InInternational Conference on Machine Learning,pp\. 48154–48185\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample \(2023a\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- H\. Touvron, L\. Martin, K\. R\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. M\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. S\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. M\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. H\. M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023b\)LLaMA 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in Neural Information Processing Systems30\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- Z\. Wan, X\. Wu, Y\. Zhang, Y\. Xin, C\. Tao, Z\. Zhu, X\. Wang, S\. Luo, J\. Xiong, and M\. Zhang \(2025\)D2O: dynamic discriminative operations for efficient generative inference of large language models\.InInternational Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.1.1),[5th item](https://arxiv.org/html/2607.08057#A5.I5.i5.p1.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.6.6.6.6.6.6.6.6.6.2),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.17.17.17.1)\.
- G\. Wang, Z\. Liu, B\. Hsieh, S\. Zhuang, J\. Gonzalez, T\. Darrell, and I\. Stoica \(2021\)sensAI: convnets decomposition via class parallelism for fast inference on live data\.Proceedings of Machine Learning and Systems3,pp\. 664–679\.Cited by:[Appendix B](https://arxiv.org/html/2607.08057#A2.p1.1)\.
- H\. Wang, L\. Han, K\. Xu, and A\. Srivastava \(2025a\)SQuat: subspace\-orthogonal KV cache quantization\.arXiv preprint arXiv:2503\.24358\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.19.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.19.1)\.
- Z\. Wang, B\. Jin, Z\. Yu, and M\. Zhang \(2024a\)Model tells you where to merge: adaptive KV cache merging for LLMs on long\-context tasks\.arXiv preprint arXiv:2407\.08454\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.30.1),[§5\.1\.3](https://arxiv.org/html/2607.08057#S5.SS1.SSS3.p1.1.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.2.2.2.2.2.2.2.2.2.4)\.
- Z\. Wang, S\. Li, Y\. Zhou, X\. Li, R\. Gu, N\. Cam\-Tu, C\. Tian, and S\. Zhong \(2024b\)Revisiting SLO and goodput metrics in LLM serving\.arXiv preprint arXiv:2410\.14257\.Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p2.1)\.
- Z\. Wang, Y\. Sun, H\. Wang, B\. Jing, X\. Shen, X\. Dong, Z\. Hao, H\. Xiong, and Y\. Song \(2025b\)Reasoning\-enhanced domain\-adaptive pretraining of multimodal large language models for short video content governance\.InConference on Empirical Methods in Natural Language Processing: Industry Track,S\. Potdar, L\. Rojas\-Barahona, and S\. Montella \(Eds\.\),pp\. 1104–1112\.Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p1.1)\.
- B\. Wu, S\. Liu, Y\. Zhong, P\. Sun, X\. Liu, and X\. Jin \(2024a\)LoongServe: efficiently serving long\-context large language models with elastic sequence parallelism\.InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles,pp\. 640–654\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.54.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1)\.
- B\. Wu, Y\. Zhong, Z\. Zhang, S\. Liu, F\. Liu, Y\. Sun, G\. Huang, X\. Liu, and X\. Jin \(2023\)Fast distributed inference serving for large language models\.Advances in Neural Information Processing Systems\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.7.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p1.1)\.
- W\. Wu, Z\. Pan, C\. Wang, L\. Chen, Y\. Bai, T\. Wang, K\. Fu, Z\. Wang, and H\. Xiong \(2025\)TokenSelect: efficient long\-context inference and length extrapolation for LLMs via dynamic token\-level KV cache selection\.InConference on Empirical Methods in Natural Language Processing,pp\. 21275–21292\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.51.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p3.1)\.
- X\. Wu, R\. Xian, T\. Guan, J\. Liang, S\. Chakraborty, F\. Liu, B\. M\. Sadler, D\. Manocha, and A\. Bedi \(2024b\)On the safety concerns of deploying LLMs/VLMs in robotics: highlighting the risks and vulnerabilities\.InVision and Language for Autonomous Driving and Robotics Workshop,Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p1.1)\.
- C\. Xiao, P\. Zhang, X\. Han, G\. Xiao, Y\. Lin, Z\. Zhang, Z\. Liu, and M\. Sun \(2024a\)InfLLM: training\-free long\-context extrapolation for LLMs with an efficient context memory\.Advances in Neural Information Processing Systems37,pp\. 119638–119661\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.9.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.InInternational Conference on Machine Learning,pp\. 38087–38099\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.4.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.3.1),[§5\.1\.1](https://arxiv.org/html/2607.08057#S5.SS1.SSS1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024b\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.17.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I5.i2.p1.1),[1st item](https://arxiv.org/html/2607.08057#S5.I3.i1.p1.2),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.7.7.7.3)\.
- W\. Xiao, R\. Bhardwaj, R\. Ramjee, M\. Sivathanu, N\. Kwatra, Z\. Han, P\. Patel, X\. Peng, H\. Zhao, Q\. Zhang, F\. Yang, and L\. Zhou \(2018\)Gandiva: introspective cluster scheduling for deep learning\.InUSENIX Symposium on Operating Systems Design and Implementation,pp\. 595–610\.Cited by:[Appendix B](https://arxiv.org/html/2607.08057#A2.p1.1)\.
- Y\. Xiong, H\. Wu, C\. Shao, Z\. Wang, R\. Zhang, Y\. Guo, J\. Zhao, K\. Zhang, and Z\. Pan \(2024\)LayerKV: optimizing large language model serving with layer\-wise KV cache management\.arXiv preprint arXiv:2410\.00428\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.49.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1)\.
- F\. Xu, T\. Goyal, and E\. Choi \(2025a\)RefreshKV: updating small KV cache during long\-form generation\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 24878–24893\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.43.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p3.1)\.
- J\. Xu, R\. Zhang, C\. Guo, W\. Hu, Z\. Liu, F\. Wu, Y\. Feng, S\. Sun, C\. Shao, Y\. Guo, J\. Zhao, K\. Zhang, M\. Guo, and J\. Leng \(2024a\)vTensor: flexible virtual tensor management for efficient LLM serving\.arXiv preprint arXiv:2407\.15309\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.31.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- M\. Xu, W\. Yin, D\. Cai, R\. Yi, D\. Xu, Q\. Wang, B\. Wu, Y\. Zhao, C\. Yang, S\. Wang, Q\. Zhang, Z\. Lu, L\. Zhang, S\. Wang, Y\. Li, Y\. Liu, X\. Jin, and X\. Liu \(2024b\)A survey of resource\-efficient LLM and multimodal foundation models\.arXiv preprint arXiv:2401\.08092\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.8.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p1.1.3)\.
- Y\. Xu, Z\. Jie, H\. Dong, L\. Wang, X\. Lu, A\. Zhou, A\. Saha, C\. Xiong, and D\. Sahoo \(2025b\)ThinK: thinner key cache by query\-driven pruning\.InInternational Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.26.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.8.8.8.8.8.8.8.8.8.2)\.
- S\. Yan, Y\. Wang, K\. Zhao, P\. Shi, Z\. Zhao, Y\. Zhang, and J\. Li \(2025a\)HeMoRa: unsupervised heuristic consensus sampling for robust point cloud registration\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 1363–1373\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- X\. Yan, Z\. Li, T\. Zhang, L\. Kong, Y\. Zhang, and X\. Yang \(2025b\)ReCalKV: low\-rank KV cache compression via head reordering and offline calibration\.arXiv preprint arXiv:2505\.24357\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.33.1),[Table 4](https://arxiv.org/html/2607.08057#S5.T4.2.2.2.2.2.2.2.2.2.2)\.
- D\. Yang, X\. Han, Y\. Gao, Y\. Hu, S\. Zhang, and H\. Zhao \(2024a\)PyramidInfer: pyramid KV cache compression for high\-throughput LLM inference\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 3258–3270\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.42.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.9.9.9.2)\.
- J\. Y\. Yang, B\. Kim, J\. Bae, B\. Kwon, G\. Park, E\. Yang, S\. J\. Kwon, and D\. Lee \(2024b\)No token left behind: reliable KV cache compression via importance\-aware mixed precision quantization\.arXiv preprint arXiv:2402\.18096\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.12.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.6.1)\.
- P\. Yang, N\. Akhtar, J\. Jiang, and A\. Mian \(2026\)Attribution\-guided model rectification of unreliable neural network behaviors\.arXiv preprint arXiv:2603\.15656\.Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p4.1)\.
- P\. Yang, N\. Akhtar, M\. Shah, and A\. Mian \(2024c\)Regulating model reliance on non\-robust features by smoothing input marginal density\.InEuropean Conference on Computer Vision,pp\. 329–347\.Cited by:[§6\.2](https://arxiv.org/html/2607.08057#S6.SS2.p4.1)\.
- P\. Yang, N\. Akhtar, Z\. Wen, and A\. Mian \(2023a\)Local path integration for attribution\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 3173–3180\.Cited by:[3rd item](https://arxiv.org/html/2607.08057#A5.I5.i3.p1.1)\.
- P\. Yang, N\. Akhtar, Z\. Wen, M\. Shah, and A\. S\. Mian \(2023b\)Re\-calibrating feature attributions for model interpretation\.InInternational Conference on Learning Representations,Cited by:[3rd item](https://arxiv.org/html/2607.08057#A5.I5.i3.p1.1)\.
- X\. Yang, Q\. Hu, J\. Li, F\. Li, Y\. Zhu, Y\. Zhou, Q\. Lin, J\. Dai, Y\. Kong, J\. Zhang, G\. Xu, and Q\. Liu \(2025\)Beluga: a CXL\-based memory architecture for scalable and efficient LLM KVCache management\.arXiv preprint arXiv:2511\.20172\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.61.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1)\.
- Y\. Yang, Z\. Cao, Q\. Chen, L\. Qin, D\. Yang, H\. Zhao, and Z\. Chen \(2024d\)KVSharer: efficient inference via layer\-wise dissimilar KV cache sharing\.arXiv preprint arXiv:2410\.18517\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.51.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.4.4.4.4.4.4.4.4.4.2)\.
- L\. Ye, Z\. Tao, Y\. Huang, and Y\. Li \(2024\)ChunkAttention: efficient self\-attention with prefix\-aware KV cache and two\-phase partition\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 11608–11620\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.43.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- Z\. Ye, L\. Chen, R\. Lai, W\. Lin, Y\. Zhang, S\. Wang, T\. Chen, B\. Kasikci, V\. Grover, A\. Krishnamurthy, and L\. Ceze \(2025\)FlashInfer: efficient and customizable attention engine for LLM inference serving\.Proceedings of Machine Learning and Systems\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.30.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p3.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- D\. Yoon, Y\. Min, H\. Kim, S\. H\. Noh, and J\. Kim \(2025\)TraCT: disaggregated LLM serving with CXL shared memory KV cache at rack\-scale\.arXiv preprint arXiv:2512\.18194\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.58.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p2.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p1.1)\.
- B\. Yu and Y\. Chai \(2025\)EvolKV: evolutionary KV cache compression for LLM inference\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 1673–1689\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.52.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.33.1)\.
- C\. Yu, T\. Wang, Z\. Shao, L\. Zhu, X\. Zhou, and S\. Jiang \(2024\)TwinPilots: a new computing paradigm for GPU\-CPU parallel LLM inference\.InACM International Systems and Storage Conference,pp\. 91–103\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.45.1)\.
- L\. Yu, J\. Lin, and J\. Li \(2025\)Stateful large language model serving with pensieve\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 144–158\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.22.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Yuan, H\. Gao, D\. Dai, J\. Luo, L\. Zhao, Z\. Zhang, Z\. Xie, Y\. Wei, L\. Wang, Z\. Xiao, Y\. Wang, C\. Ruan, M\. Zhang, W\. Liang, and W\. Zeng \(2025\)Native sparse attention: hardware\-aligned and natively trainable sparse attention\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 23078–23097\.Cited by:[§2](https://arxiv.org/html/2607.08057#S2.SS0.SSS0.Px2.p3.1)\.
- Z\. Yuan, Y\. Shang, Y\. Zhou, Z\. Dong, C\. Xue, B\. Wu, Z\. Li, Q\. Gu, Y\. J\. Lee, Y\. Yan, B\. Chen, G\. Sun, and K\. Keutzer \(2024\)LLM inference unveiled: survey and roofline model insights\.arXiv preprint arXiv:2402\.16363\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.3.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p1.1.3),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.3.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.
- Y\. Yue, Z\. Yuan, H\. Duanmu, S\. Zhou, J\. Wu, and L\. Nie \(2024\)WKVQuant: quantizing weight and key/value cache for large language models gains more\.arXiv preprint arXiv:2402\.12065\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.11.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.5.1)\.
- A\. C\. Yüzügüler, J\. Zhuang, and L\. Cavigelli \(2025\)PRESERVE: prefetching model weights and KV\-cache in distributed LLM serving\.arXiv preprint arXiv:2501\.08192\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.32.1),[4th item](https://arxiv.org/html/2607.08057#A5.I1.i4.p1.1),[Table 2](https://arxiv.org/html/2607.08057#S3.T2.14.14.14.4),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px2.p1.1)\.
- A\. Zandieh, M\. Daliri, and I\. Han \(2025\)QJL: 1\-bit quantized JL transform for KV cache quantization with zero overhead\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 25805–25813\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.16.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.17.1)\.
- H\. Zeng, Y\. Li, R\. Niu, C\. Yang, and S\. Wen \(2025a\)Enhancing spatiotemporal prediction through the integration of mamba state space models and diffusion transformers\.Knowledge\-Based Systems\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- S\. Zeng, X\. Chang, M\. Xie, X\. Liu, Y\. Bai, Z\. Pan, M\. Xu, X\. Wei, and N\. Guo \(2025b\)Futuresightdrive: thinking visually with spatio\-temporal cot for autonomous driving\.arXiv preprint arXiv:2505\.17685\.Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p1.1)\.
- S\. Zeng, D\. Qi, X\. Chang, F\. Xiong, S\. Xie, X\. Wu, S\. Liang, M\. Xu, and X\. Wei \(2025c\)JanusVLN: decoupling semantics and spatiality with dual implicit memory for vision\-language navigation\.arXiv preprint arXiv:2509\.22548\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- Y\. Zeng, W\. Yu, Z\. Li, T\. Ren, Y\. Ma, J\. Cao, X\. Chen, and T\. Yu \(2025d\)Bridging the editing gap in LLMs: FineEdit for precise and targeted text modifications\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 2193–2206\.Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1)\.
- H\. Zhang, X\. Ji, Y\. Chen, F\. Fu, X\. Miao, X\. Nie, W\. Chen, and B\. Cui \(2025a\)PQCache: product quantization\-based KVCache for long context LLM inference\.Proceedings of the ACM on Management of Data3\(3\),pp\. 1–30\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.35.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Zhang, W\. Zhang, C\. Tan, X\. Li, and Q\. Sun \(2024a\)YOLO\-PPA based efficient traffic sign detection for cruise control in autonomous driving\.InInternational Conference on Industrial Automation and Robotics,pp\. 8–13\.Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p1.1)\.
- M\. Zhang, Z\. Fang, T\. Wang, S\. Lu, X\. Wang, and T\. Shi \(2025b\)CCMA: a framework for cascading cooperative multi\-agent in autonomous driving merging using large language models\.Expert Systems with Applications,pp\. 127717\.Cited by:[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p2.1.1)\.
- M\. Zhang, H\. Sun, J\. Wang, S\. Li, W\. Ning, Q\. Qi, Z\. Zhuang, and J\. Liao \(2025c\)ClusterAttn: KV cache compression under intrinsic attention clustering\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 14451–14473\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.42.1),[§5\.1\.3](https://arxiv.org/html/2607.08057#S5.SS1.SSS3.p1.1.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.9.9.9.9.9.9.9.9.9.2)\.
- R\. Zhang, K\. Wang, L\. Liu, S\. Wang, H\. Cheng, C\. Zhang, and Y\. Shen \(2024b\)LoRC: low\-rank compression for LLMs KV cache with a progressive compression strategy\.arXiv preprint arXiv:2410\.03111\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.47.1),[Table 4](https://arxiv.org/html/2607.08057#S5.T4.6.6.6.6.6.6.6.6.6.4)\.
- R\. Zhang and Z\. Bian \(2025\)Overview of multimodal generation for recommender systems\.Journal of Frontiers of Computer Science and Technology19\(12\),pp\. 3224–19\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- R\. Zhang, C\. Liu, Y\. Su, R\. Li, X\. Huang, X\. Li, and P\. S\. Yu \(2025d\)A comprehensive survey on multimodal RAG: all combinations of modalities as input and output\.Authorea Preprints\.External Links:[Document](https://dx.doi.org/10.36227/techrxiv.176341513.38473003/v2),[Link](https://www.techrxiv.org/doi/full/10.36227/techrxiv.176341513.38473003/v2)Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. T\. Diab, X\. Li, X\. V\. Lin, T\. Mihaylov, M\. Ott, S\. Shleifer, K\. Shuster, D\. Simig, P\. S\. Koura, A\. Sridhar, T\. Wang, and L\. Zettlemoyer \(2022\)OPT: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- T\. Zhang, J\. Yi, Z\. Xu, and A\. Shrivastava \(2024c\)KV cache is 1 bit per channel: efficient large language model inference with coupled quantization\.Advances in Neural Information Processing Systems37,pp\. 3304–3331\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.5.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I4.i2.p1.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.15.1),[§5\.1\.1](https://arxiv.org/html/2607.08057#S5.SS1.SSS1.p3.1)\.
- Y\. Zhang, Y\. Hu, R\. Zhao, J\. C\. Lui, and H\. Chen \(2025e\)DiffKV: differentiated memory management for large language models with parallel KV compaction\.InACM SIGOPS Symposium on Operating Systems Principles,pp\. 431–445\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.50.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.23.23.23.2)\.
- Y\. Zhang, Y\. Du, G\. Luo, Y\. Zhong, Z\. Zhang, S\. Liu, and R\. Ji \(2024d\)CaM: cache merging for memory\-efficient LLMs inference\.InInternational Conference on Machine Learning,pp\. 58840–58850\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.34.1),[Table 5](https://arxiv.org/html/2607.08057#S5.T5.3.3.3.3.3.3.3.3.3.2)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. W\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.1.1),[2nd item](https://arxiv.org/html/2607.08057#A5.I5.i2.p1.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.1.1.1.1)\.
- J\. Zhao, Z\. Fang, S\. Li, S\. Yang, and S\. He \(2024a\)BUZZ: beehive\-structured sparse KV cache with segmented heavy hitters for efficient LLM inference\.arXiv preprint arXiv:2410\.23079\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.53.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.12.12.12.4)\.
- W\. X\. Zhao, K\. Zhou, J\. Li, T\. Tang, X\. Wang, Y\. Hou, Y\. Min, B\. Zhang, J\. Zhang, Z\. Dong, Y\. Du, C\. Yang, Y\. Chen, Z\. Chen, J\. Jiang, R\. Ren, Y\. Li, X\. Tang, Z\. Liu, P\. Liu, J\. Nie, and J\. Wen \(2023\)A survey of large language models\.arXiv preprint arXiv:2303\.18223\.Cited by:[§1](https://arxiv.org/html/2607.08057#S1.p1.1)\.
- Y\. Zhao, C\. Lin, K\. Zhu, Z\. Ye, L\. Chen, S\. Zheng, L\. Ceze, A\. Krishnamurthy, T\. Chen, and B\. Kasikci \(2024b\)Atom: low\-bit quantization for efficient and accurate LLM serving\.Proceedings of Machine Learning and Systems6,pp\. 196–209\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.20.1),[Table 3](https://arxiv.org/html/2607.08057#S4.T3.3.1.8.1)\.
- Y\. Zhao, D\. Wu, and J\. Wang \(2024c\)ALISA: accelerating large language model inference via sparsity\-aware KV caching\.InACM/IEEE Annual International Symposium on Computer Architecture,pp\. 1005–1017\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.24.1),[§4\.1](https://arxiv.org/html/2607.08057#S4.SS1.SSS0.Px1.p1.1)\.
- R\. Zhen, J\. Li, Y\. Ji, Z\. Yang, T\. Liu, Q\. Xia, X\. Duan, Z\. Wang, B\. Huai, and M\. Zhang \(2025\)Taming the titans: a survey of efficient LLM inference serving\.InInternational Natural Language Generation Conference,pp\. 522–541\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.6.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p1.1.3),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.6.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. Gonzalez, C\. W\. Barrett, and Y\. Sheng \(2024\)SGLang: efficient execution of structured language model programs\.Advances in Neural Information Processing Systems37,pp\. 62557–62583\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.10.1),[§3\.1](https://arxiv.org/html/2607.08057#S3.SS1.p2.1),[§5\.2\.1](https://arxiv.org/html/2607.08057#S5.SS2.SSS1.p1.1)\.
- Y\. Zhong, S\. Liu, J\. Chen, J\. Hu, Y\. Zhu, X\. Liu, X\. Jin, and H\. Zhang \(2024\)DistServe: disaggregating prefill and decoding for goodput\-optimized large language model serving\.InUSENIX Symposium on Operating Systems Design and Implementation,pp\. 193–210\.Cited by:[Table 9](https://arxiv.org/html/2607.08057#A4.T9.1.25.1),[3rd item](https://arxiv.org/html/2607.08057#A5.I3.i3.p1.1),[§3\.3\.1](https://arxiv.org/html/2607.08057#S3.SS3.SSS1.p1.1),[§4\.2](https://arxiv.org/html/2607.08057#S4.SS2.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024a\)WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Cited by:[item 3](https://arxiv.org/html/2607.08057#A7.I2.i3.p1.1),[§G\.1](https://arxiv.org/html/2607.08057#A7.SS1.p2.1.1)\.
- X\. Zhou, W\. Wang, M\. Zeng, J\. Guo, X\. Liu, L\. Shen, M\. Zhang, and L\. Ding \(2025\)DynamicKV: task\-aware adaptive KV cache compression for long context LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 8042–8057\.Cited by:[Table 10](https://arxiv.org/html/2607.08057#A4.T10.1.53.1),[Table 6](https://arxiv.org/html/2607.08057#S5.T6.24.24.24.2)\.
- Z\. Zhou, X\. Ning, K\. Hong, T\. Fu, J\. Xu, S\. Li, Y\. Lou, L\. Wang, Z\. Yuan, X\. Li, S\. Yan, G\. Dai, X\. Zhang, Y\. Dong, and Y\. Wang \(2024b\)A survey on efficient inference for large language models\.arXiv preprint arXiv:2404\.14294\.Cited by:[Table 8](https://arxiv.org/html/2607.08057#A2.T8.1.5.1),[Appendix C](https://arxiv.org/html/2607.08057#A3.p1.1.3),[Table 1](https://arxiv.org/html/2607.08057#S1.T1.3.1.5.1),[§1](https://arxiv.org/html/2607.08057#S1.p4.1.1)\.

## Appendix APreliminaries on LLMs

LLMs are built from stacked Transformer blocks, each with multi\-head self\-attention \(MHSA\) and feed\-forward network \(FFN\)\. These blocks are sequential, where the output of one block serves as the input to the next\. For theii\-th attention head, MHSA applies learned projections to the input featuresXXto get queries, keys, and values:

Qi=X​WQi,Ki=X​WKi,Vi=X​WVi\.Q\_\{i\}=XW^\{Q\_\{i\}\},K\_\{i\}=XW^\{K\_\{i\}\},V\_\{i\}=XW^\{V\_\{i\}\}\.Then the self\-attention operation is applied to each tuple \(QiQ\_\{i\},KiK\_\{i\},ViV\_\{i\}\) and get the output ofZiZ\_\{i\}:

Zi=attention​\(Qi,Ki,Vi\)=softmax​\(Qi​Ki⊤dk\)​Vi,Z\_\{i\}=\\mathrm\{attention\}\(Q\_\{i\},K\_\{i\},V\_\{i\}\)=\\mathrm\{softmax\}\(\\frac\{Q\_\{i\}K\_\{i\}^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\)V\_\{i\},wheredkd\_\{k\}is the dimension of the keys\. Finally, outputs of all the attention heads are concatenated:

Z=concat​\(Z1,Z2,…,Zh\)​WO,Z=\\mathrm\{concat\}\(Z\_\{1\},Z\_\{2\},\.\.\.,Z\_\{h\}\)W^\{O\},whereWOW^\{O\}is the trainable parameters\. Following this, the output of MHSA is fed into the FFN module, which applies two linear transformations with a nonlinear activation \(e\.g\., ReLU\):

FFN​\(x\)=ReLU​\(x​W1\+b1\)​W2\+b2,\\mathrm\{FFN\}\(x\)=\\mathrm\{ReLU\}\(xW\_\{1\}\+b\_\{1\}\)W\_\{2\}\+b\_\{2\},whereW1W\_\{1\},W2W\_\{2\},b1b\_\{1\}, andb2b\_\{2\}are learnable parameters of the FFN\. These modules together enable contextualized autoregressive modeling in LLMs\.

## Appendix BDesign of Our Taxonomy

Our taxonomy follows a system behavior\-oriented view of sKis introduced in §[2](https://arxiv.org/html/2607.08057#S2)and respects the domain boundary\. Specifically, we classify techniques by their operational impact along temporal, spatial, and structural dimensions\. This behavior\-oriented perspective follows established practice in machine learning systems researchXiaoet al\.\([2018](https://arxiv.org/html/2607.08057#bib.bib170)\); Rajbhandariet al\.\([2020](https://arxiv.org/html/2607.08057#bib.bib169)\); Wanget al\.\([2021](https://arxiv.org/html/2607.08057#bib.bib172)\); Jianget al\.\([2022](https://arxiv.org/html/2607.08057#bib.bib167)\); Qiuet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib171)\); Jianget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib168)\); Liet al\.\([2026](https://arxiv.org/html/2607.08057#bib.bib208)\)and aligns closely with how serving systems are actually built and optimized in practice, allowing diverse methods to be interpreted under a unified framework\.

For example, many methods perform KV cache*selection*by identifying tokens \(i\.e\., KV entries\) that are more or less important for future computation\. In our taxonomy, we do not treat selection itself as a category\. In contrast, we classify methods based on the system action taken after selection: \(i\) If unimportant KV entries are permanently discarded to free GPU memory, the method is categorized as KV cache eviction \(cf\. §[5\.2\.2](https://arxiv.org/html/2607.08057#S5.SS2.SSS2)\) under the structural dimension; \(ii\) If unimportant KV entries are offloaded to secondary storage \(e\.g\., CPU RAM\) for possible future retrieval and reload, the method falls under memory hierarchy KV orchestration \(cf\. §[4\.1](https://arxiv.org/html/2607.08057#S4.SS1)\) under the spatial dimension; and \(iii\) If the tokens are retained in GPU memory but excluded from computation, the method is considered token\-level scheduling, which is categorized as KV\-centric scheduling \(cf\. §[3\.1](https://arxiv.org/html/2607.08057#S3.SS1)\) under the temporal dimension\.

Table 8:Comparison of scope and taxonomy with existing surveys related to efficient LLM inference or serving\.SurveyKV\-centricServing onlyNo retrainSystem metricsOrganizing principleMiaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib7)\)✓✓Algorithm\-, system\-levelYuanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib4)\)✓✓Optimization layer \(parameter\-, algorithm\-, system\-, hardware\-level\)Liet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib5)\)✓✓✓System component \(KV cache and memory, computation, cloud deployment, emerging research fields\)Zhouet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib3)\)✓✓Optimization layer \(data\-, model\-, system\-level\)Zhenet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib11)\)✓✓✓Serving scale \(instance\-, cluster\-level, emerging scenarios\)Baiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib9)\)✓Lifecycle \(architecture design, pre\-training, fine\-tuning, inference, system design\)Xuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib10)\)✓Optimization layered \(architecture, algorithm, systems\)Shiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib12)\)✓✓Lifecycle \(training, deploy, post\-training\)Liet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib13)\)✓✓✓Optimization layer \(token\-, model\-, system\-level\)Liuet al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib14)\)✓✓✓KV compression types \(selective token, quantization, attention compression, hybrid\)This survey \(sKis\)✓✓✓✓System behaviors \(temporal, spatial, structural dimensions\)
## Appendix CRelated Surveys

Several recent surveys have covered the areas of efficient LLM inference and serving\.Miaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib7)\)explored both algorithmic innovations and system architectures for efficient LLM serving,Yuanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib4)\)analyzed LLM inference techniques through a Roofline\-based framework,Zhouet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib3)\)organized efficient LLM inference methods across data\-, model\-, and system\-level optimizations,Liet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib5)\)examined system\-level enhancements for LLM inference serving,Zhenet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib11)\)reviewed recent advances across different LLM serving scenarios, whileBaiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib9)\)andXuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib10)\)focused on resource\-efficient LLMs\.However, thesegeneral surveystypically treat KV cache optimization as a minor component within the broader pipelines\.

In contrast, dedicated surveys that focus on the KV cache remain rare\.Shiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib12)\)adopted a lifecycle\-based taxonomy spanning training\-stage, deploy\-stage, and post\-training optimizations\.Liet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib13)\)categorized KV cache management strategies into token\-level, model\-level, and system\-level optimizations\.Liuet al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib14)\)focused on compression strategies of the KV cache, such as selective token strategies, quantization, and attention compression\. TheseKV\-specific surveysare closest to our topic\. However, they mostly organize by lifecycle stages or abstraction levels, leaving the serving\-time behaviors of KV caches unexamined\.

Different from the above surveys, we concentrate exclusively on the sKis scope \(i\.e\., serving\-time, KV\-centric, system metrics, no retraining or architecture change\) and aim to provide a deeper understanding within this scope\. By classifying methods according to their impact along temporal, spatial, and structural dimensions, our survey enables cross\-behavior and behavior×\\timesobjective analysis, which complements prior surveys and clarifies actionable research gaps\. Table[8](https://arxiv.org/html/2607.08057#A2.T8)shows a summary\.

## Appendix DSupplementary Paper Categorization

Table 9:Full mapping of representative methods reviewed in this paper to their corresponding sKis categories\. Methods are chronologically ordered with publication venues\.MethodsVenueTaxonomy of sKisKV\-centric schedulingPipelining and overlappingHardware\-aware executionMemory hierarchy KV orchestrationCompute device KV orchestrationKV cache compressionKV cache retention managementSmoothQuantXiaoet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib101)\)ICML●FlexGenShenget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib27)\)ICML◐◐●●vLLMKwonet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib26)\)SOSP◐●FastServeWuet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib32)\)NeurIPS◐●●H2OZhanget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib117)\)NeurIPS●ScissorhandsLiuet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib121)\)NeurIPS●TetriInferHuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib34)\)arXiv●●◐RoCoRen and Zhu \([2024](https://arxiv.org/html/2607.08057#bib.bib125)\)arXiv●WKVQuantYueet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib86)\)arXiv●MiKVYanget al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib87)\)arXiv●FastDecodeHe and Zhai \([2024](https://arxiv.org/html/2607.08057#bib.bib30)\)arXiv●●◐◐QAQDonget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib88)\)arXiv●AttAccParket al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib74)\)ASPLOS●●FastGenGeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib118)\)ICLR●StreamingLLMXiaoet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib122)\)ICLR●PrebleSrivatsaet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib72)\)ICLR●◐◐KeyformerAdnanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib119)\)MLSys●AtomZhaoet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib93)\)MLSys●PromptCacheGimet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib54)\)MLSys●PyramidKVCaiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib126)\)arXiv●SplitwisePatelet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib31)\)ISCA◐●●ALISAZhaoet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib39)\)ISCA●◐◐DistServeZhonget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib33)\)OSDI◐●●Infinite\-LLMLinet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib35)\)arXiv◐●●InfiniGenLeeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib58)\)OSDI●CachedAttentionGaoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib50)\)ATC◐●●◐◐LazyLLMFuet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib42)\)arXiv●KVMergerWanget al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib142)\)arXiv●vTensorXuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib40)\)arXiv●KIVILiuet al\.\([2024d](https://arxiv.org/html/2607.08057#bib.bib91)\)ICML●CHAIAgarwalet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib138)\)ICML●CaMZhanget al\.\([2024d](https://arxiv.org/html/2607.08057#bib.bib143)\)ICML●MuxServeDuanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib52)\)ICML●●◐◐QuestTanget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib60)\)ICML●SparQAttentionRibaret al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib75)\)ICML●DéjàVuStratiet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib70)\)ICML●●◐◐CacheGenLiuet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib29)\)SIGCOMM◐●●DecoQuantLiuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib98)\)ACL●NACLChenet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib129)\)ACL●PyramidInferYanget al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib128)\)ACL●ChunkAttentionYeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib44)\)ACL●InstInferPanet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib68)\)arXiv◐●●◐TwinPilotsYuet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib36)\)SYSTOR◐●◐GEARKanget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib89)\)arXiv●LoRCZhanget al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib108)\)arXiv●SKVQDuanmuet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib94)\)COLM●LayerKVXionget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib49)\)arXiv●◐●◐CCompPark and Egger \([2024](https://arxiv.org/html/2607.08057#bib.bib38)\)PACT●●◐KVSharerYanget al\.\([2024d](https://arxiv.org/html/2607.08057#bib.bib144)\)arXiv●LAMPSShahoutet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib51)\)arXiv●◐◐BUZZZhaoet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib133)\)arXiv●LoongServeWuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib71)\)SOSP●◐◐◐EigenAttentionSaxenaet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib115)\)EMNLP●TOVAOrenet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib130)\)EMNLP●VATPGuoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib134)\)EMNLP●L2KVDevotoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib127)\)EMNLP●FastSwitchShenet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib48)\)arXiv◐◐●●
Continued on next page

- •●= primary category with main analysis in the paper;◐= secondary category omitted or only briefly mentioned in our paper to maintain focused classification\.

Table 10:Continued from previous pageMethodsVenueTaxonomy of sKisKV\-centric schedulingPipelining and overlappingHardware\-aware executionMemory hierarchy KV orchestrationCompute device KV orchestrationKV cache compressionKV cache retention managementKVQuantHooperet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib92)\)NeurIPS●CQZhanget al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib90)\)NeurIPS●ZipCacheHeet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib102)\)NeurIPS●SnapKVLiet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib120)\)NeurIPS●MiniCacheLiuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib145)\)NeurIPS●InfLLMXiaoet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib59)\)NeurIPS●◐RadixAttentionZhenget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib47)\)NeurIPS●●LokiSinghaniaet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib63)\)NeurIPS●ArkValeChenet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib184)\)NeurIPS●◐MemServeHuet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib45)\)arXiv●MooncakeQinet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib69)\)FAST●◐●◐IMPRESSChenet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib80)\)FAST●◐QJLZandiehet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib85)\)AAAI●VQ\-LLMLiuet al\.\([2025d](https://arxiv.org/html/2607.08057#bib.bib96)\)HPCA●xKVChanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib114)\)arXiv●SQuatWanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib95)\)arXiv●vAttentionPrabhuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib43)\)ASPLOS◐●PAPIHeet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib73)\)ASPLOS●PensieveYuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib55)\)EuroSys◐◐●◐AsyncKVDonget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib66)\)arXiv●◐●PaluChanget al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib111)\)ICLR●CAKEQinet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib116)\)ICLR●D2OWanet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib124)\)ICLR●●ThinKXuet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib139)\)ICLR●MagicPIGChenet al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib62)\)ICLR●OmniKVHaoet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib81)\)ICLR◐●QoQLinet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib97)\)MLSys●FlashInferYeet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib46)\)MLSys●◐◐●NeoJianget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib67)\)MLSys●●◐PRESERVEYüzügüleret al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib64)\)arXiv●◐●ReCalKVYanet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib113)\)arXiv●ClusterKVLiuet al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib61)\)DAC●PQCacheZhanget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib100)\)SIGMOD◐●◐ShadowKVSunet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib109)\)ICML◐●●SepLLMChenet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib56)\)ICML●CommVQLiet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib103)\)ICML●LaCacheShiet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib79)\)ICML●SpeCacheJieet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib182)\)ICML◐●RocketKVBehnamet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib137)\)ICML●◐ClusterAttnZhanget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib77)\)ACL●RefreshKVXuet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib78)\)ACL●OTTSuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib104)\)ACL●KVPRJianget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib65)\)ACL●◐◐SlimInferLonget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib180)\)arXiv◐●RAGCacheJinet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib76)\)TOCS◐●KVComposeAkulovet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib135)\)arXiv●LMCacheChenget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib183)\)arXiv◐◐●●◐DiffKVZhanget al\.\([2025e](https://arxiv.org/html/2607.08057#bib.bib41)\)SOSP◐●TokenSelectWuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib53)\)EMNLP●◐EvolKVYu and Chai \([2025](https://arxiv.org/html/2607.08057#bib.bib136)\)EMNLP●DynamicKVZhouet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib131)\)EMNLP●RetrievalAttentionLiuet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib57)\)NeurIPS●Ada\-KVFenget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib132)\)NeurIPS●NSNQuantSonet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib99)\)NeurIPS●KVFlowPanet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib181)\)NeurIPS◐◐●●TraCTYoonet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib84)\)arXiv◐●●CXL\-SpecKVLiu and Yu \([2026](https://arxiv.org/html/2607.08057#bib.bib83)\)FPGA●◐●◐ChanMixLiao and Wen \([2026](https://arxiv.org/html/2607.08057#bib.bib105)\)ICLR●BelugaYanget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib82)\)SIGMOD●
Continued on next page

- •●= primary category with main analysis in the paper;◐= secondary category omitted or only briefly mentioned in our paper to maintain focused classification\.

Table[9](https://arxiv.org/html/2607.08057#A4.T9)provides a supplementary mapping of all surveyed methods across the full taxonomy of 7 subcategories under 3 major optimization dimensions\. The finer\-grained categories in this table include \(i\) KV\-centric scheduling \(cf\. §[3\.1](https://arxiv.org/html/2607.08057#S3.SS1)\), \(ii\) pipelining and overlapping \(cf\. §[3\.2](https://arxiv.org/html/2607.08057#S3.SS2)\), \(iii\) hardware\-aware execution \(cf\. §[3\.3](https://arxiv.org/html/2607.08057#S3.SS3)\), \(iv\) memory hierarchy KV orchestration \(cf\. §[4\.1](https://arxiv.org/html/2607.08057#S4.SS1)\), \(v\) compute device KV orchestration \(cf\. §[4\.2](https://arxiv.org/html/2607.08057#S4.SS2)\), \(vi\) KV cache compression \(cf\. §[5\.1](https://arxiv.org/html/2607.08057#S5.SS1)\), and \(vii\) KV cache retention management \(cf\. §[5\.2](https://arxiv.org/html/2607.08057#S5.SS2)\)\.

As discussed in §[2](https://arxiv.org/html/2607.08057#S2), each method is primarily discussed under one or two key optimization categories that reflect its main contributions\. These categories are denoted as primary category \(●\) in Tab\.[9](https://arxiv.org/html/2607.08057#A4.T9)\. However, some methods also touch upon additional optimization aspects that are not covered or elaborated in the main sections\. For example, to support its “hardware\-aware execution” design of decoupling prefill and decode phases across heterogeneous devices, SplitwisePatelet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib31)\)incorporates a fine\-grained layer\-wise transmission strategy that transmits the KV cache from the prefill node to the decode node and overlaps such KV cache transmission with the computation in the prefill phase\. They serve as enabling mechanisms that make the decoupled strategy feasible and link Splitwise to the “device\-level KV transfer” and “pipelining and overlapping” categories\. We summarize these omitted associations in Tab\.[9](https://arxiv.org/html/2607.08057#A4.T9), denoted by◐, to provide a more complete mapping for readers interested in cross\-cutting techniques\.

##### Venue Diversity\.

The sKis methods span a broad range of research communities as shown in Tab\.[9](https://arxiv.org/html/2607.08057#A4.T9)\. The publication venues include top\-tier machine learning and artificial intelligence conferences \(e\.g\., ICLR, ICML, NeurIPS, AAAI\), natural language processing venues \(e\.g\., ACL, EMNLP, COLM\), systems and architecture conferences \(e\.g\., ASPLOS, ISCA, HPCA, FAST, ATC, EuroSys, OSDI, SOSP, SC, SIGCOMM, DAC, FPGA\), and interdisciplinary forums such as MLSys and SIGMOD\. We also include some impactful arXiv preprints\. This diversity underscores the inherently cross\-cutting nature of KV cache optimization and highlights the growing recognition of this topic across various research communities\.

## Appendix ETakeaways

Through a comprehensive literature review of sKis, we discovered takeaways across several domains\.

### E\.1Scheduling and Overlapping

KVS and OVLP directly target runtime stalls\. KVS prioritizes limited resources for the most reusable and latency\-sensitive work; OVLP is also a type of scheduling, aligning compute with data transfer to fill pipeline bubbles\.

Takeaway:

- ✓KVS is a multi\-objective optimization problem\. Modern schedulers often prioritize KV usage over time rather than FLOPS, and KV reuse\-driven scheduling is the default paradigm\.
- ✓KVS is enhanced by prediction\. Lightweight predictors plus a robust policy outperform traditional FCFS or SJF schemesHuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib34)\); Qinet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib69)\); Shahoutet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib51)\)\.
- ✓The key to OVLP is to perform at the true bottleneck with asymmetric pipelines\. For example, keep compute\-bound prefill on GPU, and overlap memory\-bound decode attention and KV with I/O or collective communication\.
- ✓Preferring recompute to transferJianget al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib65)\), or prefetching KV caches into L2 during collectivesDonget al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib66)\); Yüzügüleret al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib64)\), can substantially reduce pipeline bubbles, especially when bandwidth is the bottleneck\.

### E\.2Hardware\-aware Execution

HAE improves throughput, reduces mean/tail latency, and extends servable context without retraining, by decoupling phases and mapping execution to hardware capabilities\.

Takeaway:

- ✓Compute should follow hardware capabilities\. When executing on a given device, it is critical to specialize kernels, tiling, and memory layouts to that device\.
- ✓Create KV locality within the device rather than moving KV across devices\. It is effective to keep hot KV caches close to the compute\.
- ✓Compute\-intensive prefill and memory\-bound decode benefit from phase\-specific execution mappings \(cf\. §[3\.3\.2](https://arxiv.org/html/2607.08057#S3.SS3.SSS2)\)\.
- ✓HAE should adapt to the access granularity and parallelism of the target device\.

### E\.3Placement and Migration

MHO and CDO govern where KV caches reside across the memory hierarchy and how they transfer during serving\. They act directly on interconnect bandwidth bottlenecks, with GPU memory relief emerging as a by\-product of tiering and offloading\.

Takeaway:

- ✓It is a common MHO pattern to keep only future\-useful KV caches on the GPU, demote the rest to CPU or SSD, and reload guided by attention cues\. Cost models can be effectively used to choose CPU, GPU, SSD pathsShenget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib27)\); Jinet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib76)\)\.
- ✓Most MHO and CDO solutions overlap I/O transfers with compute or collectives to hide latency, although they often serve OVLP as a secondary category\.
- ✓Under interconnect bottlenecks, co\-adaptation of transfer paths, precisions, or decoding strategies can reduce TTFT and SLO violations compared with static schemesZhonget al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib33)\); Liuet al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib29)\); Shenet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib48)\)\.
- ✓Migration granularity and path should align with attention access patterns and device access units\.
- ✓Prefetch\-evict co\-design is rare\. The field would benefit from a unified objective that jointly accounts for prefetch deadlines and eviction risk\.

### E\.4KV Cache Compression

KV caches can quickly overwhelm the memory capacity of GPUs and pose bandwidth pressure as context length or batch size increases, since the size of the KV cache scales linearly with these two factors\. Consequently, prior works have proposed various approaches to directly compress the KV cache, such as quantization, low\-rank approximation, and structural compression\.

Takeaway:

- ✓Outliers dominate performance at low bitwidths or ranksSuet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib104)\)\. Isolating outliers \(e\.g\., higher bitwidths\) for value\-level compression methods prevents worst\-case error explosions\.
- ✓Recent advances trend toward applying vector quantization \(VQ\)Gray \([1984](https://arxiv.org/html/2607.08057#bib.bib177)\)for KV cache quantization, and they often reach very low\-bit \(i\.e\., 1\-2 bits\) quantization with modest quality lossZhanget al\.\([2024c](https://arxiv.org/html/2607.08057#bib.bib90)\); Liuet al\.\([2025d](https://arxiv.org/html/2607.08057#bib.bib96)\); Sonet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib99)\); Liet al\.\([2025a](https://arxiv.org/html/2607.08057#bib.bib103)\)\.
- ✓KVCC has been developed mostly at the algorithm level, while system\-level integration is thin \(cf\. Fig\.[6](https://arxiv.org/html/2607.08057#S6.F6)\)\. Thus, memory reductions often fail to translate into lower mean/tail latency or higher throughput unless KVCC is co\-designed with execution, migration, and runtime control, which is consistent with recent observations in DeltaKVHaoet al\.\([2026](https://arxiv.org/html/2607.08057#bib.bib161)\)\.

We further discuss the co\-design of KVCC with execution, migration, and runtime control \(the last akeaway\) as follows: \(i\) Co\-design with execution: quantization/de\-quantization and low\-rank updates can be fused into attention kernels or overlapped with compute, so compression overhead does not re\-introduce stalls in the decode pipeline; \(ii\) Co\-design with migration: aligning compressed packing units with device access units ensures that memory footprint reductions translate into fewer, fully utilized transfer chunks that fit overlap windows\. \(iii\) Co\-design with runtime control: exposing tunable parameters \(e\.g\., bitwidth, rank, sparsity\) to the runtime and adjusting them under SLOs remains an opportunity beyond static configurations\.

### E\.5KV Cache Eviction

KV cache eviction decides which past tokens remain resident under tight memory and bandwidth budgets, so that long contexts can be served\. It operates in both phases and trades memory and transfer cost against utility to the attention compute\.

Takeaway:

- ✓KV cache eviction is important in both prefill and decode\. The former focuses on the KV cache to be computed, while the latter focuses on the KV cache that has been computed\.
- ✓Most systems retain a small recent window, a tiny set of “attention sink” anchor tokensXiaoet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib122)\); Guet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib123)\), and a few “heavy hitters”Zhanget al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib117)\)\.
- ✓Token importance should not be judged by attention scores alone\. KV norms provide strong and low\-overhead signalsGuoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib134)\); Devotoet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib127)\)\. We also note broader links to interpretability work on importance scoringSundararajanet al\.\([2017](https://arxiv.org/html/2607.08057#bib.bib173)\); Yanget al\.\([2023a](https://arxiv.org/html/2607.08057#bib.bib175),[b](https://arxiv.org/html/2607.08057#bib.bib176)\)\.
- ✓It is effective to use heterogeneous budgets across layers or heads, rather than a uniform upper bound, as shown in Tab\.[6](https://arxiv.org/html/2607.08057#S5.T6)\. For example, shallow layers often deserve larger retention, while deeper layers emphasize global semantics and tolerate more sparsity\.
- ✓Pairing KV cache eviction with similarity\-based recall or merge is stronger than hard deletion, preserving salient context under tight budgets and improving long\-context consistencyWanet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib124)\)\.

## Appendix FBehavior\-behaviorCo\-design AffinityComputation

Below we detail the compute procedure of behavior pairs’ normalized co\-occurrence strengths, which are reflected by the edge thicknesses in Fig\.[6](https://arxiv.org/html/2607.08057#S6.F6)\.

Letℬ=\\mathcal\{B\}=\{KVS, OVLP, HAE, MHO, CDO, KVCC, KVRM\} denote the set of behaviors and𝒫\\mathcal\{P\}the set of papers\. Forp∈𝒫p\\in\\mathcal\{P\}andi∈ℬi\\in\\mathcal\{B\}, let the categorical label beℓp,i∈\{𝖯,𝖲,𝖭𝖠\}\\ell\_\{p,i\}\\in\\\{\\mathsf\{P\},\\mathsf\{S\},\\mathsf\{NA\}\\\}, which means primary category \(●\), secondary category \(◐\), or no category\. Eachℓp,i\\ell\_\{p,i\}can be observed from Tab\.[9](https://arxiv.org/html/2607.08057#A4.T9)\. We map labels to numeric weights byω​\(ℓp,i\)=𝟙\[ℓp,i=𝖯\]\+α​𝟙\[ℓp,i=𝖲\]\\omega\(\\ell\_\{p,i\}\)=\\mathbbm\{1\}\_\{\[\\ell\_\{p,i\}=\\mathsf\{P\}\]\}\+\\alpha\\mathbbm\{1\}\_\{\[\\ell\_\{p,i\}=\\mathsf\{S\}\]\}withα=0\.5\\alpha=0\.5, where𝟙\[⋅\]\\mathbbm\{1\}\_\{\[\\cdot\]\}is the indicator function that equals 1 when the stated condition holds and 0 otherwise\.

KVSOVLPHAEMHOCDOKVCCKVRMKVS–2\.54\.7541\.7506\.5OVLP2\.5–9\.258\.56\.52\.752\.25HAE4\.759\.25–3\.75101\.253\.25MHO48\.53\.75–33\.55\.75CDO1\.756\.5103–1\.252\.75KVCC02\.751\.253\.51\.25–1\.75KVRM6\.52\.253\.255\.752\.751\.75–Table 11:Raw \(pre\-normalization\) co\-occurrence matrix that encodes the weighted co\-occurrence strength between system behaviors across papers\.##### Constructing raw co\-occurrence\.

The raw co\-occurrence matrixC∈ℝ\|ℬ\|×\|ℬ\|C\\in\\mathbb\{R\}^\{\|\\mathcal\{B\}\|\\times\|\\mathcal\{B\}\|\}aggregates pairwise co\-appearance strength, as shown in Tab\.[11](https://arxiv.org/html/2607.08057#A6.T11)\. Each cellCi​jC\_\{ij\}of the behavior pairi,ji,jis defined by summing the per\-paper products of their weights:

Ci​j=∑p∈𝒫ω​\(ℓp,i\)​ω​\(ℓp,j\)\.C\_\{ij\}=\\sum\_\{p\\in\\mathcal\{P\}\}\\omega\(\\ell\_\{p,i\}\)\\omega\(\\ell\_\{p,j\}\)\.

##### Constructing normalizedco\-design affinity\.

While the raw co\-occurrence matrixCCcaptures absolute overlap, it is biased by marginal popularity, because the behaviors with larger research density tend to have largerCi​jC\_\{ij\}\. We therefore normalizeCCusing the Tanimoto coefficient\. We define the per\-behavior squared weightQiQ\_\{i\}:

Qi=∑p∈𝒫wp,i2\.Q\_\{i\}=\\sum\_\{p\\in\\mathcal\{P\}\}w\_\{p,i\}^\{2\}\.Then the Tanimoto\-normalizedco\-design affinitymatrixS∈ℝ\|ℬ\|×\|ℬ\|S\\in\\mathbb\{R\}^\{\|\\mathcal\{B\}\|\\times\|\\mathcal\{B\}\|\}reflects relative co\-occurrence strength on a\[0,1\]\[0,1\]scale, as shown in Tab\.[12](https://arxiv.org/html/2607.08057#A6.T12)\. Each cell inSi​jS\_\{ij\}of the behavior pairi,ji,jis defined as the ratio of their shared weighted presence to their squared union:

Si​j=Ci​jQi\+Qj−Ci​j\.S\_\{ij\}=\\frac\{C\_\{ij\}\}\{\\,Q\_\{i\}\+Q\_\{j\}\-C\_\{ij\}\\,\}\.Compared toCi​jC\_\{ij\}, this score controls marginal sizes and is visualized in Fig\.[6](https://arxiv.org/html/2607.08057#S6.F6)\. We draw an undirected edge between behaviorsiiandjjiffSi​j\>θS\_\{ij\}\>\\theta, where we set the thresholdθ=0\.14\\theta=0\.14; edges below the threshold are omitted to reduce clutter\. Edge thickness is proportional toSi​jS\_\{ij\}\.

KVSOVLPHAEMHOCDOKVCCKVRMKVS–0\.090\.160\.110\.0700\.14OVLP0\.09–0\.420\.300\.380\.060\.05HAE0\.160\.42–0\.100\.530\.020\.06MHO0\.110\.300\.10–0\.100\.060\.10CDO0\.070\.380\.530\.10–0\.030\.06KVCC00\.060\.020\.060\.03–0\.02KVRM0\.140\.050\.060\.100\.060\.02–Table 12:Normalizedco\-design affinitymatrix that encodes relative co\-occurrence strength between behaviors across papers\. Scores greater than the thresholdθ=0\.14\\theta=0\.14arehighlightedand visualized in Fig\.[6](https://arxiv.org/html/2607.08057#S6.F6)\.

## Appendix GExtended Discussion on Challenges

Due to space constraints, this section complements §[6\.2](https://arxiv.org/html/2607.08057#S6.SS2)with further discussion on open challenges\.

### G\.1Trustworthy sKis

As discussed inC3, efficiency optimizations typically account for average quality loss, but trustworthiness is rarely measured or attributed\. Trust risks are especially concerning in high\-stakes and safety\-critical scenariosWanget al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib204)\); Zenget al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib201)\); Wuet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib202)\); Shen and Zhang \([2026](https://arxiv.org/html/2607.08057#bib.bib211)\); Liet al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib209)\); Zhanget al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib203)\); Sunmolaet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib200)\)where errors can have severe consequences\.

One representative example \(also related to our discussion inC3\) is that KV cache eviction and compression can compromise*quality robustness*\. They may drop rare but critical tokens with low accumulated attention \(e\.g\., an exception clause in a contract, or a high value in a financial limit\), which can lead to catastrophic errors on a small subset of inputs while the system still appears efficient and accurate on average\. This failure mode can be amplified by distribution shift in workloads, such as in autonomous agent workloadsZhouet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib198)\); Zhanget al\.\([2025b](https://arxiv.org/html/2607.08057#bib.bib199)\), where statistically sparse tokens become logically important\. Optimizations tuned to the original distribution may prune these sparse critical dependencies, causing agents to hallucinate success\. Moreover, given that modern LLMs can be sensitive to prompt\-level variations such as tone/politeness\(Caiet al\.,[2025](https://arxiv.org/html/2607.08057#bib.bib205)\), such structural methods may interact with these variations and affect robustness to prompt\-level shifts\.

Trustworthiness risks extend beyond robustness to reliability, privacy, and safety, and can arise from diverse sKis behaviors\. For instance, temporal asynchrony may expose stale KV and introduce nondeterminism, harming*reliability*; cross\-tier migration can leave residual KV state or transfer KV in plaintext, harming*privacy*\.

A key gap is that many methods only measure average metrics on relatively easy workloads, but rarely consider quality lower bound, recall SLO, or semantic violation metrics, so such worst\-case failures remain invisible\. A promising direction is to consider trustworthy metrics and integrate runtime mechanisms, such as violation detectors, recovery policies, and potentially targeted rectification mechanismsYanget al\.\([2026](https://arxiv.org/html/2607.08057#bib.bib160)\), to provide a quality lower bound under stress\.

### G\.2Intermediate Semantics for Behaviors

We here provide additional discussion of intermediate semantics as a supplement toC5\.

Intuitively, intermediate semantics for sKis behaviors aim to bridge the gap between binary decisions \(e\.g\., “retain” vs\. “evict”\); examples include “reclaimable on GPU”, “compressed on GPU”, “compressed on CPU”, “summarized on CPU/SSD”, etc\. In this way, a co\-optimization strategy can be formalized as a transition between these states\. For instance, the compress\-then\-offload strategy first transitions a KV unit from a “keep” state to “compressed on GPU”, then to “compressed on CPU”\. This creates a low\-fidelity resident state that trades precision for I/O bandwidth\. Similarly, lazy eviction transitions a KV unit to “reclaimable on GPU” with a grace period to allow cheap recovery before the final transition to permanent eviction \(i\.e\., state “evict”\)\. These concrete examples show how future work can co\-optimize eviction, compression, and migration by exploiting intermediate semantics\.

### G\.3Benchmarking for sKis

Here, we focus on system\-performance benchmarking during serving, which measures actual performance metrics like latency, throughput, service\-level objectives \(SLOs\), KV cache memory and bandwidth, and energy\. In contrast, task or quality benchmarks focus on datasets and accuracy metrics\. In our survey they serve only as quality gates and are not primary evaluation objectives\. We refer readers to the surveyLiet al\.\([2024b](https://arxiv.org/html/2607.08057#bib.bib13)\)for details\.

#### G\.3\.1Review of sKis Benchmarking Practices

Many popular inference frameworks or systems, such as vLLMKwonet al\.\([2023](https://arxiv.org/html/2607.08057#bib.bib26)\), TensorRT\-LLMNVIDIA \([2023](https://arxiv.org/html/2607.08057#bib.bib191)\), and DeepSpeed\-InferenceAminabadiet al\.\([2022](https://arxiv.org/html/2607.08057#bib.bib192)\), provide benchmark scripts that measure system metrics for their local checks but remain framework\-specific\. We therefore view them as systems under test rather than the benchmark itself\. In this section, we survey benchmarking efforts that provide a platform\- and framework\-agnostic way to obtain system measurements\. They primarily fall into two categories: client\-side tools and benchmark suites\.

##### Client\-side tools\.

They define and enforce metric semantics\. Using one tool across systems yields directly comparable numbers\. LLMPerfRay \([2024](https://arxiv.org/html/2607.08057#bib.bib187)\)targets API benchmark and provides system metric measurement on service endpoints\. NVIDIA NIM benchmarking guideNVIDIA \([2025b](https://arxiv.org/html/2607.08057#bib.bib185)\)defines the common metrics of time to first token \(TTFT\), end\-to\-end request latency, inter\-token latency \(ITL\), tokens per second \(TPS\), and requests per second \(RPS\)\. The companion tool GenAI\-PerfNVIDIA \([2025a](https://arxiv.org/html/2607.08057#bib.bib186)\)emits the defined metrics and implements the stable\-window analysis across OpenAI\-API\-compatible backends\. However, although client\-side tools offer specific metrics for LLM\-based applications, we find inconsistent metric definitions and measurements across different tools\.

##### Benchmark suites\.

They are standardized packages of workloads, procedures, and reporting rules that specify what to run, how to run it, and what to report\. Such suites typically cover multiple systems and hardware and enable reproducible comparisons\. MLPerf InferenceReddiet al\.\([2020](https://arxiv.org/html/2607.08057#bib.bib190)\)emphasizes inference system comparison, and its v5\.0 includes LLM scenarios with accuracy validation\. LLM\-Inference\-BenchChitty\-Venkataet al\.\([2024](https://arxiv.org/html/2607.08057#bib.bib189)\)evaluates the inference performance of the LLaMA model family across a variety of hardware platforms\. BALIJurkschatet al\.\([2025](https://arxiv.org/html/2607.08057#bib.bib188)\)measures LLM inference across six frameworks or acceleration approaches\. It divides inference into three measured stages: setup, tokenize, and generate, and supports two settings: a technical setting with a fixed number of tokens, and a prompt\-to\-answer setting that includes tokenization\.

#### G\.3\.2Benchmark Design Principles

Building on the review, we offer a concise design checklist for sKis to improve comparability across systems\. Here, we do not aim to define a complete, reproducible benchmark suite\. Instead, we distill a set of benchmark design principles covering metrics, workloads, and reporting standards that future community benchmarks should instantiate\.

##### Metrics\.

Besides standard metrics, we recommend that future sKis benchmark suites report the following two types of important metrics:

1. 1\.Trustworthy metricsthat reflect the reliability of the serving system in satisfying SLOs, such as tail latency \(P90, P95, P99 latency\), SLO violation rate \(% of requests \> P99 target\), goodput \(throughput meeting SLOs\), recall SLO \(success rate of certain semantic segments\), and semantic violation rate\.
2. 2\.KV\-related resource metricsthat measure the utilization of resources, such as KV cache memory footprint \(as % of total GPU memory\), average effective KV bitwidth \(for compression methods\), KV\-related interconnect I/O \(the volume of KV transferred across memory tiers\), KV hit rate in memory tiers, KV\-related stalls \(% of time spent waiting for KV transfers\), effective bandwidth utilization \(useful KV transfer ratio\), and energy efficiency \(Joules per token/request\)\.

##### Workloads\.

A future sKis benchmark should cover the following three workload types to stress temporal, spatial, and structural KV behaviors:

1. 1\.Multi\-tenant or bursty online serving workloads to test the stability of temporal scheduling under high concurrency\.
2. 2\.Long\-context task workloads to test KV cache placement and migration when memory and I/O become bottlenecks\.
3. 3\.Heterogeneous workloads, such as agent workloadsZenget al\.\([2025c](https://arxiv.org/html/2607.08057#bib.bib197)\); Zhouet al\.\([2024a](https://arxiv.org/html/2607.08057#bib.bib198)\), structured editingZenget al\.\([2025d](https://arxiv.org/html/2607.08057#bib.bib206)\), or domain\-shifted workloads\(Tianet al\.,[2024](https://arxiv.org/html/2607.08057#bib.bib165),[2026](https://arxiv.org/html/2607.08057#bib.bib166); Liet al\.,[2023](https://arxiv.org/html/2607.08057#bib.bib195),[2025b](https://arxiv.org/html/2607.08057#bib.bib207); Yanet al\.,[2025a](https://arxiv.org/html/2607.08057#bib.bib196); Zenget al\.,[2025a](https://arxiv.org/html/2607.08057#bib.bib212)\), to test the robustness of structural KV cache optimizations against distribution shift\.

##### Reporting standards\.

In addition to the basic information like model, hardware, and configuration, we recommend the following reporting standards for sKis benchmarks:

1. 1\.Performance under graduated context lengths to validate scalability\.
2. 2\.Accuracy vs\. memory curves for structural methods to reveal the trade\-offs\.
3. 3\.Detailed hardware and topology setups, especially for temporal and spatial methods\.