保留还是不保留：学习解耦式大语言模型服务系统中的 KV 缓存保留策略
摘要
提出 KVLearn——一种面向分离式 LLM 服务架构的基于学习的 KV 缓存保留框架，通过前缀复用预测、成本感知保留分数和自适应阈值控制，端到端首 token 延迟最多降低 56%，节点间 KV 传输量最多减少 53%。
arXiv:2610.28637v1 Announce Type: new
Abstract: Disaggregated \textsc{LLM} serving separates prefill and decode across node pools, placing a network fabric between \textsc{KV}-cache computation and consumption. This breaks a key assumption of classical cache policies: a miss no longer incurs only local recomputation, but also network transfer to the decode node. Likewise, admitting a block to the global cache requires an additional transfer, so poor retention decisions waste both memory and bandwidth. We present \textbf{KVLearn}, a learning-based framework for cost-aware \textsc{KV}-cache retention in disaggregated serving. KVLearn includes: (i) a lightweight Prefix Reuse Predictor (\textsc{PRP}) that estimates reuse probability from structural and temporal prefix features without modifying model weights; (ii) a Cost-Aware Retention Score (\textsc{CARS}) that converts reuse probability into keep/admit decisions using recomputation, transfer, and storage costs; and (iii) an Adaptive Threshold Controller (\textsc{ATC}) that adjusts admission thresholds online from observed hit rates and memory pressure. We integrate KVLearn into a globally disaggregated serving topology and evaluate it on text and multimodal workloads, including large image/video-derived \textsc{KV} blocks with heterogeneous reuse patterns. KVLearn reduces end-to-end time-to-first-token by up to 56% versus No-Cache, 38% versus LRU-Pool, and 33% versus Mooncake-style baselines, while cutting inter-node \textsc{KV} transfer volume by up to 53% versus LRU-Pool. On MM-Session, throughput remains within ${\sim}5%$ of oracle. Code is available at https://github.com/FastLM/KVLearn.
查看缓存全文
缓存时间: 2026/10/22 16:10
# To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems
Source: [https://arxiv.org/abs/2610.28637](https://arxiv.org/abs/2610.28637)
[View PDF](https://arxiv.org/pdf/2610.28637)
> Abstract:Disaggregated \\textsc\{LLM\} serving separates prefill and decode across node pools, placing a network fabric between \\textsc\{KV\}\-cache computation and consumption\. This breaks a key assumption of classical cache policies: a miss no longer incurs only local recomputation, but also network transfer to the decode node\. Likewise, admitting a block to the global cache requires an additional transfer, so poor retention decisions waste both memory and bandwidth\. We present \\textbf\{KVLearn\}, a learning\-based framework for cost\-aware \\textsc\{KV\}\-cache retention in disaggregated serving\. KVLearn includes: \(i\) a lightweight Prefix Reuse Predictor \(\\textsc\{PRP\}\) that estimates reuse probability from structural and temporal prefix features without modifying model weights; \(ii\) a Cost\-Aware Retention Score \(\\textsc\{CARS\}\) that converts reuse probability into keep/admit decisions using recomputation, transfer, and storage costs; and \(iii\) an Adaptive Threshold Controller \(\\textsc\{ATC\}\) that adjusts admission thresholds online from observed hit rates and memory pressure\. We integrate KVLearn into a globally disaggregated serving topology and evaluate it on text and multimodal workloads, including large image/video\-derived \\textsc\{KV\} blocks with heterogeneous reuse patterns\. KVLearn reduces end\-to\-end time\-to\-first\-token by up to 56% versus No\-Cache, 38% versus LRU\-Pool, and 33% versus Mooncake\-style baselines, while cutting inter\-node \\textsc\{KV\} transfer volume by up to 53% versus LRU\-Pool\. On MM\-Session, throughput remains within $\{\\sim\}5%$ of oracle\. Code is available at[this https URL](https://github.com/FastLM/KVLearn)\.
## Submission history
From: Dong Liu \[[view email](https://arxiv.org/show-email/678094e3/2610.28637)\] **\[v1\]**Mon, 10 Aug 2026 10:46:26 UTC \(89 KB\)相似文章
@akshay_pachaar：我会学习的 12 种 KV 缓存技术，用于优化 LLM 推理。LLM 每次生成一个 token，但每个新 token ……
该推文总结了优化 LLM 推理的 12 种 KV 缓存技术，包括滑动窗口、跨层共享、MLA、量化、前缀复用、分页、驱逐和 GQA/MQA 等，帮助开发者根据具体瓶颈选择合适的组合。
面向智能体的缓存工程
提出 AKV 的缓存工程方案，让 LLM 智能体在不修改轨迹的前提下直接操作 KV 缓存（drop/repos），并配合协同设计的 serving 引擎，在降低成本与延迟的同时提升基准分数。
当 KV 缓存比 GPU 更慢：为上下文并行预填充扩展远程 KV 层级
该论文研究长上下文 LLM 服务中远程 KV 缓存层级的性能瓶颈，提出按并行 rank 分片 MLA KV 存储、异步加载等优化，将 p99 延迟从 83.32 秒降至 0.67 秒，并显著提升预填充吞吐。
KV-Repurpose：用于评估非前缀 KV 缓存复用的测试平台
本文提出 KV-Repurpose，一个基于 vLLM 的开源测试平台，用于评估超越精确前缀匹配的分段 KV 缓存复用；在服从 Zipf 分布的检索增强生成（RAG）负载下，其词元级缓存命中率达到 63.9%，远高于前缀缓存的 3.3%，并显著降低首词元生成时间（TTFT）。
KV Packet: 免重计算的上下文无关KV缓存用于大语言模型
KV Packet 提出了一种免重计算的缓存复用框架，用于大语言模型。该框架使用可训练的软标记适配器来弥合上下文不连续性，消除了开销，同时在 Llama-3.1 和 Qwen2.5 上的性能与完全重计算基线相当。