Tag
This paper investigates how LLMs determine whether their answers come from a supplied document or parametric knowledge by intervening on the KV cache after answer generation. It finds that source reports are causally tied to cached Q+A state rather than the document cache, with steering directions flipping nearly all reports on smaller models.
The discussion points out that DeepSeek's open-source DeepInfer is an inference framework deeply customized for DeepSeek 4.1F. Its KV cache is extremely optimized at just 890 bytes per token, meaning task scheduling never needs to account for VRAM capacity. In contrast, the Kimi series models have KV cache overhead 15–30x higher, resulting in poor inference performance and user experience.
提出 KVLearn——一种面向分离式 LLM 服务架构的基于学习的 KV 缓存保留框架，通过前缀复用预测、成本感知保留分数和自适应阈值控制，端到端首 token 延迟最多降低 56%，节点间 KV 传输量最多减少 53%。
A paper proposing 'cache engineering', letting LLM agents directly manipulate their KV cache via drop and repos operations without invalidating the trajectory, co-designed with the AKV serving engine, achieving 27.7% cost reduction, 1.2x throughput, and 7.1% further savings via recursive self-improvement of cache policies.
swyx argues the market hasn't priced in that every GPU sold gained ~30% in lifetime value, citing Latent.Space's breakdown of DeepSeek's production numbers: 155T tokens/day with high cache hit rates and ~80% GPU cost margins.
Simon Mo of Inferact presented the state of the vLLM project at PyTorchCon 2026, highlighting KV cache management and kernel optimizations across TPU, AMD, and NVIDIA hardware to address LLM performance and scalability challenges.
PyTorch Conference North America in San Jose (Oct 20-21) will feature a poster by Bongwoo Bak of Moreh on KV cache block placement, covering vLLM offloading strategies between GPU, CPU, and shared storage. The post also promotes conference registration tiers and pricing.
提出 AKV 的 cache engineering 方案，让 LLM agent 在不修改轨迹的前提下直接操作 KV cache（drop/repos），并配合协同设计的 serving 引擎，在降低成本与延迟的同时提升基准分数。
AWS introduces a push-based NIXL connector in vLLM that moves KV cache transfer coordination off the critical path for disaggregated prefill-decode inference, delivering up to 3x better TTFT and 30% better TPOT on EC2 P5en instances with EFA.
Blockway 团队发布了 Volundr 32B 模型及自研 Agens Engine 推理引擎（MIT 开源），通过让 72 层中的 54 层使用固定大小状态替代增长的 KV cache，使 262K 长上下文能在 16 GB GPU 预算内运行，并演示了 110K token 冷读约 69 秒、后续追问 0.5 秒回答的实测。
A hands-on review of Nemotron 3.5 Bolt, noting its efficient KV cache, 2-bit/NVFP4 quantization, and Mamba2 hybrid attention that make it well-suited for small multi-user setups on limited hardware, despite only mediocre intelligence by current standards.
Neuralwatt 分析一个月云端流量发现，agent harness 如何构建 prompt 决定了大量 GPU 计算量：缓存命中率差异导致未缓存输入在 1.6% 到 11.1% 之间，一次缓存失效大致会让请求能耗翻倍，且这些问题多数可通过避免编辑已发送内容来修复。
A survey paper that reframes long-context compression as a memory-management problem for LLMs, unifying prompt compression, KV-cache management, retrieval augmentation, and agent memory into a single information-lifecycle framework with a budgeted cost model.
RoLaQ提出了针对多头潜注意力（MLA）模型的2位KV缓存联合旋转与范围校准的离线量化方法，在DeepSeek-V2-Lite上实现3.96倍BF16 KV容量和2.51倍解码吞吐提升。
DLite提出一种无历史上下文的投机解码框架，通过复用目标模型的隐状态进行局部预测并省去历史KV缓存维护，在agentic工作负载下将接受长度提升至2-6，较SOTA方法DSpark提升最高70%。
A community benchmark evaluated 21 Qwen3.8 27B model/KV quantization combinations (189 benchmarks) that fit in 16GB VRAM, finding q5_1/q5_1 is usually optimal and that upping model quant at q4_0/q4_0 beats lower quants at full f16 KV cache.
Titan-Llama 提出一种后训练方法，通过分层记忆架构和测试时参数更新，将预训练注意力模型改造为具备外部神经记忆模块的长上下文高效推理模型，在 QuALITY 和 LongHealth 上显著提升准确率并降低推理延迟与显存占用。
论文提出联合子空间稀疏注意力（JSSA），通过学习共享低维查询-键子空间并设计免gather的索引注意力核，在128K上下文下实现最高6.43倍解码吞吐提升，同时保持接近全KV的精度。
提出 First Passage Attention（FPA），用路径极值替代单调衰减累积，从而在压缩 KV 缓存的同时实现完美的长程检索；在 110M 语言模型上以约 25% 缓存达到 8 倍训练长度的 passkey 检索。
该论文研究长上下文 LLM 服务中远程 KV 缓存层级的性能瓶颈，提出按并行 rank 分片 MLA KV 存储、异步加载等优化，将 p99 延迟从 83.32 秒降至 0.67 秒，并显著提升预填充吞吐。