When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

Hugging Face Daily Papers Papers
kv-cache llm-serving context-parallelism inference-optimization distributed-systems latency

Summary

该论文研究长上下文 LLM 服务中远程 KV 缓存层级的性能瓶颈，提出按并行 rank 分片 MLA KV 存储、异步加载等优化，将 p99 延迟从 83.32 秒降至 0.67 秒，并显著提升预填充吞吐。

Reusing the attention KV cache across requests is a standard way to cut prefill cost in long-context LLM serving, and production systems increasingly back the per-engine cache with a remote tier shared by every engine. We study such a tier in a disaggregated deployment of GLM-5.3, an FP8 mixture-of-experts model with multi-head latent attention (MLA), on 8xB300 nodes running prefill context parallelism (PCP8). Enabling the remote tier (LMCache with a Valkey store) left median time-to-first-token unchanged but raised the 16k-token p99 from 3.96 s to 17.3 s. Replaying a production request stream on an isolated engine, we find that a remote hit took 0.88 s against 0.31 s to recompute the prompt, and that synchronous loading turned a hit-heavy stream into an unstable queue: p99 83.32 s with the GPUs 7% busy. A two-term cost model explains both: a hit pays off only if the tier delivers more than B* = sR (2.53 GB/s for this model and GPU), and a fixed per-hit engine cost (T0 ~ 0.33 s) means that hits on prompts shorter than about 17k tokens cannot beat recompute at any bandwidth. Guided by the model, we (i) shard the MLA KV store across PCP ranks, with a proof that the existing minimum-over-ranks lookup returns the longest cached prefix and a load protocol that can never desynchronize collectives; (ii) partition the remote tier by rank, one server instance per rank with no coordination, which raises GET bandwidth 6.3x to 10.76 GB/s; (iii) add asynchronous loading to the sharded store; and (iv) fix three client bugs that caused unbounded stalls. On the hit-heavy stream, with 98.5% of prompt tokens served from the remote tier, p99 falls from 83.32 s to 0.67 s. The sharded store raises prefill throughput by 14%, and by 34% together with an upstream kernel fix. All changes are submitted upstream to LMCache.
Original Article
View Cached Full Text

Cached at: 10/16/26, 11:03 AM

Paper page - When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

Source: https://huggingface.co/papers/2610.21500

Abstract

Remote KV caching for long-context LLM serving can severely degrade latency unless the store is sharded by parallel rank, loaded asynchronously, and paired with high-bandwidth retrieval, yielding large throughput gains.

Reusing theattention KV cacheacross requests is a standard way to cut prefill cost in long-context LLM serving, and production systems increasingly back the per-engine cache with aremote tiershared by every engine. We study such a tier in a disaggregated deployment of GLM-5.3, an FP8mixture-of-expertsmodel withmulti-head latent attention(MLA), on 8xB300 nodes runningprefill context parallelism(PCP8). Enabling theremote tier(LMCachewith a Valkey store) left median time-to-first-token unchanged but raised the 16k-token p99 from 3.96 s to 17.3 s. Replaying a production request stream on an isolated engine, we find that a remote hit took 0.88 s against 0.31 s to recompute the prompt, and that synchronous loading turned a hit-heavy stream into an unstable queue: p99 83.32 s with the GPUs 7% busy. A two-term cost model explains both: a hit pays off only if the tier delivers more than B* = sR (2.53 GB/s for this model and GPU), and a fixed per-hit engine cost (T0 ~ 0.33 s) means that hits on prompts shorter than about 17k tokens cannot beat recompute at any bandwidth. Guided by the model, we (i) shard the MLA KV store across PCP ranks, with a proof that the existing minimum-over-ranks lookup returns the longest cached prefix and a load protocol that can never desynchronize collectives; (ii) partition theremote tierby rank, one server instance per rank with no coordination, which raises GET bandwidth 6.3x to 10.76 GB/s; (iii) addasynchronous loadingto thesharded store; and (iv) fix three client bugs that caused unbounded stalls. On the hit-heavy stream, with 98.5% of prompt tokens served from theremote tier, p99 falls from 83.32 s to 0.67 s. Thesharded storeraises prefill throughput by 14%, and by 34% together with an upstream kernel fix. All changes are submitted upstream toLMCache.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.21500

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.21500 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.21500 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.21500 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

arXiv cs.CL

本文提出 KV-Repurpose，一个基于 vLLM 的开源测试平台，用于评估超越精确前缀匹配的分段 KV-cache 复用；在 Zipfian RAG 负载下，其 token 级缓存命中率达到 63.9%，远高于前缀缓存的 3.3%，并显著降低 TTFT。