@Alacritic_Super: If you are building production LLM applications, learn LLM Caching. Caching can reduce latency, GPU utilization, and AP…
Summary
This article emphasizes the importance of LLM caching in production systems to reduce latency, GPU utilization, and costs, and introduces LMCache, an open-source KV cache management layer for scalable LLM inference.
View Cached Full Text
Cached at: 07/12/26, 02:57 PM
If you are building production LLM applications, learn LLM Caching.
Caching can reduce latency, GPU utilization, and API costs often by an order of magnitude for repeated requests.
Types of LLM caching: Prompt Cache Prefix Cache KV Cache (Key-Value Cache) Semantic Cache Response Cache Embedding Cache Tool Call Cache Retrieval (RAG) Cache HTTP/API Cache Distributed Cache
Why it matters: Lower Time-to-First-Token (TTFT) Higher throughput Reduced token costs Better GPU utilization Faster agent workflows Lower inference latency Improved user experience
Learn these concepts alongside: vLLM PagedAttention Continuous Batching Speculative Decoding Quantization KV Cache Management Distributed Inference Cache Eviction Policies (LRU, LFU, TTL) Redis & Memcached
Resources
LMCache https://github.com/LMCache/LMCache
vLLM Documentation — https://docs.vllm.ai
LangChain Caching — https://python.langchain.com/docs/integrations/llm_caching/…
Redis — https://redis.io
Anthropic Prompt Caching — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching…
OpenAI Prompt Caching — https://platform.openai.com/docs/guides/prompt-caching…
The fastest token is the one you don’t have to generate. Smart caching is one of the highest-ROI optimizations for production AI systems.
LMCache/LMCache
Source: https://github.com/LMCache/LMCache
A KV Cache Management Layer for Scalable LLM Inference
Blog | Documentation | Join Slack | Community Meeting | Roadmap
⭐ If LMCache helps you serve LLMs faster and cheaper, give us a star — it helps more teams discover the project.
Updates
- [2026/05] 🔥 Agentic workload benchmark on AMD MI300X (blog).
- [2026/04] 🔥 LMCache’s new multiprocess (MP) architecture release (blog).
- [2026/03] LMCache at GTC 2026 (post).
- [2026/01] LMCache multi-node P2P CPU memory sharing, from experimental feature to production (blog).
More
- [2025/11] LMCache x CoreWeave accelerate efficient LLM inference for Cohere (blog).
- [2025/10] LMCache joins the PyTorch Foundation and Tensormesh unveiled (blog, PyTorch).
- [2025/09] NVIDIA Dynamo integrates LMCache, accelerating LLM inference (blog).
- [2025/08] 🎉 LMCache hits 5,000+ GitHub stars (blog).
- [2025/08] LMCache supports gpt-oss (20B/120B) on day 1 (blog).
- [2025/07] Get faster LLM inference and cheaper responses with LMCache and Redis (Redis blog).
- [2025/07] LMCache extends its turbo-boost to multimodal models in vLLM V1 (blog).
- [2025/06] LLM Production Stack goes cross-hardware: AMD, Arm and Ascend (blog).
About
LMCache is a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality. As a result, LMCache reduces TTFT (time-to-first-token) and improves throughput, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG).
LMCache is vendor-neutral. It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches.
Key features
-
Engine-independent deployment: LMCache, as a standalone daemon process, manages KV cache independently from the inference engine process, so that KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines).
-
Persistent, tiered KV cache offloading and reuse: Move KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends, enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT.
-
Production-level KV cache observability: LMCache provides a rich set of KV cache observability metrics, including typical Kubernetes metrics (health monitoring, performance diagnostics), KV-cache-specific metrics (request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance), management metrics (user-specific usage), and more.
-
Pluggable storage and transport backends: Easily integrate remote storage and KV transfer backends through a unified interface, enabling KV cache offloading and sharing across storage providers. Through this interface, LMCache supports storage backends including CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS.
-
Non-prefix KV reuse: Extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt. This leverages CacheBlend to selectively recompute tokens for quality recovery.
-
PD disaggregation and KV transfer: Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL.
-
Pluggable KV transformation: A simple interface for researchers to write compression, token dropping, and custom serialization through a flexible SERDE interface.
LMCache is becoming an integral layer in the LLM inference ecosystem, with community-driven integration with serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers:
Getting Started
To use LMCache, simply install lmcache from your package manager, e.g. pip:
pip install lmcache
For more setup options and examples, see:
Contributing
We welcome and value contributions and collaborations. Join us in improving LMCache. Check out the Contributing Guide or join our Slack community to get started.
Adoption and Partnerships
LMCache has a growing community of developers, researchers, industry adopters, and partners building the next generation of efficient LLM inference systems.
As an independent open-source project, LMCache is becoming the de-facto standard for KV Cache management in LLM inference. Its continued development and community work are supported in part by Tensormesh.
Citation
LMCache builds on research in KV cache management, including cache reuse, offloading, compression, and serving optimization. If you use LMCache in your research, please cite the LMCache paper and related work.
@article{cheng2025lmcache,
title={LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference},
author={Cheng, Yihua and Liu, Yuhan and Yao, Jiayi and An, Yuwei and Chen, Xiaokun and Feng, Shaoting and Huang, Yuyang and Shen, Samuel and Du, Kuntai and Jiang, Junchen},
journal={arXiv preprint arXiv:2510.09665},
year={2025}
}
Related papers
@inproceedings{liu2024cachegen,
title={Cachegen: Kv cache compression and streaming for fast large language model serving},
author={Liu, Yuhan and Li, Hanchen and Cheng, Yihua and Ray, Siddhant and Huang, Yuyang and Zhang, Qizheng and Du, Kuntai and Yao, Jiayi and Lu, Shan and Ananthanarayanan, Ganesh and others},
booktitle={Proceedings of the ACM SIGCOMM 2024 Conference},
pages={38--56},
year={2024}
}
@inproceedings{yao2025cacheblend,
title={Cacheblend: Fast large language model serving for rag with cached knowledge fusion},
author={Yao, Jiayi and Li, Hanchen and Liu, Yuhan and Ray, Siddhant and Cheng, Yihua and Zhang, Qizheng and Du, Kuntai and Lu, Shan and Jiang, Junchen},
booktitle={Proceedings of the twentieth European conference on computer systems},
pages={94--109},
year={2025}
}
License
The LMCache codebase is licensed under Apache License 2.0. See the LICENSE file for details.
Similar Articles
@h100envy: Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference op…
An ex-vLLM core contributor explains how to reduce LLM inference cost by 10x using LMCache with KV cache offloading to CPU/SSD/remote storage, a technique used by production stacks like Bloomberg.
LMCache/LMCache
LMCache is an open-source KV cache management layer for LLM inference that reduces time-to-first-token and improves throughput by enabling persistent storage and reuse of KV cache across serving engines.
@techNmak: Your LLM inference is burning 50% of its compute on work it has already done. If you're running RAG or Multi-Turn Chat,…
LMCache is an open-source library that makes KV cache persistent and shareable across requests, eliminating recomputation in RAG and multi-turn chat workloads, achieving up to 15x throughput gain and 3-10x reduction in time-to-first-token.
@Alacritic_Super: The biggest bottleneck in LLM inference isn't arithmetic but it's moving data. A single multiply-accumulate operation i…
An educational thread explaining that the main bottleneck in LLM inference is data movement, not computation, and highlighting techniques like quantization, KV cache optimization, and FlashAttention to reduce memory traffic.
@akshay_pachaar: https://x.com/akshay_pachaar/status/2074502882812952666
A practitioner's guide to KV cache management, introducing the open-source LMCache architecture that cuts input token costs by 90% and speeds up LLM inference by up to 14x by eliminating redundant context processing in agentic workflows.