every inference engine makes the same mistake.

an inference engine like vLLM or SGLang is the software sitting between your request and the model weights. its job is to load the model onto the GPU, run your prompt through it, and stream tokens back as fast as the hardware allows.

everything it does well comes down to keeping the GPU busy doing math.

as context windows grew, engines took on cache management too, and it went into that same process. the KV cache is the model’s saved understanding of tokens it has already read, and storing it, loading it back, and shuttling it between GPU and CPU all compete with attention for the same resources.

so the two take turns.

Google’s TurboQuant shows how tight that constraint is. it shrinks the KV cache to 3 bits per value with no loss in accuracy, so cached blocks take up less GPU memory and travel faster out to CPU RAM or disk.

the shrinking is not free, though. something has to compress each block on the way out and decompress it on the way back, and that work runs on the same GPU that is generating tokens.

switch TurboQuant on and inference throughput drops 20%+ compared to leaving it off. the compression is not the problem, and the process it runs in is.

LMCache is an open-source solution that moves cache management into a separate process running alongside the engine. three things follow from that split.

→ the handoff is tiny. the engine sends the block IDs it needs, which are a few identifiers rather than gigabytes of data, and all the actual movement of cached vectors happens on the other side while the engine keeps computing attention.

→ both processes share GPU memory directly. passing cached state between two GPUs normally costs several copies of the data, and here they read and write the same region, so a cached document stops being duplicated per worker.

→ lookups run in parallel. cached blocks live in four places, GPU memory first, then CPU RAM, then local SSD, then cloud storage. checking them one at a time means the slowest one sets your floor, so LMCache queries all four at once and streams from whichever answers first.

that’s the real shift. cache work is I/O-heavy and inference is compute-heavy, and keeping them in one process means every improvement to your caching costs you inference time.

on H200s with Qwen3-235B and 50 concurrent users, separating them gives 14x faster time-to-first-token and 4x faster decoding. startup drops from over three minutes to about 30 seconds.

@lmcache directly plugs into vLLM, SGLang, and TensorRT-LLM, runs on NVIDIA and AMD, and survives a crash on either side, with the engine falling back to uncached inference until the cache process reconnects.

check it out on GitHub: https://github.com/LMCache/LMCache

(don’t forget to star )

the diagram below traces one prompt through the whole path. i wrote the full breakdown of how LMCache works, and the article is quoted below.

LMCache/LMCache

Source: https://github.com/LMCache/LMCache

Updates

• [2026/05] 🔥 Agentic workload benchmark on AMD MI300X (blog).
• [2026/04] 🔥 LMCache’s new multiprocess (MP) architecture release (blog).
• [2026/03] LMCache at GTC 2026 (post).
• [2026/01] LMCache multi-node P2P CPU memory sharing, from experimental feature to production (blog).

About

LMCache is a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality. As a result, LMCache reduces TTFT (time-to-first-token) and improves throughput, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG).

LMCache is vendor-neutral. It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches.

Key features

• Engine-independent deployment: LMCache, as a standalone daemon process, manages KV cache independently from the inference engine process, so that KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines).

• Persistent, tiered KV cache offloading and reuse: Move KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends, enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT.

• Production-level KV cache observability: LMCache provides a rich set of KV cache observability metrics, including typical Kubernetes metrics (health monitoring, performance diagnostics), KV-cache-specific metrics (request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance), management metrics (user-specific usage), and more.

• Pluggable storage and transport backends: Easily integrate remote storage and KV transfer backends through a unified interface, enabling KV cache offloading and sharing across storage providers. Through this interface, LMCache supports storage backends including CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS.

• Non-prefix KV reuse: Extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt. This leverages CacheBlend to selectively recompute tokens for quality recovery.

• PD disaggregation and KV transfer: Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL.

• Pluggable KV transformation: A simple interface for researchers to write compression, token dropping, and custom serialization through a flexible SERDE interface.

LMCache is becoming an integral layer in the LLM inference ecosystem, with community-driven integration with serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers:

Getting Started

To use LMCache, simply install lmcache from your package manager, e.g. pip:

pip install lmcache

For more setup options and examples, see:

• Installation
• Quickstart
• LMCache Recipes
• CLI Reference
• Benchmarking Guide
• Production Deployment

Contributing

We welcome and value contributions and collaborations. Join us in improving LMCache. Check out the Contributing Guide or join our Slack community to get started.

Adoption and Partnerships

LMCache has a growing community of developers, researchers, industry adopters, and partners building the next generation of efficient LLM inference systems.

As an independent open-source project, LMCache is becoming the de-facto standard for KV Cache management in LLM inference. Its continued development and community work are supported in part by Tensormesh.

Citation

LMCache builds on research in KV cache management, including cache reuse, offloading, compression, and serving optimization. If you use LMCache in your research, please cite the LMCache paper and related work.

@article{cheng2025lmcache,
  title={LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference},
  author={Cheng, Yihua and Liu, Yuhan and Yao, Jiayi and An, Yuwei and Chen, Xiaokun and Feng, Shaoting and Huang, Yuyang and Shen, Samuel and Du, Kuntai and Jiang, Junchen},
  journal={arXiv preprint arXiv:2510.09665},
  year={2025}
}

@inproceedings{liu2024cachegen,
  title={Cachegen: Kv cache compression and streaming for fast large language model serving},
  author={Liu, Yuhan and Li, Hanchen and Cheng, Yihua and Ray, Siddhant and Huang, Yuyang and Zhang, Qizheng and Du, Kuntai and Yao, Jiayi and Lu, Shan and Ananthanarayanan, Ganesh and others},
  booktitle={Proceedings of the ACM SIGCOMM 2024 Conference},
  pages={38--56},
  year={2024}
}

@inproceedings{yao2025cacheblend,
  title={Cacheblend: Fast large language model serving for rag with cached knowledge fusion},
  author={Yao, Jiayi and Li, Hanchen and Liu, Yuhan and Ray, Siddhant and Cheng, Yihua and Zhang, Qizheng and Du, Kuntai and Lu, Shan and Jiang, Junchen},
  booktitle={Proceedings of the twentieth European conference on computer systems},
  pages={94--109},
  year={2025}
}

License

The LMCache codebase is licensed under Apache License 2.0. See the LICENSE file for details.