Your KV Caching Is Broken

A practitioner’s guide to KV cache management. How a modern caching architecture cuts input token costs by 90% and speeds up LLM inference by up to 14x.

Researchers at Stanford studied how AI agents actually spend their inference budgets. The finding that stood out is that about 62% of what gets sent to an agent on every call is just repeated content. The same system prompts, the same tool definitions, the same documents, fed in again and again.

So every time the agent takes a single step, you hand it everything from scratch. All the instructions, all the context, all the background. Even if it just processed the exact same information one second ago.

The expensive part isn’t the thinking or the reasoning. It’s the AI re-reading the same notes over and over.

The agent loop re-sending the same prompt payload (system prompt, tool definitions, documents) on every step,

The agent loop re-sending the same prompt payload (system prompt, tool definitions, documents) on every step,

Per-token prices dropped 80% between 2023 and 2026, with GPT-4 class models falling from $30 to $0.40 per million tokens. But agentic workflows consume 5 to 30x more tokens per task than a standard chatbot query, because every step re-sends all that context. So even though each token got cheaper, the total bill went up, since volume outran the price cuts.

Uber learned this the hard way. Rolling out Claude Code across their engineering org burned through their entire 2026 AI budget in four months. Gartner now forecasts that 40% of AI agent projects will be cancelled by 2027 because of cost overruns alone.

The industry is optimizing the wrong variable. Making tokens cheaper doesn’t help if most of those tokens shouldn’t exist in the first place.

So, in this article, we’re going to learn about a new** open-source architecture **that moves cache management out of the inference engine entirely. Teams running it are seeing up to 14x faster time-to-first-token, and most practitioners haven’t caught up to it yet, so understanding it now puts you ahead of nearly everyone running inference today.

Diagram showing how LMCache works

Diagram showing how LMCache works

What happens when you send a prompt

Every time you hit a model with a prompt, it runs every token through the attention mechanism. For each token, the model computes a Key vector and a Value vector across every attention layer. These K and V vectors capture how the model “understands” each token’s relationship to every other token in the context.

This collection of K and V vectors is called the KV cache.

How the attention mechanism produces the KV cache

How the attention mechanism produces the KV cache

The computation scales quadratically with input length. Double the context and attention computation roughly quadruples. At 4K tokens (a simple chatbot turn) this is cheap. At 128K tokens (an agent loop with tools, documents, and history), it’s expensive.

Quadratic compute cost curve across 4K, 32K, and 128K context lengths: double the context, quadruple the cost

Quadratic compute cost curve across 4K, 32K, and 128K context lengths: double the context, quadruple the cost

A single MI300X GPU generates roughly 15 TB of KV cache per day. Most of it gets thrown away after each request.

The KV cache for your system prompt is identical every time you send it. The KV cache for a document you uploaded is identical every time a user asks about it. But the model re-derives that same understanding from scratch, every single time.

Think of it like re-reading a textbook from page 1 every time someone asks a follow-up about chapter 7. You already understood chapters 1 through 6, but you have no way to save and reuse that understanding. So you start over.

What prefix caching solves and where it stops

The industry noticed this waste and built a feature called prompt caching to deal with it.

It works like this. If two consecutive requests share the same opening tokens (the “prefix”), the provider stores the KV cache from the first request and reuses it on the second. The model skips recomputing those tokens and only processes what’s new.

The savings are real. Anthropic’s implementation gives a 90% cost reduction on cached input tokens. Hit rates of 60 to 85% are achievable for stable workloads. For teams with stable system prompts and tool definitions, this is the single highest-leverage optimization available today.

But prefix caching has a hard ceiling.

The cached portion must be an exact, byte-for-byte prefix of the new request. Change anything in the cached region (even a single character) and you get a full cache miss.

This breaks in three common scenarios:

Three prefix caching scenarios: when it works and when it doesn’t

Three prefix caching scenarios: when it works and when it doesn’t

• RAG with multiple documents: You cached document A alone and document B alone. A new query needs both. The second document’s cached KV state is invalid because it was computed without awareness of the first document.

• Document order changes: The same three documents appear in different orders across requests. Every permutation is a cache miss, even though the documents themselves are identical.

• Growing conversation history: Each new turn changes the full context after the prefix. Earlier cached states beyond the stable prefix become useless.

Alibaba Cloud’s production data confirms the limitation. 10% of KV cache blocks serve 77% of all hits. Most cached content never gets reused because the rigid prefix-matching rule prevents it.

Prefix caching is a meaningful optimization, but it only helps when your context has a long, unchanging beginning. Many real-world workloads don’t look like that.

The caching performance tax

There’s a second problem with how caching works today, and it affects every KV cache tool available.

Every KV cache library runs inside the inference engine’s process. That means cache operations (storing, loading, moving KV tensors around) and the actual inference computation share the same resources. They can’t run at the same time.

When the engine is busy managing cache, it stops doing inference. When it’s doing inference, cache operations wait.

Inference and cache I/O taking turns inside a single process

Inference and cache I/O taking turns inside a single process

Think of it like a restaurant where the chef also has to run to the storage room to fetch ingredients between every dish. The cooking and the fetching can’t happen simultaneously, so the kitchen slows down even though the chef is perfectly capable.

The impact is measurable. Google’s TurboQuant, a recent KV cache quantization technique, compresses the cache to 3 bits per value with zero accuracy loss. But when it runs inside the inference engine, it causes 20%+ inference slowdown.

The compression works perfectly, but the fact that it runs inside the same process as inference negates the benefit.

Cache management and inference serving are fundamentally different workloads. One is I/O-heavy (moving large tensors between GPU, CPU, and storage). The other is compute-heavy (matrix multiplications on GPU).

Forcing both into the same process is like running a database and a web server in the same thread. It works until load hits, and then everything fights for the same resources.

LMCache and the disaggregated approach

LMCache is an open-source project that takes a fundamentally different approach. Instead of running cache management inside the inference engine, it runs as a completely separate process alongside it.

Going back to the restaurant analogy, this is like hiring a dedicated runner for the storage room. The chef never leaves the kitchen. The runner handles all the fetching and storing independently. Both work at full speed without waiting on each other.

Traditional single-process design vs LMCache’s disaggregated architecture

Traditional single-process design vs LMCache’s disaggregated architecture

In practice, LMCache connects to the inference engine through shared GPU memory. The engine just tells LMCache “here are the block IDs I need” (tiny messages, almost no data).

All the heavy work of actually moving KV tensors between GPU, CPU, and storage happens inside LMCache’s own process. The inference engine doesn’t even notice it’s happening.

This separation produces three concrete wins.

• No resource contention: Cache I/O never blocks inference, and inference never blocks cache I/O. The 20% throughput loss from running optimization techniques inside the engine disappears.

• Zero-copy sharing across GPUs: In the traditional setup, sharing cached data between two GPUs requires multiple memory copies. LMCache lets both GPUs read and write the same memory region directly, skipping those copies entirely.

• Multi-tier parallel loading: Cached data can live across GPU memory, CPU RAM, local SSD, and remote storage. Traditional approaches check these one by one, bottlenecking on the slowest tier. LMCache checks all of them simultaneously and streams data from wherever it finds a match, in parallel.

The four storage tiers (GPU memory, CPU RAM, local SSD, cloud storage) with sequential checking vs LMCache’s parallel lookup across all tiers at once

The four storage tiers (GPU memory, CPU RAM, local SSD, cloud storage) with sequential checking vs LMCache’s parallel lookup across all tiers at once

The performance difference is significant. On H200 GPUs with the Qwen3-235B model and 50 concurrent users, LMCache delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. Startup time drops from over 3 minutes to about 30 seconds.

And the economic math even works out even at very low cache hit rates. A cached prompt only needs to be reused two to three times per week (about 1% hit rate) for the system to pay for itself. At 10% hit rate on a 1,000-node deployment, the savings reach roughly $29M over three years.

LMCache integrates with all major inference engines (vLLM, SGLang, TensorRT-LLM) and supports both NVIDIA and AMD GPUs.

Solving the prefix problem with CacheBlend

LMCache’s architecture solves the performance side of caching. But remember the prefix caching ceiling from earlier. You cached document A and document B separately, and now a query needs both. The second document’s cache is broken because it was computed in isolation.

The LMCache team’s research paper CacheBlend, which won the EuroSys 2025 Best Paper Award, directly addresses this limitation.

The problem comes down to one thing. When you cache two documents independently, neither one “knows” about the other. So when you stitch their cached states together, the model can’t jointly understand them. The connections between the two documents were never computed.

But it turns out that in modern transformer models, most tokens primarily attend to their own local context. Only a small fraction of tokens have strong connections across document boundaries.

CacheBlend exploits this by identifying just those few tokens and selectively recomputing only them. Everything else gets reused as-is from the independent caches.

Prefix caching recomputing everything after the first document vs CacheBlend reusing all cached documents

Prefix caching recomputing everything after the first document vs CacheBlend reusing all cached documents

The result is 2 to 4x faster processing for multi-document queries (the kind you see in RAG applications) without any quality loss. Instead of recomputing everything from scratch when documents are combined, CacheBlend recovers the missing cross-document understanding at a fraction of the cost.

For teams building RAG systems, multi-document Q&A, or agents that accumulate context from multiple sources, this turns every document in the knowledge base into a reusable cached asset, regardless of what order it appears in or what other documents sit alongside it.

Built for production

LMCache isn’t a research prototype. It ships with the infrastructure that production teams expect.

• Prometheus and OpenTelemetry integration for tracking cache hit rates and I/O performance.

• Kubernetes operator for deployment

• CLI for debugging and benchmarking.

The fault tolerance design is worth noting.

If the inference engine crashes, LMCache preserves all cached data on CPU and storage, so recovery doesn’t start cold.

If LMCache itself crashes, the inference engine enters a downgrade mode where caching is disabled but inference continues normally, and it reconnects automatically when the cache process recovers.

Neither failure takes the whole system down.

The bigger picture

AI applications now generate roughly 15 TB of KV cache per GPU per day, and most of it gets thrown away. The infrastructure to manage, store, and reuse that data is being built right now.

For practitioners running agentic workloads on self-hosted models, KV cache management isn’t a future optimization. It’s a cost structure decision you’re making today, whether you realize it or not.

LMCache is at the forefront of it. It’s 100% open-source.

**LMCache GitHub →
**(don’t forget to star 🌟)

Those who like learning from videos, I will be doing a video deep dive on this architecture soon.

That’s all for today, thanks for reading.

Cheers! :)