@_avichawla: DeepSeek did it again! They just open-sourced DeepInfer, the inference stack behind cache hits that return the first to…

DeepSeek did it again! They just open-sourced DeepInfer, the inference stack behind cache hits that return the first token up to 26x faster. When a request repeats a long prefix, like a system prompt or earlier chat turns, the server can reuse the KV cache it already computed instead of running prefill again. Ideally, you'd expect that cache to sit in GPU memory, since disk is supposed to be far too slow for anything on the request path. In practice, DeepSeek keeps it on SSDs. When DeepSeek launched this, a 128K-token prompt that hit the cache got its first token in 500ms instead of 13s. And in one day of production traffic last year, 56% of all input tokens were served straight from disk. So disk isn't the bottleneck here as long as what you store is small enough, and the storage is wide enough. Let's understand how this works. 1) Why reuse the KV cache at all: Before generating anything, an LLM reads the whole prompt and computes a key and a value for every token in every layer. This is prefill, and on long prompts it's the slow part. Many requests start the same way. For instance, a chat resends every earlier turn or an agent resends the same system prompt and tool definitions on every call. If those KV are saved, the next request with the same prefix loads them and only computes the new tokens. 2) Why GPU memory is predominantly used: Every token stores KV in every layer, so on a standard model a long prompt's KV cache runs into many GBs. Reading that much from disk can take longer than recomputing it on the GPU. So engines keep it in GPU memory. But GPU memory is small, so old prefixes get evicted fast and many repeats miss. 3) DeepSeek made the KV cache small: DeepSeek-V2 introduced Multi-head Latent Attention (MLA). Instead of storing separate KV for every attention head, MLA stores one small compressed vector per token per layer, and the keys and values are rebuilt from it during attention. In the V2 paper, this cut the KV cache by 93.3% compared to DeepSeek's earlier 67B model. And MLA shrank the KV cache enough to store it on low-cost disks. Fewer bytes per cached token implied fewer bytes to read on every hit, and it made loading from SSD faster than recomputing. 4) 3FS reads it from many SSDs at once: One SSD would still be too slow, so the cache lives on 3FS, DeepSeek's own distributed file system. 3FS splits files into chunks and spreads them across thousands of SSDs on many storage nodes. A read pulls chunks from many drives in parallel, so their bandwidth adds up. When a request arrives, the system matches its prefix against cached blocks, fetches those blocks from 3FS into host memory, and copies them to the GPU layer by layer while the model is already running. And since every prefill machine can reach every cached prefix, a request doesn't have to go back to the machine that served it last. Disk is also cheap, so the cache can stay for days. As per the DeepInfer docs, cached prefixes are retained for 3 days for agent traffic and 7 days for chat. Disk caching works when the KV cache is small, and the storage is wide. DeepSeek got the first from its attention design and the second from 3FS. This is also evident from the price. On deepseek-flash today, a cache hit costs 50x less than a miss. If you use the API, keep the stable parts of your prompt (system prompt, tools, documents) at the front, since only an exact match from the very first token counts as a hit. The visual below depicts what we discussed. That said, prefix caching is only one of the caches in LLM serving. I covered all four in an article recently, with code for each. Read it below.