@YRSM_Simon: Found a bug in vLLM local deployment of DeepSeek V4 Flash. Same session responds instantly, but switching to another session and back triggers a full re-prefill, and the KV cache fails. Stayed up all night, 100% reproducible: A first...

X AI KOLs Following News
deepseek-v4 vllm kv-cache bug cache-eviction local-deployment

Summary

Discovered a KV cache invalidation bug in DeepSeek V4 Flash with vLLM local deployment: after switching sessions, prefix cache hit rate drops to 0, causing re-prefill, with latency exploding from ~1 second to ~100 seconds. Root cause suspected to be vLLM calling _maybe_evict_cached_block() when reusing free queue blocks, which removes the cached index.

Discovered a bug in vLLM local deployment of DeepSeek V4 Flash. The same session responds instantly, but switching to another session and switching back triggers a full re-prefill — KV cache invalidated. Stayed up all night, 100% reproducible: A first time: 0% cache hit → ~100s A second time: ~100% cache hit → ~1s B first time: 0% cache hit → ~100s Switch back to A: 0% cache hit → ~100s again GPU memory not full (KV pool <10%), no preemption happened, A and B clearly fit in cache simultaneously—all ruled out. The problem seems to be not capacity but cache management logic. DeepSeek V4's chat template makes the first 256 tokens of every request nearly identical. After a request finishes, its KV blocks go into the free queue and could be reused as prefix cache. But on the next allocation, vLLM directly takes those blocks for reuse and calls _maybe_evict_cached_block(), which deletes the cached index (hash). This bug severely impacts concurrent sessions. Hope the community can fix it soon @vllm_project
Original Article
View Cached Full Text

Cached at: 08/10/26, 07:33 AM

Found a bug in local deployment of deepseek v4 flash with vllm.

Same session responds instantly, but switch to another session and switch back — it goes straight back to Prefill, and the KV cache is invalidated.

Stayed up all night debugging, 100% reproducible:

A first time: 0% hit → ~100s A second time: ~100% hit → ~1s B first time: 0% hit → ~100s Switch back to A: 0% hit → ~100s again

VRAM isn’t full (KV pool <10%), no preemption occurred, and A and B could clearly fit in the cache simultaneously — all ruled out. Looks like the problem isn’t capacity, but the cache management logic.

DeepSeek V4’s chat template makes the first 256 tokens of every request almost identical. After a request finishes, the KV blocks go into the free queue and could be reused as prefix cache.

But the next time space is allocated, vLLM directly grabs those blocks for reuse and casually calls _maybe_evict_cached_block() — deleting the cached index (hash) in the process.

This bug severely impacts multi-session concurrency. Hope the community can fix it soon @vllm_project

Similar Articles

@karminski3: Magic! DeepSeekV4 context memory compressed to 1/10! Everyone knows DeepSeekV4 supports 1M context and is heavily optimized. To actually use 1M context, VRAM usage is only about 10GB (compared to DeepSeek-V3.2 which needs about…

X AI KOLs Following

FlashMemory-DeepSeek-V4 proposes a novel inference paradigm called Lookahead Sparse Attention (LSA), which uses a neural memory indexer to actively predict future context needs, compressing physical KV cache usage to 13.5% of full context baseline while improving average accuracy by 0.6%. This method adopts a decoupled training strategy that allows independent training of the indexer without loading the base model, significantly reducing training cost.

FlashMemory DeepSeek-V4 Retriever (GitHub Repo)

TLDR AI

Introduces FlashMemory DeepSeek-V4 Retriever, a lightweight model that sparsifies DeepSeek-V4's CSA KV-cache by predicting which chunks will be attended to next, keeping only ~10-15% on-device while matching full-attention performance.