Author released an open-source tool called cache-pressure that benchmarks how well local LLM inference engines actually retain KV cache contexts under pressure, allowing users to verify real cache capacity against advertised claims.
Hello, I'm a bit obsessed with cache management on local LLMs. For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate the fixes. That led me to create a tool with a simple protocol that allows you to pinpoint how well the cache is actually managed on your deployment: Runs a probe to calibrate the expectations (what's a cache hit vs a cache miss in your setup) Hydrates X stable contexts of Y tokens each in order to completely fill the cache Runs cache-hit validation on the reverse order (last added is the first validated) until a cache-miss is found It's better to let this run alone to get a real value. Understand this will evict all your current cached context, so don't do it alongside real work. My results This is the result from my A/B test, control (my previous prod) vs my fixed prod. aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix-schedfix (pre-fix image, retention 4096): ── Retention under pressure ── capacity: 2,023,924 tokens retained contexts: 27/80 retained tokens: 1,052,025 retained % capacity: 51.98% oldest evicted: context #52 (older contexts evicted) With the dedupe + boundfix patches applied (retention 0): ── Retention under pressure ── capacity: 2,047,043 tokens retained contexts: 77/80 retained tokens: 3,000,048 retained % capacity: 146.56% oldest evicted: context #2 (older contexts evicted) How this can matter to you This allows you to exactly know how much tokens your cache actually holds. For most of us, cache management is a black box; this allows you to get ground truth. And yes, with my fixes, the 2M advertised cache translates to 3M retained tokens. Review the code if you doubt the claim, it's all open source: cache-pressure (and my ds4 prefix cache fixes : ds4-prefix-cache-fixes). The engine's own advertised number is wrong, and this tool finds the real value. This tool is for: 1. All of you needing to validate a setup or compare different inference engines ; 2. for inference engine maintainer to help validate changes in cache management It works under one big assumption though: most recent contexts should be preserved as much as possible. What I noticed while testing the tool is basically: vLLM good, other engines need better config. I'm mainly using vLLM so I spent lots of hours tweaking the config to get the best results, so for other engines it's up to you to decide if the above assumption fits your need (feels obvious to me it should, but I don't know what you all need of course), and how to achieve it with configuring your inference engines. How to launch 1. Clone the repo git clone https://github.com/co-l/cache-pressure 2. Install requirements pip install -r requirements.txt 3. Run the tool python3 bench/cache_pressure.py --base-url http://my-server:8000/v1 \ --kv-size <advertised_cache> I've tested it against vLLM, ninfer, llama.cpp and SGLang ; so you might need to tweak the probe so it works with your setup. 4. Interpret the results ── Retention under pressure ── capacity: 2,023,924 tokens retained contexts: 27/80 retained tokens: 1,052,025 <--- retained % capacity: 51.98% <--- oldest evicted: context #52 (older contexts evicted) The retained tokens and retained % capacity are the measured cumulative values that resisted cache eviction under pressure. Note: this post was 100% human written, the repo is 100% AI-generated under my guidance and review.
This paper introduces LaProx, a novel KV Cache eviction strategy for long-context LLM inference that reformulates the problem as an output-aware matrix multiplication approximation, achieving high performance with only 5% cache usage.
CompressKV proposes a semantic-retrieval-guided KV-cache compression method for GQA-based LLMs, identifying Semantic Retrieval Heads to retain critical tokens. It achieves over 97% full-cache performance using only 3% of the KV cache on LongBench tasks.
This empirical study examines practical online KV cache compaction for LLM agents, comparing token eviction and attention matching methods under different proxy query sources. It finds that delaying compaction to use future agent queries recovers performance, and token eviction preserves accuracy while reducing KV cache by 80%.
This paper presents a workload-aware benchmark comparing KV-cache compression techniques (quantization, pruning, merging) on long-context LLM serving tasks, finding that compression ratio alone is a poor predictor of performance and advocating for workload-aware selection.
LMCache is an open-source KV cache management layer for LLM inference that reduces time-to-first-token and improves throughput by enabling persistent storage and reuse of KV cache across serving engines.