@ClementDelangue: is this useful?
Summary
Buun introduces VBR (Variable Bit Rate), a new KV cache format that dynamically quantizes layers to optimize quality within VRAM constraints, available now on master.
View Cached Full Text
Cached at: 07/15/26, 09:49 AM
is this useful?
buun (@spiritbuun): Introducing a new KV format: VBR (Variable Bit Rate) VBR dynamically quantizes your KV cache layer-by-layer as your session grows, giving you the highest quality possible within your VRAM constraints. This is my dream format. Pushed to master. Available now. 1/15 🧵
Similar Articles
Spiritbuun's VBR (Variable Bit Rate) KV cache — first impressions
An appreciation post for Spiritbuun's llama.cpp fork, highlighting its new Variable Bit Rate (VBR) KV cache feature that dynamically degrades cache precision to fit VRAM, enabling larger auto-fitted context sizes with minimal speed loss.
KVarN: Native vLLM backend for KV-cache quantization by Huawei
Huawei CSL releases KVarN, a native vLLM attention backend for KV-cache quantization that delivers 3-5x more KV-cache capacity and up to ~1.3x the throughput of FP16, with no calibration required. It claims up to ~2.4x the throughput of TurboQuant while maintaining FP16-level accuracy on models like Qwen3-32B.
PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression
PolyKV is a layer-wise KV cache compression framework that assigns heterogeneous eviction policies and non-uniform budgets per layer, significantly improving over uniform baselines on LongBench with LLaMA-3.1-8B and Qwen3-8B.
Here are my KV cache quantization benchmarks: TurboQuant is overrated but saved by TCQ, q5 deserves more attention, and symmetric q8 might be a waste of VRAM
A detailed benchmark comparing KV cache quantization methods (TurboQuant, TCQ, q4, q5, q8) using PPL and KLD metrics on Qwen 3.6 27B, finding that TCQ improves low-bit quantization, asymmetric KV beats symmetric at same size, and q8 is often overkill. Includes analysis and data in linked article.
@QuixiAI: LESSON LEARNED: Always use BF16 kv cache. I was using turboquant. Yeah the VRAM consumption sucks - so another trick is…
The author shares a technical lesson on using BF16 KV cache instead of turboquant for AI model optimization and implements CPU offloading in SlimServe for Qwen models to manage VRAM consumption.