Can we already use Google's TurboQuant (TQ) for KV Cache in llama-server? Or are we waiting for a PR?

Reddit r/LocalLLaMA Tools
quantization kv-cache llama-cpp turboquant vram compression

Summary

Community discussion on whether Google's TurboQuant compression can already be applied to KV cache in llama-server or if implementation is still pending.

Hey everyone, Ever since the day Google announced [TurboQuant](https://www.google.com/url?sa=E&q=https%3A%2F%2Fresearch.google%2Fblog%2Fturboquant-redefining-ai-efficiency-with-extreme-compression%2F), I've been following the news about its extreme compression capabilities without noticeable quality degradation. I see it mentioned constantly on this sub, but despite all the discussions, I'm honestly still a bit confused: is it actually applicable for us right now? And if so, how? I recently saw an article/post where someone applied this TQ quantization directly to the **model weights**. They managed to get Qwen3.5-27B running at near-Q4\_0 quality, making it about 10% smaller, which finally allowed it to fit comfortably on a 16GB card (specifically an RTX 5060 Ti). This is huge for us with consumer GPUs. However, since TurboQuant was initially heavily pitched for its efficiency with context and memory, my main question is about the **KV Cache**. As we know, context length is the real VRAM killer. So my doubts are: 1. **Can we currently apply TQ quantization to the KV cache when using llama-server (llama.cpp)?** 2. If yes, how do we enable it? Is there already a CLI flag similar to --cache-type q4\_0 / --cache-type q8\_0? 3. Or is this strictly limited to model weights right now, and we are still waiting for an official PR/release from the llama.cpp team to implement TQ for the KV cache? I'd love to hear if anyone has tested this or knows the current development status. Thanks!
Original Article

Similar Articles

Dynamic KV Cache Quantization and Load-on-demand mmproj/MTP: my llama.cpp wishlist

Reddit r/LocalLLaMA

A developer has implemented a proof-of-concept PR for llama.cpp that adds dynamic KV cache quantization via an HTTP endpoint, allowing users to requantize their KV cache on-demand without fully reloading the model. The post also outlines a wishlist including load-on-demand mmproj/MTP swapping and an automatic --fit flag for context optimization.