An appreciation post for Spiritbuun's llama.cpp fork, highlighting its new Variable Bit Rate (VBR) KV cache feature that dynamically degrades cache precision to fit VRAM, enabling larger auto-fitted context sizes with minimal speed loss.
Well, this is an appreciation post on the spiritbuun llama.cpp fork. Some weeks ago I was testing forks and configurations to find which one was the best for my secondary model on my 3060, and the winning combo turned out to be Spiritbuun's fork + CUDA + mudler's Apex I-Compact quantization for the Qwen3.6-35B-A3B model. See https://www.reddit.com/r/LocalLLaMA/comments/1tq0h1p/qwen3635ba3bapex_128k_ctx_on_rtx_3060_12gb_37_ts/ Then Spiritbuun shipped turbo8, and I switched my key cache to it. Same speed, double precision. Good. But now he has incorporated a feature that I think deserves a mention in r/LocalLLama. What is VBR? Instead of a fixed KV quantization tier for your whole context, VBR lets you set a floor (e.g. turbo3_tcq at 3.25 bits/value) and it sets an entry tier (f16). As context grows, cache degrades step by step through the turbo(turbo8, turbo4, 3_tcq, 2_tcq, 1_tcq) ladder to stay within a VRAM budget. So you can just do: llama-server -m model.gguf -ct vbr and as the developer said: "That single flag is the whole product: it derives a KV VRAM budget from whatever is left after weights and compute, advertises the largest context that fits at the floor tier (capped at the model's training length), and degrades tiers on the fly as context fills. No context length to guess, no codec to pick. Run with -v to watch the VBR degrade #… steps fire." The result: you get dynamic context instead of a fixed n_ctx. The budget is computed from remaining VRAM at startup, and the degrade controller kicks in during decode when you're running low. And you can adjust the degrade floor, and more things. My launch command is: /root/buun-llama-cpp/build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf \ --host 0.0.0.0 --port 8000 \ --no-mmap --mlock \ -ctk turbo8 -ctv vbr --vbr-floor turbo3_tcq \ --jinja --reasoning-budget 1024 \ --flash-attn on -ngl 99 --n-cpu-moe 20 You can see I pinned the key cache to turbo8, and set the degrade floor to turbo3_tcq My setup RTX 3060 12GB Model: Qwen3.6-35B-A3B-APEX-MTP-I-Compact Config: -ctk turbo8 -ctv vbr --vbr-floor turbo3_tcq --flash-attn on Numbers (bench.sh from club-3090, 5 runs) Metric turbo8+vbr turbo8+turbo4 (fixed) Decode TPS 50.85 52.83 TTFT 93ms 178ms VRAM used 10.2 GiB 11 GiB CV 0.6% 0.4% So VBR is only ~4% slower than matched turbo8/turbo4, but you get nearly 2x faster TTFT and lower VRAM usage. The context budget was auto-fitted to 216k tokens to fit in the remaining VRAM. What I liked The degrade controller is smooth — no visible drops in TPS when it kicks in TTFT being cut in half is noticeable in real use The fork is solid. Spiritbuun has put a ton of work into this, from the TurboQuant rotation matrices to the VMM pool that maps physical pages on demand One thing to note Hit a bug with the fused f16<->t8 asymmetric kernels that caused output corruption in turbo8-K / f16-V (the baseline before degrade if you use vbr) configs. Reverted the commit and everything's clean. Spiritbuun's aware of it. The fork is worth checking out if you're pushing local inference on limited VRAM. Fork: https://github.com/spiritbuun/llama.cpp
Buun introduces VBR (Variable Bit Rate), a new KV cache format that dynamically quantizes layers to optimize quality within VRAM constraints, available now on master.
A user shares their experience offloading the KV cache to RAM in llama.cpp, achieving comparable speeds while freeing VRAM for larger models and context windows, suggesting this trade-off is often worthwhile.
BeeLlama.cpp v0.4.1 introduces KVarN, KV precision tail, and improved KV cache support, with benchmarks showing that using tail 1024 allows kvarn5 and q6_0 to match q8_0 precision while using less VRAM.
A developer has implemented a proof-of-concept PR for llama.cpp that adds dynamic KV cache quantization via an HTTP endpoint, allowing users to requantize their KV cache on-demand without fully reloading the model. The post also outlines a wishlist including load-on-demand mmproj/MTP swapping and an automatic --fit flag for context optimization.