Tag
Buun introduces VBR (Variable Bit Rate), a new KV cache format that dynamically quantizes layers to optimize quality within VRAM constraints, available now on master.
An appreciation post for Spiritbuun's llama.cpp fork, highlighting its new Variable Bit Rate (VBR) KV cache feature that dynamically degrades cache precision to fit VRAM, enabling larger auto-fitted context sizes with minimal speed loss.