[llama.cpp] Asymmetric KV q8/q4 cache: current caveats and discussion in GGML repo
Summary
Discusses caveats of using asymmetric KV cache quantization in llama.cpp, where mismatched q8/q4 types cause prompt processing on CPU instead of GPU, and a proposed fix via compilation flags.
Similar Articles
Dynamic KV Cache Quantization and Load-on-demand mmproj/MTP: my llama.cpp wishlist
A developer has implemented a proof-of-concept PR for llama.cpp that adds dynamic KV cache quantization via an HTTP endpoint, allowing users to requantize their KV cache on-demand without fully reloading the model. The post also outlines a wishlist including load-on-demand mmproj/MTP swapping and an automatic --fit flag for context optimization.
kv-cache : avoid kv cells copies by ggerganov · Pull Request #24277 · ggml-org/llama.cpp
This pull request by ggerganov optimizes kv-cache in llama.cpp to avoid unnecessary copies of kv cells, improving inference performance. It is a contribution to the open-source LLM inference library llama.cpp.
llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch
A llama.cpp PR proposes switching quantized-KV decode from the VEC to TILE SYCL kernel on Intel Battlemage, reportedly delivering up to ~169% faster decode at 118K context. The PR is open with caveats and limited independent verification.
DeepSeek-V4-Flash (MXFP4): compute buffer scales ~3x just from KV cache quant type (f16 vs q8_0) — anyone else seeing this? Llama.cpp
A user reports that changing the KV cache quant type from f16 to q8_0 in DeepSeek-V4-Flash (MXFP4) causes the compute buffer to scale roughly 3x, asking if others have observed this behavior, specifically using llama.cpp.
ICYM: llama.cpp b9455 --SM Tensor KV Cache Fix is MERGED
llama.cpp version b9455 merges a fix for `-sm tensor` KV cache quantization on multi-GPU setups, addressing a shape information loss issue when flattening tensors.