DeepSeek-V4-Flash (MXFP4): compute buffer scales ~3x just from KV cache quant type (f16 vs q8_0) — anyone else seeing this? Llama.cpp
Summary
A user reports that changing the KV cache quant type from f16 to q8_0 in DeepSeek-V4-Flash (MXFP4) causes the compute buffer to scale roughly 3x, asking if others have observed this behavior, specifically using llama.cpp.
Similar Articles
I merged fixes for quantized KV cache into my DeepSeek V4 branch
Merged fixes for quantized KV cache into the DeepSeek V4 branch of llama.cpp, with benchmark perplexity results for f16, q8_0, and q4_0 cache types.
You really should not quantize KV Cache for DeepSeek V4 Flash
A technical post warns against quantizing the KV cache for DeepSeek V4 Flash, showing significant quality degradation in perplexity, KL divergence, and token probabilities compared to Qwen 397B.
[llama.cpp] Asymmetric KV q8/q4 cache: current caveats and discussion in GGML repo
Discusses caveats of using asymmetric KV cache quantization in llama.cpp, where mismatched q8/q4 types cause prompt processing on CPU instead of GPU, and a proposed fix via compilation flags.
DeepSeek V4 Flash on a Single AMD MI300X
This repository provides configuration, patches, and tuning to run the DeepSeek V4 Flash 304B checkpoint on a single AMD MI300X in production, achieving 168 tok/s decode without quantization. It includes correctness overlays for vLLM ROCm, AITER tuning tables, and a hybrid KV cache strategy.
DeepSeek V4 @ IQ3XXS on M1 Ultra 128GB- 16 tok/s in LM Studio after patch
A GitHub patch allows running DeepSeek V4 Flash in LM Studio on 128GB Macs by sideloading antirez's llama.cpp fork, working around struct-layout drift, decoding splits, and code-signing issues.