Quantizing MTP KV Cache = free lunch?

Reddit r/LocalLLaMA Tools
multi-token-prediction kv-cache quantization llama-cpp qwen inference-optimization vram

Summary

Quantizing the Multi-Token Prediction (MTP) KV cache to q8_0 in llama.cpp for Qwen models reduces VRAM usage without affecting inference speed or acceptance rate, effectively providing a 'free lunch' for memory-constrained setups.

With the MTP llama.cpp implementation in the Qwen3.6/3.5 models more VRAM is required for the MTP layer. However, many people don't realize this layer comes with its own KV cache which can also be quantized: -cache-type-k-draft q8_0 -cache-type-v-draft q8_0 **So is it free lunch thus allowing us to fit slightly more context?** From a short benchmark on Qwen3.7-27B-Q8\_0 it certainly seems so: `--spec-type draft-mtp --spec-draft-n-max 3` Aggregate: { "n_requests": 9, "total_predicted": 1404, "total_draft": 1302, "total_draft_accepted": 957, "aggregate_accept_rate": 0.735, "wall_s_total": 49.46 } `--spec-type draft-mtp --spec-draft-n-max 3` \-cache-type-k-draft q8\_0 -cache-type-v-draft q8\_0 Aggregate: { "n_requests": 9, "total_predicted": 1404, "total_draft": 1302, "total_draft_accepted": 957, "aggregate_accept_rate": 0.735, "wall_s_total": 49.32 } Also tested with tensor parallelism: `-sm tenor --spec-type draft-mtp --spec-draft-n-max 3` Aggregate: { "n_requests": 9, "total_predicted": 1404, "total_draft": 1294, "total_draft_accepted": 959, "aggregate_accept_rate": 0.7411, "wall_s_total": 38.42 } `-sm tensor --spec-type draft-mtp --spec-draft-n-max 3 -cache-type-k-draft q8_0 -cache-type-v-draft q8_0` Aggregate: { "n_requests": 9, "total_predicted": 1404, "total_draft": 1294, "total_draft_accepted": 959, "aggregate_accept_rate": 0.7411, "wall_s_total": 38.29 } Let me know if I'm coping or if you have other experiences. Tested on 2xMi50 32GBs @ PCIe 4.0 x 8
Original Article

Similar Articles

Dynamic KV Cache Quantization and Load-on-demand mmproj/MTP: my llama.cpp wishlist

Reddit r/LocalLLaMA

A developer has implemented a proof-of-concept PR for llama.cpp that adds dynamic KV cache quantization via an HTTP endpoint, allowing users to requantize their KV cache on-demand without fully reloading the model. The post also outlines a wishlist including load-on-demand mmproj/MTP swapping and an automatic --fit flag for context optimization.

MTP for Qwen3.6-35B-A3B on 6GB VRAM laptop: not worth it

Reddit r/LocalLLaMA

Benchmarks of Multi-Token Prediction (MTP) support in llama.cpp for the Qwen3.6-35B-A3B model on a 6GB VRAM laptop show that MTP is not worth using due to significantly slower prompt processing outweighing minor generation speed gains. The author found that using q4_0 quantization for the draft KV cache saves VRAM without hurting quality.