@populartourist: llama.cpp release b9235 added some new toys for boosting inference. Benchmarked Qwen3.6 27B on an RTX 5090 with llama.c…
Summary
llama.cpp release b9235 introduces speculative n-gram tuning, achieving up to ~7x throughput improvement on Qwen3.6 27B on an RTX 5090, with the k4v96 configuration showing the best sustained performance in 10k and 70k token tests.
View Cached Full Text
Cached at: 05/20/26, 04:33 PM
llama.cpp release b9235 added some new toys for boosting inference.
Benchmarked Qwen3.6 27B on an RTX 5090 with llama.cpp, using speculative n-gram tuning across 10k generated tokens tests.
Increasing –spec-ngram-map-k4v-size-m scaled decode throughput (predicted_per_second) up to ~7x faster accepted output token generation.
A follow-up 7x50k token generation tests on k4v64 and k4v96 samples confirmed the sustained 10k-token performance, making k4v96 winner.
k4v128 was tested too, but less stable against k4v96 in the 7x50k token run, so it was removed from the charts.
Real-world results remain anecdotal, albeit k4v96 showed a much lower acceptance rate than traditional –spec-draft-n-max 3 while still producing faster evaluation speeds - so the trade-off seems to be worth it.
Flags in comments below for the k4v96 tested sample.
-
–spec-type - draft-mtp,ngram-mod,ngram-map-k4v
- --spec-draft-n-max - "3" - --spec-draft-p-min - "0.0" - --spec-ngram-mod-n-match - "24" - --spec-ngram-mod-n-min - "48" - --spec-ngram-mod-n-max - "64" - --spec-ngram-map-k4v-size-n - "16" - --spec-ngram-map-k4v-size-m - "96" - --spec-ngram-map-k4v-min-hits - "1"
I’d prefer 1M
As far I’m aware, if you can run Qwen3.6 27B or 35B-A3B with MTP - this simply tunes the prediction of MTP - so it should technically make it faster.
Only one way to know is to test it yourself.
Similar Articles
Llama.cpp PR 8% speed boost
A llama.cpp PR moves sampling from CPU to GPU, yielding 8% faster tokens on an RTX 5090 and ~4% on a Tesla P40 for Qwen3.6-35B inference.
I tested all llama.cpp's speculative decoding methods on Qwen 3.6 27B: MTP ~2.7x, DFlash ~3.7x, n-gram stack ~6x on real coding. Local AI win. My findings on RTX 6000 PRO.
Comprehensive benchmarks of llama.cpp's speculative decoding methods on Qwen 3.6 27B show n-gram stacking on DFlash achieves up to 6x speedup on iterative coding tasks, with ngram-mod providing most of the gain and zero VRAM cost.
Qwen 3.6 27B on 24GB VRAM setup: backend comparisons, quant choice and settings (llama.cpp, ik_llama.cpp, BeeLlama, vllm)
The article compares llama.cpp backends for running Qwen 3.6 27B on an RTX 3090 24GB, finding ik_llama.cpp with IQ4_KS quantization yields the best performance (1261 tok/s prefill, 72.9 tok/s decode).
New: Llama.cpp adaptive speculation for faster inference
Llama.cpp introduces adaptive speculation to dynamically adjust token prediction for faster inference, achieving up to 50% speed improvement, particularly for models like Qwen3.8.
BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)
BeeLlama.cpp is a performance-focused fork of llama.cpp that introduces DFlash speculative decoding and TurboQuant KV-cache compression, enabling high-speed local inference of large models like Qwen 3.6 27B on consumer hardware.