@populartourist: llama.cpp release b9235 added some new toys for boosting inference. Benchmarked Qwen3.6 27B on an RTX 5090 with llama.c…

X AI KOLs Following Tools

Summary

llama.cpp release b9235 introduces speculative n-gram tuning, achieving up to ~7x throughput improvement on Qwen3.6 27B on an RTX 5090, with the k4v96 configuration showing the best sustained performance in 10k and 70k token tests.

llama.cpp release b9235 added some new toys for boosting inference. Benchmarked Qwen3.6 27B on an RTX 5090 with llama.cpp, using speculative n-gram tuning across 10k generated tokens tests. Increasing --spec-ngram-map-k4v-size-m scaled decode throughput (predicted_per_second) up to ~7x faster accepted output token generation. A follow-up 7x50k token generation tests on k4v64 and k4v96 samples confirmed the sustained 10k-token performance, making k4v96 winner. k4v128 was tested too, but less stable against k4v96 in the 7x50k token run, so it was removed from the charts. Real-world results remain anecdotal, albeit k4v96 showed a much lower acceptance rate than traditional --spec-draft-n-max 3 while still producing faster evaluation speeds - so the trade-off seems to be worth it. Flags in comments below for the k4v96 tested sample.
Original Article
View Cached Full Text

Cached at: 05/20/26, 04:33 PM

llama.cpp release b9235 added some new toys for boosting inference.

Benchmarked Qwen3.6 27B on an RTX 5090 with llama.cpp, using speculative n-gram tuning across 10k generated tokens tests.

Increasing –spec-ngram-map-k4v-size-m scaled decode throughput (predicted_per_second) up to ~7x faster accepted output token generation.

A follow-up 7x50k token generation tests on k4v64 and k4v96 samples confirmed the sustained 10k-token performance, making k4v96 winner.

k4v128 was tested too, but less stable against k4v96 in the 7x50k token run, so it was removed from the charts.

Real-world results remain anecdotal, albeit k4v96 showed a much lower acceptance rate than traditional –spec-draft-n-max 3 while still producing faster evaluation speeds - so the trade-off seems to be worth it.

Flags in comments below for the k4v96 tested sample.

  • –spec-type - draft-mtp,ngram-mod,ngram-map-k4v

    - --spec-draft-n-max
    - "3"
    - --spec-draft-p-min
    - "0.0"
    
    - --spec-ngram-mod-n-match
    - "24"
    - --spec-ngram-mod-n-min
    - "48"
    - --spec-ngram-mod-n-max
    - "64"
    
    - --spec-ngram-map-k4v-size-n
    - "16"
    - --spec-ngram-map-k4v-size-m
    - "96"
    - --spec-ngram-map-k4v-min-hits
    - "1"
    

I’d prefer 1M

As far I’m aware, if you can run Qwen3.6 27B or 35B-A3B with MTP - this simply tunes the prediction of MTP - so it should technically make it faster.

Only one way to know is to test it yourself.

Similar Articles

Llama.cpp PR 8% speed boost

Reddit r/LocalLLaMA

A llama.cpp PR moves sampling from CPU to GPU, yielding 8% faster tokens on an RTX 5090 and ~4% on a Tesla P40 for Qwen3.6-35B inference.