New: Llama.cpp adaptive speculation for faster inference
Summary
Llama.cpp introduces adaptive speculation to dynamically adjust token prediction for faster inference, achieving up to 50% speed improvement, particularly for models like Qwen3.8.
Similar Articles
Llama.cpp PR 8% speed boost
A llama.cpp PR moves sampling from CPU to GPU, yielding 8% faster tokens on an RTX 5090 and ~4% on a Tesla P40 for Qwen3.6-35B inference.
@populartourist: llama.cpp release b9235 added some new toys for boosting inference. Benchmarked Qwen3.6 27B on an RTX 5090 with llama.c…
llama.cpp release b9235 introduces speculative n-gram tuning, achieving up to ~7x throughput improvement on Qwen3.6 27B on an RTX 5090, with the k4v96 configuration showing the best sustained performance in 10k and 70k token tests.
A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s
A llama.cpp pull request adds an x86 VNNI implementation for Q2_0 × Q8_0 dot products, delivering 3.0–3.6x CPU-only speedups on Bonsai models with bit-for-bit kernel correctness and 99.2% token agreement.
Llama.cpp DSpark PC Tree Fork (up to 3%-29.5% faster!)
An implementation of DSpark PC Tree in a llama.cpp fork shows performance gains up to 29.5% faster inference on Qwen 3.0 models based on benchmark results.
BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)
BeeLlama.cpp is a performance-focused fork of llama.cpp that introduces DFlash speculative decoding and TurboQuant KV-cache compression, enabling high-speed local inference of large models like Qwen 3.6 27B on consumer hardware.