@hank_aibtc: https://x.com/ClementDelangue/status/2058672394865111544/video/1… Local LLM speed ceiling broken again! llama.cpp natively supports MTP (Multi-Token Prediction): - No extra draft model needed…
Summary
llama.cpp natively supports Multi-Token Prediction (MTP) without requiring an extra draft model. By leveraging the model's built-in prediction head, local models like Qwen3.6-27B achieve 1.7x+ speedup, making 27B models run smoothly on consumer GPUs.
View Cached Full Text
Cached at: 05/26/26, 11:06 AM
https://x.com/ClementDelangue/status/2058672394865111544/video/1…
The ceiling for local large model speed has been shattered again!
llama.cpp now natively supports MTP (Multi-Token Prediction):
- No separate draft model required
- Uses the model’s built-in prediction heads
- Qwen3.6-27B easily achieves 1.7x+ speedup
Now a 27B model runs smoothly on consumer-grade GPUs for daily use! Privacy, security, and zero cost—all in one.
Just add two flags:
--spec-type draft-mtp --spec-draft-n-max 2
Who wants to test how many tok/s a 4090/5090 can hit?
Similar Articles
@victormustar: llama.cpp with MTP support makes local models fast enough to use as daily drivers Qwen3.6-27B dense generation (on A10G…
llama.cpp adds MTP support for Qwen3.6 models, boosting generation speed by 78% on A10G hardware, making local models viable as daily drivers.
@NFTCPS: Attention to those running large models locally! Someone has transformed llama.cpp into a performance beast — BeeLlama.cpp. With the same VRAM, inference speed triples and context capacity expands 7.5x. This isn't a slide deck; it's real benchmark data. It stuffs three top-tier optimizations into one codebase: DFlash speculative decoding…
BeeLlama.cpp is a fork of llama.cpp that integrates DFlash speculative decoding, TurboQuant/TCQ KV-cache compression, and adaptive draft control, achieving up to 3x faster inference and 7.5x context expansion on the same hardware.
@NFTCPS: Running large models locally – looking at the hundreds of GB of weights and VRAM requirements, most people are discouraged immediately, and I was too. Colibri changes the game by treating VRAM, RAM, and disk as a unified hierarchy, streaming weights from disk on demand. Written in pure C with zero dependencies, it already has over 25,000 stars. Here are a few points: …
Introducing colibri, an open-source inference engine written in pure C that unifies VRAM, RAM, and disk as a hierarchical structure to stream large model weights, supporting various cutting-edge MoE models to run locally on consumer hardware, lowering the barrier to using large models.
llama.cpp MTP speculative simplified for July 2026 big wins on dense models, underwhelming on MoE
An analysis of native MTP speculative decoding in llama.cpp shows significant speedups (1.4x-2.2x) for dense models like Qwen3.6-27B, but underwhelming results on MoE architectures, where gains are minimal due to already low per-step overhead.
@NFTCPS: 4GB VRAM running 70B large model? It actually works! AirLLM did a clever trick — layered inference, not loading the whole model into VRAM at once, but layer by layer, compute and discard, squeezing the giant into a small GPU. The best part: 100% open source, freebie warning https://github.com/0xSo…
AirLLM is a fully open-source tool that uses layered inference (loading and releasing VRAM layer by layer) to enable 70B large language models to run on GPUs with only 4GB VRAM, without quantization, distillation, or pruning. It already supports running Llama3.1 405B on 8GB VRAM.