MTP is all about acceptance rate

Reddit r/LocalLLaMA News
mtp benchmark gemma4 mlx-vlm performance apple-silicon

Summary

A user benchmarked MTP (Multi-Token Prediction) on Gemma 4 with mlx-vlm on M4 Max Studio, finding it excellent for code generation (1.53x faster, 66% acceptance) but detrimental for JSON output (50% slower, only 8% acceptance) and neutral for long-form prose, suggesting MTP benefits vanish when acceptance drops below 50%.

So I was very excited about the MTP stuff especially since Gemma4 has become my "daily driver" for some stuff. I grabbed the latest mlx-vlm and did some tests and found it disappointing. | Workload | MTP off | MTP on | Result | Draft accept rate | |---|---|---|---|---| | Code generation | 75 tok/s | 114.8 tok/s | 1.53× faster | 66% of slots | | Long-form prose | 75 tok/s | 71.1 tok/s | 0.95× (wash) | 31% of slots | | JSON output | 51.3 tok/s | 25.6 tok/s | **0.50× slower** | 8% of slots | - Code generation was the typical "Write some python functions to do X" - Long form prose was "Write an 800 word essay on paper money in the Tang Dynasty" - JSON output was my core use case where I'm handing the LLM a list of items, asking it to group them by similarity according to some rules and then get them back in a structured output*. So if you want to use it for local coding, MTP is great. If you're not, maybe not so hot. My regression testing seems to indicate that once token acceptance dips below 50% the overhead kills the benefit. All this on an M4 Max Studio w/Gemma4-26b-a4b *Bonus for you hackers: Gemma's JSON structure instruction following is pretty good and I find using structured output to be about a 20% hit to token generation. It is faster to just accept a little bit of sloppy JSON and massage it at runtime; so all this is with json_schema off which mlx-vlm doesn't support for spec-decode anyway
Original Article

Similar Articles

MTP+GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 - llama.cpp

Reddit r/LocalLLaMA

A user benchmarks token generation speed on llama.cpp with the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 flag, comparing performance with and without MTP (Multi-Token Prediction). Results show a significant speedup from 49 tok/s to 64 tok/s when MTP is enabled on an RTX5090 with a Qwen3.6-27B model.

MTP for Qwen3.6-35B-A3B on 6GB VRAM laptop: not worth it

Reddit r/LocalLLaMA

Benchmarks of Multi-Token Prediction (MTP) support in llama.cpp for the Qwen3.6-35B-A3B model on a 6GB VRAM laptop show that MTP is not worth using due to significantly slower prompt processing outweighing minor generation speed gains. The author found that using q4_0 quantization for the draft KV cache saves VRAM without hurting quality.