Getting the most out of MTP

Reddit r/LocalLLaMA Tools
mtp multi-token-prediction performance-tuning benchmarks gemma qwen

Summary

A guide on optimizing MTP (Multi-Token Prediction) performance by tuning n_max parameter, with benchmark results for various models like Gemma-31b and Qwen on P100 and V100 GPUs.

If you want to get the most out of MTP. You have to run some tests / benchmarks to do so. Turning it on with defaults will get improvements, but for many models and card combinations, you are leaving a lot of performance on the table if you don't tune n_max. Can be easily missing out on 50-100% of the possible performance on some models. I ran some benchmarks against the various models I am using on my hardware (p100 + 2xV100) and there are some pretty big differences between model families. Below are some of the results I got. Full details, some other models, including impacts to VRAM and scripts to run the benchmarks are on github here: https://github.com/bradrlaw/ai-server/blob/main/docs/benchmarking.md Gemma-31b scaled nicely with more n-max Qwen benefited most from a middle setting Everyone's favorite scaled well The smaller Gemma model behaved opposite of the larger one
Original Article

Similar Articles

MTP is all about acceptance rate

Reddit r/LocalLLaMA

A user benchmarked MTP (Multi-Token Prediction) on Gemma 4 with mlx-vlm on M4 Max Studio, finding it excellent for code generation (1.53x faster, 66% acceptance) but detrimental for JSON output (50% slower, only 8% acceptance) and neutral for long-form prose, suggesting MTP benefits vanish when acceptance drops below 50%.

More Qwen3.6-27B MTP success but on dual Mi50s

Reddit r/LocalLLaMA

The article benchmarks the Qwen3.6-27B model using Multi-Token Prediction (MTP) and tensor parallelism on dual Mi50 GPUs, demonstrating significant speedups via llama.cpp.