@jakevin7: Recently I've been reading about GLM 5.2 and found some interesting things to share. GLM-5.2 uses MTP (Multi-Token Prediction) to accelerate inference: a lightweight "draft model" quickly predicts multiple tokens, then the main model verifies them all at once; if accepted, it skips the decoding steps.
Summary
GLM-5.2 adopts MTP (Multi-Token Prediction) technology to accelerate inference and fixes a training-inference discrepancy in GLM-5.1's MTP that caused KV cache mixing issues.
View Cached Full Text
Cached at: 06/20/26, 08:22 PM
Recently, I’ve been reading the documentation for GLM 5.2 and found some interesting things to share.
GLM-5.2 uses MTP (Multi-Token Prediction) to accelerate inference: a lightweight “draft model” first predicts multiple tokens quickly, then the large model verifies them all at once; if accepted, it skips the decoding steps.
There’s an interesting bug fix in the documentation: GLM-5.1’s MTP had a “training-inference discrepancy” starting from the 2nd step, causing the KV cache to mix results from the target model and the MTP model, resulting in a low acceptance rate. GLM-5.2 resolves this through IndexShare + KVShare.
Similar Articles
@karminski3: Local deployment of GLM-5.2 with vLLM finally gets fast! Good news for local GLM-5.2 deployment! As we know, GLM-5.2 now comes with a built-in MTP head for speculative decoding. However, this only works with the bf16 original precision GLM-5.2, which...
Community efforts, including a hybrid quantization approach by dnhkng, have enabled vLLM and SGLang to support GLM-5.2 with MTP heads, boosting local inference speed from 2 token/s to over 43 token/s on dual GH200 hardware. The challenge involved managing DSA-based MTP and quantization compatibility.
Multi-Token Prediction (MTP) for LLaMA.cpp - Gemma 4 speedup by 40%
A new implementation of Multi-Token Prediction (MTP) in llama.cpp achieves a 40% speedup for Gemma 4 models, tested on a MacBook Pro M5Max. The post provides links to quantized GGUF models and the patched source code.
@_yucheng_lu: MTP makes autoregressive LLMs fast. Can the same trick work for diffusion LMs? Had a fun collaboration with @modal expl…
Introduces Multi-Token Residual Prediction (MRP), a technique that accelerates diffusion language model inference by predicting residuals between adjacent denoising steps, achieving up to 1.56x speedup in SGLang and recovering up to +16 accuracy points in aggressive decoding settings.
Multi-Token Prediction Done Right
Soft-MTP 提出一种 KL 梯度恢复技巧，去除多token预测训练中的噪声残差梯度，在无额外开销下改进困惑度、投机解码与推理能力。
I tested MTP on vLLM and llama.cpp for Gemma 4 & Qwen 3.6 — 3.34x faster inference, here are my findings RTX 6000 PRO.
Benchmarks of Multi-Token Prediction (MTP) on Gemma 4 31B and Qwen 3.6 27B using vLLM and llama.cpp show up to 3.34x faster inference, with optimal speculative token counts varying by model and engine.