mtp

A user asks why Multi-Token Prediction (MTP) speculative decoding in llama.cpp yields lower throughput (~15 tok/s) than standard decode (~30 tok/s) for Ornith 1.5 9B, despite a high 0.95 draft acceptance rate, noting halved memory bandwidth and GPU utilization.

作者发布了 Qwen3.8-27B Uncensored (Heretic + MTP) 的 UD-IQ4_XS 量化版本，使其能在 16 GB 显存的 RTX 4080 上以 55-68 tok/s 运行，并提供 12 GB / 24 GB 版本及 KLD 质量对比与速度基准测试。

A cafe-llama.cpp update unlocks Qwen's Multi-Token Prediction potential for speculative decoding, roughly doubling decoding speed (~25 to 50+ tok/s on Qwen3.8-Flash-Next, ~40 to 100 tok/s on Qwen3.6-35B Q8) with just the --spec-draft-n-max 4 flag.

A developer stripped antirez's ds4 (DwarfStar) inference engine down to a Metal-only Qwen3.8 Flash Next build, cutting the codebase from 85k to 45k lines and yielding ~10% faster decode speeds with bit-exact output, plus a reusable tooling repo called StarForge for maintaining such forks.

TensorFold 0.6.2 is an Apache 2.0 open-source LLM inference engine claiming over 2x faster token throughput than MLX on Apple Silicon (61 vs 27 tok/s on an M3 Ultra) via multi-token-prediction speculative decoding, with parallel concurrent requests and OpenAI-compatible API.

MTP (multi-token prediction) support work for Qwen Flash Next in llama.cpp has been restarted, with GGUF quants available on Hugging Face and an open PR, though it is still work in progress.

This pull request enables CUDA graph for MTP draft in llama.cpp to improve performance of LLM inference on GPUs.

The article details the merging of Multi-Token Prediction (MTP) support into ik_llama.cpp, which significantly enhances token generation speed for the Qwen3.8-Flash-Next model, with benchmarks showing up to double the speed on high-end hardware.

A user reported slower inference speeds with MTP on the Gemma 4 12B QAT model when using the Vulkan backend, but discovered that switching to ROCm significantly improved performance, achieving 60t/s with MTP.

MTP has been released for the Qwen3.8-Flash-Next-GGUF model, providing detailed instructions on integration with inference tools like llama.cpp, vLLM, and Ollama for deployment.

The article details custom optimizations for running the Qwen3.8-Flash-Next AI model on Mac M1 Max hardware, including SSD streaming, custom quantizations, and a sparse attention mechanism to improve performance.

Benchmark results for running Qwen3.8-Flash-Next with MTP on Strix Halo hardware using the Vulkan backend in llama.cpp, including performance metrics and observations on output quality.

llama.cpp now supports MTP (Multi-Token Prediction) for GLM-Air models, enhancing inference capabilities for large language models in C/C++.

The article details an experiment achieving 50 tokens per second inference with Qwen3.8-27B at 256K context on a 24GB GPU using Multi-Token Prediction and custom optimizations.

This article presents a quantized version of the Qwen3.8-27B model using INT4 AutoRound quantization with working MTP for speculative decoding, achieving significant inference speedups on GPUs.

llama.cpp has added support for MTP/DSpark on DeepSeek V4 Flash, enabling local inference with the popular C/C++ LLM runtime.

An analysis of native MTP speculative decoding in llama.cpp shows significant speedups (1.4x-2.2x) for dense models like Qwen3.6-27B, but underwhelming results on MoE architectures, where gains are minimal due to already low per-step overhead.

A guide on optimizing MTP (Multi-Token Prediction) performance by tuning n_max parameter, with benchmark results for various models like Gemma-31b and Qwen on P100 and V100 GPUs.

This paper likely discusses the application of Multi-Task Prompting (MTP) to Mixture of Experts (MoE) models, exploring how MTP can improve performance or efficiency in MoE architectures.

Running Qwen3.6 27B on an RTX 5090, achieving 6.4k tokens per second after tuning MTP and cache settings, demonstrating optimization techniques for inference.