speculative-decoding

A detailed benchmark of DeepSeek V4 Flash on a dual GH200 workstation, comparing SGLang and vLLM at 1M context with DSpark speculative decoding, finding SGLang faster at ~317 vs ~276 tok/s.

Introduces OoO-Spec, a method that accelerates LLM tool calling by computing semantic slots out of order via a small sidecar model, achieving up to 5.34x speedup over autoregressive decoding and outperforming existing drafter methods across multiple targets and benchmarks.

This paper introduces SparseSpec-L, a training-free self-speculative decoding framework for long-context LLM inference that uses a dynamically sparsified and recallable KV cache along with an entropy-based controller for speculation length, achieving up to 2.79x speedup over autoregressive decoding.

This paper proposes functional reconstruction for converting MHA/GQA checkpoints into MLA draft models for speculative decoding, directly optimizing attention modules to preserve token acceptance. It reports consistent improvements across 192 configurations involving Llama/Qwen models and multiple conversion methods.

This paper analyzes lossy verification schemes in speculative decoding, categorizing them into truncation-based and collaborative verification, and identifies pitfalls and principles for maintaining generation quality.

OpenAI deployed GPT-5.6 Sol, achieving 20% lower serving costs and 15%+ better token-generation efficiency through improved GPU kernels and speculative decoding.

Recent versions of llama.cpp now automatically load MTP tensors for draft-mtp architectures, even if speculative decoding is not enabled, potentially increasing VRAM usage for users with bundled MTP blocks in their GGUF files.

AngelSpec introduces a unified training and inference framework for speculative decoding that jointly optimizes autoregressive multi-token prediction and block-parallel diffusion drafters to handle heterogeneous real-world workloads. Experiments on the Hy3 model series show up to 2.4x speedup over autoregressive decoding and 11.8% higher throughput than DFlash.

Kimi K3, a 2.8-trillion-parameter multimodal MoE model with 1M context window, now has day-0 vLLM support. This guide details how vLLM serves K3's novel architecture including Kimi Delta Attention, Attention Residuals, and speculative decoding achieving up to 370 tok/s.

Cohere Labs releases North-Mini-Code-1.0-eagle, a draft model for speculative decoding to accelerate code generation. It features three dense transformer layers with sliding window attention and is compatible with fp8/w4a4 target models.

Adds DSpark speculative decoding support to llama.cpp via pull request, enhancing inference performance.

PRESTO introduces a prefix-aligned tree drafting framework for diffusion speculative decoding, achieving up to 1.5x speedup on dedicated diffusion drafters and 1.12x on self-speculative diffusion LLMs.

Qwen3.6-27B model shows improved speculative decoding performance when using heavier quantization levels, enhancing inference efficiency.

Introduces ADSD, a prompt-suffix attack that causes acceptance collapse in speculative decoding by forcing the draft model to propose tokens the target model is unlikely to accept, increasing inference time while preserving task quality.

Baseten details how it built the fastest API for GLM-5.2, achieving over double the launch-day performance and introducing a latency-optimized Fast version for coding and agents, with further improvements planned.

An analysis of native MTP speculative decoding in llama.cpp shows significant speedups (1.4x-2.2x) for dense models like Qwen3.6-27B, but underwhelming results on MoE architectures, where gains are minimal due to already low per-step overhead.

SonicSampler presents a unified suite of tile-aware Triton kernels that vertically fuse the entire LLM sampling pipeline, supporting dynamic per-request behaviors and speculative verification, achieving up to 16x speedup over state-of-the-art baselines.

oMLX 0.5.2 release adds live menu bar activity, a reorganized Models menu, Bonsai low-bit kernels, and improved performance with custom Metal kernels and native speculative decoding, making it the fastest way to run MLX models on Mac.

A detailed benchmark of Unsloth's Qwen3.6-27B NVFP4 model on RTX 5090 GPUs, showing MTP (multi-token prediction) gives large speedups for single requests at short context but becomes detrimental under batch concurrency or long contexts.

DogukanUrker demonstrates running Gemma 4 12B with full 262,144 context at ~100 tok/s on a single RTX 3060 using speculative decoding and KV cache splitting, achieving nearly full GPU utilization without CPU offload.