What is Speculative Decoding? (trending on paperswithco.de) [R]

Reddit r/MachineLearning Tools
speculative-decoding llm-inference inference-optimization sglang draft-model parallel-decoding papers-with-code

Summary

Speculative decoding is an inference optimization technique that uses a fast draft model to propose future tokens verified in parallel by a larger model, improving LLM generation speed. The article highlights its trending status on Papers with Code and a recent SGLang blog post about state-of-the-art latencies using DFlash models.

A method that is currently trending on [Papers with Code](https://paperswithcode.co/) is Speculative Decoding. https://preview.redd.it/dm4nh4t71o7h1.png?width=3082&format=png&auto=webp&s=b6468668667d4bcfb6c9248d3af7fd09f21fe0da Speculative decoding is an inference optimization technique that uses a fast, small "draft" model to quickly propose several future tokens, which are then verified in parallel by a larger, slower "target" model. This process significantly speeds up token generation for large language models (LLMs) by allowing multiple tokens per step without sacrificing output quality. SGLang, one of the most popular frameworks for running LLMs alongside vLLM, just released a blog post detailing how they achieve state-of-the-art latencies for LLM inference serving using Modal and Z.ai's DFlash speculative decoding models. Learn more at [https://paperswithcode.co/methods/speculative-decoding](https://paperswithcode.co/methods/speculative-decoding). You can also find all the papers that cite the original paper that introduced this technique. SGLang's blog: [https://www.lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/](https://www.lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/) Let me know which other methods I should add! Cheers, Niels from HF
Original Article

Similar Articles

Speculative Decoding with a Speculative Vocabulary

arXiv cs.CL

This paper proposes SpecVocab, a method for selecting a per-step vocabulary subset for the draft model in speculative decoding, achieving higher acceptance length and up to 8.1% throughput improvement over EAGLE-3.

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

arXiv cs.CL

AngelSpec introduces a unified training and inference framework for speculative decoding that jointly optimizes autoregressive multi-token prediction and block-parallel diffusion drafters to handle heterogeneous real-world workloads. Experiments on the Hy3 model series show up to 2.4x speedup over autoregressive decoding and 11.8% higher throughput than DFlash.

Speculative Decoding Across Languages

arXiv cs.CL

This paper compares three strategies to improve speculative decoding efficiency for non-English languages, finding that task-specific distillation improves acceptance rates but generalizes poorly, while n-gram draft models offer consistent speed-ups despite lower acceptance rates.