@charles_irl: If you're interested in speculative decoding, take some time to grok this chart! And read the article from @haoailab.ht…
Summary
A roofline model from the LLM Engineer's Almanac estimates speedups from speculative decoding for different draft lengths across models and hardware, with a note that it may underestimate benefits when overhead is significant.
View Cached Full Text
Cached at: 07/07/26, 01:21 AM
If you’re interested in speculative decoding, take some time to grok this chart! And read the article from @haoailab.https://modal.com/llm-almanac/spec-dec-roofline…
LLM Engineer’s Almanac - Spec Dec Roofline Model (Speedup ratio)
Source: https://modal.com/llm-almanac/spec-dec-roofline
γ*=16 (max), 1.6x speedup
Sequence length4,096 tok/seq
5122k8k32k131k
Acceptance probability75%
Relative cost per token10%
Acceptance probability89%
Relative cost per block10%
This modeling system usesroofline analysisto estimate the speedups from speculative decoding for different draft lengths applied to different models running on different hardware. It is only a model! It tends to underestimate the benefit whenoverheadis a major contributor to latency, e.g. small batch sizes on small models.
The roofline model used here was inspired by the work ofFergus FinnofDoubleword. In particular, the implementation was derived usinghis DeepSeek-V4 Flash B200 optimal draft length estimatoras a reference.
Similar Articles
@charles_irl: spec is all u need
Yong Quan highlights that better speculative decoders can unlock near-linear throughput gains in LLM inference, as presented at a Modal workshop by Charles.
@charles_irl: On Friday, we released six new state-of-the-art drafters for accelerated inference. We also put out a blog post on why …
On Friday, we released six new state-of-the-art drafters for accelerated inference, along with a blog post on speculative decoding and a roofline model tool to estimate speedups.
What is Speculative Decoding? (trending on paperswithco.de) [R]
Speculative decoding is an inference optimization technique that uses a fast draft model to propose future tokens verified in parallel by a larger model, improving LLM generation speed. The article highlights its trending status on Papers with Code and a recent SGLang blog post about state-of-the-art latencies using DFlash models.
@_avichawla: Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effe…
Researchers introduced DFlash, a technique using block diffusion models for speculative decoding that accelerates LLM inference by up to 8.5x without accuracy loss. It is already integrated with major frameworks like vLLM and SGLang.
[Research] JetSpec: Speculative Decoding with Parallel Tree Drafting Enables up to 9.64x Lossless LLM Inference Speedup with more than 1000TPS
JetSpec introduces parallel tree drafting for speculative decoding, achieving up to 9.64x end-to-end speedup on LLM inference while maintaining lossless accuracy, with throughput reaching ~1000 TPS on a single B200 GPU.