@charles_irl: If you're interested in speculative decoding, take some time to grok this chart! And read the article from @haoailab.ht…
Summary
A roofline model from the LLM Engineer's Almanac estimates speedups from speculative decoding for different draft lengths across models and hardware, with a note that it may underestimate benefits when overhead is significant.
View Cached Full Text
Cached at: 07/07/26, 01:21 AM
If you’re interested in speculative decoding, take some time to grok this chart! And read the article from @haoailab.https://modal.com/llm-almanac/spec-dec-roofline…
LLM Engineer’s Almanac - Spec Dec Roofline Model (Speedup ratio)
Source: https://modal.com/llm-almanac/spec-dec-roofline
γ*=16 (max), 1.6x speedup
Sequence length4,096 tok/seq
5122k8k32k131k
Acceptance probability75%
Relative cost per token10%
Acceptance probability89%
Relative cost per block10%
This modeling system usesroofline analysisto estimate the speedups from speculative decoding for different draft lengths applied to different models running on different hardware. It is only a model! It tends to underestimate the benefit whenoverheadis a major contributor to latency, e.g. small batch sizes on small models.
The roofline model used here was inspired by the work ofFergus FinnofDoubleword. In particular, the implementation was derived usinghis DeepSeek-V4 Flash B200 optimal draft length estimatoras a reference.
Similar Articles
@charles_irl: spec is all u need
Yong Quan highlights that better speculative decoders can unlock near-linear throughput gains in LLM inference, as presented at a Modal workshop by Charles.
@charles_irl: On Friday, we released six new state-of-the-art drafters for accelerated inference. We also put out a blog post on why …
On Friday, we released six new state-of-the-art drafters for accelerated inference, along with a blog post on speculative decoding and a roofline model tool to estimate speedups.
What is Speculative Decoding? (trending on paperswithco.de) [R]
Speculative decoding is an inference optimization technique that uses a fast draft model to propose future tokens verified in parallel by a larger model, improving LLM generation speed. The article highlights its trending status on Papers with Code and a recent SGLang blog post about state-of-the-art latencies using DFlash models.
@akshay_pachaar: Speculative Decoding Clearly Explained (done right, it can make LLM inference 4x faster) A local LLM typically produces…
这篇帖子通俗解释了投机解码（Speculative Decoding）的工作原理——草稿模型预测、目标模型并行验证并以概率方式接受或纠正——并介绍了开源引擎 Uzu，它在 Apple Silicon 上结合并行起草、连贯草稿路径与免回滚验证，使 4-bit Qwen3.5 9B 的推理速度从 llama.cpp 的 22 tokens/s 提升到 92–117 tokens/s。
@_avichawla: Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effe…
Researchers introduced DFlash, a technique using block diffusion models for speculative decoding that accelerates LLM inference by up to 8.5x without accuracy loss. It is already integrated with major frameworks like vLLM and SGLang.