@charles_irl: spec is all u need
Summary
Yong Quan highlights that better speculative decoders can unlock near-linear throughput gains in LLM inference, as presented at a Modal workshop by Charles.
View Cached Full Text
Cached at: 06/29/26, 08:44 PM
spec is all u need
Yong Quan is at sf (@yongquanYQ): a lot of LLM infra performance work is incremental
but one standout/reminder from today’s @modal workshop by @charles_irl:
better speculative decoders can unlock near-linear throughput gains, unlike many optimizations that only yield single-digit improvements! something to try
Similar Articles
@charles_irl: Speculation Is All You Need. In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFla…
Modal and Z Lab release six new DFlash speculative decoding draft models for Qwen 3.x, achieving over 1000 tokens per second on a B200 and arguing that speculative decoding is the most impactful inference optimization.
@charles_irl: If you're interested in speculative decoding, take some time to grok this chart! And read the article from @haoailab.ht…
A roofline model from the LLM Engineer's Almanac estimates speedups from speculative decoding for different draft lengths across models and hardware, with a note that it may underestimate benefits when overhead is significant.
@lmsysorg: New blog: The next generation of speculative decoding: DFlash and Spec V2 DFlash + Spec V2 hit >4.3X baseline throughpu…
New research on DFlash and Spec V2 speculative decoding methods achieves >4.3X baseline throughput for LLM inference, released as the default speculative decoding engine in SGLang.
What is Speculative Decoding? (trending on paperswithco.de) [R]
Speculative decoding is an inference optimization technique that uses a fast draft model to propose future tokens verified in parallel by a larger model, improving LLM generation speed. The article highlights its trending status on Papers with Code and a recent SGLang blog post about state-of-the-art latencies using DFlash models.
@_avichawla: Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effe…
Researchers introduced DFlash, a technique using block diffusion models for speculative decoding that accelerates LLM inference by up to 8.5x without accuracy loss. It is already integrated with major frameworks like vLLM and SGLang.