@lmsysorg: New blog: The next generation of speculative decoding: DFlash and Spec V2 DFlash + Spec V2 hit >4.3X baseline throughpu…
Summary
New research on DFlash and Spec V2 speculative decoding methods achieves >4.3X baseline throughput for LLM inference, released as the default speculative decoding engine in SGLang.
View Cached Full Text
Cached at: 06/15/26, 05:04 PM
New blog: The next generation of speculative decoding: DFlash and Spec V2
DFlash + Spec V2 hit >4.3X baseline throughput for LLM inference, now the default speculative decoding engine in SGLang! Together with @modal and http://z-lab.ai, our jointly-released DFlash drafter for Qwen 3.5 397B-A17B beats both baseline and native MTP in every setting we benchmarked:
4.3X baseline & 1.5X native MTP throughput (concurrency 1, HumanEval, 8xB200) Block diffusion drafter: a full token block in one forward pass KV injection: target-model features fed into every draft layer’s KV cache for higher acceptance Spec V2 overlap scheduler: +33% end-to-end
Read the code, deploy a DFlash server, and start experimenting!
Z Lab
Source: https://z-lab.ai/
We focus onmaking AI smaller, faster, and more efficientthrough full-stack innovations:
- 🧠Algorithm: Designing efficient model architectures and approximations (e.g., sparsity, compression).
- ⚙️System: Building hardware-aware system support to accelerate emerging AI workloads.
- 🚀Application: Working with real-world use cases in generative AI, robotics, and scientific discovery.
We are part of theUCSD ML Systems Groupand theUCSD Center for Visual Computing.
News
- Jan 2026ParoQuantis accepted to ICLR 2026! ParoQuant enables efficient reasoning LLM inference through pairwise rotation quantization.
- Jan 2026DFlashis released! DFlash uses block diffusion for speculative decoding, enabling efficient and high-quality parallel drafting.
- Jun 2025SparseVILAis accepted to ICCV 2025! SparseVILA decouples visual token sparsity for efficient vision-language model inference.
- Jun 2025SparseLoRAis accepted to ICML 2025! SparseLoRA applies contextual sparsity to skip unnecessary computations during fine-tuning, achieving up to 2.2× compute reduction.
Highlights
Similar Articles
@_avichawla: Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effe…
Researchers introduced DFlash, a technique using block diffusion models for speculative decoding that accelerates LLM inference by up to 8.5x without accuracy loss. It is already integrated with major frameworks like vLLM and SGLang.
@charles_irl: Speculation Is All You Need. In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFla…
Modal and Z Lab release six new DFlash speculative decoding draft models for Qwen 3.x, achieving over 1000 tokens per second on a B200 and arguing that speculative decoding is the most impactful inference optimization.
What is Speculative Decoding? (trending on paperswithco.de) [R]
Speculative decoding is an inference optimization technique that uses a fast draft model to propose future tokens verified in parallel by a larger model, improving LLM generation speed. The article highlights its trending status on Papers with Code and a recent SGLang blog post about state-of-the-art latencies using DFlash models.
@modal: We worked with @lmsysorg and http://z-lab.ai to - integrate DFlash spec into @sgl_project - make it faster with overlap…
Modal collaborated with LMSys and Z Lab to integrate DFlash speculative decoding into SGLang, achieving up to 4.3x throughput improvement over baseline and 1.5x over native multi-token prediction for large language models.
DFlash and Spec V2 Decoding (14 minute read)
Z Lab, SGLang, and Modal release DFlash, a new speculative decoding model for Qwen 3.5 397B-A17B that uses block diffusion and KV injection to achieve over 4x throughput improvement over baseline and 1.5x over native MTP.