DStill: Trajectory Consistency Distillation for Accelerated One-Step Drafting in Block Diffusion Speculative Decoding

arXiv cs.CL Papers
speculative-decoding diffusion-models consistency-distillation llm-inference acceleration

Summary

DStill 是一个两阶段轨迹一致性蒸馏框架，将块扩散投机解码中多步精修的 drafter 蒸馏为单步 drafting，在三个模型族上相较单步 DFlash 最高提升 28% 接受长度和 25% 加速，较自回归解码最高达 6.45 倍加速。

arXiv:2610.27379v1 Announce Type: new Abstract: Block-diffusion speculative decoding accelerates large language model inference by drafting multiple tokens in parallel. However, one-step drafting in discrete diffusion is limited by token factorization error: each token is predicted from the same masked block, without conditioning on the tokens selected at other positions in that step. This can produce inconsistent drafts and reduce acceptance length. Iterative refinement mitigates this limitation by conditioning later predictions on previously committed tokens, though additional passes increase drafting cost. To address this trade-off between draft fidelity and inference latency, we introduce DStill, a two-stage framework that distills a target-conditioned drafter's own multi-step refinement into one-step drafting. We first train the drafter to complete partially revealed blocks, then use its refinement trajectories to train a student to predict the final completion in one pass. Using a curriculum that gradually transitions from partial to fully masked inputs, we successfully transfer the multi-step refinement capabilities into one step. Across reasoning, math, code benchmarks on three model families DStill improves acceptance length by up to 28% and speedup by up to 25% over one-step DFlash, achieving up to 6.45x speedup over auto-regressive decoding.
Original Article
View Cached Full Text

Cached at: 10/21/26, 07:05 AM 

# DStill: Trajectory Consistency Distillation for Accelerated One-Step Drafting in Block Diffusion Speculative Decoding
Source: [https://arxiv.org/abs/2610.27379](https://arxiv.org/abs/2610.27379)
[View PDF](https://arxiv.org/pdf/2610.27379)

> Abstract:Block\-diffusion speculative decoding accelerates large language model inference by drafting multiple tokens in parallel\. However, one\-step drafting in discrete diffusion is limited by token factorization error: each token is predicted from the same masked block, without conditioning on the tokens selected at other positions in that step\. This can produce inconsistent drafts and reduce acceptance length\. Iterative refinement mitigates this limitation by conditioning later predictions on previously committed tokens, though additional passes increase drafting cost\. To address this trade\-off between draft fidelity and inference latency, we introduce DStill, a two\-stage framework that distills a target\-conditioned drafter's own multi\-step refinement into one\-step drafting\. We first train the drafter to complete partially revealed blocks, then use its refinement trajectories to train a student to predict the final completion in one pass\. Using a curriculum that gradually transitions from partial to fully masked inputs, we successfully transfer the multi\-step refinement capabilities into one step\. Across reasoning, math, code benchmarks on three model families DStill improves acceptance length by up to 28% and speedup by up to 25% over one\-step DFlash, achieving up to 6\.45x speedup over auto\-regressive decoding\.

## Submission history

From: Shreyas Kulkarni \[[view email](https://arxiv.org/show-email/8839ae57/2610.27379)\] **\[v1\]**Mon, 19 Oct 2026 23:05:16 UTC \(70 KB\)

Similar Articles

DFlash: Block Diffusion for Flash Speculative Decoding

Papers with Code Trending

DFlash is a new speculative decoding framework that uses a lightweight block diffusion model for parallel token drafting, achieving over 6x acceleration compared to autoregressive methods. It significantly outperforms existing state-of-the-art methods like EAGLE-3 while maintaining high output quality.

One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting

arXiv cs.LG

该论文提出 Packed-SS，一种自投机解码调度方案，通过让扩散起草器在验证当前草稿时提前一个调用构建下一草稿，实现单次模型调用完成验证与起草，输入规模随投机宽度线性增长，并在 Nemotron 3B/8B 与 SetDLM 1.7B 上取得更高吞吐量。

DFlash 2: Keep Drafting Parallel

Reddit r/LocalLLaMA

DFlash 2 improves speculative decoding by predicting tokens in parallel, achieving over 20% more output per verification pass with minimal latency, and is integrated into major inference engines like SGLang and vLLM.

DEdit: Iterative Draft Editing for Speculative Decoding

arXiv cs.CL

DEdit introduces a diffusion-based drafter for speculative decoding that iteratively edits drafts via token-to-token predictions, using a ProposalMix training scheme to repair errors while preserving correct tokens. It achieves macro-average speedups of 5.72× and 5.97× on Qwen3-4B and Qwen3-8B across seven benchmarks.