DStill：面向块扩散投机解码中单步草稿加速的轨迹一致性蒸馏
摘要
DStill 是一个两阶段轨迹一致性蒸馏框架，将块扩散投机解码中多步精修的 drafter 蒸馏为单步 drafting，在三个模型族上相较单步 DFlash 最高提升 28% 接受长度和 25% 加速，较自回归解码最高达 6.45 倍加速。
arXiv:2610.27379v1 Announce Type: new
Abstract: Block-diffusion speculative decoding accelerates large language model inference by drafting multiple tokens in parallel. However, one-step drafting in discrete diffusion is limited by token factorization error: each token is predicted from the same masked block, without conditioning on the tokens selected at other positions in that step. This can produce inconsistent drafts and reduce acceptance length. Iterative refinement mitigates this limitation by conditioning later predictions on previously committed tokens, though additional passes increase drafting cost. To address this trade-off between draft fidelity and inference latency, we introduce DStill, a two-stage framework that distills a target-conditioned drafter's own multi-step refinement into one-step drafting. We first train the drafter to complete partially revealed blocks, then use its refinement trajectories to train a student to predict the final completion in one pass. Using a curriculum that gradually transitions from partial to fully masked inputs, we successfully transfer the multi-step refinement capabilities into one step. Across reasoning, math, code benchmarks on three model families DStill improves acceptance length by up to 28% and speedup by up to 25% over one-step DFlash, achieving up to 6.45x speedup over auto-regressive decoding.
查看缓存全文
缓存时间: 2026/10/21 07:05
# DStill: Trajectory Consistency Distillation for Accelerated One-Step Drafting in Block Diffusion Speculative Decoding Source: [https://arxiv.org/abs/2610.27379](https://arxiv.org/abs/2610.27379) [View PDF](https://arxiv.org/pdf/2610.27379) > Abstract:Block\-diffusion speculative decoding accelerates large language model inference by drafting multiple tokens in parallel\. However, one\-step drafting in discrete diffusion is limited by token factorization error: each token is predicted from the same masked block, without conditioning on the tokens selected at other positions in that step\. This can produce inconsistent drafts and reduce acceptance length\. Iterative refinement mitigates this limitation by conditioning later predictions on previously committed tokens, though additional passes increase drafting cost\. To address this trade\-off between draft fidelity and inference latency, we introduce DStill, a two\-stage framework that distills a target\-conditioned drafter's own multi\-step refinement into one\-step drafting\. We first train the drafter to complete partially revealed blocks, then use its refinement trajectories to train a student to predict the final completion in one pass\. Using a curriculum that gradually transitions from partial to fully masked inputs, we successfully transfer the multi\-step refinement capabilities into one step\. Across reasoning, math, code benchmarks on three model families DStill improves acceptance length by up to 28% and speedup by up to 25% over one\-step DFlash, achieving up to 6\.45x speedup over auto\-regressive decoding\. ## Submission history From: Shreyas Kulkarni \[[view email](https://arxiv.org/show-email/8839ae57/2610.27379)\] **\[v1\]**Mon, 19 Oct 2026 23:05:16 UTC \(70 KB\)
相似文章
DFlash：用于快速投机解码的块扩散
DFlash 是一种新的投机解码框架，它使用轻量级的块扩散模型进行并行标记起草，与自回归方法相比，实现了超过 6 倍的加速。在保持高输出质量的同时，其性能显著优于现有的最先进方法（如 EAGLE-3）。
基于分解引导的扩散草稿推测解码训练与推理方法
DScout 将扩散草稿器的训练与推理分解为前缀接受度和延迟权衡两个维度，从而提升推测解码的吞吐量，在 Qwen3-4B 和 Qwen3-8B 上分别实现了 3.84 倍和 4.14 倍的平均加速。
领先一次调用：采用扩散草拟的打包式自推测解码
该论文提出了 Packed-SS，一种自投机解码调度方案。该方案让扩散起草器在验证当前草稿时，提前一次调用构建下一份草稿，从而在单次模型调用中完成验证与起草。其输入规模随投机宽度线性增长，并在 Nemotron 3B/8B 和 SetDLM 1.7B 上实现了更高的吞吐量。
DFlash 2：保持并行起草
DFlash 2 通过并行预测令牌改进推测解码，在最小延迟下实现每次验证通过时输出增加超过20%，并集成到主要推理引擎如 SGLang 和 vLLM 中。
DEdit：用于投机解码的迭代草稿编辑
DEdit 提出了一种基于扩散的草稿生成器，用于投机解码，通过 token 到 token 的预测迭代式地编辑草稿，并采用 ProposalMix 训练方案在修复错误的同时保留正确的 token。在七个基准测试上，该方法在 Qwen3-4B 和 Qwen3-8B 上分别实现了 5.72× 和 5.97× 的宏平均加速。