Tag
Stanford HazyResearch 提出 Next-Concept Modeling (NCM),一种既非自回归也非扩散的粗到细并行文本生成方法,据称比强扩散基线快 10–31 倍且训练 tokens 更少;不过实验规模仅到 725M 参数,能否扩展至通用 LLM 规模仍待验证。
This paper proposes a lightweight evolutionary heuristic scheduler to optimize denoising trajectories in diffusion large language models, addressing failure modes like EOS Overflow and Proximal Bias, and outperforming baselines on reasoning and planning benchmarks.
The paper proposes Informed Masking (IM), a structure-aware perturbation method for reinforcement learning in diffusion large language models that prioritizes downstream tokens, improving performance on math and planning benchmarks.
The paper proposes Independent Token Sampling (ITS), a query-efficient framework for detecting training data in diffusion large language models by selecting weakly dependent tokens to reduce approximation errors, achieving improved performance over state-of-the-art baselines.
The paper introduces Consistency Forcing (CForce), a distillation technique for diffusion large language models that improves parallel decoding by aligning early-stage predictions with later stages, enhancing speed-quality trade-offs.
Proposes Ripple-Pivot Search, a training-free decoding method for diffusion large language models that proactively commits mid-entropy pivot positions to reduce uncertainty and accelerate parallel decoding, achieving 4-10x speedup.
SLIM-RL introduces a risk-budgeted random-masking reinforcement learning method for diffusion LLMs that avoids trajectory slicing, achieving state-of-the-art results on math and code benchmarks with significantly fewer training samples.
A new post highlights a drawback of diffusion LLMs: bidirectional attention causes keys and values to drift across steps, breaking KV caching. However, generation quality is robust to slight KV drift, and research has focused on maximizing stale KV reuse without quality degradation.
WaveFilter proposes a training-free, wavelet-guided KV cache filtering framework for diffusion large language models that enhances long-context capability by precisely identifying key tokens and constructing sparse caches, improving performance on complex long-context tasks.
This paper introduces Temporal-Spatial Parallel Decoding (TSPD) and Confidence Extrapolation (CE) to accelerate inference in diffusion-based large language models by dynamically deciding when tokens have converged and forecasting logit trends, reducing unnecessary denoising steps while preserving output quality.
This paper introduces WINO and WINO+, methods that enable revokable parallel decoding in diffusion LLMs and distill efficient denoising trajectories, significantly improving the quality-speed trade-off.
This paper introduces DARE, a method for improving the inference efficiency of Diffusion Large Language Models by reusing cached key-value and output activations to reduce computational redundancy with negligible quality loss.