Looped Uniform Diffusion Language Models

Hugging Face Daily Papers Papers
diffusion-models language-models transformers training-efficiency text-generation

Summary

该论文研究了在均匀扩散语言模型（UDLM）中循环早期 Transformer 层的方法，在不改变训练目标和采样器的前提下，以显著更少的训练 FLOPs 达到相同的验证 NLL，并在推理时降低生成困惑度。

Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context. This challenging training objective makes UDLMs more FLOP-intensive to train than MDLMs, but it yields stronger downstream reasoning performance. Recently, selective looping, which recursively applies a few early Transformer layers within each denoising step, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget. We investigate looping in UDLMs while keeping the training objective and sampler unchanged. We find that looping significantly improves training efficiency: looped UDLMs reach the validation NLL of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. We also find that, at a matched NFE budget, looped UDLMs reach up to 9.8% lower generation perplexity than the non-looped UDLM, with most gains achieved at moderate loop counts. Finally, we study different loop-count distributions and find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.
Original Article
View Cached Full Text

Cached at: 10/22/26, 07:36 PM

Paper page - Looped Uniform Diffusion Language Models

Source: https://huggingface.co/papers/2610.30066

Abstract

Looping early Transformer layers improves training and inference efficiency of uniform diffusion language models without altering their objective or sampler.

Uniform diffusion language models(UDLMs) generate text by iterativelydenoisingsequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereasmasked diffusion language models(MDLMs) always condition on a clean context. This challenging training objective makes UDLMs more FLOP-intensive to train than MDLMs, but it yields stronger downstream reasoning performance. Recently,selective looping, which recursively applies a few earlyTransformer layerswithin eachdenoisingstep, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget. We investigate looping in UDLMs while keeping the training objective and sampler unchanged. We find that looping significantly improves training efficiency: looped UDLMs reach thevalidation NLLof a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. We also find that, at a matched NFE budget, looped UDLMs reach up to 9.8% lowergeneration perplexitythan the non-looped UDLM, with most gains achieved at moderate loop counts. Finally, we study differentloop-count distributionsand find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.30066 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.30066 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.30066 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Less Uniform Discrete Diffusion is More Powerful and Scalable

arXiv cs.CL

This paper proposes LUDI, a less uniform diffusion language modeling framework that fixes over-uniform training objectives and condition-target confusion in uniform diffusion LMs, enabling a 7B-scale UDLM with 3x-token-per-step speedup over autoregressive decoding and competitive complex reasoning performance.

On-Policy Residual Learning for Diffusion Language Models

arXiv cs.LG

该论文提出ORL（On-policy Residual Learning）框架，通过熵收缩目标与异步残差捕获等系统级协同设计，使扩散语言模型（DLM）在相近质量下将吞吐量提升至约6.9×，在单请求解码速度上超越自回归模型的推测解码方案。

One Model to Loop Them All: Elastic Latent Recurrence for Language Models

arXiv cs.CL

NVIDIA 提出 Elastic Latent Recurrence (ELR) 训练配方，使循环（looped）语言模型在单次循环时达到与同规模非循环模型相当的精度，同时每次额外循环都能以更多推理计算换取精度提升，最终产出可在 1-4 循环间弹性的单一检查点 Koi-4B。

Looped Diffusion Transformer: the same blocks run several times inside each denoising step, and loop depth can be changed at sampling time [R]

Reddit r/MachineLearning

介绍 Looped Diffusion Transformer 论文：通过在每个去噪步骤内重复运行共享的 transformer 块，并结合深度监督和自调制注意力，使 260M 模型以 4.9 倍更少的推理计算超越 6.5 倍大的模型，且循环深度可在采样时调节；作者还用 MIT 开源权重在 M5 Pro 上实测了不同深度的效果。