LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Hugging Face Daily Papers Papers
diffusion-models language-modeling continuous-diffusion flow-matching generative-models

Summary

LangFlow presents the first continuous diffusion language model that rivals discrete diffusion approaches, challenging the long-held belief that continuous diffusion is inferior for language modeling. The work introduces key ingredients like optimal Gumbel-based noise scheduling and demonstrates competitive perplexity and transfer learning performance compared to discrete diffusion baselines.

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Original Article Export to Word Export to PDF
View Cached Full Text

Cached at: 04/20/26, 08:29 AM

Paper page - LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Source: https://huggingface.co/papers/2604.11748 Continuous diffusion has dominated image and video generation β€” but in language modeling, it has long been considered inferior to discrete diffusion.

We challenge this belief with πŸš€LangFlow: the first continuous diffusion language model that rivals β€” and even surpasses β€” discrete diffusion.

πŸ“ Blog:https://caradryanl.github.io/blog/2026/langflow/ πŸ’» GitHub:https://github.com/nealchen2003/LangFlow πŸ’‘ HuggingFace:https://huggingface.co/Continuous-Rivals-Discrete πŸ“„ Arxiv:https://arxiv.org/abs/2604.11748

LangFlow shows for the first time that continuous diffusion can rival discrete counterparts on language modeling: πŸ” On LM1B and OpenWebText, both our perplexity (PPL) and generative perplexity (Gen. PPL) match or surpass the best discrete diffusion models. πŸ” On zero-shot transfer, LangFlow outperforms the best discrete diffusion on 3 out of 7 and autoregressive baselines on 4 out of 7 benchmarks.

LangFlow connects embedding-space DLMs to Flow Matching. It predicts clean token probabilities from noisy token embeddings, and then derives the embedding-space flow in closed form. Endorsed by Bregman divergence, we train the model via the cross-entropy loss.

However, training such continuous diffusion on language has long been a struggle. We thereby reveal several crucial ingredients:

πŸ“Œ The noise schedule should make the information gain per unit time uniform. Under this principle, the optimal noise scheduler for language is a Gumbel distribution, greatly different from that for images.

πŸ“Œ Self-conditioning significantly improves likelihood and sample quality, with effects substantially different from discrete diffusion. Disabling it during comparison with discrete DLMs is unfair. The protocol of training and evaluating continuous DLMs should be rectified!

πŸ“Œ Using ODE sampling, LangFlow can naturally derive a novel ODE-based likelihood bound from Flow Matching, enabling principled evaluation by perplexity, the core metric of language modeling.

The potential of continuous DLMs extends far beyond just performance. They open the door for all continuous diffusion techniques to be introduced into language modeling: – One-step generation, such as Consistency Models – Guided generation, such as CFG – Unified multimodal generation, such as protein structure-sequence co-design

🌱 LangFlow suggests: continuous diffusion is a viable and promising paradigm for language modeling!

I’m grateful to be part of such an amazing team pushing this forward. A huge thank you to our great advisors Ge Liu and Jiaxuan You, and to our amazing collaborators Chumeng Liang, Yuxin (Neal) Chen, Ruihan Guo, Chaoran Cheng.

Similar Articles

CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

arXiv cs.CL

CRoCoDiL proposes a continuous and robust conditioned diffusion approach for language that shifts masked diffusion models into a continuous semantic space, achieving superior generation quality and 10x faster sampling speeds compared to discrete methods like LLaDA.

Continuous Latent Diffusion Language Model

Hugging Face Daily Papers

Cola DLM is a hierarchical latent diffusion language model that uses text-to-latent mapping and conditional decoding to achieve efficient, non-autoregressive text generation.

Diffusion Model as a Generalist Segmentation Learner

Hugging Face Daily Papers

This paper introduces DiGSeg, a framework that repurposes pretrained diffusion models for state-of-the-art semantic and open-vocabulary segmentation by leveraging latent space conditioning and text-guided alignment.