Hierarchical Continuous Diffusion Language Models

Hugging Face Daily Papers Papers
diffusion-models language-models continuous-diffusion variational-inference structured-reasoning generative-models

Summary

The paper proposes Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single denoising process derived from a variational bound on token likelihood. HC-DLM improves over discrete and continuous diffusion baselines on Sudoku, Countdown, and LM1B tasks.

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.
Original Article
View Cached Full Text

Cached at: 10/02/26, 04:28 AM

Paper page - Hierarchical Continuous Diffusion Language Models

Source: https://huggingface.co/papers/2610.02193

Abstract

Discretediffusionlanguagemodelsofferacompellingalternativetoautoregressivegenerationfortasksdemandingbidirectionalreasoningandglobalconstraintsatisfaction.Yettheyshareastructuralbottleneck:whendecodinginparallel,eachtokenissampledindependentlyfromitsmarginal,severingthestatisticaldependenciesamongthetokensdecodedtogether.Continuousdiffusionlanguagemodelsavoidthisbydenoisingasharedcontinuousstate,buttheirdenoiserseesonlythatstate,sonothingtiesittoavalidtokenconfigurationuntilitisfinallydecoded.Toaddressthis,weproposeHierarchicalContinuousDiffusionLanguageModels(HC-DLM),whichcouplediscretetokengenerationwithacontinuouslatenttrajectoryinasingle,principleddenoisingprocess,whosetrainingobjectiveisderivedfromavariationalboundonthetokenlikelihood.Incontrasttorecentmethodsthatattachcontinuouscontexttoaself-containeddiscretechain,HC-DLMmakesthelatenttheonlypersistentgenerativestate:tokensarereadoutfromitateverystepandfeedbackasascaffoldforthenextlatentupdate.Onstructuredreasoning(Sudoku),mathematicalplanning(Countdown)andlanguagemodeling(LM1B),HC-DLMimprovesoverdiscreteandcontinuousdiffusionbaselinesatmatchedmodelsize,inpuzzleaccuracyonSudokuandCountdownandingenerativeperplexityonLM1B.Projectpage:https://hc-dlm.github.io/.

View arXiv pageView PDFProject pageGitHub1Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.02193 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.02193 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.02193 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Hierarchical Masked Diffusion Language Models

arXiv cs.CL

The paper introduces Hierarchical Masked Diffusion Language Models (HMDLM), which incorporate position-level priority hierarchies into masked diffusion language models, achieving lower perplexity on LM1B and WikiHow and improved Sudoku solution accuracy over existing diffusion baselines.

Continuous Diffusion Language Models (CDLM's)

Hacker News Top

The article discusses the resurgence of continuous diffusion models for language generation, highlighting recent research and historical context that challenges the dominance of autoregressive language models.

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Hugging Face Daily Papers

LangFlow presents the first continuous diffusion language model that rivals discrete diffusion approaches, challenging the long-held belief that continuous diffusion is inferior for language modeling. The work introduces key ingredients like optimal Gumbel-based noise scheduling and demonstrates competitive perplexity and transfer learning performance compared to discrete diffusion baselines.

Token Time Continuous Diffusion for Language Modeling

Hugging Face Daily Papers

Introduces Token Time Continuous Diffusion (TTCD), a new diffusion language model that operates in continuous space with per-token times, outperforming discrete models at high speedups in conditional generation and Sudoku solving.

TextLDM: Language Modeling with Continuous Latent Diffusion

Hugging Face Daily Papers

This paper introduces TextLDM, a method that adapts visual latent diffusion transformers for language modeling by mapping discrete tokens to continuous latents. It demonstrates that this approach, enhanced by representation alignment, matches GPT-2 performance and unifies visual and text generation architectures.