Tag
NVIDIA introduces Sigma, the first large-scale (3B/8B) continuous diffusion language model, which denoises Gaussian-corrupted token embeddings over steerable low-dimensional ODE/SDE trajectories, achieving performance parity with discrete diffusion and autoregressive models on math and coding benchmarks while enabling embedding-space steering for quality-diversity trade-offs and efficient few-step distillation.
The paper proposes Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single denoising process derived from a variational bound on token likelihood. HC-DLM improves over discrete and continuous diffusion baselines on Sudoku, Countdown, and LM1B tasks.
The article discusses the resurgence of continuous diffusion models for language generation, highlighting recent research and historical context that challenges the dominance of autoregressive language models.
Introduces Discrete Stochastic Localization (DSL), a continuous-state diffusion framework for non-autoregressive text generation that uses unit-sphere token embeddings and a timestep-invariant denoiser, achieving better distributional faithfulness than masked discrete diffusion models on OpenWebText.
Introduces Token Time Continuous Diffusion (TTCD), a new diffusion language model that operates in continuous space with per-token times, outperforming discrete models at high speedups in conditional generation and Sudoku solving.
LangFlow presents the first continuous diffusion language model that rivals discrete diffusion approaches, challenging the long-held belief that continuous diffusion is inferior for language modeling. The work introduces key ingredients like optimal Gumbel-based noise scheduling and demonstrates competitive perplexity and transfer learning performance compared to discrete diffusion baselines.