Token Time Continuous Diffusion for Language Modeling
Summary
Introduces Token Time Continuous Diffusion (TTCD), a new diffusion language model that operates in continuous space with per-token times, outperforming discrete models at high speedups in conditional generation and Sudoku solving.
View Cached Full Text
Cached at: 07/20/26, 09:41 AM
Paper page - Token Time Continuous Diffusion for Language Modeling
Source: https://huggingface.co/papers/2607.14106
Abstract
Inthispaperweintroducetokentimecontinuousdiffusion(TTCD),anewdiffusionlanguagemodelwhich(a)operatesincontinuousspace,deterministicallymappingGaussiannoisetoafinaltokencanvaswithnofurthersampling,andcrucially(b)incorporatesanewnotionofper-tokentimes,withsometokensproceedingfromnoisetotokenatafasterratethanothers.ContinuousspacemodelinghelpsTTCDavoidtheparallelsamplingofmultipletokens,whichisakeysourceofinaccuracyathighspeedupsformodelsthatiteratepurelyindiscretespace.Thenotionofper-tokentimeshelpsTTCDtobettermodelconditionalgeneration,allowsformoresuretokenstoproceedatafasterrate,andallowsfordifferentiatedinter-tokeninfluencesduringrefinement.TTCDoutperformsdiscretemodelsathighspeedups.Wetraina160MparameterTTCDmodelonOpenWebText,andthenself-distillit;wefindthatathighspeedupswearecomparableinunconditionalgenerationquality,andoutperforminconditionalgeneration,severalexistingmodelsofsimilarsizetrained,onthesamedata,andself-distilled.WeachievesimilargainsinSudokusolvingaswell.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.14106 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.14106 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.14106 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Hierarchical Continuous Diffusion Language Models
The paper proposes Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single denoising process derived from a variational bound on token likelihood. HC-DLM improves over discrete and continuous diffusion baselines on Sudoku, Countdown, and LM1B tasks.
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
LangFlow presents the first continuous diffusion language model that rivals discrete diffusion approaches, challenging the long-held belief that continuous diffusion is inferior for language modeling. The work introduces key ingredients like optimal Gumbel-based noise scheduling and demonstrates competitive perplexity and transfer learning performance compared to discrete diffusion baselines.
Continuous Diffusion Language Models (CDLM's)
The article discusses the resurgence of continuous diffusion models for language generation, highlighting recent research and historical context that challenges the dominance of autoregressive language models.
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
Set Diffusion introduces a new class of language models that interpolates between autoregressive and diffusion models by factorizing token generation over flexible-position, flexible-length token sets. This enables faster decoding and flexible token ordering, achieving better speed-quality tradeoffs on reasoning, summarization, and unconditional generation tasks.
TextLDM: Language Modeling with Continuous Latent Diffusion
This paper introduces TextLDM, a method that adapts visual latent diffusion transformers for language modeling by mapping discrete tokens to continuous latents. It demonstrates that this approach, enhanced by representation alignment, matches GPT-2 performance and unifies visual and text generation architectures.