Token Time Continuous Diffusion for Language Modeling
Summary
Introduces Token Time Continuous Diffusion (TTCD), a new diffusion language model that operates in continuous space with per-token times, outperforming discrete models at high speedups in conditional generation and Sudoku solving.
View Cached Full Text
Cached at: 07/20/26, 09:41 AM
Paper page - Token Time Continuous Diffusion for Language Modeling
Source: https://huggingface.co/papers/2607.14106
Abstract
Inthispaperweintroducetokentimecontinuousdiffusion(TTCD),anewdiffusionlanguagemodelwhich(a)operatesincontinuousspace,deterministicallymappingGaussiannoisetoafinaltokencanvaswithnofurthersampling,andcrucially(b)incorporatesanewnotionofper-tokentimes,withsometokensproceedingfromnoisetotokenatafasterratethanothers.ContinuousspacemodelinghelpsTTCDavoidtheparallelsamplingofmultipletokens,whichisakeysourceofinaccuracyathighspeedupsformodelsthatiteratepurelyindiscretespace.Thenotionofper-tokentimeshelpsTTCDtobettermodelconditionalgeneration,allowsformoresuretokenstoproceedatafasterrate,andallowsfordifferentiatedinter-tokeninfluencesduringrefinement.TTCDoutperformsdiscretemodelsathighspeedups.Wetraina160MparameterTTCDmodelonOpenWebText,andthenself-distillit;wefindthatathighspeedupswearecomparableinunconditionalgenerationquality,andoutperforminconditionalgeneration,severalexistingmodelsofsimilarsizetrained,onthesamedata,andself-distilled.WeachievesimilargainsinSudokusolvingaswell.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.14106 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.14106 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.14106 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
LangFlow presents the first continuous diffusion language model that rivals discrete diffusion approaches, challenging the long-held belief that continuous diffusion is inferior for language modeling. The work introduces key ingredients like optimal Gumbel-based noise scheduling and demonstrates competitive perplexity and transfer learning performance compared to discrete diffusion baselines.
Continuous Diffusion Language Models (CDLM's)
The article discusses the resurgence of continuous diffusion models for language generation, highlighting recent research and historical context that challenges the dominance of autoregressive language models.
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
Set Diffusion introduces a new class of language models that interpolates between autoregressive and diffusion models by factorizing token generation over flexible-position, flexible-length token sets. This enables faster decoding and flexible token ordering, achieving better speed-quality tradeoffs on reasoning, summarization, and unconditional generation tasks.
TextLDM: Language Modeling with Continuous Latent Diffusion
This paper introduces TextLDM, a method that adapts visual latent diffusion transformers for language modeling by mapping discrete tokens to continuous latents. It demonstrates that this approach, enhanced by representation alignment, matches GPT-2 performance and unifies visual and text generation architectures.
Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion
This paper introduces a diffusion language model that treats text as a continuous process over binary bitstreams, using entropy-gated stochastic sampling to close the performance gap with autoregressive models. It achieves state-of-the-art results on LM1B and OWT benchmarks while reducing memory footprint.