@Letian_Wang_6: Language got AR; everything else got diffusion - an unstable equilibrium. A question that won't leave me alone lately: …
Summary
A researcher muses on the unstable equilibrium where language uses autoregressive models while other modalities use diffusion, and speculates that a unified multimodal architecture depends on the order each modality reveals information. He seals a bet on this idea.
View Cached Full Text
Cached at: 08/07/26, 12:56 PM
Language got AR; everything else got diffusion - an unstable equilibrium. A question that won’t leave me alone lately: as LLMs, video, and robotics all mature, who absorbs whom? What does a truly native unified multimodal architecture look like, not AR-for-text stitched onto diffusion-for-pixels/actions?
I’ve come to believe the answer hinges on a design axis almost nobody names explicitly: not which loss each modality gets, but in what order each modality should reveal its information. Get that axis right, and the rest of the stack collapses into something almost embarrassingly simple.
Sealing my current bet here: 5dd31d98f71c2a6d525e2032e3afda7c277204cc537f95316ad2810ce0e8eec5
Ask me in 1 year :)
Similar Articles
Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding
The paper introduces Nemotron-Labs-Diffusion, a tri-mode language model that unifies autoregressive, diffusion, and self-speculation decoding, achieving superior throughput and efficiency compared to existing models.
I built a diffusion language model from scratch. It writes flawless sentences that mean nothing, and that is the interesting part.
The author built Joey, a 170M parameter masked diffusion language model from scratch, trained on FineWeb-Edu and fine-tuned on DailyDialog, achieving fluent but incoherent sentences due to capacity limitations. The project highlights the differences from autoregressive LLMs and the lessons learned from building and debugging the system.
Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion
This paper introduces a diffusion language model that treats text as a continuous process over binary bitstreams, using entropy-gated stochastic sampling to close the performance gap with autoregressive models. It achieves state-of-the-art results on LM1B and OWT benchmarks while reducing memory footprint.
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
Set Diffusion introduces a new class of language models that interpolates between autoregressive and diffusion models by factorizing token generation over flexible-position, flexible-length token sets. This enables faster decoding and flexible token ordering, achieving better speed-quality tradeoffs on reasoning, summarization, and unconditional generation tasks.
@volokuleshov: New blog post: How to Build a Diffusion Language Model. Diffusion LLMs went from open problem to reality in 2 years (Me…
A comprehensive blog post by Volodymyr Kuleshov's Cornell group explains how to build diffusion language models, covering core techniques like masked diffusion, iterative refinement, variable-length generation, controllable generation, fast samplers, and RL post-training, using open-source models such as Mercury, Gemma Diffusion, and Nemotron Diffusion as examples.