@Letian_Wang_6: Language got AR; everything else got diffusion - an unstable equilibrium. A question that won't leave me alone lately: …
Summary
A researcher muses on the unstable equilibrium where language uses autoregressive models while other modalities use diffusion, and speculates that a unified multimodal architecture depends on the order each modality reveals information. He seals a bet on this idea.
View Cached Full Text
Cached at: 08/07/26, 12:56 PM
Language got AR; everything else got diffusion - an unstable equilibrium. A question that won’t leave me alone lately: as LLMs, video, and robotics all mature, who absorbs whom? What does a truly native unified multimodal architecture look like, not AR-for-text stitched onto diffusion-for-pixels/actions?
I’ve come to believe the answer hinges on a design axis almost nobody names explicitly: not which loss each modality gets, but in what order each modality should reveal its information. Get that axis right, and the rest of the stack collapses into something almost embarrassingly simple.
Sealing my current bet here: 5dd31d98f71c2a6d525e2032e3afda7c277204cc537f95316ad2810ce0e8eec5
Ask me in 1 year :)
Similar Articles
Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding
The paper introduces Nemotron-Labs-Diffusion, a tri-mode language model that unifies autoregressive, diffusion, and self-speculation decoding, achieving superior throughput and efficiency compared to existing models.
@BetaTomorrow: #DeepManifoldInterpretation Paper: On the Entropy Calibration of Language Models Author: Steven Cao, Gregory Valiant, a…
The paper 'On the Entropy Calibration of Language Models' interprets rising entropy as increasing diffusion of accessible pathways in autoregressive generation, proposing that scaling has limited benefits due to heavy-tailed data and suggesting a pathway-aware decoding alternative.
I built a diffusion language model from scratch. It writes flawless sentences that mean nothing, and that is the interesting part.
The author built Joey, a 170M parameter masked diffusion language model from scratch, trained on FineWeb-Edu and fine-tuned on DailyDialog, achieving fluent but incoherent sentences due to capacity limitations. The project highlights the differences from autoregressive LLMs and the lessons learned from building and debugging the system.
@radixark: The real world is multimodal. For AI to understand and recreate it, models need to learn across modalities. In our late…
Radixark shares a blog post about how Miles supports multimodal learning for AI models with a shared post-training design for vision-language models and diffusion models.
Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion
This paper introduces a diffusion language model that treats text as a continuous process over binary bitstreams, using entropy-gated stochastic sampling to close the performance gap with autoregressive models. It achieves state-of-the-art results on LM1B and OWT benchmarks while reducing memory footprint.