@0xLogicrw: MIT's Kai-Ming He team has released a new language model, ELF (Embedded Language Flows). They bypass current autoregressive architectures by directly applying their expertise in diffusion models from the visual domain to text generation. Specifically: the entire generation process is embedded in a continuous vector space, converting it back to...

MIT's Kai-Ming He team has released a new language model called ELF (Embedded Language Flows). They bypass current autoregressive architectures by directly applying their expertise in diffusion models from the visual domain to text generation. Specifically: the entire generation process is embedded in a continuous vector space, with the data only being converted back to discrete tokens at the final step. Image pixels can change color smoothly, making them naturally suitable for diffusion model denoising. Text, however, does not work this way; words are disconnected, with no middle ground between "cat" and "dog." To bridge this gap, previous text diffusion models struggled: they either forced vocabulary lookups at every denoising step or attached a separate decoder outside the model. ELF completely severs this entanglement: it ignores the middle stages, allowing the model to quietly perform denoising calculations in continuous space, converting vectors back to text only at the very last moment using the same network. This approach not only works but is extremely data-efficient. In evaluations on OpenWebText, the 105M-parameter ELF-B model reduced perplexity (Gen. PPL) to approximately 24.1 using only 32 sampling steps, outperforming several baseline models. More importantly, it consumed only about 45B training tokens, whereas comparable competitors typically require over 500B, effectively reducing training volume by 90%. This result demonstrates that the inherent discreteness of language does not block the path for continuous diffusion. Once this approach matures, future large models could potentially break free from the architectural constraints of generating text only "left-to-right, one word at a time."