Bulding my own Diffusion Language Model from scratch was easier than I thought [P]

Reddit r/MachineLearning 04/21/26, 05:23 PM Tools

Summary

Developer shares a minimalist 7.5M-parameter diffusion language model trained from scratch on Shakespeare, releasing the code as a learning resource.

Since I felt like I was relying on Claude Code a lot recently, I wanted to see how hard it is to implement a diffusion language model from scratch without the help of AI-Generated code. So I built one while waiting for the training for my master's thesis. This is what I got after a few hours of training on my MacBook Air M2. I trained on the tiny Shakespeare dataset from Karpathy and prompted "to be, " To be, fo hend! First her sense ountier to Jupits, be horse. Words of wisdom! The model has around 7.5M Params and vocabulary size is 66 (65 chars + \[MASK\]. I definitely did not train long enough, but I ran out of time for this one. Projects like these help me make sense of big scary words like (discrete) diffusion, encoder, decoder, tokenizer. Maybe this encourages someone :) Check out the code here if you're interested: [https://github.com/Encrux/simple\_dlm](https://github.com/Encrux/simple_dlm) Thanks for reading! Be horse.

Original Article

Similar Articles

I built a diffusion language model from scratch. It writes flawless sentences that mean nothing, and that is the interesting part.

Reddit r/AI_Agents

The author built Joey, a 170M parameter masked diffusion language model from scratch, trained on FineWeb-Edu and fine-tuned on DailyDialog, achieving fluent but incoherent sentences due to capacity limitations. The project highlights the differences from autoregressive LLMs and the lessons learned from building and debugging the system.

@volokuleshov: New blog post: How to Build a Diffusion Language Model. Diffusion LLMs went from open problem to reality in 2 years (Me…

X AI KOLs Timeline

A comprehensive blog post by Volodymyr Kuleshov's Cornell group explains how to build diffusion language models, covering core techniques like masked diffusion, iterative refinement, variable-length generation, controllable generation, fast samplers, and RL post-training, using open-source models such as Mercury, Gemma Diffusion, and Nemotron Diffusion as examples.

Diffusion Language Models: An Experimental Analysis

arXiv cs.AI

A systematic experimental analysis evaluating eight state-of-the-art Diffusion Language Models across multiple benchmarks, analyzing trade-offs between generation quality and computational efficiency.

Improved Large Language Diffusion Models

arXiv cs.CL

iLLaDA is an 8B parameter masked diffusion language model with fully bidirectional attention, trained from scratch on 12T tokens. It shows broad improvements over LLaDA and remains competitive with Qwen2.5 7B on several benchmarks. The model and code are open-sourced.

I developed a 270 million parameter language model entirely from scratch as an independent research project

Reddit r/LocalLLaMA

A 270M parameter language model trained from scratch on English Wikipedia and instruction-tuned for conversational AI, developed as an independent research project.