I trained a 0.5M model on 1B tokens of Fineweb-edu dataset.
Summary
A personal project where a 0.5M parameter language model was trained on 1 billion tokens from the Fineweb-edu dataset.
Similar Articles
I trained a 1B-parameter LLM from scratch on 20B tokens for about $200
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens for about $200, using fineweb-edu for pretraining and LoRA finetuning on OpenHermes. The project includes open-source code, model weights, and a demo website.
I trained a 75M parameter LLM from scratch on 18B tokens and it beats a model almost double its size
Trained a 75M parameter LLM called KeyLM from scratch on 18B tokens, achieving competitive instruction-following scores against larger models while using fewer parameters and less data.
I developed a 270 million parameter language model entirely from scratch as an independent research project
A 270M parameter language model trained from scratch on English Wikipedia and instruction-tuned for conversational AI, developed as an independent research project.
[NEW] Supra-50M Released!
SupraLabs released Supra-50M, a compact 50M-parameter causal language model with base and instruct versions, trained on 20B tokens from fineweb-edu, achieving competitive benchmarks against larger models like GPT-2 and SmolLM.
I built a diffusion language model from scratch. It writes flawless sentences that mean nothing, and that is the interesting part.
The author built Joey, a 170M parameter masked diffusion language model from scratch, trained on FineWeb-Edu and fine-tuned on DailyDialog, achieving fluent but incoherent sentences due to capacity limitations. The project highlights the differences from autoregressive LLMs and the lessons learned from building and debugging the system.