A developer shares lessons learned while building Apex-2, a small open LLM trained on ~80B tokens for about $2,000, covering GPU bottlenecks, DiLoCo-style distributed training with GH200 instances, and aggressive MinHash deduplication of FineWeb-Edu and DCLM.
Hi everyone, thank you so much for all the interest in my model. It's more than I expected. Here is a short summary of the trial and error I went through while building Apex-2. 1. GPUs were always the bottleneck I planned to train on about 1T tokens, but in the end I could only train on about 80B. FineWeb-Edu alone is about 1.3T tokens, and I clearly underestimated the scale: a single H100 was not enough. This project really showed me why so much money goes into GPUs and VRAM. 2. DiLoCo Within the same region, running two separate instances worked better for me. Instead of a 2x H100 instance, I used two GH200 instances and merged the models every fixed number of steps. Each GPU reached about 40% MFU. A 2x H100 instance costs more per GPU (about $4.19/hour, vs $2.29/hour for a GH200). With two GH200 instances, each at about 40% MFU and merging every 350 steps, training ran about 1.9x faster than on one GPU, at a lower price. (The data-center network between the instances probably helped; a merge usually took less than a minute.) 3. Deduplicating FineWeb-Edu and DCLM When I deduplicated the whole corpus at once (MinHash, near-duplicates included), 57% of my FineWeb-Edu sample and 34% of DCLM turned out to be duplicates. FineWeb-Edu is only deduplicated within each Common Crawl snapshot, so pages that were crawled again in later snapshots remain. With a bigger budget this might not matter, but I had to get the most out of very little compute, so I removed them. (Note: the FineWeb authors reported that deduplicating across snapshots did not improve their results, so this is a trade-off rather than a free win.) For the MoE architecture I followed the Mixtral paper (https://arxiv.org/abs/2401.04088). The whole project cost about $2,000. I also write down my thoughts on LLMs here, if you're interested: https://github.com/DW-dev-UE/LLM-from-scratch/blob/main/ThinkingLab/ThinkingLab.en.md I didn't plan to share this model on Reddit, so I'm afraid I don't remember many of the smaller mistakes 😭 I'm now building a 21B-parameter MoE model, and I'll share the lessons and mistakes from that one as I go. Thank you again for your interest! If I get the chance, I'd love to join a lab and help build LLMs for everyone.
YOON1v released Apex-2, a from-scratch decoder-only Mixture-of-Experts LLM with 3.87B total / 1.45B active parameters, pretrained on 86.5B tokens and SFT-tuned, with full code, architecture writeup, and benchmark results (HumanEval 43.9) released on Hugging Face and GitHub.
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens for about $200, using fineweb-edu for pretraining and LoRA finetuning on OpenHermes. The project includes open-source code, model weights, and a demo website.
A tweet highlights Stanford's 2-hour course on building LLMs from scratch, covering tokenization, decoding, training pipelines, and architecture, noting that Anthropic pays high salaries for such expertise.
A developer created a 250M parameter quantized LLM from scratch, trained on 30B tokens, deployable in 60 MB on CPU with a novel disk-based long context system supporting up to 100M tokens.
The author shares learnings from training a 160M parameter LLM from scratch, experimenting with architectures like multi-token prediction and hierarchical reasoning models. They emphasize the importance of fast iteration, simplifying ideas, and understanding why architectures work.