Training an LLM from scratch on 1800's texts (160GB dataset)
Summary
The author details training an LLM from scratch on 160GB of 1800s English texts, with a 500M parameter evaluation model and plans for a 2B model, showing promising results for historical Q&A.
Similar Articles
Making a vintage LLM from scratch
The author documents their journey of building a 340M parameter LLM from scratch, trained exclusively on pre-1900 texts, including custom datasets, training scripts, and open-sourcing the model and code.
I trained a 1B-parameter LLM from scratch on 20B tokens for about $200
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens for about $200, using fineweb-edu for pretraining and LoRA finetuning on OpenHermes. The project includes open-source code, model weights, and a demo website.
I trained a 75M parameter LLM from scratch on 18B tokens and it beats a model almost double its size
Trained a 75M parameter LLM called KeyLM from scratch on 18B tokens, achieving competitive instruction-following scores against larger models while using fewer parameters and less data.
Toward LLMs Beyond English-Centric Development
This paper demonstrates that LLMs are heavily biased toward English, and shows that continual pre-training does not offer cost advantages over training from scratch for adapting models to other languages, especially for cultural understanding.
I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB [R]
A developer created a 250M parameter quantized LLM from scratch, trained on 30B tokens, deployable in 60 MB on CPU with a novel disk-based long context system supporting up to 100M tokens.