I pretrained and post trained a 500M parameter LLM and 330M parameter Image generator from scratch
Summary
The author details the process of pretraining and post-training a 500M parameter language model and a 330M parameter image generator entirely from scratch.
Similar Articles
I trained a 75M parameter LLM from scratch on 18B tokens and it beats a model almost double its size
Trained a 75M parameter LLM called KeyLM from scratch on 18B tokens, achieving competitive instruction-following scores against larger models while using fewer parameters and less data.
@tom_doerr: Trains billion-parameter LLMs from scratch on a single GPU https://github.com/FareedKhan-dev/train-llm-from-scratch…
A GitHub repository provides scripts to train billion-parameter language models from scratch on a single GPU using PyTorch, based on the Transformer architecture.
I developed a 270 million parameter language model entirely from scratch as an independent research project
A 270M parameter language model trained from scratch on English Wikipedia and instruction-tuned for conversational AI, developed as an independent research project.
Making a vintage LLM from scratch
The author documents their journey of building a 340M parameter LLM from scratch, trained exclusively on pre-1900 texts, including custom datasets, training scripts, and open-sourcing the model and code.
I trained a 1B-parameter LLM from scratch on 20B tokens for about $200
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens for about $200, using fineweb-edu for pretraining and LoRA finetuning on OpenHermes. The project includes open-source code, model weights, and a demo website.