Build a modern LLM from scratch. Every line commented. Explained like we are five. https://t.co/DAROHuZkSU

raiyanyahya/how-to-train-your-gpt

Source: https://github.com/raiyanyahya/how-to-train-your-gpt

🧠 How to Train Your GPT

A guide to building a world-class language model from absolute scratch. Taught like you’re five. Built like you’re an engineer.

I made this with the goal of learning something I didn’t understand completely. Specifically the attention part. I use AI a lot to understand key concepts and verifying them.

📖 What Is This?

This is a 12-chapter, 3,900+ line interactive textbook that teaches you how to build, train and run a modern language model from absolute scratch. The same family of architecture behind ChatGPT, Claude, LLaMA and Mistral.

You won’t just read about Transformers. You’ll write every line yourself: tokenizer, embeddings, attention, training loop, inference engine. Every single line annotated to explain what it does and why it’s there.

🤔 Why This Exists

Most ML tutorials fall into one of two traps:

The goal: After finishing, you won’t just know that attention “works”. You’ll understand the variance argument behind 1/√d_k. How RoPE captures relative position through rotation. Why pre-norm beats post-norm for deep networks. And exactly where every gradient flows during backpropagation.

👥 Who Is This For?

🔧 Prerequisites: Python basics (variables, functions, classes, pip install). That’s it. No calculus, no linear algebra, no PyTorch experience required. We teach those as we go.

🗺️ Chapters

⭐ Start with Chapter 0 and read sequentially. Each builds on the previous.

🏗️ What You’ll Build

💎 ~860 lines of core model code, ~2,600 lines of explanation and diagrams

🏛️ Architecture

This guide implements the latest publicly-documented decoder-only Transformer:

ℹ️ GPT-4 and Claude architectures are proprietary/undisclosed. This teaches the best publicly-confirmed architecture: what LLaMA 3, Mistral and Qwen 2.5 use.

🚀 Quick Start

# 1. Clone
git clone https://github.com/raiyanyahya/how-to-train-your-gpt.git
cd how-to-train-your-gpt

# 2. Create environment
python -m venv gpt_env
source gpt_env/bin/activate          # Mac/Linux
# gpt_env\Scripts\activate           # Windows

# 3. Install dependencies (CPU version — for GPU see below)
pip install torch tiktoken datasets numpy matplotlib --index-url https://download.pytorch.org/whl/cpu

# 4. Verify GPU (optional but recommended)
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}')"

# 5. Start reading!
open chapters/00_overview.md

Run the training script:

python main.py

This uses the tiny config (d_model=256, 4 layers) by default. Training takes a few minutes on CPU. For the GPT-2 scale config (151M params, 768 dims, 12 layers), edit the config in main.py and uncomment the larger configuration.

💻 The default config uses a tiny model (d_model=256, 4 layers, 17M params) that runs in minutes on CPU. For the full GPT-2 scale (151M params, 768 dims, 12 layers), edit the config in main.py and uncomment the larger configuration. You’ll need a GPU for that one.

📓 Jupyter Notebooks

Alongside the textbook, each chapter has a companion notebook you can run live. These strip away the explanations and give you pure, clean code that executes from top to bottom. If the textbook teaches you why, the notebooks let you see it happen.

We’re going to run this whole project on a very small dataset so you can watch training happen in minutes rather than weeks. Every notebook is self-contained — open it, run all cells, and you’ll see the model learn in real time.

# Install everything you need
pip install jupyter tiktoken torch numpy datasets matplotlib --index-url https://download.pytorch.org/whl/cpu

# Start with chapter 2 (tokenization)
jupyter notebook notebooks/02_tokenization.ipynb

Notebooks live in the notebooks/ directory, one per chapter. Open any of them and hit Cell → Run All.

📖 How to Read

Each chapter follows the same 4-step structure:

💡 Tip: Lost in the code? Jump back to the analogy. Confused by the math? Skip to the worked example.

✨ What Makes This Different

🎯 Skills You’ll Gain

• ✅ Explain how GPT-4 tokenizes text using BPE
• ✅ Understand why RoPE, RMSNorm and SwiGLU replaced older techniques
• ✅ Compute attention scores manually for a 3-token sentence
• ✅ Debug a Transformer training loop (loss spikes, flat lines, overfitting)
• ✅ Choose sampling parameters (temperature, top_k, top_p) for different use cases
• ✅ Understand why KV caching is critical for production inference
• ✅ Read modern ML papers with confidence (you’ll recognize every component)

🔮 Next Steps After Finishing

📁 File Structure

📦 how-to-train-your-gpt/
├── 📄 README.md              ← You are here
└── 📂 chapters/
    ├── 🏠 00_overview.md     ← What is a GPT? Why build one?
    ├── 🔧 01_setup.md        ← Install tools, GPU vs CPU, venv basics
    ├── 🔪 02_tokenization.md ← BPE walkthrough, EOS tokens, emoji handling
    ├── 🧊 03_embeddings.md   ← How numbers become meaning, king − man + woman
    ├── 📍 04_positional_encoding.md ← RoPE math, numerical example, theta
    ├── 🧠 05_attention.md    ← ⭐ THE CORE (713 lines). Q,K,V, scaling, causal mask
    ├── 🧱 06_transformer_block.md ← RMSNorm, SwiGLU, residuals, pre-norm vs post
    ├── 🏗️ 07_gpt_model.md    ← Complete 151M model, weight tying, logits explained
    ├── 🏋️ 08_training.md     ← Cross-entropy, backprop, AdamW, cosine warmup
    ├── 🎤 09_inference.md    ← KV cache, temperature, top-k/p, beam search
    ├── 📜 10_full_script.md  ← Runnable main.py
    └── 📊 11_glossary.md     ← Architecture provenance, parameter breakdown

"Any sufficiently explained technology is indistinguishable from magic. Until you build it yourself."

_{⭐ Star this repo if you found it useful | 🐛 Issues & PRs welcome | 📖 Happy learning!}