Stanford CS336: Language Modeling from Scratch – 18 Complete Lectures · Open-Source Assignments · Zero-Cost Self-Study

Stanford CS336: Language Modeling from Scratch

CS336 is a 5-credit graduate course at Stanford University, jointly taught by Tatsunori Hashimoto and Percy Liang (Spring 2026). The course focuses on building language models from scratch. As the course homepage aptly states, this course is to language models what an operating systems course is to operating systems: rather than teaching you “how to use” large models, it guides you through building a complete language model pipeline from start to finish, covering data collection and cleaning, Transformer model construction, training, and evaluation prior to deployment.

Stanford CS336 | Language Modeling from Scratch
From cs336.stanford.edu

This positioning leads to two defining characteristics:

1. Massive engineering workload. The instructors explicitly state that the assignment code volume “far exceeds that of typical AI courses,” with almost no provided scaffolding. Students must write their own tokenizer, build their own model, implement the FlashAttention2 kernel, and configure distributed training from scratch. “Solid Python proficiency” is listed as the primary prerequisite, even before deep learning experience.

2. Comprehensive coverage of the entire LLM lifecycle, not just modeling. From the syllabus, only lectures 3 and 4 focus on “model architecture.” The rest are devoted to systems (GPUs/kernels/parallelism), data, Scaling Laws, inference, evaluation, and post-training. This distribution itself reflects the post-2024 industry consensus on “what it takes to build a large model”: model architecture is only a small part of the equation; compute efficiency and data quality are the major components.

Course Backbone: Five Assignments Forming a Complete LLM Pipeline

The course’s five programming assignments (Assignment 1–5) form a progressive, complete chain, interleaved with lectures:

These five assignments together constitute the complete production pipeline for a modern LLM, from raw internet data to a conversational model. Assignment 4 is particularly noteworthy: genuinely cleaning Common Crawl into training data is a step most NLP courses skip entirely but is one of the most time-consuming in industry.

YouTube Lectures: Complete Recordings of All 18 Sessions

YouTube
Stanford CS336: Language Modeling from Scratch | Spring 2026
This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing th…

The playlist, published on the Stanford Online @StanfordOnline channel, contains 18 videos (17 lectures + 1 guest lecture), each lasting between 1 hour 15 minutes and 1 hour 29 minutes, with content corresponding directly to the syllabus:

Module 1: Model Foundations (Lectures 1–4)

1. Overview, Tokenization
2. PyTorch (including FLOPs, memory, arithmetic intensity)
3. Architectures (designs and hyperparameters)
4. Attention Alternatives (including MoE)

Module 2: Systems & Efficiency (Lectures 5–8)
5. GPUs, TPU fundamentals
6. Kernels, Triton, XLA (operator kernel programming)
7–8. Parallelism (two lectures: data, tensor, pipeline parallelism, etc.)

Module 3: Scaling & Inference (Lectures 9–11)
9. Scaling Laws
10. Inference (deployment efficiency)
11. Scaling Laws (continued, including data-centric scaling)

Module 4: Evaluation & Data (Lectures 12–14)
12. Evaluation (model assessment methodologies)
13. Data: Sources, Datasets
14. Data (filtering, deduplication, mixing, synthetic data)

Module 5: Post-Training & Alignment (Lectures 15–17 + Guest)
15. Mid/Post-Training (SFT/RLHF intro)
16. Post-Training - RLVR (Reinforcement Learning with Verifiable Rewards, the core training method for current reasoning models)
17. Alignment - Multimodality
• Guest Lecture: Dan Fu

A detail: the syllabus originally listed another guest lecture by Daniel Selsam (June 1), but it was not included in the playlist. The final public release comprises 17 lectures plus Dan Fu’s talk—18 videos in total. Recordings were uploaded in batches starting mid-April, aligned with the semester progress.

Notable Course Design Choices

Strict limits on AI tool usage. The Honor Code explicitly permits using LLMs for low-level or conceptual questions but forbids using AI to solve problems directly, prohibits Copilot or similar AI autocompletion, and disallows referencing existing online implementations. In a course that “teaches people to train AI,” restricting AI-assisted writing is logically consistent—the course’s entire value lies in hands-on implementation.

Well-paved path for self-learners. The course lists B200 GPU rental prices from multiple cloud providers (Modal, Lambda, RunPod, Nebius, Together—about $5–7.5 per hour) and offers practical advice: debug on CPU first, only switch to GPU for actual training and benchmarking. The course is sponsored by Modal for compute resources.

Guest lineup reflects course vision. Guest lecturer Dan Fu is a Stanford PhD and founding member of Lambda Lab, focusing on efficient systems and model training.

Evaluation of the Course

CS336 is currently the most publicly accessible course that covers the full landscape of large language model engineering with the deepest implementation depth. It assumes you can already train small models (requiring prerequisites like CS224N/CS229 and other deep learning courses) and builds upon that to address the true differentiators—systems optimization, Scaling Laws, data processing, and post-training RL—at an implementable depth. Lectures 6–8 (kernels and parallelism) and Lecture 16 (RLVR) in particular cover content that most university courses either ignore or address only at the literature-survey level.

For learners aiming to systematically build LLM engineering skills, the lectures, five assignments, and course materials are all publicly available—a rare resource for self-study. However, one should honestly assess the barrier: without solid PyTorch and systems architecture fundamentals (concepts like memory hierarchy and arithmetic intensity), the risk of dropping out midway is significant.