@kuririrn: Studying to grasp the recent trends in scratch development of LLMs Stanford CS336 Lang. Modeling from Scratch | Spring …

Studying to grasp the recent trends in scratch development of LLMs Stanford CS336 Lang. Modeling from Scratch | Spring 2025 | Lec. 3: Architectures, Hyperparameters

TL;DR: This course dives deep into the evolution of architectures from the original Transformer to modern LLMs, focusing on convergent evolutions such as pre-normalization, RMS normalization, rotary position embeddings (RoPE), and provides data-driven recommendations for hyperparameter selection.

Course Overview & Background

You may have noticed I’m not as creative as Percy when it comes to lecturing. So you’ll see PowerPoint slides rather than executable Python slides, but you’ll also find PDF files on the website. I’ve named this lecture “Everything You Didn’t Want to Know About LM Architecture and Training” because we’re going to dive into details that most other courses won’t tell you — like “what should my hyperparameters be?” and some minor administrative matters. Also, if you’re working on assignments, we’re updating them (mainly small errors), so make sure to pull the latest updates as you go.

Where We Start: A Quick Review of Transformers

We’ll first quickly review the Transformer, then give two variants of the standard Transformer. One comes from the standard Transformer (like you saw in 224n), and the other is a modern consensus variant that you’ll actually implement in the assignment. Then, we’ll take a more data-driven perspective on Transformer architectures: many LLMs have been trained, and you can read all those papers to try to understand what changed and what is common. Through this almost “evolutionary” analysis, we try to identify the elements that make Transformers actually work. The best way to learn is by doing, but this lecture is about learning from others’ experience.

Architecture Changes: From Original Transformer to Modern Consensus

Pre-Normalization vs Post-Normalization

The original Transformer paper used post-normalization: adding layer normalization after multi-head attention and after the feed-forward network, then returning to the residual stream. Very early on, people realized that moving layer normalization to before each subcomponent (pre-normalization) works better in many ways. Essentially all modern LMs use pre-normalization. One exception is OPT 350M — they messed up and it was abandoned during training; it’s an interesting finding from my architecture survey.

The argument: pre-normalization is more stable. If you want to use post-normalization, you need careful learning rate warm-up for stable training. Early work by Salazar et al. 2020 showed that post-normalization (blue curve) causes more loss spikes; while pre-normalization keeps gradient magnitudes constant and overall gradient norms are smaller. Today, pre-normalization and other normalization techniques are used as stability aids when training large neural networks.

Double Normalization (A Newer Variant)

Someone in 224N this year asked: “Why must layer normalization be placed at the front? Why can’t it be after the feed-forward network?” Sure you can. Recently, some have simply added layer normalization both before and after the block. Grok and GMA 2 both adopt this approach: normalize both before and after. Some only normalize after attention or after the feed-forward. This is actually an interesting variation — pre-normalization dominated for a long time, but now it’s shifting. Some have evaluated this method and found it more stable and friendly for training larger models.

Why Is Layer Normalization Inside the Residual Path Bad?

An intuitive argument: residual connections provide an identity mapping from the top of the network to the bottom, which makes gradient propagation very easy. Placing layer normalization in the middle may interfere with this gradient behavior. Looking back at the earlier figure, that’s what you’d expect to see.

RMS Normalization: One of the Convergent Changes

In the original Transformer, people used Layer Normalization (LayerNorm): subtract the empirical mean, divide by standard deviation, then scale and shift. Many models work well with LayerNorm. But many models have now switched to RMS Normalization — dropping the mean adjustment and also omitting the bias term (β). The LLaMA family, PaLM, Chinchilla, T5 have all moved to RMS normalization. Two reasons: first, there is no significant difference (models trained with RMS normalization are just as good as those with LayerNorm); second, it’s faster and equally good.

What makes it faster? Without subtracting the mean, there are fewer operations; without adding a bias term β, you need to load fewer parameters from memory into the compute unit. Someone might think: in 224n you told me only matrix multiplications matter for runtime, this isn’t matrix multiplication, so why do I care? There’s a nice paper (Ivan et al. 2023, title roughly “Memory Movement Is All You Need”) that profiled all components of a Transformer: tensor contractions (matrix multiplies) account for 99.88% of Transformer FLOPs, but normalization operations account for only 0.17% of FLOPs yet consume about 25% of runtime. One important reason is memory movement. So architecture design can no longer consider only FLOPs; you have to carefully think about memory access.

Convergent Evolution of Position Embeddings

I put together a small table tracking from the original Transformer (2017) to the latest models (2025). You’ll see certain architectural changes being explored — for the position embeddings column: people used absolute positions, relative positions, RoPE, ALiBi, etc., but roughly from 2023 onward, everyone only uses RoPE (Rotary Position Embeddings). Neural architectures are almost convergently evolving. Your assignment requires implementing RoPE.

Other Details: Activation Functions and Bias

Feed-forward layers now use what’s called SwiGLU (pronounced “swiggloo” in the assignment). Linear layers no longer use bias terms. This is also common practice in modern LLMs.

Hyperparameter Selection

Once the architecture is set, you still need to choose hyperparameters, such as hidden dimension, MLP inner projection dimension, number of attention heads, vocabulary size, etc. You can’t just pick arbitrarily — you should choose them in a reasonably intelligent way. These will be covered in the subsequent sections.

Summary

We’ve seen a convergent evolution from the original Transformer to a LLaMA-style baseline — pre-normalization, RMS normalization, RoPE, SwiGLU, no bias. But people still do various extra things (like double normalization, parallel/serial layers). Understanding the motivations behind these changes (stability, memory movement, performance) is the core of this lecture.

Source: @kuririrn - Studying to grasp the recent trends in scratch development of LLMs Stanford CS336 (https://www.youtube.com/watch?v=ptFiH_bHnJw)