Stanford University CS 312 “Deep Learning Alchemy” Fall New Course

Professors @tatsu_hashimoto and @kothasuhas lead the course, primarily addressing a common issue: most core knowledge in deep learning is empirical, theoretical explanations fall short, and reading papers can’t replace hands-on practice.

The course uses two creeds as its fulcrum: alchemy must be learned by hand; prediction is understanding—to accurately predict results before running experiments is true mastery.

Course website:
https://deep-learning-alchemy.github.io

Main course schedule
Seven units: hyperparameter tuning and scaling laws → optimizers and hyperparameter invariants (NQM, μP, Muon, weight decay) → loss landscapes → training stability → architecture capacity → data-efficient generalization (switching to DNA modality) → undisclosed unit. The reading list serves as a map of empirical deep learning, with multiple authors (μP, Muon, weight decay series) right in the team.

Homework code, quizzes, and lecture notes are all publicly available, course videos released by @StanfordOnline, and self-learners can use Modal to rent GPUs for full reproduction. It can be seen as a sequel to CS 336 (building LMs from scratch): that course teaches how to build models, this one teaches how to understand model behavior.

Stanford CS 312 | Deep Learning Alchemy

Source: https://deep-learning-alchemy.github.io/ Example Assignment Problem: Start from the baseline depth 8 transformer that we pre-train on 614M tokens. This transformer (implementation provided below) features multiple components such as prenorm, attention, MLP, residuals, and positional embeddings. In this problem, we want to understand how much each component improves loss. Explore the following directions.1. Perform leave-one-out ablations on the role of each component in our current model. 2. ... 3. ... 4. Understand the role of the residual connection. Can you come up with different schemes that improve the loss benefit of the residual connection?

Relevant code snippet``` class Block(nn.Module): def init(self, width, …): super().init() self.attn_norm = RMSNorm(width) self.attn = Attention(width, …) self.mlp_norm = RMSNorm(width) self.mlp = MLP(width, …)

def update(self, x):
    attn = self.attn(self.attn_norm(x))
    mlp = self.mlp(self.mlp_norm(x + attn))
    return attn + mlp

def forward(self, x):
    return x + self.update(x)

class Transformer(nn.Module): def init(self, vocab_size, depth=8, …): super().init() self.token_embed = nn.Embedding(vocab_size, …) self.position_embed = nn.Embedding(…) self.blocks = nn.ModuleList( [Block(…) for _ in range(depth)] ) self.final_norm = RMSNorm(…) self.lm_head = nn.Linear(…, vocab_size)

def forward(self, tokens):
    positions = torch.arange(
        tokens.size(1), device=tokens.device
    )
    x = self.token_embed(tokens)
    x = x + self.position_embed(positions)
    for block in self.blocks:
        x = block(x)
    return self.lm_head(self.final_norm(x))

**Example Quiz Question \(Difficulty Easy\)**: We train each of the following depth\-8 transformer variants using the same training recipe and hyperparameters\. Rank the variants from lowest to highest final validation loss\. Use <, \>, or = between the letters; use = when the losses are within 0\.01\.1. Baseline
2. No prenorm
3. No residual
4. No MLP
5. No attention
6. No positional embeddings

No prenorm diff```
@@ -10,4 +10,4 @@ class Block.update
 def update(self, x):
-    attn = self.attn(self.attn_norm(x))
+    attn = self.attn(x)
-    mlp = self.mlp(self.mlp_norm(x + attn))
+    mlp = self.mlp(x + attn)
     return attn + mlp

No residual diff``` @@ -16,2 +16,2 @@ class Block.forward def forward(self, x):

• return x + self.update(x)

• return self.update(x)

No MLP diff```
@@ -10,4 +10,3 @@ class Block.update
 def update(self, x):
     attn = self.attn(self.attn_norm(x))
-    mlp = self.mlp(self.mlp_norm(x + attn))
-    return attn + mlp
+    return attn

No attention diff``` @@ -10,4 +10,3 @@ class Block.update def update(self, x):

• attn = self.attn(self.attn_norm(x))
• mlp = self.mlp(self.mlp_norm(x + attn))
• return attn + mlp

• mlp = self.mlp(self.mlp_norm(x))
• return mlp

No positional embeddings diff```
@@ -22,2 +22,1 @@ Transformer.__init__
         self.token_embed = nn.Embedding(vocab_size, ...)
-        self.position_embed = nn.Embedding(...)
@@ -30,8 +29,4 @@ Transformer.forward
     def forward(self, tokens):
-        positions = torch.arange(
-            tokens.size(1), device=tokens.device
-        )
         x = self.token_embed(tokens)
-        x = x + self.position_embed(positions)
         for block in self.blocks:
             x = block(x)

Example Quiz Question (Difficulty Hard): We train the default 8 layer transformer as well as a variant where instead of having the residual connection come from the previous layer, we take 0.5 contribution from the previous layer and 0.5 contribution from two layers before (except for the first layer). For each method, we take the best loss over six learning rates {10−4, 3 × 10−4, 10−3, 3 × 10−3, 10−2, 3 × 10−2}. What is your prediction of the loss difference between each method (Lmixed residual−Ldefault)?Core diff``` @@ -34,3 +34,9 @@ Transformer.forward x_two_back = None for i, block in enumerate(self.blocks):

• x = block(x)

• update = block.update(x)
• residual = (

•    x if i == 0
  

•    else 0.5 * x + 0.5 * x_two_back
  

• )
• x_two_back, x = x, residual + update

*Note: both of these questions are hard, do not get discouraged if you do not have intuition for them without running experiments from the assignment\!*

> **Tatsunori Hashimoto (@tatsu_hashimoto):**
> Deep learning is often taught as a sanitized collection of facts and math. But work at the frontier of AI is messy and empirical.
> 
> Our new class, Deep Learning Alchemy, aims to teach the art and science of deep learning in the hands-on spirit of Language Modeling From Scratch.