@tatsu_hashimoto: I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning in…

X AI KOLs Timeline News
deep-learning optimization education transformers scaling-laws hyperparameters

Summary

A Stanford CS312 (Deep Learning Alchemy) instructor posts exam-style prediction questions testing deep learning intuition, asking whether learning-rate or weight-decay rescaling can compensate for an 8× batch size increase in an AdamW-trained Transformer.

I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition. First up: optimization. If we increase an LM's batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓ This quiz is part of our optimizer hyperparameter assignment and unit designed by @wen_kaiyue Setup: We train an 8-layer, ~35M-parameter Transformer on 614.4M tokens using AdamW. At batch size 64, the best reference run has peak LR 0.003, weight decay (WD) 0.2, and final validation loss 2.9178. Now increase the batch size from 64 to 512, which gives 8× fewer optimizer steps. Everything else stays fixed except the changes below. Which recipe gives the lowest final validation loss? A. Keep LR and WD unchanged B. LR ×√8 C. LR ×8 D. WD ×8 E. LR ×√8 and WD ×√8 What would you predict, and why?
Original Article
View Cached Full Text

Cached at: 10/30/26, 09:20 PM

I’m posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition.

First up: optimization. If we increase an LM’s batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓

This quiz is part of our optimizer hyperparameter assignment and unit designed by @wen_kaiyue

Setup: We train an 8-layer, ~35M-parameter Transformer on 614.4M tokens using AdamW. At batch size 64, the best reference run has peak LR 0.003, weight decay (WD) 0.2, and final validation loss 2.9178.

Now increase the batch size from 64 to 512, which gives 8× fewer optimizer steps. Everything else stays fixed except the changes below.

Which recipe gives the lowest final validation loss?

A. Keep LR and WD unchanged

B. LR ×√8

C. LR ×8

D. WD ×8

E. LR ×√8 and WD ×√8

What would you predict, and why?

Similar Articles