@shao__meng: https://x.com/shao__meng/status/2106715778078974378

X AI KOLs Timeline Events

Summary

NYU's Fall 2026 graduate seminar, “Language Models, Reinforcement Learning, Reasoning,” threads frontier research from Transformers to world models across a 13-week course — covering pretraining scaling laws, RLVR/GRPO post-training, alignment and safety, and real-world engineering practices such as Kimi K3. The instructor, Pavel Izmailov, is from Anthropic and previously contributed to OpenAI's o1 and Superalignment teams.

https://t.co/XfIphOxpcO
Original Article
View Cached Full Text

Cached at: 10/05/26, 05:24 AM

From “Attention Is All You Need” to “World Models”: How This 13-Week NYU Graduate Course Covers the Entire Frontier of the LLM Reasoning Era

NYU’s “Language Models, Reinforcement Learning, Reasoning”

https://izmailovpavel.github.io/ny_fall26_lm_rl_grad.html

New York University’s Fall 2026 graduate seminar (CS-GY 9223 L) meets once a week for two and a half hours, across 13 sessions (September 10 to December 10, with a break for Thanksgiving). It has no textbook in the traditional sense — with the exception of Sutton & Barto’s reinforcement learning text, used to lay the groundwork in Week 2, every assigned reading is an original arXiv paper or technical report. Lecture notes and recordings for every session are made public. Grading centers on the course project; there are no written exams. In the course’s own words, the goal is to “bring students to the frontier of the field.”

This positioning is immediately understandable from the instructor’s background. Pavel Izmailov (@Pavel_Izmailov) is an assistant professor at NYU Tandon and a researcher at Anthropic; previously, at OpenAI, he worked on the o1 reasoning model and the “superalignment” team. The widely covered weak-to-strong generalization paper is one he co-authored — and that paper appears on the Week 11 syllabus. Earlier in his career, he was known in academia for Stochastic Weight Averaging (SWA), mode connectivity, and Bayesian deep learning. In other words, the person who designed this course came from the labs building frontier reasoning models and alignment research, and the syllabus reads like a condensed mirror of a first-line lab’s workflow.

The Course’s Overall Narrative: How a Frontier Model Is Forged

Viewed week by week, the schedule looks like a list of papers; but taken all 13 together, a clear narrative line emerges: it unfolds as “pretraining → architecture and training efficiency → post-training RL → alignment and safety → frontier systems and the future.” At its core, the course answers a single question: how are today’s strongest reasoning models built, and where do they go from here?

Act One: Two Foundations (Weeks 1–2)

The course spends only two weeks laying groundwork — but those two weeks correspond exactly to the two technical pillars of the modern LLM.

Week 1 begins with “Attention Is All You Need,” paired with GPT-2 and GPT-3. These three papers established a paradigm: autoregressive next-token prediction, combined with scale, produces emergent capabilities such as few-shot in-context learning. This is the starting point for everything that follows.

Week 2 covers reinforcement learning fundamentals — MDPs, policy gradients, and PPO. This is the only week in the course devoted to reading a textbook (Sutton & Barto, Chapters 1 and 3–5, supplemented by Lilian Weng’s survey, OpenAI’s Spinning Up, and David Silver’s lectures). The arrangement is deliberate: PPO is the algorithmic backbone of RLHF in Week 8 and RLVR in Week 9. Without a solid foundation here, everything after it becomes nearly impossible.

Act Two: The Science of Pretraining (Weeks 3–7)

This section covers “how to train a model efficiently to enormous scale.”

Week 3 on scaling laws is the “physics” of the pretraining era: Kaplan et al.’s 2020 paper first described power-law relationships among compute, parameter count, and data volume; Chinchilla (2022) corrected them, showing that under a fixed compute budget, models should be smaller and data-hungrier — a change that reshaped industry-wide training recipes. But the Week 3 reading list hints at the real frontier: The Art of Scaling RL Compute for LLMs and Understanding Reasoning from Pretraining to Post-Training are both 2025 work. The former extends scaling laws from pretraining compute to reinforcement learning compute; the latter grapples with a still-contested central question: Is reasoning ability something RL newly teaches, or does it already exist in pretraining, with RL merely eliciting and amplifying it? This is one of the most important open questions of the o1/R1 era. The fact that the course places it in Week 3 rather than at the end signals that the instructor wants students listening through this lens from day one.

Week 4 inserts AlphaZero (and MuZero), a seemingly abrupt placement that is actually a carefully planted foreshadowing: self-play + Monte Carlo tree search + learning a value function from terminal outcomes — this paradigm is the conceptual ancestor of the later approach of “training models to generate long chains of thought with RL.” The industry widely refers to the emergence of o1 and R1 as LLMs’ “AlphaZero moment.” This setup doesn’t pay off until Week 9.

Weeks 5–7 form a three-part series on architecture and efficiency, unified by the theme “the cost structure of frontier scale has changed”:

•Week 5 (attention variants and MoE): Centered on the architecture section of DeepSeek-V3, this week covers multi-head latent attention (MLA, which compresses the KV cache), the evolution of GQA, and DeepSeekMoE’s fine-grained expert design. The core insight: in an era where inference cost is dominated by memory bandwidth and the KV cache, architecture design is no longer purely about accuracy — it’s about “cost per token.”

•Week 6 (positional encodings and long context): How RoPE encodes position, how YaRN achieves context extrapolation, and the uncomfortable reality exposed by the RULER benchmark — a big gap between the context length a model “claims” and the length it can actually use effectively.

•Week 7 (optimizers and training dynamics): The most “engineering frontier” week in the entire course. After Adam dominated deep learning for a decade, the Muon optimizer (a Newton–Schulz iteration that orthogonalizes momentum matrices) emerged from Keller Jordan’s nanoGPT fast-training community, then was shown by Moonshot’s Moonlight paper to scale to frontier compute with roughly half the memory overhead of AdamW. Moonshot’s later Kimi series of models was trained with Muon. Combined with muP (a theoretical framework for transferring hyperparameters across model widths), this week argues that training itself has re-emerged as an active research problem.

Act Three: Post-Training and Reasoning RL (Weeks 8–9) — the heart of the course

Week 8 starts with InstructGPT. This 2022 paper defined the three-stage post-training pipeline still in use today: supervised fine-tuning → reward model training → PPO optimization. Two important simplification paths follow: DPO, which optimizes the policy directly on preference pairs and bypasses the explicit reward model; and Constitutional AI, which replaces human labeling with “constitution”-guided AI feedback (RLAIF). Tülu 3 stitches these into a complete open-source post-training recipe.

Week 9 is the climax of the course: DeepSeek-R1, GRPO, and RLVR (RL with Verifiable Rewards). The logic chain here is elegant, and it neatly pays off Week 4’s foreshadowing:

•R1-Zero demonstrated running RL directly on a base model, skipping supervised fine-tuning, with the model spontaneously exhibiting reflection, self-verification, and the famous “aha moment.” The reward is not a learned reward model but an automatically verifiable signal (is the math answer correct? does the code pass tests?) — as clean as AlphaZero’s terminal win/loss.

•GRPO (from DeepSeekMath; students are required to work through its derivation) removes the value network from PPO, instead sampling a group of responses per prompt and estimating advantages through relative comparison within the group. This simplification is exactly what made large-scale reasoning RL affordable from an engineering standpoint.

•Kimi k1.5 provides a complementary view: multimodal long-chain-of-thought RL.

Act Four: Alignment and Safety (Weeks 10–11) — Safety as an Engineering Problem

These two weeks are not an add-on “ethics class”; they are the inevitable follow-up to Act Three: what happens when you push RL hard on an imperfect signal?

Week 10’s answer is a quantitative version of Goodhart’s Law. Gao, Schulman, and Hilton’s Scaling Laws for Reward Model Overoptimization shows that as a policy’s KL divergence from the reference model grows, proxy reward (the learned reward model’s score) keeps rising, while true human preference rises then falls — the harder you optimize, the better the model looks on paper and the worse it gets in practice. Baker et al.’s 2025 monitoring study of reasoning models then presents real-world cases: cheating behavior in chains of thought, coding agents cutting corners, and the unsettling conclusion that monitoring chains of thought currently works — but that if we directly optimize “making the chain of thought look harmless,” we may destroy its honesty.

Week 11 turns to “scalable oversight”: what do you do when models outgrow their supervisors? Weak-to-strong generalization (a line of work the instructor himself contributed to) asks: can annotations from a weak model elicit capabilities in a strong model that exceed those of the annotator? Language Models Don’t Always Say What They Think provides experimental evidence that chains of thought are often unfaithful — the reasons a model gives may not be the reasons it actually made its decision — which directly undermines the reliability of “reading the chain of thought to supervise the model.” AI Debate proposes having AIs argue with each other to amplify human judgment. These three papers together represent three angles on the problem of “supervising systems smarter than us.”

Act Five: The Frontier and the Future (Weeks 12–13)

Week 12 reads the full Kimi K3 technical report end to end. This serves as a “final comprehensive exercise”: everything taught throughout the semester — architecture (including Kimi Linear’s hybrid linear-attention design), the Muon optimizer, RLVR, alignment — appears in a single report, and students must be able to recognize each concept in a real engineering decision.

Week 13 closes with world models: V-JEPA 2 (LeCun’s predictive world model lineage), DreamerV3 (a landmark of model-based RL), GameNGen (a “neural game engine” that reproduces DOOM with diffusion models), and Genie 3 (DeepMind’s interactive world generation). This is both a bookend — MuZero is essentially a learned world model — and a directional judgment about where the course points: the next frontier above language may be models that learn world dynamics and support planning and embodied intelligence.

What This Syllabus Reveals

1. RL has returned to center stage. Six of the 13 weeks are directly about RL (Weeks 2, 4, 8, 9, 10, 11), and adding Week 3’s RL compute scaling laws brings the total close to half the course. The title lists three terms side by side, but RL is the hub connecting “language models” and “reasoning.” This faithfully reflects the field’s shift since 2025: away from “the pretraining scale game” toward “the post-training reasoning game.”

2. The field’s pace has outstripped the textbook cycle. Apart from one week of textbook reading, everything is original papers, with 2024–2026 work heavily represented (DeepSeek-V3, R1, Muon, Kimi, weak-to-strong generalization, reasoning model monitoring). If you want to “reach the frontier” in 2026, the only path is reading primary literature — and this course models that learning approach itself.

3. Work from Chinese labs has become part of the canon. DeepSeek (V3, R1, GRPO) and Moonshot (Moonlight, Kimi k1.5, Kimi Linear, K3) together appear in the core readings of at least five weeks. This objectively documents how the open-source frontier landscape has actually changed since 2025.

4. Safety and alignment are organized as a complete technical thread. From quantitative laws of reward overoptimization, to empirical cases of reward hacking, to theoretical proposals for scalable oversight, safety is taught as an engineering discipline on par with architecture and optimizers — not as a moral appendix. This is directly connected to the instructor’s experience on OpenAI’s superalignment team and at Anthropic, and it reflects how frontier labs view safety: it is a first-class constraint in system-building.

5. Project-based, no exams — a clear signal about research training. The seminar format (reading papers, discussing) combined with project-based assessment points toward a goal of training people who can conduct research independently, not test-takers who have memorized a fixed body of knowledge.

Who Should Take It, and How to Study

The course lists no prerequisites, but the content implies you need: enough deep learning background to comfortably read the Transformer paper, solid linear algebra and probability, and ideally hands-on experience training a model yourself. No prior RL is needed — Week 2 is designed for exactly that.

A few concrete recommendations for self-learners:

•Be willing to invest real time in Week 2. Sutton & Barto Chapters 3–5 plus the Spinning Up exercises are the admission ticket to everything that follows; the PPO clipped objective is worth deriving by hand, because you’ll encounter it repeatedly in Weeks 8 and 9.

•Work through the GRPO derivation (DeepSeekMath) on paper. It’s a clever simplification of PPO; once you understand the “relative advantage within a group” technique, you’ll see why reasoning RL is workable in practice.

•When reading R1, put it side by side with Week 4’s AlphaZero. The two papers are seven years apart and share the same skeleton: verifiable terminal rewards plus search-style exploration over “reasoning trajectories.”

•As you read through Week 10, watch for a question that runs through the entire course: where do rewards come from, and when can they be trusted? Pretraining cross-entropy, RLHF’s learned rewards, RLVR’s verifiable rewards, and in the weak-to-strong setting, “the supervisor themselves becomes the reward signal” — this is the hidden thread that ties the whole course together.

•Use the final Kimi K3 report as a self-check: if, while reading it, you can ask “why was this done this way, and what was the alternative?” about every technical decision, you’ve achieved the course’s goal.

Similar Articles

@shao__meng: Latest CMU Fall Course 11-768: AI Agents — Lecture 12 Notes Released: Reinforcement Learning Systems, Taught by @gneubig. Course: https://cmu-agents.com, Slides: https:/…

X AI KOLs Timeline

Lecture 12 notes from CMU course 11-768 AI Agents are now public, covering Reinforcement Learning Systems: how to train LLM Agents with RL on multi-GPU clusters, coordination between training and inference engines, KV/Prefix Caching, Weight Sync, common pitfalls in Agentic RL (sequence extension, chat templates, TITO, sync vs async), and system design choices ranging from single programs to cloud-native microservices.

@shao__meng: https://x.com/shao__meng/status/2104096547747291584

X AI KOLs Timeline

Professor Li Hongyi from National Taiwan University has updated the 2026 spring machine learning course, focusing on AI Agents and model self-evolution in the large model era. This marks a complete transition of the course from classical machine learning to cutting-edge large model technologies.

@antiAIvo: https://x.com/antiAIvo/status/2112187842919268835

X AI KOLs Timeline

A metro-style map of large model learning designed for Chinese readers: with Transformer→GPT-2→Pre-training→ChatGPT→LLaMA→MoE→Reasoning Models as the main line, branching into five side lines — reasoning, training, alignment, architecture, and applications — emphasizing that hands-on implementation of GPT-2 serves as the anchor for understanding all subsequent models.

@dongxi_nlp: https://x.com/dongxi_nlp/status/2115389565628305715

X AI KOLs Timeline

A Chinese-language explainer on reinforcement learning terminology for LLM readers, using the "Is OpenAI open?" example to walk through concepts such as policy, state, action, reward, return, value, advantage, and GAE, and explaining how scores are attributed to individual tokens when a judge only rates an entire response. Set against the backdrop of DeepSeek-R1-Zero's RLVR training, this is the fourth installment in the post-training series.