@shao__meng: 9 episodes, 230 minutes — @jbhuang0604's video series "AI Training and Inference" explains the fundamentals of foundation model training and inference, covering the entire training/inference stack: single-GPU memory → thousand-GPU parallelism → cheap fine-tuning → inference acceleration → small-model distillation → inference RL…

X AI KOLs Timeline News
education training-inference distributed-training fine-tuning reinforcement-learning video-series

Summary

A blogger shares Jia-Bin Huang's 9-episode, 230-minute video series "AI Training and Inference", which systematically covers the training and inference architecture of foundation models: from ZeRO/FSDP memory optimization, five-dimensional parallelism, LoRA variants, speculative decoding, on-policy distillation, GRPO reinforcement learning, to AdamW/Muon optimizers and LLM serving.

9 episodes, 230 minutes — @jbhuang0604's video series "AI Training and Inference" explains the fundamentals of foundation model training and inference, covering the entire training/inference stack: single-GPU memory → thousand-GPU parallelism → cheap fine-tuning → inference acceleration → small-model distillation → inference RL → optimizers (AdamW→Muon) **The through-line of this training/inference series** First solve "the model doesn't fit on a single GPU", then scale up to the ten-thousand-GPU level with five-dimensional parallelism; then the LoRA family brings adaptation costs down; on the inference side, speculative decoding speeds things up without changing the output distribution; capability compression goes through on-policy distillation; reasoning ability follows the RL line from PPO to GRPO; and finally it comes back to optimizer baselines and Muon. The serving side (KV cache, scheduling, etc.) is covered in the last episode, Episode 9. https://youtube.com/playlist?list=PLXrQ2OzD3YKA… **1. ZeRO & FSDP (~23 min)** How floating-point and mixed precision consume memory; data parallelism replicates optimizer states and gradients. ZeRO-1/2 shard the optimizer and gradients, ZeRO-3 shards the weights too; FSDP implements the same sharding schedule module-by-module in PyTorch. **2. Five-Dimensional Parallelism (~40 min)** When pure data parallelism isn't enough, partition along data / tensor / sequence·context / pipeline / expert dimensions. Row- and column-wise splits for tensor parallelism with all-reduce; context parallelism and ring attention; pipelines from GPipe, 1F1B to zero-bubble / DualPipe; all-to-all for MoE; with Llama 3's 16,384-GPU layout as an example, plus a three-step "cheat sheet" for arranging parallelism strategies. **3. LoRA and Variants (~21 min)** Freeze W0, learn a low-rank ΔW = BA; typically initialize A randomly and B with zeros. LoRA+ uses different learning rates for A and B; QLoRA quantizes the backbone with NF4 etc. while keeping the adapter in high precision (the subtitles claim 70B-scale fine-tuning on a single 48GB GPU, as stated by the speaker); VeRA shares frozen random bases and only trains scaling vectors; DoRA decomposes magnitude and direction. **4. Speculative Decoding (~25 min)** A small draft model guesses ahead while the large model verifies in parallel; rejection sampling guarantees the same distribution as the target model. Then expands into draft trees, Medusa, MTP, the EAGLE series, and DFlash / DFlash 2. **5. On-Policy Distillation (~31 min)** Soft labels and "dark knowledge" → sequence-level distillation; exposure bias; on-policy RL rewards are sparse. On-policy distillation: the student generates its own trajectories while still receiving dense guidance from the teacher; also emphasizes KL direction; applications mentioned include reasoning, personalization, data reuse, and self-distillation. **6. GRPO (~23 min)** From REINFORCE, baselines/values, GAE, Actor-Critic, TRPO/PPO (especially clipping) to GRPO: sample a group of answers for the same question, normalize rewards within the group as advantages, share the advantage across the whole response, and skip training a separate value model; add KL relative to a reference policy. Dr. GRPO removes length normalization and standard-deviation division, mitigating biases like "correct answers get shorter" and "advantages on hard problems get amplified". The video connects it to DeepSeek-R1-style reasoning training. **7. AdamW (~7 min)** SGD → momentum → RMSProp → Adam; Adam entangling weight decay with adaptive step sizes hurts generalization; AdamW decouples weight decay, becoming the long-term default. **8. Muon / MuonClip (~18 min)** Momentum matrices of linear layers are near low-rank, so rarely-visited directions get weak updates; Muon orthogonalizes the momentum matrix via Newton–Schulz etc. (the speaker claims roughly 2x compute efficiency relative to AdamW, as stated in the video). At scale it pairs weight decay with learning rates scaled by matrix size; QK-clip addresses exploding attention logits, extended to MuonClip in the MLA setting. **9. LLM Serving (~42 min)** Members-only content — if you've watched the first 8 episodes and feel you need it, feel free to study it on your own.
Original Article
View Cached Full Text

Cached at: 10/31/26, 05:24 AM

9 episodes, 230 minutes — @jbhuang0604’s video series “AI Training and Inference” lays out the fundamentals of foundation model training and inference, covering the entire training-and-inference stack: single-GPU memory → thousand-GPU parallelism → cheap fine-tuning → inference acceleration → small-model distillation → inference RL → optimizers (AdamW→Muon).

The through-line of this training & inference series

First solve “the model doesn’t fit on a single GPU,” then scale out to ten-thousand-GPU level with five-dimensional parallelism; the LoRA family then brings adaptation costs down; on the inference side, speculative decoding speeds things up without changing the output distribution; capability compression goes through on-policy distillation; inference capability follows the RL line from PPO to GRPO; and finally we return to the optimizer baseline and Muon. Serving-side topics (KV cache, scheduling, etc.) come last in Episode 9.

https://youtube.com/playlist?list=PLXrQ2OzD3YKA…

  1. ZeRO & FSDP (~23 min) How floating-point and mixed precision consume memory; data parallelism replicates optimizer states and gradients. ZeRO-1/2 shard the optimizer and gradients; ZeRO-3 additionally shards the weights. FSDP is the same sharding schedule implemented module-by-module in PyTorch.

  2. Five-dimensional parallelism (~40 min) When pure data parallelism isn’t enough, split along data / tensor / sequence-context / pipeline / expert. Tensor parallelism’s row-and-column splits and all-reduce; context parallelism and ring attention; pipelines from GPipe and 1F1B to zero-bubble / DualPipe; MoE’s all-to-all — using Llama 3’s 16,384-GPU layout as an example, plus a three-step “cheat sheet” for arranging parallelism.

  3. LoRA and variants (~21 min) Freeze W₀, learn a low-rank ΔW = BA; typically initialize A randomly and B at zero. LoRA+ gives A and B different learning rates; QLoRA quantizes the backbone with NF4 etc. while keeping adapters at full precision (the subtitles claim 70B-scale fine-tuning on a single 48 GB GPU — per the speaker’s statement); VeRA shares frozen random bases and only trains scaling factors; DoRA decomposes magnitude and direction.

  4. Speculative decoding (~25 min) A small draft model guesses ahead; the large model verifies in parallel; rejection sampling guarantees the same distribution as the target model. Then goes deeper into draft trees, Medusa, MTP, the EAGLE series, and DFlash / DFlash 2.

  5. On-policy distillation (~31 min) Soft labels and “dark knowledge” → sequence-level distillation; exposure bias; on-policy RL’s sparse rewards. On-policy distillation: the student uses its own trajectories yet still receives the teacher’s dense guidance — with emphasis on KL direction. Applications mentioned include reasoning, personalization, data reuse, and self-distillation.

  6. GRPO (~23 min) From REINFORCE, baselines/value functions, GAE, Actor-Critic, TRPO/PPO (especially clipping) to GRPO: sample a group of answers for the same question, normalize rewards within the group as advantages, share a single advantage across the whole sequence, and skip training a separate value model; add KL relative to a reference policy. Dr. GRPO drops length normalization and the standard-deviation division, reducing biases like “correct answers get shorter” and “harder questions get inflated advantages.” The video ties this to DeepSeek-R1-style reasoning training.

  7. AdamW (~7 min) SGD → momentum → RMSProp → Adam; Adam’s entanglement of weight decay with adaptive step sizes hurts generalization; AdamW decouples weight decay and became the long-standing default.

  8. Muon / MuonClip (~18 min) Momentum matrices of linear layers are approximately low-rank, so rare directions receive weak updates; Muon orthogonalizes the momentum matrix using Newton–Schulz etc. (the speaker claims roughly 2× the compute efficiency relative to AdamW — per the video’s statement). At scale, it’s paired with decay and learning rates scaled by matrix size; exploding attention logits are handled with QK-clip, extended to MuonClip in the MLA setting.

  9. LLM Serving (~42 min) Members-only content — if, after watching the first 8 episodes, you feel you need it, go study it yourself.

@shao__meng: 9 episodes, 230 minutes — @jbhuang0604’s video series “AI Training and Inference” explains the fundamentals of foundation model training and inference, covering the entire train-and-infer stack: single-GPU memory → thousand-GPU parallelism → cheap fine-tuning → inference acceleration → small-model distillation → inference R…

Channel: @shao__meng Source: https://www.youtube.com/playlist?list=PLXrQ2OzD3YKA

Jia-Bin Huang (@jbhuang0604): I’m building a playlist/course for LLM training and inference (WIP). https://t.co/0JdUqjiiQm

So far, we have common topics like distributed training, fine-tuning, RL, OPD, serving, spec. decoding, and basic optimizers.

What else should I include to make this more complete? 🤔

Similar Articles

@snowboat84: https://x.com/snowboat84/status/2065215177029787705

X AI KOLs Timeline

This article is the middle part of the AI Engineering Landscape series, detailing core techniques such as inference optimization, model slimming (quantization, distillation, pruning, MoE), and speculative decoding, while reviewing the latest advances from hardware to the engineering stack.

@tanzhengmc97: https://x.com/tanzhengmc97/status/2066531753762656730

X AI KOLs Timeline

Explained the operating principles of large models in easy-to-understand language, including word vectors, Transformer attention mechanism, next-word prediction training, and emergent abilities, suitable for beginners to understand basic AI concepts.

@antiAIvo: https://x.com/antiAIvo/status/2112187842919268835

X AI KOLs Timeline

A metro-style map of large model learning designed for Chinese readers: with Transformer→GPT-2→Pre-training→ChatGPT→LLaMA→MoE→Reasoning Models as the main line, branching into five side lines — reasoning, training, alignment, architecture, and applications — emphasizing that hands-on implementation of GPT-2 serves as the anchor for understanding all subsequent models.

@Etudecn: Martin, a technical expert and presenter from IBM Technology channel, thoroughly explains the seven most brain-burning terms in the AI world through this video. Agents, large reasoning models, vector databases. How RAG makes models more accurate. Why the MCP protocol can unify external tools. And why MoE architecture is so...

X AI KOLs Timeline

Martin, an expert from IBM Technology channel, explains in a simple and profound way seven key concepts in the AI world through the video: Agents, large reasoning models, vector databases, RAG, MCP protocol, MoE architecture, and artificial superintelligence. It is considered the most information-dense AI concept explainer of the year.