@shao__meng: Latest CMU Fall Course 11-768: AI Agents — Lecture 12 Notes Released: Reinforcement Learning Systems, Taught by @gneubig. Course: https://cmu-agents.com, Slides: https:/…
Summary
Lecture 12 notes from CMU course 11-768 AI Agents are now public, covering Reinforcement Learning Systems: how to train LLM Agents with RL on multi-GPU clusters, coordination between training and inference engines, KV/Prefix Caching, Weight Sync, common pitfalls in Agentic RL (sequence extension, chat templates, TITO, sync vs async), and system design choices ranging from single programs to cloud-native microservices.
View Cached Full Text
Cached at: 10/03/26, 06:53 AM
CMU’s New Fall Course 11-768: AI Agents
Lecture 12 Slides Now Public: Reinforcement Learning Systems
Lecturer: @gneubig Course: https://cmu-agents.com Slides: https://cmu-agents.com/slides/lecture-12-rl-systems.pdf…
Lecture 12 focuses on reinforcement learning (RL) systems, and lays out clearly what needs to be solved at the systems level to actually run RL training for an LLM Agent on a real multi-GPU cluster.
@gneubig argues that the essence of LLM RL systems is forcibly mashing together two engines whose design philosophies are diametrically opposed — the training engine and the inference engine — and making them cooperate. And the long-horizon, multi-turn nature of Agentic RL amplifies every one of these tensions.
Optimizing Each Side: Inference vs. Training
Four techniques on the inference side:
· KV Caching: trade memory for recompute. The scale is worth remembering: under bf16, Llama-3-8B needs 128KB per token, so a single 32k-token trajectory is roughly 4GB, and 64 concurrent trajectories mean 256GB.
· Prefix Caching (RadixAttention and the like): especially important for Agentic RL, because in GRPO, multiple rollouts within a group share the same task prefix, and a single rollout’s history can be reused across subsequent turns.
· Continuous Batching (Orca’s iteration-level scheduling): schedule at the granularity of a single decode step; as soon as a sequence finishes, it yields the slot to a queued request. Usually paired with chunked prefill or prefill/decode disaggregation.
· Speculative Decoding: a cheap drafter guesses k tokens and the policy verifies them in parallel. An RL-specific pitfall: since the policy changes at every step, a frozen drafter’s acceptance rate decays, so the drafter must be trained online with auxiliary objectives (SFT/distillation).
Two techniques on the training side:
· Activation Checkpointing: store only each layer’s input and recompute layer by layer during backprop. The cost is roughly one extra forward pass (≈+33% compute); selective recomputation (attention only) is far cheaper.
· Prefix Sharing (the training dual of inference-side prefix caching): in GRPO’s long-prompt batches, 80–95% of tokens are repeated prefixes. A tree attention mask computes them only once, and each branch’s gradients accumulate back into the prefix. Snowflake ZoRRo reports up to 6× speedups for actor updates; AReaL-DTA compresses multi-turn τ2-bench rollouts 9.43× with a prefix tree, and per-path training via DFS is up to 8.31× faster than dense.
The “Seam” Between the Two Engines
· Weight Sync: full synchronization of a 1T-parameter model is around 2TB. But roughly 99% of bf16 parameters are unchanged between RL updates, so syncing only the delta (PULSE) slashes this dramatically. Note that after syncing, tensors must be re-sharded according to the inference engine’s parallel layout.
· R3 (Rollout Routing Replay): for MoE models, tiny numerical differences between the training and inference engines can select different experts, producing inconsistent logprobs on the two sides. The fix is to record expert selections during rollout and force-replay them during training, which requires the inference engine to expose routing decisions to the training side.
The Most Common Pitfalls in Agentic RL
Sequence Extension. Compare two harnesses: A appends every turn to the full history; B compresses old interactions into a summary S every two turns. From the training batch’s perspective, A is a single concatenated data point (action tokens carry a loss mask and broadcast advantage Â), while B yields two data points both containing the Task — the repeated task tokens produce no loss but still incur context-compute cost. Conclusion: keep exact prefixes and keep concatenating for as long as feasible (this is also what the Tinker docs recommend).
Chat Template hazards. The default Qwen3/3.5 template strips reasoning content from history, which directly breaks sequence extension. This is why Agent RL frameworks (Tinker, PrimeRL) typically ship with custom templates/renderers.
TITO (Token-In, Token-Out). Trajectory data should be passed between engines as tokens, not strings. Small transformations in string space (e.g., a template adding a whitespace) cause the two sides to tokenize differently; tokenization is a one-to-one mapping, but detokenization is many-to-one, so a string round-trip is lossy.
Sync vs. Async. Sync RL colocates training and inference engines on the same GPUs. Async RL (Pipeline RL, AReaL) partitions GPUs and overlaps inference with weight updates, but requires staleness/off-policy corrections. Since Agentic RL is often rollout-bound (long trajectories), the inference side can be given more GPUs (e.g., 3:1), provided the inference framework supports continuous batching and in-flight weight updates (SGLang already does).
Complex harnesses (sub-agents, context compression, the RAO/RLM direction) further increase systems complexity.
Systems Design Choices (from single-program to cloud-native)
Harness-agnostic design. Agent harnesses vary widely (Codex, Claude Code, OpenHands, Pi, and more). Writing bespoke rollout code for each harness is an anti-pattern. The right approach (as Agent Lightning does): deploy a man-in-the-middle/proxy service that emulates an inference API (compatible with chat-completions, responses, anthropic, and other protocols), so a new harness can be plugged in with zero code changes.
SPMD single-program mode (the AReaL <1.0 approach): one Python program runs on every rank, dist. new_group partitions training and inference groups, and coordination logic is embedded in the program. Directly controllable, but inflexible.
Everything-as-a-Service + single controller: a lightweight orchestrator (a training loop on CPUs) plus independent services for inference/agents/training/weight-sync/environment rewards, communicating over standard protocols (SGLang/vLLM APIs, the Tinker training API, the Open Reward Standard). The payoff is threefold: separation of concerns (researchers can iterate on algorithms quickly without heavy dependencies), modularity (any implementation behind the same protocol can be swapped in, hosted or self-hosted), and elastic scaling across heterogeneous clusters (an 8×H100 + 4×Mi350 mixed deployment, cf. AstraFlow).
A microservices architecture also naturally unlocks three advanced capabilities: parallel multi-policy training (heterogeneous multi-agent rollouts), post-deployment online RL (Cursor’s production instance: Tab and Composer produce a new checkpoint every 5 hours from real user interactions), and multi-tenant multi-LoRA training (8-way concurrent throughput beats 8-way sequential, requiring special kernels like Grouped-GEMM/SGMV; Tinker is a pioneer on this path, with a very clean API — one training client per training run, one sampling client per LoRA checkpoint).
meng shao (@shao__meng): A weekend is a good time to settle in and systematically study the fundamentals
CMU’s new Fall course 11-768: AI Agents
Lecturers: @dan_fried @gneubig Course: https://t.co/9VlmFdAsCd Video:
Similar Articles
@shao__meng: Weekends are perfect for settling down to systematically learn foundational knowledge CMU Fall Latest Course 11-768: AI…
CMU offers a new fall 2026 course on AI Agents taught by Daniel Fried and Graham Neubig, covering topics like tool calling, reinforcement learning training, and human-computer interaction.
@shao__meng: https://x.com/shao__meng/status/2106715778078974378
NYU's Fall 2026 graduate seminar, “Language Models, Reinforcement Learning, Reasoning,” threads frontier research from Transformers to world models across a 13-week course — covering pretraining scaling laws, RLVR/GRPO post-training, alignment and safety, and real-world engineering practices such as Kimi K3. The instructor, Pavel Izmailov, is from Anthropic and previously contributed to OpenAI's o1 and Superalignment teams.
@alacheng: Learning Agents from Scratch — a great free online course. It doesn't just cover LLM API calls; it covers memory systems, multi-agent systems, agentic RL, and more, plus practical code for a small agent. Easy to understand yet in-depth. https://github.com/Haozhe-X…
The open-source bilingual AI Agent textbook "agent_learning" has been released — 23 chapters covering everything from scratch: memory systems, RAG, multi-agent systems, Agentic RL, and production deployment, along with runnable reference Agent code and tests.
@shao__meng: University of Michigan's Latest AI Programming Course "Applied Agentic Software Engineering" Lecture 14 Notes on "Subag…
密歇根大学最新AI编程课程《Applied Agentic Software Engineering》第14讲笔记发布,围绕子智能体(Subagents)展开,讲解为何需要子智能体、其工作机制、新鲜与分叉上下文的核心权衡,以及与团队、交接等概念的区别。
@gneubig: This Fall at CMU we're teaching a new course on AI Agents! The goal is that you learn how to create a scaffold, build e…
Carnegie Mellon University is offering a new fall course on AI Agents, covering scaffolding, evals, and training agentic LLMs using reinforcement learning, balancing theory and practice.