@shao__meng: Latest CMU Fall Course 11-768: AI Agents — Lecture 12 Notes Released: Reinforcement Learning Systems, Taught by @gneubig. Course: https://cmu-agents.com, Slides: https:/…

X AI KOLs Timeline Events

Summary

Lecture 12 notes from CMU course 11-768 AI Agents are now public, covering Reinforcement Learning Systems: how to train LLM Agents with RL on multi-GPU clusters, coordination between training and inference engines, KV/Prefix Caching, Weight Sync, common pitfalls in Agentic RL (sequence extension, chat templates, TITO, sync vs async), and system design choices ranging from single programs to cloud-native microservices.

CMU's latest fall course 11-768: AI Agents Lecture 12 notes released: Reinforcement Learning Systems Taught by: @gneubig Course: https://cmu-agents.com Slides: https://cmu-agents.com/slides/lecture-12-rl-systems.pdf… Lecture 12 focuses on Reinforcement Learning (RL) Systems, explaining clearly what the systems layer actually needs to solve to get LLM Agent RL training running on a real multi-GPU cluster. @gneubig's view: the essence of LLM RL systems is forcing two engines with completely opposite architectural orientations — the training engine and the inference engine — to work together in coordination. And the long trajectories and multi-turn characteristics of Agentic RL amplify all of these tensions further. # Optimizations on Each Side: Training and Inference Four techniques on the inference side: · KV Caching: trading memory for recomputation. The scale is worth remembering — under Llama-3-8B bf16, it's 128KB per token; a single 32k-token trajectory is about 4GB, and 64 concurrent rollouts would be 256GB. · Prefix Caching (RadixAttention, etc.): especially important for Agentic RL, because multiple rollouts within a GRPO group share the same task prefix, and the history of a given rollout can be reused in subsequent turns. · Continuous Batching (Orca-style iteration-level scheduling): scheduling at the granularity of a single decode step, where completed sequences immediately yield to queued requests; usually combined with chunked prefill or prefill/decode disaggregation. · Speculative Decoding: a cheap drafter guesses k tokens, and the policy verifies them in parallel in one pass. A pitfall specific to RL: the policy changes at every step, so the acceptance rate of a frozen drafter decays — the drafter must be trained online with auxiliary objectives (SFT/distillation). Two techniques on the training side: · Activation Checkpointing: storing only each layer's input and recomputing layer by layer during the backward pass. The cost is roughly one extra forward pass (≈+33% compute); selective recomputation (attention only) is much cheaper. · Prefix Sharing (the training-side dual of inference-side prefix caching): in GRPO long-prompt batches, 80–95% of tokens are repeated prefixes. Using a tree attention mask, they're computed only once, and gradients from each branch are accumulated back to the prefix. Snowflake's ZoRRo reports up to 6× speedup for actor updates; AReaL-DTA compresses multi-turn τ²-bench rollouts 9.43× using a prefix tree, and DFS path-by-path training achieves up to 8.31× versus dense batching. # The "Seam" Between the Two Engines · Weight Sync: a full sync of a 1T-parameter model is about 2TB. However, roughly 99% of bf16 parameters don't change between RL updates — syncing only the delta (as in PULSE) can dramatically reduce the volume. Note that after syncing, the weights must be re-sharded according to the inference engine's parallel layout. · R3 (Rollout Routing Replay): with MoE models, small numerical differences between the training and inference engines may select different experts, causing logprobs to diverge on the two sides. The solution is to record the expert selection during rollout and force a replay during training — this requires the inference engine to expose its routing decisions to the training side. # The Easiest Pitfalls to Step In in Agentic RL Sequence Extension. Compare two harnesses: A appends each turn to the full history; B compresses old interactions into a summary S every two turns. From the training batch's perspective, A is a single concatenated data point (with loss masks and broadcast advantages  on the action tokens), while B is two data points that both contain Task — the repeated task tokens produce no loss but still incur the context computation cost. The conclusion: keep exact prefixes and continuous concatenation as much as feasible (which is also what the Tinker documentation recommends). The Chat Template trap. Qwen3/3.5's default template strips reasoning content from the history, directly breaking sequence extension. So Agent RL frameworks (Tinker, PrimeRL) typically ship custom templates/renderers. TITO (Token-In, Token-Out). Trajectory data should be passed between engines as tokens rather than strings. Small transformations in string space (e.g., a whitespace added by the template) can lead to inconsistent tokenization on the two sides; tokenization is a one-to-one mapping, while detokenization is many-to-one — a string round-trip is lossy. Sync vs Async. Sync RL colocates the training and inference engines on the same set of GPUs; Async RL (Pipeline RL, AReaL) splits GPUs, overlapping inference with weight updates, but requires staleness / off-policy corrections. Since Agentic RL is often rollout-bound (trajectories are long), more GPUs can be allocated to the inference side (e.g., 3:1), provided the inference framework supports continuous batching and in-flight weight updates (SGLang already does). Complex harnesses (sub-agents, context compression, in the direction of RAO/RLM) further increase systems complexity. # System Design Choices (From Single Program to Cloud-Native) Harness-Agnostic design. Agent harnesses come in many flavors (Codex, Claude Code, OpenHands, Pi, etc.), and writing custom rollout code for each harness is an anti-pattern. The right approach (as in Agent Lightning): deploy a man-in-the-middle / proxy service that mimics an inference API (compatible with protocols such as chat-completions, responses, anthropic, etc.), so a new harness can integrate with zero code changes. SPMD single-program mode (as in AReaL <1.0): one Python program runs on each rank, using dist.new_group to partition training and inference groups, with coordination logic embedded in the program. Directly controllable, but inflexible. Everything-as-a-Service + a single controller: a lightweight orchestrator (training loop on CPU) plus independent services for inference, Agent, training, weight syncing, environment rewards, and so on, communicating over standard protocols (SGLang/vLLM APIs, Tinker training API, Open Reward Standard). The benefits are triple: separation of concerns (researchers can iterate on algorithms quickly without heavy dependencies), modularity (swap implementations as long as they share the same protocol, whether hosted or self-deployed), and elastic scaling on heterogeneous clusters (mixed deployment of 8×H100 + 4×Mi350, see AstraFlow). A microservice architecture also naturally gives rise to three advanced capabilities: multi-policy parallel training (heterogeneous multi-agent rollouts), online RL after deployment (a Cursor production example: Tab and Composer produce new checkpoints every 5 hours from real user interactions), and multi-tenant multi-LoRA training (8-way concurrent throughput outperforms 8 sequential runs, requiring special kernels such as Grouped-GEMM/SGMV; Tinker is a pioneer on this path — its API design is very clean, with one training client per training run and one sampling client per LoRA checkpoint).
Original Article
View Cached Full Text

Cached at: 10/03/26, 06:53 AM

CMU’s New Fall Course 11-768: AI Agents

Lecture 12 Slides Now Public: Reinforcement Learning Systems

Lecturer: @gneubig Course: https://cmu-agents.com Slides: https://cmu-agents.com/slides/lecture-12-rl-systems.pdf…

Lecture 12 focuses on reinforcement learning (RL) systems, and lays out clearly what needs to be solved at the systems level to actually run RL training for an LLM Agent on a real multi-GPU cluster.

@gneubig argues that the essence of LLM RL systems is forcibly mashing together two engines whose design philosophies are diametrically opposed — the training engine and the inference engine — and making them cooperate. And the long-horizon, multi-turn nature of Agentic RL amplifies every one of these tensions.

Optimizing Each Side: Inference vs. Training

Four techniques on the inference side:

· KV Caching: trade memory for recompute. The scale is worth remembering: under bf16, Llama-3-8B needs 128KB per token, so a single 32k-token trajectory is roughly 4GB, and 64 concurrent trajectories mean 256GB.

· Prefix Caching (RadixAttention and the like): especially important for Agentic RL, because in GRPO, multiple rollouts within a group share the same task prefix, and a single rollout’s history can be reused across subsequent turns.

· Continuous Batching (Orca’s iteration-level scheduling): schedule at the granularity of a single decode step; as soon as a sequence finishes, it yields the slot to a queued request. Usually paired with chunked prefill or prefill/decode disaggregation.

· Speculative Decoding: a cheap drafter guesses k tokens and the policy verifies them in parallel. An RL-specific pitfall: since the policy changes at every step, a frozen drafter’s acceptance rate decays, so the drafter must be trained online with auxiliary objectives (SFT/distillation).

Two techniques on the training side:

· Activation Checkpointing: store only each layer’s input and recompute layer by layer during backprop. The cost is roughly one extra forward pass (≈+33% compute); selective recomputation (attention only) is far cheaper.

· Prefix Sharing (the training dual of inference-side prefix caching): in GRPO’s long-prompt batches, 80–95% of tokens are repeated prefixes. A tree attention mask computes them only once, and each branch’s gradients accumulate back into the prefix. Snowflake ZoRRo reports up to 6× speedups for actor updates; AReaL-DTA compresses multi-turn τ2-bench rollouts 9.43× with a prefix tree, and per-path training via DFS is up to 8.31× faster than dense.

The “Seam” Between the Two Engines

· Weight Sync: full synchronization of a 1T-parameter model is around 2TB. But roughly 99% of bf16 parameters are unchanged between RL updates, so syncing only the delta (PULSE) slashes this dramatically. Note that after syncing, tensors must be re-sharded according to the inference engine’s parallel layout.

· R3 (Rollout Routing Replay): for MoE models, tiny numerical differences between the training and inference engines can select different experts, producing inconsistent logprobs on the two sides. The fix is to record expert selections during rollout and force-replay them during training, which requires the inference engine to expose routing decisions to the training side.

The Most Common Pitfalls in Agentic RL

Sequence Extension. Compare two harnesses: A appends every turn to the full history; B compresses old interactions into a summary S every two turns. From the training batch’s perspective, A is a single concatenated data point (action tokens carry a loss mask and broadcast advantage Â), while B yields two data points both containing the Task — the repeated task tokens produce no loss but still incur context-compute cost. Conclusion: keep exact prefixes and keep concatenating for as long as feasible (this is also what the Tinker docs recommend).

Chat Template hazards. The default Qwen3/3.5 template strips reasoning content from history, which directly breaks sequence extension. This is why Agent RL frameworks (Tinker, PrimeRL) typically ship with custom templates/renderers.

TITO (Token-In, Token-Out). Trajectory data should be passed between engines as tokens, not strings. Small transformations in string space (e.g., a template adding a whitespace) cause the two sides to tokenize differently; tokenization is a one-to-one mapping, but detokenization is many-to-one, so a string round-trip is lossy.

Sync vs. Async. Sync RL colocates training and inference engines on the same GPUs. Async RL (Pipeline RL, AReaL) partitions GPUs and overlaps inference with weight updates, but requires staleness/off-policy corrections. Since Agentic RL is often rollout-bound (long trajectories), the inference side can be given more GPUs (e.g., 3:1), provided the inference framework supports continuous batching and in-flight weight updates (SGLang already does).

Complex harnesses (sub-agents, context compression, the RAO/RLM direction) further increase systems complexity.

Systems Design Choices (from single-program to cloud-native)

Harness-agnostic design. Agent harnesses vary widely (Codex, Claude Code, OpenHands, Pi, and more). Writing bespoke rollout code for each harness is an anti-pattern. The right approach (as Agent Lightning does): deploy a man-in-the-middle/proxy service that emulates an inference API (compatible with chat-completions, responses, anthropic, and other protocols), so a new harness can be plugged in with zero code changes.

SPMD single-program mode (the AReaL <1.0 approach): one Python program runs on every rank, dist. new_group partitions training and inference groups, and coordination logic is embedded in the program. Directly controllable, but inflexible.

Everything-as-a-Service + single controller: a lightweight orchestrator (a training loop on CPUs) plus independent services for inference/agents/training/weight-sync/environment rewards, communicating over standard protocols (SGLang/vLLM APIs, the Tinker training API, the Open Reward Standard). The payoff is threefold: separation of concerns (researchers can iterate on algorithms quickly without heavy dependencies), modularity (any implementation behind the same protocol can be swapped in, hosted or self-hosted), and elastic scaling across heterogeneous clusters (an 8×H100 + 4×Mi350 mixed deployment, cf. AstraFlow).

A microservices architecture also naturally unlocks three advanced capabilities: parallel multi-policy training (heterogeneous multi-agent rollouts), post-deployment online RL (Cursor’s production instance: Tab and Composer produce a new checkpoint every 5 hours from real user interactions), and multi-tenant multi-LoRA training (8-way concurrent throughput beats 8-way sequential, requiring special kernels like Grouped-GEMM/SGMV; Tinker is a pioneer on this path, with a very clean API — one training client per training run, one sampling client per LoRA checkpoint).

meng shao (@shao__meng): A weekend is a good time to settle in and systematically study the fundamentals

CMU’s new Fall course 11-768: AI Agents

Lecturers: @dan_fried @gneubig Course: https://t.co/9VlmFdAsCd Video:

Similar Articles

@shao__meng: https://x.com/shao__meng/status/2106715778078974378

X AI KOLs Timeline

NYU's Fall 2026 graduate seminar, “Language Models, Reinforcement Learning, Reasoning,” threads frontier research from Transformers to world models across a 13-week course — covering pretraining scaling laws, RLVR/GRPO post-training, alignment and safety, and real-world engineering practices such as Kimi K3. The instructor, Pavel Izmailov, is from Anthropic and previously contributed to OpenAI's o1 and Superalignment teams.

@alacheng: Learning Agents from Scratch — a great free online course. It doesn't just cover LLM API calls; it covers memory systems, multi-agent systems, agentic RL, and more, plus practical code for a small agent. Easy to understand yet in-depth. https://github.com/Haozhe-X…

X AI KOLs Timeline

The open-source bilingual AI Agent textbook "agent_learning" has been released — 23 chapters covering everything from scratch: memory systems, RAG, multi-agent systems, Agentic RL, and production deployment, along with runnable reference Agent code and tests.