The Hands-on Modern RL tutorial everyone has been waiting for is finally available in English PDF download link: https://github.com/walkinglabs/hands-on-modern-rl/releases/tag/v0.1.5…

walkinglabs/hands-on-modern-rl

Source: https://github.com/walkinglabs/hands-on-modern-rl

Hands-On Modern RL

A hands-on modern reinforcement learning course

A practice-first guide to modern RL, from classic control to LLM post-training, RLVR, and multimodal agents.

English · 中文

WeChat Group (微信)

Course Preview · Overview · News · Contents · Course Outline · Experiment Code · Quick Start · Contributing

Course Preview

A clear learning map From the preface and foundations to frontier topics, the chapter tree and page outline help you navigate quickly.	Line-by-line code focus Key PPO, DPO, and GRPO implementations include code maps that connect formulas to readable code.
Training metric visualization Real curves, metric explanations, and failure signals sit together so you can debug while running experiments.	LLM post-training pipelines RLHF, DPO, GRPO, RLVR, and related topics are tied together through flows, artifacts, and cases.
Agentic RL experiment DeepCoder-style GRPO training curves connect tool-use agents, response length, and reward dynamics in a reproducible lab.	Atari game experiment Atari Pong gameplay screenshots and DQN training notes show how pixel-based agents turn screens into decisions.

We hope this open course gives more learners the courage to climb toward the frontier of intelligence and solve more of the hard problems on the path to AGI.

The course is evolving quickly. We recommend focusing on chapters that are not marked as under construction; chapters still in progress may contain mistakes, and corrections or suggestions are welcome.

Help Wanted

Because compute resources are limited, we are seeking GPU support. If you can help with GPU access, please contact [email protected].

Contents

• Course Preview
• Overview
• Course Outline
• Experiment Code
• Quick Start
• Contributing
• Citation
• License

Overview

Hands-On Modern RL is an open course for learning modern reinforcement learning through practice. Instead of the usual “formula first, black-box API later” route, this course takes a practice-first path: learners begin with runnable code and observable training behavior, then use those concrete traces to understand states, value functions, policy gradients, reward modeling, credit assignment, and the rest of the mathematical structure behind RL.

The course spans classic control and connects directly to current AI frontiers, including large language model (LLM) post-training, preference alignment with DPO and GRPO, reinforcement learning with verifiable rewards (RLVR), multi-turn tool-use agents, Agentic RL, and vision-language model (VLM) reinforcement learning.

The goal is to provide a solid ladder: from solving CartPole for the first time to building modern post-training and agent systems.

Design Principles

The course is organized around these engineering and teaching principles:

1. Practice before formalism. Each major topic starts from experiments, metrics, failure cases, or implementation details, then introduces the mathematical abstraction.
2. Theory explains behavior. MDPs, Bellman equations, policy gradients, GAE, PPO clipping, DPO objectives, and GRPO-style group advantages are introduced as tools for explaining what the code does.
3. Modern RL goes beyond classic RL. The course covers classic control and deep RL, then moves into RLHF, preference optimization, RLVR, VLM reinforcement learning, and multi-turn agent training.
4. Debugging is first-class. Training collapse, reward hacking, KL drift, entropy decay, OOM failures, and evaluation blind spots are treated as core material.
5. Readable systems beat black boxes. Examples favor explicit implementations, inspectable metrics, and clear experiment boundaries so learners can modify and extend them.

Audience

This course is for learners who want to understand reinforcement learning by building and inspecting working systems.

It is especially useful for:

• Machine learning engineers moving from supervised learning into RL.
• Researchers and students preparing to read modern RL and alignment papers.
• LLM practitioners interested in RLHF, DPO, GRPO, RLVR, and post-training systems.
• Builders of tool-use agents, web agents, code agents, and evaluation pipelines.
• Self-learners who prefer code, experiments, and visual intuition before dense derivations.

Recommended background:

• Python programming experience.
• Basic PyTorch familiarity.
• Introductory linear algebra, probability, and calculus for machine learning.
• Ability to read papers and trace open-source training scripts.

The course includes math review appendices, so full mathematical fluency is not required on day one.

Learning Goals

After completing the course, learners should be able to:

• Implement and explain the core RL loop: environment interaction, trajectory collection, reward feedback, policy updates, and evaluation.
• Connect MDPs, value functions, Bellman equations, TD learning, policy gradients, and advantage estimation to concrete training behavior.
• Read and modify implementations of DQN, REINFORCE, Actor-Critic, PPO, DPO, GRPO, and related methods.
• Reason about LLM post-training pipelines, including SFT, reward modeling, PPO-style RLHF, DPO-family methods, and RLVR training.
• Understand multi-turn interaction and credit assignment, and build tool-use, trajectory-synthesis, and Agentic RL systems.
• Extend reinforcement learning ideas to VLMs, embodied intelligence, multi-agent self-play, and other frontier areas.
• Diagnose common RL failure modes and design reasonable algorithms, engineering evaluations, and debugging workflows for new RL problems.

Current Status

This repository is an active courseware project. Content is being expanded chapter by chapter, with emphasis on correctness, runnable examples, and a stable learning path.

• Course site: walkinglabs.github.io/hands-on-modern-rl
• Source content: docs/
• Runnable examples: code/
• Local verification: npm run verify
• License: CC BY-NC-SA 4.0

Issues and pull requests are welcome for typo fixes, conceptual corrections, reproducibility improvements, references, and focused course extensions.

News

Note: This course was created with AI assistance and has not yet been fully reviewed. It may contain factual mistakes or code that does not run as expected. Issues and pull requests are very welcome.

• [2026-05-13] 🚀 Major Upgrade: LLM and Traditional RL Hands-on Labs: Added reproducible training examples for Agentic RL (Deep Research / rLLM) and Traditional RL (Actor-Critic continuous control). Includes complete code and fine-tuning analysis for building an Agentic training system from scratch, along with new VLM RL (GeoQA geometry reasoning) hands-on experiments!
• [2026-05-02] Initial browsable open-source release for testing and feedback.

Roadmap

The course is under active development. Planned milestones:

• 2026-05-02: Initial open-source browsable release for community testing and feedback.
• 2026-05-10: Publish a first stable minor version, fix early typos, and stabilize Part 1 and Part 2 content and code.
• Late May 2026: Improve reproducible LLM RL experiments and add a full RLVR hands-on module with evaluation.
• Early June 2026: Deliver Agentic RL projects step by step, from single-tool use to complex Deep Research trajectory synthesis.
• Late June 2026: Add Unity-based embodied RL environments and trainable project examples.
• July 2026 and later: Expand multimodal frontier content with full VLM RL or Diffusion RL hands-on cases.

Course Outline

The course is divided into four parts plus appendices. The README keeps only the main modules; the online site contains the full chapter tree, diagrams, code references, and detailed navigation.

Preface

Part 1: Foundations by Practice

Part 2: Core Theory and Methods

Part 3: LLM-era RL

Part 4: Frontier and Advanced Systems

Appendices

Experiment Code

The code/ directory contains runnable examples aligned with course chapters. Each chapter’s code is intentionally compact so it can be inspected, run, and modified independently.

See code/README.md for a code index and chapter-specific dependency notes.

Recommended Learning Path

A practical path through the repository:

1. Read the course guide and run the CartPole example.
2. Skim the DPO chapter early, even before finishing all theory, to anchor the motivation for LLM post-training.
3. Study Chapters 03-07 in order; this is the conceptual core.
4. After understanding policy gradients and PPO, return to RLHF, DPO, GRPO, and RLVR.
5. Use the debugging and engineering appendices whenever a training run behaves strangely.
6. Treat frontier chapters as extensions: VLM reinforcement learning, Agentic RL, continuous control, multi-agent systems, and test-time reasoning.

Quick Start

Read Online

Published course site:

https://walkinglabs.github.io/hands-on-modern-rl/

Run the Documentation Site Locally

Requirements:

• Node.js >= 18.0.0
• npm

git clone https://github.com/walkinglabs/hands-on-modern-rl.git
cd hands-on-modern-rl
npm install
npm run dev

Then open the local VitePress URL shown in the terminal, usually:

http://localhost:5173

Verify the Site

Before submitting a pull request that changes documentation structure, theme code, navigation, build scripts, or generated assets, run:

npm run verify

This checks formatting, lints the VitePress theme, builds the site, and verifies expected build artifacts.

Run Course Code

Most code examples use Python and are organized by chapter.

cd code
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For smaller installs, use chapter-specific requirements files:

pip install -r chapter01_cartpole/requirements.txt
python chapter01_cartpole/1-ppo_cartpole.py

Some chapters may require additional system libraries, GPU support, model downloads, or environment-specific setup. Start with Chapter 01 before running examples that involve LLMs, VLMs, or heavy simulators.

Repository Structure

hands-on-modern-rl/
|-- docs/                      # VitePress course content
|   |-- .vitepress/            # Site config, navigation, and theme overrides
|   |-- public/                # Static assets copied into the built site
|   |-- preface/               # Course introduction and history
|   |-- chapter*/              # Main course chapters
|   |-- appendix*/             # Supplementary material and references
|   `-- summaries/             # Part-level review and summary notes
|-- code/                      # Runnable examples aligned with chapters
|-- scripts/                   # Maintenance and verification scripts
|-- package.json               # Site scripts and dependencies
|-- AGENTS.md                  # Repository maintenance guide
`-- README.md                  # Main project overview

Development Commands

npm run dev           # Start the local documentation server
npm run build         # Build the static site
npm run preview       # Preview the built site locally
npm run format        # Format repository files with Prettier
npm run format:check  # Check formatting
npm run lint          # Lint VitePress theme code
npm run verify        # Run format check, lint, build, and artifact verification

Contributing

Contributions should make the course clearer, more accurate, easier to reproduce, or easier to navigate.

Good contributions include:

• Fixing conceptual errors, formulas, diagrams, broken links, or typos.
• Improving explanations without changing the intended learning path.
• Adding small, reproducible experiments that clarify existing chapters.
• Improving scripts, build reliability, navigation, or accessibility.
• Adding high-quality references to papers, official documentation, or widely used open-source implementations.

Please keep pull requests focused. A good PR usually changes one chapter, one experiment, one group of diagrams, or one infrastructure issue at a time.

When adding content:

1. Put course material under docs/.
2. Use kebab-case for new directories and files.
3. Prefer directory-based routes with index.md.
4. Update docs/.vitepress/config.mjs when adding navigable pages.
5. Run npm run verify before requesting review if your change touches config, theme, scripts, or generated site output.
6. Use Conventional Commits, such as docs: clarify ppo clipping or fix: repair chapter link.

For repository-specific maintenance rules, see AGENTS.md.

Other Courses

Our team has also created other courses. Take a look:

WeChat Group (微信)

For suggestions or feedback, scan the QR code to join the WeChat group (微信):

Citation

If you use this course in teaching materials, study notes, or derivative non-commercial educational work, please cite the repository:

@misc{hands_on_modern_rl,
  title        = {Hands-On Modern RL: Practice-first reinforcement learning from CartPole to LLM post-training and agentic systems},
  author       = {WalkingLabs},
  year         = {2026},
  howpublished = {\url{https://github.com/walkinglabs/hands-on-modern-rl}},
  note         = {Open courseware repository}
}

License

This course is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

You may share and adapt the material for non-commercial purposes, provided that you give appropriate credit and distribute derivative works under the same license.

Star History