@rasbt: Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable …
Summary
Sebastian Raschka's sixth 'Reasoning from Scratch' video introduces and implements Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO), covering reward design, advantage computation, GRPO loss, and MATH-500 training results.
View Cached Full Text
Cached at: 10/03/26, 03:01 PM
Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO).
00:00 Introduction 01:54 What makes a reasoning model different? 04:25 Reasoning traces and model capability 08:29 Accuracy and format rewards 11:34 Aha moments and DeepSeek-R1 training 14:41 Reasoning effort and answer length 18:38 RLHF and RLVR 23:04 GRPO vs. PPO 26:40 GRPO explained with a cooking analogy 31:43 The KL term and simplified GRPO 35:04 Loading the pretrained model 36:07 Loading the MATH training data 39:26 Sampling model responses 46:30 Computing verifiable rewards 49:55 Computing advantages 51:54 Token and sequence log probabilities 55:29 Implementing sequence log probabilities 57:37 Fixing the inference-mode error 1:02:24 Computing the GRPO loss 1:04:37 Putting the GRPO step together 1:09:19 The GRPO training loop 1:12:57 Training settings, logging, and checkpoints 1:17:24 Running training and inspecting outputs 1:19:28 Loading and evaluating checkpoints 1:22:33 MATH-500 results and training stability 1:24:05 Memory requirements and next steps
Similar Articles
RL Beyond the Verifiable (8 minute read)
An analysis discussing the limitations of reinforcement learning with verifiable rewards (RLVR) in math and coding, and the challenge of extending RL to subjective or unverifiable tasks like planning or scientific discovery. It explores techniques such as RLHF and Constitutional AI as alternatives for alignment.
Video Models Can Reason with Verifiable Rewards
VideoRLVR optimizes video diffusion models for verifiable reasoning tasks using reinforcement learning with rule-based rewards, achieving better performance than supervised methods in constraint-satisfying video generation.
@neural_avb: Lurking the Reasoning Training docs rn. Time to write a verifiers env and Unsloth/TRL that shit! Video soon if it all g…
The user is working on implementing reasoning training with verifiers using Unsloth and TRL, reporting progress on locally generating GRPO-like rollouts with a small SLM and a tiny RM, and promises a video soon.
Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
This paper systematically compares process and outcome reward structures for reinforcement learning with verifiable rewards (RLVR) in small language models for mathematical reasoning. The study finds that process-only supervision significantly improves accuracy and reasoning trace fidelity over outcome-only supervision, and analyzes failure modes.
SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning
Introduces SVR-R1, a multi-turn reinforcement learning framework that uses the model's own verification as a learning signal for multi-modal reasoning, achieving significant accuracy improvements over standard GRPO baselines on vision-language reasoning benchmarks.