A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

Hugging Face Daily Papers Papers
reinforcement-learning policy-optimization gradient-dynamics training-stability language-model reasoning

Summary

This paper analyzes token-level gradient dynamics in RLVR training, revealing how advantage sign and token probability jointly affect update stability, and introduces Winner Advantage Policy Optimization (WAPO) which performs clipped updates only on positive-advantage completions to improve stability.

Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse. We analyse this instability through token-level gradient dynamics, deriving a taxonomy that predicts how updates affect next-token probabilities and entropy. The taxonomy shows that stability depends jointly on the advantage sign and token distribution under the current policy. Motivated by this finding, we propose Winner Advantage Policy Optimization (WAPO), a simple online clipped policy-gradient objective that updates only on positive-advantage completions. Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves training stability and matches or outperforms baselines across multiple model families. Full code can be found at https://github.com/layer6ai-labs/wapo.
Original Article
View Cached Full Text

Cached at: 06/17/26, 03:35 AM

Paper page - A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

Source: https://huggingface.co/papers/2606.16154 Rather than treating training collapse as a black box, the paper analyzestoken-level gradient dynamicsand derives a simple taxonomy showing that the effect of an update depends jointly on theadvantage signand the token’s probability under the current policy. This provides an intuitive explanation for why entropy sometimes collapses so abruptly.

The resulting algorithm,Winner Advantage Policy Optimization (WAPO), is almost surprisingly simple: perform clipped policy updates only on positive-advantage completions. Despite this small change, it consistently improves training stability while matching or exceeding GRPO-style baselines across math reasoning and multi-hop QA.

Similar Articles

Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL

arXiv cs.LG

This paper introduces FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage function that dynamically schedules gradient weights during RL post-training of LLMs, achieving faster convergence and better accuracy-diversity trade-offs compared to static baselines.

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

arXiv cs.AI

The paper proposes OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization by masking gradients on answer spans, and introduces Contrast-Augmented Reward to refine reward estimation without extra rollouts. It consistently outperforms existing label-free methods and matches supervised ground-truth reward training across reasoning benchmarks.

Value-Gradient Hypothesis of RL for LLMs

arXiv cs.LG

This paper introduces the value-gradient hypothesis to explain why critic-free RL methods like PPO and GRPO work well for LLMs, showing that the actor backward pass carries a value-gradient-like signal. It derives a predictive criterion for when RL is most effective along the pretraining trajectory.