Rethinking the Divergence Regularization in LLM RL
Summary
This paper introduces DRPO, which replaces the hard mask in DPPO with a smooth advantage-weighted quadratic regularizer to improve stability and efficiency in LLM reinforcement learning by providing continuous gradient corrections beyond trust-region boundaries.
View Cached Full Text
Cached at: 06/10/26, 05:44 AM
Paper page - Rethinking the Divergence Regularization in LLM RL
Source: https://huggingface.co/papers/2606.09821
Abstract
DRPO improves LLM reinforcement learning stability by replacing hard masks with smooth regularization that provides continuous gradient corrections beyond trust-region boundaries.
Reinforcement learning(RL) has become a key component of post-traininglarge language models(LLMs). In practice, LLM RL is oftenoff-policybecause of training-inference mismatch and policy staleness, makingtrust-region controlessential for stable optimization. Mainstream methods such asPPOandGRPOapproximate this control with aratio-clippingmechanism, but theimportance ratiocan be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such asDPPOaddresses this mismatch by replacing ratio-based clipping with adivergence-based mask, yielding a trust region defined by the sampled token’s absolute probability shift. However,DPPOstill relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smoothadvantage-weighted quadratic regularizeronpolicy shift. DRPO preserves the same trust-region geometry asDPPOwhile inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training.
View arXiv pageView PDFGitHub324Add to collection
Get this paper in your agent:
hf papers read 2606\.09821
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.09821 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.09821 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.09821 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
Introduces ODRPO, a framework that decomposes discrete rewards into ordinal binary indicators to improve robustness of policy optimization in RLAIF for LLMs, achieving up to 14.8% relative improvement with minimal overhead.
Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning
This paper introduces CPPO, a method that improves reinforcement learning with verifiable rewards for LLMs by using position-weighted thresholds and cumulative prefix budgeting to address limitations of uniform token-level trust regions.
DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off
DiPO introduces a novel reinforcement learning approach for LLMs that uses perplexity-based sample partitioning to disentangle exploration and exploitation subspaces, combined with a bidirectional reward allocation mechanism for more stable policy optimization. The method demonstrates superior performance on mathematical reasoning and function calling tasks.
Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex
This paper introduces Listwise Policy Optimization (LPO), a method for RLVR that explicitly handles target projection via divergence minimization on the response simplex to improve training stability and performance in LLMs.
Self-Distilled Policy Gradient
SDPG (Self-Distilled Policy Gradient) is a new RL training framework for LLMs that combines group-relative verifier advantages with on-policy self-distillation and KL regularization to address sparse rewards and instability in RLVR training. The method uses a shared model as both student and teacher by conditioning on privileged context, showing improved stability and performance over RLVR and self-distillation baselines.