Rethinking the Divergence Regularization in LLM RL
Summary
This paper introduces DRPO, which replaces the hard mask in DPPO with a smooth advantage-weighted quadratic regularizer to improve stability and efficiency in LLM reinforcement learning by providing continuous gradient corrections beyond trust-region boundaries.
View Cached Full Text
Cached at: 06/10/26, 05:44 AM
Paper page - Rethinking the Divergence Regularization in LLM RL
Source: https://huggingface.co/papers/2606.09821
Abstract
DRPO improves LLM reinforcement learning stability by replacing hard masks with smooth regularization that provides continuous gradient corrections beyond trust-region boundaries.
Reinforcement learning(RL) has become a key component of post-traininglarge language models(LLMs). In practice, LLM RL is oftenoff-policybecause of training-inference mismatch and policy staleness, makingtrust-region controlessential for stable optimization. Mainstream methods such asPPOandGRPOapproximate this control with aratio-clippingmechanism, but theimportance ratiocan be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such asDPPOaddresses this mismatch by replacing ratio-based clipping with adivergence-based mask, yielding a trust region defined by the sampled token’s absolute probability shift. However,DPPOstill relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smoothadvantage-weighted quadratic regularizeronpolicy shift. DRPO preserves the same trust-region geometry asDPPOwhile inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training.
View arXiv pageView PDFGitHub324Add to collection
Get this paper in your agent:
hf papers read 2606\.09821
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.09821 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.09821 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.09821 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Predictive Divergence Masks for LLM RL
Proposes predictive divergence masks for LLM reinforcement learning that improve upon PPO's direction criterion by predicting whether the next policy gradient step will increase or decrease the divergence used by the trust region, leading to better alignment and improved RL training across model scales.
ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
Introduces ODRPO, a framework that decomposes discrete rewards into ordinal binary indicators to improve robustness of policy optimization in RLAIF for LLMs, achieving up to 14.8% relative improvement with minimal overhead.
Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
The paper introduces ERPO, a method that moves regularization from the action-side to the input-side by controlling query distribution, addressing the stability-exploration dilemma in LLM policy optimization, and showing improvements on mathematical reasoning benchmarks.
Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning
This paper introduces CPPO, a method that improves reinforcement learning with verifiable rewards for LLMs by using position-weighted thresholds and cumulative prefix budgeting to address limitations of uniform token-level trust regions.
DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off
DiPO introduces a novel reinforcement learning approach for LLMs that uses perplexity-based sample partitioning to disentangle exploration and exploitation subspaces, combined with a bidirectional reward allocation mechanism for more stable policy optimization. The method demonstrates superior performance on mathematical reasoning and function calling tasks.