Soft Adaptive Policy Optimization

Papers with Code Trending Papers
reinforcement-learning large-language-models policy-optimization training-stability off-policy-updates sapo

Summary

SAPO introduces a smooth, temperature-controlled gate to adaptively attenuate off-policy updates in reinforcement learning for large language models, enhancing training stability and performance compared to methods with hard clipping.

Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.
Original Article
View Cached Full Text

Cached at: 08/16/26, 03:37 PM

Paper page - Soft Adaptive Policy Optimization

Source: https://huggingface.co/papers/2511.20347 Published on Nov 25, 2025

Abstract

Soft Adaptive Policy Optimization (SAPO) enhances the stability and performance of reinforcement learning in large language models by adaptively attenuating off-policy updates with a smooth, temperature-controlled gate, leading to improved training stability and performance.

Reinforcement learning(RL) plays an increasingly important role in enhancing the reasoning capabilities oflarge language models(LLMs), yet stable and performantpolicy optimizationremains challenging.Token-level importance ratiosoften exhibit high variance-a phenomenon exacerbated inMixture-of-Experts models-leading to unstable updates. Existing group-basedpolicy optimizationmethods, such asGSPOandGRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We proposeSoft Adaptive Policy Optimization(SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuatesoff-policy updateswhile preserving useful learning signals. Compared withGSPOandGRPO, SAPO is bothsequence-coherentandtoken-adaptive. LikeGSPO, SAPO maintainssequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used inGSPO. When a sequence contains a few highly off-policy tokens,GSPOsuppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improvingsample efficiency. Relative toGRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higherPass@1 performanceunder comparable training budgets. Moreover, we employ SAPO to train theQwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.

View arXiv pageView PDFProject pageGitHub15.2kautoAdd to collection

Get this paper in your agent:

hf papers read 2511\.20347

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2511.20347 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2511.20347 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2511.20347 in a Space README.md to link it from this page.

Collections including this paper6

Browse 6 collections that include this paper

Similar Articles

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

arXiv cs.AI

This paper proposes Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method for safe reinforcement learning that incorporates local constraint sensitivity to improve safety recovery and reduce oscillations near safety boundaries, achieving higher constrained returns on navigation and locomotion benchmarks.

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Hugging Face Daily Papers

This paper presents Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for agentic tasks, outperforming GRPO and its variants on coding and reasoning benchmarks. SAO is deployed in the GLM-5.2 model's agentic RL pipeline.