Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

Hugging Face Daily Papers Papers
reinforcement-learning asynchronous trust-region staleness clipping optimization machine-learning

Summary

Introduces Staleness-Adaptive Trust Regions (SAT) to stabilize asynchronous reinforcement learning by adaptively controlling update intervals based on staleness. Evaluated on a decoupled asynchronous RL setup using Qwen3-30B-A3B-Base, achieving improved results.

Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch is critical: training-inference divergence governs approximation error in finite-horizon bounds, whereas PPO clipping only gates sampled outward updates, acting as a sampled surrogate rather than a full-policy constraint. As a result, high-staleness updates remain weakly controlled in the asynchronous regime where stale rollouts matter most. We introduce the Staleness-Adaptive Trust Region (SAT), which uses the detached sampled log-ratio as a practical staleness proxy, identifies high-mismatch tails within each batch via staleness-based kernel scaling, and contracts only the sign-selected endpoint of the nominal PPO interval. This preserves baseline behavior on ordinary tokens while enforcing more conservative updates on newly intercepted outward bands. We prove local interval containment and pointwise pessimism relative to PPO, showing how the adaptive rule reshapes update geometry under heterogeneous staleness. We evaluate SAT in a decoupled asynchronous RL setup built on Qwen3-30B-A3B-Base, using SGLang as the inference engine and Megatron for training. In this setting, SAT-GSPO w/ R3 achieves the best observed AIME24 avg@8, reaching 35.83 at lag 1 and 34.79 at lag 8, while SAT-GSPO reaches 34.17 at lag 1. Adaptive clipping and routing replay act as complementary stabilizers targeting mismatch tails and routing inconsistency, respectively. Overall, aligning clip intervals with staleness heterogeneity effectively stabilizes asynchronous RL.
Original Article
View Cached Full Text

Cached at: 07/22/26, 06:41 AM

Paper page - Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

Source: https://huggingface.co/papers/2607.18722 Published on Jul 21

·

Submitted byhttps://huggingface.co/zli12321

LZXon Jul 22

Abstract

Asynchronousreinforcementlearningimprovesthroughputbydecouplingrolloutgenerationfromoptimization,butstalenessisaninevitablebyproductcompoundedbypolicylag,enginedelays,andmixture-of-expertsrouting.Fromatrust-regionperspective,thismismatchiscritical:training-inferencedivergencegovernsapproximationerrorinfinite-horizonbounds,whereasPPOclippingonlygatessampledoutwardupdates,actingasasampledsurrogateratherthanafull-policyconstraint.Asaresult,high-stalenessupdatesremainweaklycontrolledintheasynchronousregimewherestalerolloutsmattermost.WeintroducetheStaleness-AdaptiveTrustRegion(SAT),whichusesthedetachedsampledlog-ratioasapracticalstalenessproxy,identifieshigh-mismatchtailswithineachbatchviastaleness-basedkernelscaling,andcontractsonlythesign-selectedendpointofthenominalPPOinterval.Thispreservesbaselinebehavioronordinarytokenswhileenforcingmoreconservativeupdatesonnewlyinterceptedoutwardbands.WeprovelocalintervalcontainmentandpointwisepessimismrelativetoPPO,showinghowtheadaptiverulereshapesupdategeometryunderheterogeneousstaleness.WeevaluateSATinadecoupledasynchronousRLsetupbuiltonQwen3-30B-A3B-Base,usingSGLangastheinferenceengineandMegatronfortraining.Inthissetting,SAT-GSPOw/R3achievesthebestobservedAIME24avg@8,reaching35.83atlag1and34.79atlag8,whileSAT-GSPOreaches34.17atlag1.Adaptiveclippingandroutingreplayactascomplementarystabilizerstargetingmismatchtailsandroutinginconsistency,respectively.Overall,aligningclipintervalswithstalenessheterogeneityeffectivelystabilizesasynchronousRL.

View arXiv pageView PDFGitHub9Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.18722 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.18722 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.18722 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Z.ai's Stable Asynchronous RL (13 minute read)

TLDR AI

The paper introduces Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for LLM post-training, and demonstrates that SAO consistently outperforms GRPO on agentic coding and reasoning benchmarks.

Trust Region Q Adjoint Matching

Hugging Face Daily Papers

Trust Region Q-Adjoint Matching (TRQAM) addresses instability in off-policy reinforcement learning by adaptively controlling path-space KL divergence through projected dual descent, enabling stable fine-tuning of pretrained flow policies. The method consistently outperforms prior arts on 50 OGBench tasks, achieving a 68% success rate in offline RL compared to the strongest baseline's 46%.

Safe Continual Reinforcement Learning under Nonstationarity via Adaptive Safety Constraints

arXiv cs.LG

Proposes LILAC+, a framework for safe continual reinforcement learning under nonstationarity that uses three adaptive safety mechanisms: context-based safety constraints, adaptation-speed constraints, and budget-to-state safety enforcement. Evaluations in simulated driving environments show reduced safety violations under distribution shift while maintaining competitive performance.

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

arXiv cs.LG

This paper identifies a structural failure mode in sequential fine-tuning of shared-context multi-agent LLM teams, formalized as compounding occupancy shift, and proposes TeamTR, a trust-region framework that resamples trajectories and enforces per-agent divergence control, achieving 7.1% average improvement over baselines.