Tsinghua has introduced SAO, an algorithm designed to address the off-policy drift and stability challenges that arise in async RL. As stated in the abstract, SAO has been successfully integrated into the agentic RL training pipeline for training the open GLM-5.2 model (750B-A40B).

Paper Link:

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Source: https://arxiv.org/html/2607.07508 Zhenyu Hou Yujiang Li11footnotemark:1Jie Tang Yuxiao Dong

Tsinghua University

Abstract

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

Figure 1:The performance of SAO on reasoning and coding benchmarks.The four reasoning benchmarks are evaluated in a reasoning-with-Python-tool setting, where the baseline is the Qwen3-30B-A3B SFT model; SWE-Bench Verified evaluates coding with the Qwen3-30B-A3B baseline. SAO outperforms the corresponding baseline and GRPO across all five benchmarks.## 1Introduction

Large Language Model (LLM) development is shifting from supervised pre-training toward post-training reinforcement learning (RL). Recent work in Reinforcement Learning has demonstrated that scaling RL compute together with test-time compute is a highly effective way to improve model intelligence(DeepSeek-AI,2024a; OpenAI,2024; Cobbeet al.,2021; Lightmanet al.,2023). Most LLM RL pipelines remain synchronous and interleaved: the policy generates a batch of rollouts, and optimization starts only after the entire batch is collected(Ouyang and others,2022; Rafailovet al.,2024).

For agentic and coding workloads, rollout lengths are highly variable, so short trajectories finish quickly while long ones become stragglers; as a result, large portions of the GPU cluster idle while waiting for the slowest rollouts(DeepSeek-AI,2024b; Kwonet al.,2023; Yuet al.,2022). Asynchronous RL mitigates thisimbalanced generation overheadby consuming rollouts continuously as they arrive, improving utilization and wall-clock efficiency(Mnihet al.,2016; Lianget al.,2018; Hoffman and others,2020).

However, asynchrony introduces two challenges. First, each trajectory can be generated by multiple versions of the old rollout model, which leads to more unpredictable and severe off-policy, and thus harms the training stability. Previous works(Fuet al.,2025; Noukhovitchet al.,2024)make attempts for asynchronous RL but mainly focus on efficiency optimization rather than effectiveness. Second, group-wise methods such as GRPO(Shao and others,2024; Wang and others,2022)are mismatched to asynchronous training. GRPO samples a group of responses for each prompt and uses the group-level average for advantage estimation. The group-wise sampling induces latency-driven off-policy behavior because the group has to wait for the slower one to finish before fed into training. In addition, group-wise sampling is incompatible with online or complex agentic settings where the environment often provides only a single trajectory feedback per prompt(Sutton and Barto,2018; Schulmanet al.,2017; Yao and others,2022; Nakano and others,2021).

In this paper, we propose Single-rollout Asynchronous Optimization (SAO) for agentic RL. It keeps asynchronous RL training stable and effective under policy lag while preserving the efficiency of asynchrony. Instead of group-wise sampling, such as GRPO, SAO uses single-rollout updates. To make this setting practical, it also introduces effective value-model training strategies. Our contributions are as follows:

• •To stabilize training under varied policy lag, we use token-level importance sampling strategy. It directly uses the log-probabilities from the rollout engine and applies stricter double-sided token-level clipping and masking.
• •To reduce off-policy effects, we use one single rollout sampling for each prompt instead of group-wise sampling previously populated by GRPO. To further make this setting practical in agentic RL, we improve the value model process. Specifically, we update the critic more frequent than the actor and fine-tune the value model with frozen attention.
• •To handle multi-turn agent trajectories with interleaved environment feedback, we derive a skip-observation token-level GAE estimator. It computes advantages across action-to-action boundaries. It also avoids propagating noise through observation tokens that are not generated by the model.

We evaluate SAO on agentic coding and math reasoning benchmarks, including SWE-Bench Verified(Jimenezet al.,2023), AIME2025(Balunovićet al.,2025), BeyondAIME(ByteDanceSeed,2025), HMMT(Balunovićet al.,2025), and IMOAnswerBench(Luonget al.,2025). The results demonstrate that our asynchronous RL design can stably train for around one thousand steps and achieves consistently better performance than improved GRPO. In addition, we show that the single-rollout strategy in SAO is uniquely suited for simulated online learning, where it can adapt to dynamic environmental changes.

2Preliminaries

In reinforcement learning for language models, the model is parameterized byθ\thetaas a stochastic policyπθ​(y|q)\pi_{\theta}(y|q), which generates a response sequencey=[y1,…,y|y|]y=[y_{1},\dots,y_{|y|}]given a queryqqfrom dataset𝒟\mathcal{D}. RL optimizesπθ\pi_{\theta}by maximizing a clipped surrogate objective that encourages stable policy updates. Formally, for a given batch of data, the unified optimization target is defined as:

𝔼​[1|y|​∑t=1|y|min⁡(rt​(θ)​A^t,clip​(rt​(θ),1−ϵ,1+ϵ)​A^t)]\mathbb{E}\left[\frac{1}{|y|}\sum_{t=1}^{|y|}\min\left(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right)\right] wherert​(θ)=πθ​(yt∣q,y<t)πθold​(yt∣q,y<t)r_{t}(\theta)=\frac{\pi_{\theta}(y_{t}\mid q,y_{<t})}{\pi_{\theta_{\text{old}}}(y_{t}\mid q,y_{<t})}is the probability ratio between the current and old policies,ϵ\epsilonis the clipping hyperparameter. The fundamental distinction between PPO(Schulmanet al.,2017)and GRPO(DeepSeek-AI,2024b)lies in whether to estimate the advantage functionA^t\hat{A}_{t}and the necessity of auxiliary value networks.

Proximal Policy Optimization (PPO).Standard PPO typically adopts an Actor-Critic architecture, requiring the training of a separate value function (Critic)VϕV_{\phi}, parameterized byϕ\phi, to estimate the expected return of the current state. This critic is optimized concurrently with the policy to minimize the value errorℒϕVF=𝔼​[(Vϕ​(q,y<t)−R)2]\mathcal{L}_{\phi}^{\text{VF}}=\mathbb{E}[(V_{\phi}(q,y_{<t})-R)^{2}], whereRRdenotes the cumulative reward. To balance bias and variance, PPO employs Generalized Advantage Estimation (GAE). The advantageA^tGAE\hat{A}_{t}^{\text{GAE}}is computed as an exponentially weighted sum of temporal difference errors:

A^tGAE=∑l=0|y|−t−1(γ​λ)l​δt+l\hat{A}_{t}^{\text{GAE}}=\sum_{l=0}^{|y|-t-1}(\gamma\lambda)^{l}\delta_{t+l}whereδt=rt+γ​Vϕ​(st+1)−Vϕ​(st)\delta_{t}=r_{t}+\gamma V_{\phi}(s_{t+1})-V_{\phi}(s_{t}). While effective, this approach necessitates maintaining a copy of the model parameters for the value function, essentially doubling the memory footprint during training and increasing computational overhead.

3Asynchronous Reinforcement Learning with Single Rollout

Figure 2:Overview of SAO with single rollout design. The numbers denote the generation order of trajectories. For SAO, each trajectory becomes available for training immediately upon completion. In contrast, GRPO must wait until all trajectories in a group are generated before training can begin.In this section, we introduce SAO to tackle training instability and off-policy drift in asynchronous RL training. With a simple token-level clipping strategy and single rollout as an alternative to group-wise sampling, we show that asynchronous RL can be stably scaled to thousands of training steps and achieve significant performance improvements. Figure2shows the overall design of SAO.

3.1Stabilizing Asynchronous RL via Direct Double-Sided Importance Sampling (DIS)

A primary challenge in asynchronous RL is the “policy lag” that emerges between rollout models and the training models. In decoupled PPO for LLM, importance sampling is employed to relieve off-policy bias by keeping three distinct models: the current policyπθ\pi_{\theta}, the old policyπθold\pi_{\theta_{\text{old}}}, and the rollout policyπrollout\pi_{\text{rollout}}, whereπθπθold\frac{\pi_{\theta}}{\pi_{\theta_{\text{old}}}}is used for staled off-policy correction andπθoldπrollout\frac{\pi_{\theta_{\text{old}}}}{\pi_{\text{rollout}}}for training-rollout mismatch. However, as rollout engines may undergo multiple updates during a single trajectory generation in asynchronous RL, this renders the tracking of exact behavior probabilitiesπθold\pi_{\theta_{\text{old}}}computationally prohibitive. Otherwise, we have to maintain an extensive history of model checkpoints{πθold(1),…,πθold(N)}\{\pi_{\theta_{\text{old}}^{(1)}},\dots,\pi_{\theta_{\text{old}}^{(N)}}\}, which is infeasible in practical implementation.

To resolve this, we propose a simplified yet aggressive token-level importance sampling to clip off-policy tokens. First, we directly useπrollout\pi_{\text{rollout}}as the behavior proxy andπθ\pi_{\theta}for importance sampling, i.e.,rt​(θ)=πθπrolloutr_{t}(\theta)=\frac{\pi_{\theta}}{\pi_{\text{rollout}}}, while dropping the inaccurateπθold\pi_{\theta_{\text{old}}}. This eliminates the computational overhead of separate old-policy inference by utilizing the log-probabilities generated during the rollout phase.

Second, we employ a double-sided calibration token-level masking strategy. Unlike standard PPO clipping, which clips only selected off-policy tokens with(A>0,rt​(θ)>1+ϵh)(A>0,r_{t}(\theta)>1+\epsilon_{h})or(A<0,rt​(θ)<1−ϵl)(A<0,r_{t}(\theta)<1-\epsilon_{l}), we restrict the trust region to the interval[1−ϵℓ,1+ϵh][1-\epsilon_{\ell},1+\epsilon_{h}], while tokens falling outside this range are masked from gradient computation entirely to prevent instabilities arising from extreme policy divergence. This shares similarities with the IcePop mechanismTeamet al.(2025), yet our strategy is simpler by further removingπθold\pi_{\theta_{\text{old}}}while still achieving stable training.

Formally, the optimization objective with token-level clipping can be written as:

L​(θ)=𝔼^t​[f​(rt​(θ),ϵl,ϵh)​A^t​log⁡πθ​(at|st)]L(\theta)=\hat{\mathbb{E}}_{t}\left[f(r_{t}(\theta),\epsilon_{l},\epsilon_{h})\hat{A}_{t}\log\pi_{\theta}(a_{t}|s_{t})\right](1)In this formulation, the probability ratiort​(θ)r_{t}(\theta)is computed directly from the rollout logs to circumvent the need for historical policy tracking:

rt​(θ)=exp⁡(log⁡πθ​(at|st)−log⁡πrollout​(at|st))r_{t}(\theta)=\exp\left(\log\pi_{\theta}(a_{t}|s_{t})-\log\pi_{\text{rollout}}(a_{t}|s_{t})\right)(2)Stability is further enforced via the calibration functionf​(x;ϵℓ,ϵh)f(x;\epsilon_{\ell},\epsilon_{h}):

f​(x;ϵℓ,ϵh)={x,if​1−ϵℓ<x<1+ϵh0,otherwisef(x;\epsilon_{\ell},\epsilon_{h})=\begin{cases}x,&\text{if }1-\epsilon_{\ell}<x<1+\epsilon_{h}\\ 0,&\text{otherwise}\end{cases}(3) This design circumvents the intensive need to track the historical model ensemble. By utilizing the rollout log-probabilities directly, we accept a controlled degree of off-policy bias in exchange for a substantial reduction in computational complexity and the elimination of errors associated with using a single, potentially stale, “latest” old policy model. Empirical results demonstrate that this simplified mechanism enables more aggressive clipping, which effectively regularizes the update steps and yields superior training stability in asynchronous settings.

3.2Reducing Off-Policy with Single Rollout

In asynchronous RL, an inevitable problem is off-policy. Yet current popular group-wise sampling RL algorithms, e.g., GRPO, could introduce more severe off-policy. Group-wise sampling introduces an “imbalanced generation” bias, and the group data has to wait for the “slowest” sample to finish before being fed into training. One promising solution is to replace group-wise sampling with single-rollout, where a sample is immediately fed into training upon generation.

However, single-rollout optimization inherently suffers from high variance in gradient estimation, similar to REINFORCEZhanget al.(2021). To reduce variance requires a sufficiently good value model. In this part, we focus on simple strategies to optimize value modeling to ultimately boost the policy’s performance.

Faster Value Update than Policy.We identify that the primary source of instability in single-rollout RL is the interdependence between the policy and the value function. If the value modelVϕV_{\phi}is inaccurate, the advantage estimatesA^t\hat{A}_{t}become noisy, leading to destructive policy updates. To mitigate this, we implement a Faster Value Update adapted for LLMs. We decouple the optimization frequencies of the policy and the value model. Specifically, for every single gradient update applied to the policyπθ\pi_{\theta}, we enforceKKupdates to the value networkVϕV_{\phi}(whereK>1K>1). In our experiments, we setK=2K=2. This strategy facilitates the faster adaptation of value estimates to the current policy before they are utilized for advantage computation, thereby reducing the variance.

Stabilizing Value Model Training via Parameter Freezing.In our pilot experiments, we find the instability of value model training, where the gradient norms of the value model are significantly larger than the corresponding policy model. Further decomposition shows that this instability originates primarily from the Full Attention layers, whereas the Mixture-of-Experts (MoE) layers remain relatively stable. Based on this observation, we employ a “Frozen-Attention” training strategy for the value model. During the RL training, we freeze the parameters of the attention modules inVϕV_{\phi}and optimize the MoE projections. We hypothesize that the pre-trained attention weights already possess sufficient semantic capability to attend to relevant tokens. By restricting optimization to the MoE layers, we effectively regularize the value model.

Skip-Observation Token-level GAE for Agentic Tasks.Agentic tasks present a unique challenge for token-level value estimation due to their trajectory structure:T=[a0,o0,a1,o1,…]T=[a_{0},o_{0},a_{1},o_{1},\dots], whereaia_{i}represents model actions andoio_{i}represents environment feedback. Standard Generalized Advantage Estimation (GAE) attempts to calculate the value difference between adjacent tokens. However, the transition from the end of an actionai,enda_{i,\text{end}}to the start of an observationoi,starto_{i,\text{start}}is discontinuous from the model’s perspective, as the model does not generateoio_{i}. Calculating advantage across this boundary introduces noise, as the value modelV​(oi,start)V(o_{i,\text{start}})attempts to predict the value of an external environment state.

To resolve this, we derive a “Skip-Observation” GAE. We explicitly modify the Bellman target to bypass environment feedback tokens, linking the value of the current action directly to the value of the subsequent action. Formally, letai,Na_{i,N}be the last token of actionii, andai+1,0a_{i+1,0}be the first token of the next action. We define the advantage as:

A^​(ai,N)=δ+γ​λ​A^​(ai+1,0)\hat{A}(a_{i,N})=\delta+\gamma\lambda\hat{A}(a_{i+1,0})(4) where the temporal difference residualδ\deltais calculated bridging the observation gap:

δ=rt+γ​V​(ai+1,0)−V​(ai,N)\delta=r_{t}+\gamma V(a_{i+1,0})-V(a_{i,N})(5) This formulation constrains the advantage estimation to rely purely on the model outputs, filtering out the stochasticity of environment feedback. In contrast, some works may consider using a step-level value function and GAE as an alternative to the token-level value; however, we found that a step-level value could lead to suboptimal performance, which will be shown in the experimental part. We also conduct other advantage designs for agentic traces, and the results can be found in the Appendix.

Scaling Value Pretraining.Finally, to support these mechanisms, we find it essential to scale the data used for value model pretraining. Our experiments demonstrate that the “cold start” problem in value estimation is a major bottleneck. By significantly increasing the scale of the value pretraining corpus, we provide a robust initialization point that promotes the effectiveness of our single-rollout and TTUR mechanisms from the early stages of training.

4Experiments

Table 1:Experimental Results on math reasoning benchmarks(Accuracy %).ModelAIME2025BeyondAIMEHMMT Nov 2025IMOAnswerBenchClaude-Sonnet-4.587.062.081.765.8GPT-5 High94.674.089.276.0GLM-4.795.7-93.582.0Qwen3-30B-A3Bw/ python14.610.517.37.8w/o python85.063.076.755.3SFT (w/ python)80.453.375.253.3SFT (w/o python)14.646.817.342.0GRPO (w/ python)84.254.876.055.8Qwen3-30B-A3BSAO (ours)97.374.888.374.0-SAO (w/ DIS only)94.271.586.771.3-GRPO (+ DIS)93.570.884.070.0Table 2:Experimental Results on SWE-Bench Verified (Accuracy %).ModelAccuracy (%)Qwen3-30B-A3B23.0+ GRPO (w/ DIS)27.0+ SAO (ours)29.8

Figure 3:Performance comparison between SAO and GRPO (w/ DIS) during training. It can be observed that SAO almost consistently outperforms the optimized GRPO during the training process on different benchmarks.### 4.1Experimental Setup

Training Details.For math reasoning with Python, we finetune Qwen3-30B-A3B-Thinking-2507(Yanget al.,2025a)for 3 epochs on Tool-Integrated Reasoning (TIR) data produced by GPT-OSS-120B(OpenAI,2025)and use the finetuned model to initialize the policy and value model. TIR requires the model to interleave natural-language math reasoning with Python tool calls.

For RL of agentic reasoning, we employ a batch size of 128, a group size of 1, and a max-length of 128k tokens. The policy is optimized with a learning rate of1×10−61\times 10^{-6}, with a token clipping ofϵlow=0.3\epsilon_{\text{low}}=0.3,ϵhigh=5.0\epsilon_{\text{high}}=5.0. We adopt a length-adaptive GAE(Yueet al.,2025)withλpolicy=1−1α​l\lambda_{\text{policy}}=1-\frac{1}{\alpha l}andα=1.5\alpha=1.5. The value model is trained with a learning rate of5×10−65\times 10^{-6},λcritic=1\lambda_{\text{critic}}=1, and a 10-step warmup period. We set theK=2K=2for faster value update for the value model, performing two value model updates per batch. For GRPO variants, each training batch contains 16 prompts with 8 rollout samples per prompt, yielding the same batch size of 128. For the RL of coding agent, we directly use Qwen3-30B-A3B-Thinking-2507 for training and keep almost all the hyperparameters the same as TIR, except forϵlow=0.8\epsilon_{\text{low}}=0.8andϵhigh=3.0\epsilon_{\text{high}}=3.0. For SWE-Bench Verified, we use OpenHands as the scaffold, with a maximum of 300 interaction turns and a 128k-token context budget.

Evaluation.We evaluate SAO on four math reasoning benchmarks including AIME2025, BeyondAIME(ByteDanceSeed,2025), HMMT Nov 2025(Balunovićet al.,2025)and IMOAnswerBench(Luonget al.,2025), reporting Pass@1 accuracy. All evaluations use top-p=1.0p=1.0, temperature1.01.0, and a maximum generation length of 128k tokens. Math-reasoning evaluations allow up to 50 turns to support extensive reasoning and tool calls, while SWE-Bench Verified evaluations allow up to 300 OpenHands interaction turns. To reduce variance, we report the mean performance across 16 evaluation runs for AIME2025 / HMMT / IMOAnswerBench and 4 runs for BeyondAIME.

4.2Main Results

Tables1and2summarize the performance of baselines and different training strategies. GRPO denotes the standard GRPO withclip-higherimplementationYueet al.(2025), which keeps the latest old policy for importance sampling. GRPO (w/ DIS) denotes using the proposed DIS strategy for GRPO.

As shown in Tables1and2, SAO consistently outperforms all baselines on both agentic reasoning and coding benchmarks. Standard GRPO suffers from a performance collapse at approximately 160 training steps The scores reported for these models represent their final valid performance before collapsing. Figure3illustrates the evaluation performance across training steps of SAO compared to vanilla GRPO and GRPO (w/ DIS). Vanilla GRPO tends to quickly collapse, while GRPO with DIS can achieve stable training, demonstrating the effectiveness of DIS. In addition, SAO and GRPO (w/ DIS) exhibit comparable performance in the initial stage; a distinct performance divergence occurs after approximately 400 training steps, demonstrating the effectiveness and stability of SAO.

4.3Ablation Studies

We conduct extensive ablation studies to evaluate the impact of various training configurations on the performance of SAO. The results are shown in Table4.

• •Effects of faster value update: To ablate the effects of faster value update than the policy model, we conduct experiments where the value model is updated only once per batch (critic-train-epoch=1), as opposed to the two updates per batch employed in SAO.
• •Full vs. frozen-attention value model.: To evaluate the impact of attention-freezing during RL, this variant performs full-parameter updates on the value model.
• •Vanilla VAPO and single-rollout with Running-mean baseline.Standard VAPO(Yueet al.,2025)with length-adaptive GAE and a value-based RL baseline. Besides, a single-rollout baseline that maintains a sliding window of the 8 most recent rewards for each prompt, using their mean as a baseline for advantage estimation to provide a simple alternative to parametric value models.

As shown in Table4, all examined variants exhibit a performance decline relative to the proposed SAO, validating the necessity of each design choice. Table3further summarizes the value-training strategy and critic-update settings behind the main value-model ablations. Regarding update frequency, the results indicate that a single update is insufficient for the critic to accurately track rapid policy shifts, leading to less reliable baseline estimations. The full-parameter value-training variant further suggests that frozen-attention updates help regularize critic optimization in complex reasoning tasks. In addition, the RL with running-mean reward achieves decent performance, but still lags far from our SAO, demonstrating the advantage and necessity of a well-trained value model for RL. As for the vanilla VAPO, the training also quickly collapses during training, similar to the vanilla GRPO.

(a) (b) (c)

Figure 4:Training dynamics of asynchronous single-rollout RL. (a) Explained Variance for SAO and a single-critic-update baseline. (b) Critic gradient norm during value training under full-parameter optimization and frozen-attention optimization used in SAO. (c) Token-level clip ratio during training for SAO with the proposed DIS and the VAPO baseline.

4.4Training Dynamics

We analyze the training dynamics of SAO to understand how it facilitates training stability.

Effects of Faster Value Update.Figure4(a) illustrates Explained Variance comparison between SAO and the single-critic-update baseline during training. Explained Variance assesses the alignment between the predicted valuesV​(s)V(s)and the ground-truth returnsRR, defined asE​V=1−Var​(R−V​(s))Var​(R)EV=1-\frac{\text{Var}(R-V(s))}{\text{Var}(R)}. SAO demonstrates significantly higher explained variance after approximately 400 training steps, indicating faster value convergence and better alignment with policy distribution.

Gradient of Critic Models.We examine the impact of freezing attention parameters in value training. As shown in Figure4(b), full-parameter value training exhibits significantly larger critic gradient norms, suggesting unstable optimization dynamics. In contrast, the frozen-attention strategy maintains lower and smoother gradient norms, implying improved numerical stability.

Clipped Tokens.Figure4(c) monitors the token-level clip ratio of SAO applying our proposed DIS strategy and the standard VAPO baseline without it. While VAPO maintains a near-zero clip ratio, it fails to effectively gate divergent off-policy updates, leading to a rapid training collapse at approximately 90 steps.

Table 3:Ablation results of value model training strategy and critic update frequency. We compare partial parameters, i.e., frozen-attention, with full-parameter value update in RL training, as well as the effectiveness of faster critic updates per policy step. We report Accuracy (%) for all datasetsValue TrainingStrategyCritic UpdateFrequencyAIME2025BeyondAIMESAOFrozen Attention297.374.8Single-step-updateFrozen Attention195.0069.75Full-Parameter Value TrainingFull-Parameter290.6274.50Table 4:Ablation results of value model training strategy and critic update frequency. We compare partial parameters, i.e., frozen-attention, with full-parameter value update in RL training, as well as the effectiveness of faster critic updates per policy step. We report Accuracy (%) for all datasetsAIME2025BeyondAIMESAO97.374.8SAO w/o Faster value95.069.8SAO w/o Frozen attention90.674.5Vanilla VAPO (w/o DIS)91.369.0Running mean baseline79.855.3

4.5Online Learning Simulation

Task Design.In real-world online learning environments, feedback is typically restricted to a single trajectory per prompt. This constraint is inherently incompatible with group-based optimization strategies like GRPO, which depend on relative rewards within a sample group for advantage estimation. In contrast, SAO utilizes a value-based critic to provide advantage estimation, allowing for effective policy updates from individual trajectories.

Therefore, we design a simulated online writing task to assess the adaptability of SAO in non-stationary environments. In this setting, the feedback signal is designed as the language tone of user preference. Reward criteria are sequentially adjusted to favor three distinct stylistic archetypes:cute,chuunibyou, andclassical.

Dynamic Reward Assignment.For reward signal assignment, we employ GLM-4.7(GLMet al.,2025)as an LLM-based judge to evaluate two primary dimensions: response quality and stylistic adherence. The final rewardr∈{0,1}r\in\{0,1\}is computed as:r=rquality×rstyler=r_{\text{quality}}\times r_{\text{style}}whererquality,rstyle∈{0,1}r_{\text{quality}},r_{\text{style}}\in\{0,1\}denote binary rewards.

Throughout training, the system prompt requires the model to select one stylistic archetype from a pool of four candidates: Academic, Cute, Chuunibyou, and Classical. In our actual experiments, the candidate set consists ofAcademic, Cute, Chuunibyouduring the first two phases, andClassical, Cute, Chuunibyouin the final phase.

Results.As illustrated in Figure5a, we evaluate the performance of the three candidate linguistic styles of each phase on a held-out test set . SAO demonstrates rapid policy realignment following each reward preference shift, characterized by transitions between stylistic archetypes to maintain adherence to the evolving environmental feedback.

Comparison against Running-Mean Baseline.To better understand the effectiveness of the value model in the online environment, we also adopt the Running Mean Advantage Estimation approach as the baseline. This method approximates the baselinebbby tracking a sliding window of the 128 most recent rewards, thereby facilitating advantage computation asA^=r−𝔼​[rw​i​n​d​o​w]\hat{A}=r-\mathbb{E}[r_{window}]. By decoupling advantage estimation from intra-prompt sample groups, this setup permits policy optimization in an online, single-rollout context. Figure5bdepicts the evolution of training rewards throughout the online learning process of SAO and the Running Mean baseline, where the speed and magnitude of reward recovery following stylistic shifts serve as key indicators of algorithmic adaptability. The Running Mean baseline exhibits a pronounced adaptation lag due to the inertia of its historical window, which remains temporarily biased by rewards from the preceding distribution. In contrast, SAO’s value-based critic dynamically tracks reward shifts, facilitating rapid recovery and consistently higher convergence levels. This confirms that SAO’s state-dependent baseline provides the precision necessary for effective alignment in non-stationary environments.

(a)We report the accuracy transition of three writing styles—cute, chuunibyou, and classical—on a held-out evaluation set throughout the online training process. Shaded regions indicate phase transitions where the reward preference is switched to favor a different stylistic archetype. SAO rapidly suppresses the previously dominant style and realigns its policy to the new target based on environmental feedback. (b)We compare the evolution of training rewards between SAO and a Running Mean Advantage Estimation baseline under single-rollout online learning. Shaded regions denote stylistic reward shifts. While both methods eventually recover after distribution changes, the Running Mean baseline exhibits a pronounced adaptation lag and lower stable performance.

Figure 5:Online learning simulation under changing writing-style preferences.

5Related Work

5.1Reinforcement Learning for Language Models

The standard RLHF pipeline trains a reward model from preference data and optimizes the policy with PPO(Ouyang and others,2022; Schulmanet al.,2017). To reduce the overhead and instability of value-function learning, critic-free objectives such as Group Relative Policy Optimization (GRPO)(Shao and others,2024; DeepSeek-AI,2024a)and REINFORCE-style baselines (e.g., RLOO)(Ahmadian and others,2024)have become increasingly popular. GRPO forms advantages by normalizing rewards within a prompt-level group, which improves stability in synchronous training but introduces an implicit synchronization barrier: updates must wait until all group members are generated, exacerbating staleness and off-policy drift under asynchrony.

Recent work further refines GRPO/PPO-style objectives to improve stability and variance reduction, including sequence-level importance weighting(Zhenget al.,2025), adaptive clipping strategies(Yanget al.,2025b), and smoother alternatives to hard clipping(Yueet al.,2025). However, these works focus primarily on synchronous RL, where exact importance-sampling ratios are easier to obtain. Importance sampling and clipping strategies for asynchronous RL remain less explored.

5.2Synchronous and Asynchronous RL for LLMs

Most large-scale LLM RL implementations remain synchronous and interleaved: collect a full batch of rollouts with a fixed policy snapshot, then run optimization epochs on that batch(Ouyang and others,2022). With long-tail output lengths in reasoning and tool-use, synchronous barriers cause stragglers and substantial idle time, motivating asynchronous actor–learner designs where rollout generation and learning proceed concurrently(Mnihet al.,2016; Sutton and Barto,2018). However, asynchrony introduces policy lag and off-policy drift, often requiring staleness-aware training or off-policy corrections(Espeholt and others,2018).

Several recent systems target asynchronous RL specifically for LLMs.Noukhovitchet al.(2024)study asynchronous RLHF as online-but-off-policy learning and characterize robustness tradeoffs. On the systems side, AReaL(Fuet al.,2025)fully decouples rollout from training and incorporates staleness-aware PPO-style updates for reasoning tasks. ROLL Flash provides fine-grained parallelism and rollout–train decoupling for RLVR and agentic training(Luet al.,2025). Complementary to asynchronous systems, MobileRL studies online agentic RL for mobile GUI agents and introduces difficulty-adaptive GRPO variants to improve stability and sample efficiency in multi-turn GUI environments(Xuet al.,2025). Our work complements these systems by focusing on the single-rollout setting where group-based baselines (e.g., GRPO) are structurally mismatched, and by stabilizing asynchronous learning algorithm designs.

6Conclusion

In this work, we explore the optimization of asynchronous RL on the training effectiveness and stability We proposed SAO, a single-rollout asynchronous RL strategy that addresses off-policy and instability. SAO stabilizes training with token-level importance sampling and double-sided clipping/masking, and improves generalization by replacing group-wise sampling with single-rollout enabled by stronger value-model training. On agentic reasoning and coding tasks, SAO shows consistent outperformance over GRPO baselines, and adapts effectively in simulated online learning.

References

• A. Ahmadianet al.(2024)Back to basics: revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740.Cited by:§5.1.
• M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev (2025)Matharena: evaluating llms on uncontaminated math competitions.arXiv preprint arXiv:2505.23281.Cited by:§1,§4.1.
• ByteDanceSeed (2025)BeyondAIME: advancing math reasoning evaluation beyond high school olympiads.Hugging Face.Cited by:§1,§4.1.
• K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano,et al.(2021)Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168.Cited by:§1.
• DeepSeek-AI (2024a)DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint.Cited by:§1,§5.1.
• DeepSeek-AI (2024b)DeepSeek-v3 technical report.arXiv preprint.Cited by:§1,§2.
• L. Espeholtet al.(2018)IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures.InInternational Conference on Machine Learning,Cited by:§5.2.
• W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang,et al.(2025)AReaL: a large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298.Cited by:§1,§5.2.
• T. GLM, A. Zeng, X. Lv, Q. Zheng, Z. Hou, B. Chen, C. Xie, C. Wang, D. Yin, H. Zeng, J. Zhang,et al.(2025)Glm-4.5: agentic, reasoning, and coding (arc) foundation models.arXiv preprint arXiv:2508.06471.Cited by:§4.5.
• M. Hoffmanet al.(2020)Acme: a research framework for distributed reinforcement learning.arXiv preprint arXiv:2006.00979.Cited by:§1.
• C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2023)Swe-bench: can language models resolve real-world github issues?.arXiv preprint arXiv:2310.06770.Cited by:§1.
• W. Kwon, Z. Li, S. Zhang, X. Zhuang, Y. Sheng, L. Zheng, C. Fonseca, and I. Stoica (2023)Efficient memory management for large language model serving with pagedattention.InProceedings of the 29th Symposium on Operating Systems Principles,Cited by:§1.
• E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, and I. Stoica (2018)RLlib: abstractions for distributed reinforcement learning.InInternational Conference on Machine Learning,pp. 3053–3062.Cited by:§1.
• H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023)Let’s verify step by step.arXiv preprint arXiv:2305.20050.Cited by:§1.
• H. Lu, Z. Liu, S. Xiong, Y. He, W. Gao, Y. Wu, W. Wang, J. Liu, Y. Li, H. Zhao,et al.(2025)Part ii: roll flash–accelerating rlvr and agentic training with asynchrony.arXiv preprint arXiv:2510.11345.Cited by:§5.2.
• T. Luong, D. Hwang, H. H. Nguyen, G. Ghiasi, Y. Chervonyi, I. Seo, J. Kim, G. Bingham, J. Lee, S. Mishra, A. Zhai, C. H. Hu, H. Michalewski, J. Kim, J. Ahn, J. Bae, X. Song, T. H. Trinh, Q. V. Le, and J. Jung (2025)Towards robust mathematical reasoning.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,External Links:LinkCited by:§1,§4.1.
• V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu (2016)Asynchronous methods for deep reinforcement learning.InInternational conference on machine learning,pp. 1928–1937.Cited by:§1,§5.2.
• R. Nakanoet al.(2021)WebGPT: browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332.Cited by:§1.
• M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville (2024)Asynchronous rlhf: faster and more efficient off-policy rl for language models.arXiv preprint arXiv:2410.18252.Cited by:§1,§5.2.
• OpenAI (2024)Learning to reason with llms.https://openai.com/index/learning-to-reason-with-llms.Cited by:§1.
• OpenAI (2025)Gpt-oss-120b & gpt-oss-20b model card.External Links:2508.10925,LinkCited by:§4.1.
• L. Ouyanget al.(2022)Training language models to follow instructions with human feedback.InAdvances in Neural Information Processing Systems,Vol.35,pp. 27730–27744.Cited by:§1,§5.1,§5.2.
• R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2024)Direct preference optimization: your language model is secretly a reward model.InAdvances in Neural Information Processing Systems,Vol.36.Cited by:§1.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017)Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347.Cited by:§1,§2,§5.1.
• Z. Shaoet al.(2024)DeepSeekMath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.Cited by:§1,§5.1.
• R. S. Sutton and A. G. Barto (2018)Reinforcement learning: an introduction.MIT press.Cited by:§1,§5.2.
• L. Team, A. Shen, B. Li, B. Hu, B. Jing, C. Chen, C. Huang, C. Zhang, C. Yang, C. Lin,et al.(2025)Every step evolves: scaling reinforcement learning for trillion-scale thinking model.arXiv preprint arXiv:2510.18855.Cited by:§3.1.
• X. Wanget al.(2022)Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171.Cited by:§1.
• Y. Xu, X. Liu, X. Liu, J. Fu, H. Zhang, B. Jing, S. Zhang, Y. Wang, W. Zhao, and Y. Dong (2025)MobileRL: online agentic reinforcement learning for mobile gui agents.arXiv preprint arXiv:2509.18119.External Links:LinkCited by:§5.2.
• Z. Xu and Z. Ding (2025)Single-stream policy optimization.arXiv preprint arXiv:2509.13232.Cited by:§A.2.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al.(2025a)Qwen3 technical report.arXiv preprint arXiv:2505.09388.Cited by:§4.1.
• S. Yang, C. Dou, P. Guo, K. Lu, Q. Ju, F. Deng, and R. Xin (2025b)Dcpo: dynamic clipping policy optimization.arXiv preprint arXiv:2509.02333.Cited by:§5.1.
• S. Yaoet al.(2022)React: synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629.Cited by:§1.
• G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun (2022)Orca: a distributed serving system for Transformer-based generative models.In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22),pp. 521–538.Cited by:§1.
• Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du,et al.(2025)Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118.Cited by:3rd item,§4.1,§4.2,§5.1.
• J. Zhang, J. Kim, B. O’Donoghue, and S. Boyd (2021)Sample efficient reinforcement learning with reinforce.InProceedings of the AAAI conference on artificial intelligence,Vol.35,pp. 10887–10895.Cited by:§3.2.
• C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang,et al.(2025)Group sequence policy optimization.arXiv preprint arXiv:2507.18071.Cited by:§5.1.

Appendix AAdditional Experimental Results

A.1RL with Agentic Step as Action

To mitigate the high variance inherent in token-level value predictions, we implement a step-wise GAE calculation. We define a step (denoted asSiS_{i}) as a single conversation turn and assume that all constituent tokens share a uniform learning signal within each step. We define the step-level valueV​(Si)V(S_{i})using two primary aggregation methods based on constituent token values predictions{vi,1,vi,2,…,vi,n}\{v_{i,1},v_{i,2},\dots,v_{i,n}\}:

• •Step Average.The step value is the average of all token value predictions within that step,V​(Si)=1n​∑j=1nvi,jV(S_{i})=\frac{1}{n}\sum_{j=1}^{n}v_{i,j}. In this setting, the value model is trained on all tokens.
• •Last-Token Prediction.We define the step value asV​(Si)=vi,nV(S_{i})=v_{i,n}, using only the final token of each step, assuming that the final token provides the most accurate value prediction of the step, as it encapsulates the most comprehensive information of the entire unit. During value model training, we apply a loss mask to all intermediate tokens, ensuring that only the last token of each step contributes to the optimization.

To provide a more stable learning signal, we implement a step-wise GAE calculation that shifts the advantage estimation from a token-level to a step-level granularity. We derive a single advantageA^i\hat{A}_{i}for each step based on the step-level TD errorδi=Ri+γ​V​(Si+1)−V​(Si)\delta_{i}=R_{i}+\gamma V(S_{i+1})-V(S_{i}). This advantage is then assigned uniformly to all tokens within the corresponding step, effectively smoothing out the local noise prevalent in auto-regressive generation. Furthermore, the length-adaptive GAE mechanism is modified to scale the decay factorλ\lambdabased on the total number of steps rather than the raw token length, where

λpolicy=1−1α∗step number\lambda_{\text{policy}}=1-\frac{1}{\alpha*\text{step number}} However, as illustrated by the training reward in Figure6and the performance results in Table5, both step-wise approaches underperform token-wise value training. We attribute this failure to the fact that token-level training provides a finer-grained supervision signal for both the critic and the policy, which is essential for accurately capturing the logical transitions within complex reasoning trajectories.

Figure 6:Training reward for token-level SAO training and step-level variants, where token-level shows better training rewards.Table 5:The ablation on the action granularity for value and policy model training.Step-leveldenotes that each agentstepis viewed as an action to calculate the value.Token-levelrefers to each token being viewed as an action. We report the results with the same training steps (400 steps).AIME2025BeyondAIMEStep-level (Average)85.860.5Step-level (Last-Token)87.362.8Token-level89.866.8

A.2Comparison to Other Baselines of Single-Rollout Strategies

SPOXu and Ding [2025]or directly using the historical running-mean reward as the baseline for advantage estimation are also feasible ways to achieve RL with a single rollout per prompt. However, SPO and running-mean baselines rely on prior information about training-data difficulty and achieve worse performance than SAO, as shown in the experiment section.

Appendix BLimitations and Broader Impact

Our experiments focus on large-scale agentic reasoning, coding, and simulated online writing tasks with a Qwen3-30B-A3B backbone. The conclusions therefore may not transfer directly to smaller models, non-agentic RLHF settings, or environments with dense rewards and shorter rollouts. In addition, SAO depends on a trained value model and rollout log-probabilities, so deployment requires infrastructure that can reliably preserve token-level behavior probabilities during asynchronous generation. The online learning study uses a controlled simulated preference shift; real user-facing online adaptation would require stronger safeguards, monitoring, and privacy review before deployment.

By improving the stability and efficiency of LLM reinforcement learning, this work can reduce the cost of training capable agentic systems. The same capability could also make it easier to optimize models for harmful objectives if used without appropriate data filtering, access controls, or evaluation, so responsible release and monitoring are important for any deployed system derived from this work.