Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Hugging Face Daily Papers Papers
reinforcement-learning llm asynchronous-optimization agentic-rl sao token-level-clipping grpo

Summary

This paper presents Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for agentic tasks, outperforming GRPO and its variants on coding and reasoning benchmarks. SAO is deployed in the GLM-5.2 model's agentic RL pipeline.

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).
Original Article
View Cached Full Text

Cached at: 07/09/26, 03:40 PM

Paper page - Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Source: https://huggingface.co/papers/2607.07508

Abstract

Asynchronous reinforcement learning with single-rollout optimization addresses stability issues in LLM training for complex tasks, outperforming existing methods in coding and reasoning benchmarks.

Reinforcement learning(RL) is becoming increasingly important for post-traininglarge language models(LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizonagentic tasks. Recently,asynchronous RLhas emerged as a more efficient alternative by updating the model as rollouts arrive. However, existingasynchronous RLsystems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is thatgroup-wise samplingin the widely-usedGRPOframework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges inasynchronous RL. To reduceoff-policy effectsand improve generalization, we replacegroup-wise samplingwithsingle-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-sidetoken-level clippingstrategy. SAO is able to train stably for one thousand steps and consistently outperformGRPOand its variants on agentic coding and reasoning benchmarks, such asSWE-Bench Verified,BeyondAIME, andIMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.07508

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.07508 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.07508 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.07508 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Z.ai's Stable Asynchronous RL (13 minute read)

TLDR AI

The paper introduces Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for LLM post-training, and demonstrates that SAO consistently outperforms GRPO on agentic coding and reasoning benchmarks.

@VukRosic99: How Is GLM 5.2 Trained? Tsinghua's Async RL Paper Explained The paper from Tsinghua University replaces GRPO's wait-for…

X AI KOLs Timeline

This paper from Tsinghua University introduces Single-rollout Asynchronous Optimization (SAO) for reinforcement learning post-training of LLMs. SAO replaces batch-based GRPO with single-rollout asynchronous training to reduce idle GPU time and improve stability, and it was used to train the GLM-5.2 model (750B-A40B), achieving state-of-the-art results on agentic coding and reasoning benchmarks.