AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

Papers with Code Trending Papers
reinforcement-learning large-language-models asynchronous reasoning system-optimization gpu-utilization open-source

Summary

AReaL is a fully asynchronous reinforcement learning system for LLM reasoning, achieving up to 2.57x training speedup over synchronous systems while maintaining or improving performance. It decouples generation and training to improve GPU utilization and includes optimizations like staleness-enhanced PPO.

Reinforcement learning (RL) has become a trending paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous by alternating generation and training in a batch setting, where the rollouts in each training batch are generated by the same (or latest) model. This stabilizes RL training but suffers from severe system-level inefficiency. Generation must wait until the longest output in the batch is completed before model update, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.57times training speedup compared to the best synchronous systems with the same number of GPUs and matched or even improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.
Original Article
View Cached Full Text

Cached at: 07/02/26, 03:44 PM

Paper page - AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

Source: https://huggingface.co/papers/2505.24298 Published on May 30, 2025

Abstract

AReaL, a fully asynchronous reinforcement learning system, decouples generation and training to achieve higher GPU utilization and up to 2.57x training speedup for large language models on reasoning tasks.

Reinforcement learning(RL) has become a trending paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous by alternating generation and training in a batch setting, where therolloutsin each training batch are generated by the same (or latest) model. This stabilizes RL training but suffers from severe system-level inefficiency. Generation must wait until the longest output in the batch is completed beforemodel update, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higherGPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhancedPPOvariant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.57times training speedup compared to the best synchronous systems with the same number of GPUs and matched or even improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.

View arXiv pageView PDFGitHub5.43kAdd to collection

Get this paper in your agent:

hf papers read 2505\.24298

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper6

#### inclusionAI/AReaL-boba-2-8B Text Generation• UpdatedJun 13, 2025 • 292 • 28 #### inclusionAI/AReaL-boba-2-14B Text Generation• UpdatedJun 10, 2025 • 109 • 22 #### inclusionAI/AReaL-boba-2-8B-Open Text Generation• UpdatedJun 4, 2025 • 93 • 20 #### inclusionAI/AReaL-boba-2-14B-Open Text Generation• UpdatedJun 4, 2025 • 110 • 20 Browse 6 models citing this paper## Datasets citing this paper1

#### inclusionAI/AReaL-tau2-data Preview• UpdatedMar 2 • 474 • 13

Spaces citing this paper1

Collections including this paper5

Browse 5 collections that include this paper

Similar Articles

Z.ai's Stable Asynchronous RL (13 minute read)

TLDR AI

The paper introduces Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for LLM post-training, and demonstrates that SAO consistently outperforms GRPO on agentic coding and reasoning benchmarks.

Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

Hugging Face Daily Papers

This paper systematically investigates failure modes in reinforcement learning for small language models (70-500M parameters) using PPO, identifies silent LoRA freezing, numerical overflow, and catastrophic policy collapse, and proposes a robust system with merge-and-reinitialize adapters, float32 precision, and a safety mechanism. The approach converges stably and outperforms baselines with less data.

Adaptive Latent Agentic Reasoning

arXiv cs.CL

This paper introduces Adaptive Latent Agentic Reasoning (ALAR), a dual-mode framework for LLM agents that uses compact latent reasoning for routine turns and selectively escalates to explicit chain-of-thought for harder decisions, achieving up to 84.6% token reduction while maintaining task accuracy.