“FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic LMs”

This paper argues that scalable agent RL doesn’t need multiple rollouts of the same prompt, but can learn from each trajectory as soon as it finishes.

They combine reward centering across independent prompts, sequence-level trust regions for stale trajectories, and sample-mean optimization so long failures don’t dominate training, creating stable critic-free learning from just one rollout per prompt.

This moves from group-based RL that spends compute repeatedly sampling the same prompts to asynchronous RL where every rollout covers a new prompt, immediately becomes training data, and lets agents learn from more diverse experiences with less rollout compute.

https://alphaxiv.org/abs/2609.flashreinforce-asynchronous-rl-agentic-models…

FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models

Source: https://www.alphaxiv.org/abs/2609.flashreinforce-asynchronous-rl-agentic-models

Abstract

Asynchronous reinforcement learning suits long-horizon agents with irregular rollout times, but group-relative updates require repeated rollouts of each prompt. However, group-based sampling limits prompt coverage at a fixed rollout budget and introduces group synchronization barriers. One rollout per prompt avoids these constraints but removes the variance-reducing group baseline, making updates noisier and more prone to instability on stale, variable-length trajectories. We present FlashREINFORCE, a critic-free, one-pass framework built on One-Batch REINFORCE, a Sequence Trust Region, and Sample-Mean Optimization. These components preserve prompt coverage and signed feedback while controlling policy drift and length-amplified negative updates. In our experiments, FlashREINFORCE remains stable through 6,000 updates on DeepSeek-R1-Distill-Qwen-1.5B at a policy lag of approximately four updates. On Qwen2.5-Math-1.5B, it reaches 38.0 mean accuracy across five benchmarks, outperforming the reported GRPO baseline by 1.7 percentage points with half the rollouts (256k versus 512k). Python-tool experiments extend to the 30B MoE model Qwen3-30B-A3B, with stable training at policy lag eight. On Qwen2.5-7B-Instruct, FlashREINFORCE maintains tool use where GRPO stops making tool calls and achieves 98.3%/96.5% seen/unseen ALFWorld success.

AI Overview

Why This Paper Matters

Training language models with reinforcement learning becomes more difficult when the model must reason for many steps, call tools, or interact with an environment. Rollouts can differ substantially in length: one agent may finish quickly, while another may spend thousands of tokens reasoning or waiting for tool results. In asynchronous training systems, completed trajectories are sent to the learner as soon as they are available, often after being generated by an older version of the policy.

FlashREINFORCE proposes a critic-free way to learn from these asynchronous, unevenly timed trajectories. Its central design choice is to use exactly one rollout per prompt, rather than sampling groups of responses and comparing them. It combines batch-centered rewards, sequence-level filtering of stale trajectories, and per-trajectory loss normalization.

Figure 1: Reported long-chain reasoning training remains stable while evaluation performance increases over approximately 6,000 cumulative updates.

The paper’s contribution is therefore both algorithmic and systems-oriented. It addresses how to use asynchronous data efficiently without requiring a value-function critic, synchronized response groups, or repeated optimization over the same batch.

The Challenge with Existing Reinforcement Learning Methods

Many language-model reinforcement learning methods use multiple responses to the same prompt. GRPO and RLOO, for example, estimate how good one response is relative to its sampled siblings. This relative comparison acts as a variance-reduction baseline: a response that scores better than the group average receives positive feedback, while a response that scores worse receives negative feedback.

Group sampling has two important drawbacks in long-horizon agent settings.

First, a fixed rollout budget is spent repeatedly on the same prompts. If the system generatesGGtrajectories, group-based training may cover only a fraction of the prompt diversity achievable withGGindependent prompts. Second, group methods introduce synchronization barriers. The learner may need to wait for all responses in a group even if some trajectories have already completed.

Critic-based methods avoid some of these limitations by learning a value estimate, but training a reliable critic for long sequences with sparse or terminal rewards is itself difficult. A critic can also become inaccurate as the policy changes or as the environment distribution shifts.

FlashREINFORCE instead returns to the REINFORCE score-function estimator, but modifies its use for asynchronous language-model training. It constructs a baseline from the mean reward of the current batch and uses one complete batch for one optimizer update.

One Rollout per Prompt and Signed Feedback

Suppose a batch containsBBcompleted trajectories, where trajectoryiireceives scalar rewardRiR_i. FlashREINFORCE computes the batch mean and centers every reward:

Rˉ=1B∑j=1BRj,Ai=Ri−Rˉ\bar{R}=\frac{1}{B}\sum_{j=1}^{B}R_j,\qquad A_i=R_i-\bar{R}

Here,AiA_iis used as the trajectory’s advantage. A trajectory above the batch mean receives positive feedback, while one below the mean receives negative feedback.

This distinction is particularly important for binary rewards. If successful trajectories receive reward one and failures receive reward zero, an uncentered one-sample REINFORCE update can reinforce successes but does not directly discourage failures. Batch centering gives failed trajectories a negative advantage whenever the batch contains enough successful trajectories to place the mean above zero.

The baseline is simple and does not require a separately trained value model. It also works with non-binary rewards, where the sign ofAiA_iindicates whether a trajectory performed above or below the current batch average.

The approach does not claim to solve fine-grained credit assignment. With only a terminal reward, all tokens in a trajectory initially share the same scalar advantage. The paper instead focuses on preventing this coarse signal from being amplified by stale data or by unusually long sequences.

Correcting for Stale Behavior Policies

In asynchronous training, a rollout is generated by a behavior policyμi\mu_i, while the learner updates a newer policyπθ\pi_\theta. The method records the behavior probability of every sampled token at collection time. For tokenttin trajectoryii, it computes the importance ratio

ρi,t(θ)=exp⁡[log⁡πθ(ai,t∣hi,t)−log⁡μi(ai,t∣hi,t)].\rho_{i,t}(\theta)=\exp\left[\log\pi_\theta(a_{i,t}\mid h_{i,t})-\log\mu_i(a_{i,t}\mid h_{i,t})\right].

The sampled action isai,ta_{i,t}and its history ishi,th_{i,t}. The ratio increases the influence of actions that are more likely under the learner than under the behavior policy, and decreases the influence of actions whose probability has fallen.

The paper emphasizes that the behavior probability must be stored during rollout generation. Recomputing an old probability later may not reproduce the actual sampling distribution because inference engines, routing decisions, numerical settings, or training and inference implementations may differ.

For numerical stability, the implementation computes log-ratios in float32, clamps them to the interval from−30-30to3030, and then exponentiates. Unlike PPO, FlashREINFORCE does not use tokenwise ratio clipping as its primary trust-region mechanism.

Sequence-Level Trust Regions

Importance sampling corrects action probabilities at behavior-generated histories, but it does not fully correct the fact that those histories were produced by a stale policy. The older the behavior policy, the more likely the current model is to have moved away from the distribution that generated the trajectory.

FlashREINFORCE addresses this with a sequence-level admission test. For each sampled action, it compares the behavior probabilityppwith the current-policy probabilityqq. It treats the sampled action as one outcome and all other vocabulary tokens as a second outcome, producing the Bernoulli KL proxy

di,t=plog⁡pq+(1−p)log⁡1−p1−q,Dˉi=1Ti∑t=1Tidi,t.d_{i,t}=p\log\frac{p}{q}+(1-p)\log\frac{1-p}{1-q},\qquad \bar{D}_i=\frac{1}{T_i}\sum_{t=1}^{T_i}d_{i,t}.

Here,TiT_iis the number of tokens in trajectoryii. The trajectory is accepted only when

Dˉi≤δ.\bar{D}_i\leq\delta.

The resulting binary maskmim_iis applied to the complete trajectory. If the sequence fails the gate, its token contributions are discarded; if it passes, the trajectory is retained as a coherent training example.

This differs from token-local filtering. Token masking can remove early or late tokens independently even though subsequent tokens are still evaluated at histories generated by the original behavior policy. Sequence-level rejection treats the entire stored trajectory as one off-policy object. The paper’s theoretical discussion links this choice to accumulated policy movement over the trajectory.

The gate is a practical diagnostic rather than a full guarantee. Because it only examines the probability of sampled actions and aggregates the remaining vocabulary into one category, it is a proxy for the full categorical KL. Passing the gate does not prove that the complete policy distributions are close.

Sample-Mean Optimization

The final component changes how token losses are reduced. Instead of averaging all tokens in the batch together, FlashREINFORCE first averages tokens within each trajectory and then averages trajectories:

L=−1B∑i=1BmiAi1Ti∑t=1Tistopgrad⁡(ρi,t)log⁡πθ(ai,t∣hi,t).\mathcal{L}=-\frac{1}{B}\sum_{i=1}^{B}m_iA_i\frac{1}{T_i}\sum_{t=1}^{T_i}\operatorname{stopgrad}(\rho_{i,t})\log\pi_\theta(a_{i,t}\mid h_{i,t}).

This gives each accepted trajectory approximately equal weight. Under conventional token-mean reduction, a trajectory with twice as many tokens contributes roughly twice as much. That can be undesirable when a long failed rollout receives a negative advantage: its failure signal may overwhelm shorter examples.

The method also uses one full-batch optimizer update before discarding the data. Reusing the same rollout batch over several sequential minibatch updates would cause the policy to move while the remaining examples were still off-policy. Fresh one-batch updates reduce this additional source of mismatch.

A simplified implementation looks like this:

Empirical Results

The paper evaluates mathematical reasoning, Python tool use, and interactive decision making.

On DeepSeek-R1-Distill-Qwen-1.5B, trained with 1,460 MATH problems and approximately four updates of policy lag, the method remained stable through 6,000 updates. The reported AIME24/25 mean average accuracy increased from 21.7% to 33.7%. Response length declined during training, while entropy stabilized rather than collapsing.

The sequence trust region contributed to this stability. In a continuation comparison, the trust-gated configuration reached 33.7%, compared with 30.6% without the gate. The ungated run showed higher train–inference KL and stopped improving as effectively.

On Qwen2.5-Math-1.5B, FlashREINFORCE reached a five-benchmark mean of 38.0% using 256,000 rollouts, compared with 36.3% for the reported GRPO baseline using 512,000 rollouts. The result was not uniformly better on every benchmark, but it was stronger on Minerva, AIME2025, and OlympiadBench.

Tool-use experiments provide a more direct test of long-horizon interaction. On Qwen2.5-7B-Instruct with Python, FlashREINFORCE continued making tool calls through 600 updates, averaging 3.25 calls per trajectory. GRPO stopped making tool calls at approximately 200 updates in the reported comparison. On the 30B-parameter Qwen3 mixture-of-experts model, FlashREINFORCE achieved 67.1% versus 60.3% for GRPO at a matched rollout budget, while operating with greater policy lag.

Figure 2: The reported curves show that sequence-gated FlashREINFORCE remains comparatively stable under asynchronous policy lag.

On ALFWorld, the method reached 98.3% success on seen games and 96.5% on unseen games after 12,800 trajectories. These numbers exceed the cited GRPO and C-RF results, although the paper notes that the comparisons are not necessarily fully controlled reproductions.

What Is Distinctive and What Remains Open

FlashREINFORCE differs from group-relative methods by using one independent rollout per prompt. It differs from critic-based methods by using a batch reward mean instead of a learned value estimator. It differs from PPO-style approaches by controlling drift through sequence-level admission rather than relying primarily on tokenwise ratio clipping. Finally, its sample-mean reduction prevents sequence length from determining a trajectory’s weight.

Ablations support all three design choices. Removing negative feedback sharply damaged tool-use learning and caused tool calls to disappear. Token-mean reduction produced longer responses, more truncation, and fewer tool calls. Token-local trust masking was unstable compared with sequence-level gating.

Several limitations qualify the conclusions. Terminal rewards still provide coarse token credit assignment. The KL gate is only a sampled-action proxy, and sample-mean normalization intentionally changes the weighting of the standard REINFORCE objective. Entropy-based filtering provides modest optional gains but becomes harmful when applied aggressively. The evaluation also covers a limited set of environments, and some baselines come from prior work with different systems and synchronization conditions.

Overall, the paper presents a compact recipe for asynchronous agent training: collect independent trajectories, center rewards across a fresh batch, reject sequences that have drifted too far from their behavior policy, and give each retained trajectory equal weight. Its main value is showing how these choices can make single-rollout, critic-free reinforcement learning usable for long-horizon language-model behavior.

Rethinking the trust region in LLM reinforcement learning

This work directly motivates FlashREINFORCE’s sequence-level trust mechanism and divergence-based admission rule. The paper specifically draws on DPPO’s analysis of importance sampling, divergence control, and token-versus-sequence trust decisions.

Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in LLM reinforcement learning. arXiv preprint arXiv:2602.04879, 2026. 4, 6, 11, 12, 17

Trust region masking for long-horizon LLM reinforcement learning

Trust Region Masking is a closely related approach to sequence-level admission under policy drift in long-horizon language-model RL. FlashREINFORCE explicitly uses its accumulated-drift perspective and cites it as a key motivation for rejecting complete trajectories rather than individual tokens.

Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, and Baoxiang Wang. Trust region masking for long-horizon LLM reinforcement learning. arXiv preprint arXiv:2512.23075, 2026. 4, 12, 22, 26

When speed kills stability: Demystifying RL collapse from the training–inference mismatch

This citation is central to FlashREINFORCE’s treatment of asynchronous training–inference mismatch and sequence-level filtering. Seq-MIS provides a directly related precedent for combining importance correction with trajectory-level stability control.

Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demystifying RL collapse from the training–inference mismatch. Technical article, 2025. URL https://richardli.xyz/rl-collapse. Introduces Sequence-Level Masked Importance Sampling (Seq-MIS). 4, 12

IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures

IMPALA is the foundational asynchronous actor–learner reference for using importance weighting when rollout workers and the learner operate on different policy snapshots. FlashREINFORCE extends this off-policy asynchronous paradigm to critic-free, single-rollout language-model training.

Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Victor Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, 2018. 12

Variance reduction techniques for gradient estimates in reinforcement learning

This paper provides the theoretical foundation for action-independent baselines and variance reduction in policy-gradient methods. FlashREINFORCE’s batch-mean reward centering is introduced specifically to recover variance reduction when sibling-rollout baselines and critics are unavailable.

Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5:1471–1530, 2004. 1, 12

Audio

FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models

FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models

Transcript

Similar papers

Show moreShow less

Discussion