SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

# SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Source: [https://arxiv.org/html/2608.19842](https://arxiv.org/html/2608.19842)
Lang Feng11footnotemark:1Affiliation:Bo AnAffiliation:Nanyang Technological University, Singaporedyliang@stu\.xmu\.edu\.cn,lang005@e\.ntu\.edu\.sg,ylliu@xmu\.edu\.cn,Yunlong LiuThanks:Corresponding authorAffiliation:Xiamen University, China

###### Abstract

Agentic reinforcement learning \(RL\) has become a critical stage in the post\-training of large language models\. Existing critic\-free, group\-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization \(PPO\) and achieving strong performance on long\-horizon interactive tasks\. Despite their success, recent studies revealed three limitations: \(1\) Lack explicit value generalization and effective temporal credit assignment; \(2\) Suffer from potential advantage collapse in long\-horizon complex tasks; \(3\) Require a costly trade\-off between sampling budget and policy performance\. In this work, we proposeSingle\-rolloutAutoregressivePolicyOptimization \(SAPO\), a low\-memory and compute\-efficient framework in which the policy and value functions share a single autoregressive backbone\. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on\-policy SARSA objectives\. To robustly estimate the contribution of each turn, we further introduce a trajectory\-level generalized advantage estimator that combinesλ\\lambda\-returns with batch normalization\. Experiments across ALFWorld and WebShop with Qwen2\.5\-1\.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean\+15\.1and\+12\.1percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per\-iteration runtime by33\.2%over PPO\. \(Going On\)

## 1Introduction

Recent advances in agentic reinforcement learning \(RL\) have transformed large language models from passive text generators into agents capable of long\-horizon interaction such as reasoning, tool use, web search, and code execution tasks\. Large\-scale RL post\-training has produced substantial improvements in mathematical reasoning and coding\[[17](https://arxiv.org/html/2608.19842#bib.bib1),[5](https://arxiv.org/html/2608.19842#bib.bib2),[7](https://arxiv.org/html/2608.19842#bib.bib3)\], while interactive systems such as Search\-R1 demonstrate that policies can learn to acquire external information through multi\-turn actions\[[8](https://arxiv.org/html/2608.19842#bib.bib4)\]\. These successes suggest that learning directly from environmental feedback is becoming a critical pathway toward increasingly autonomous and general\-purpose language agents\.

Most agentic RL approaches build on proximal policy optimization \(PPO\) or its critic\-free, group\-relative variants\. Conventional PPO provides token\- or turn\-level credit assignment through a learned value function and generalized advantage estimation \(GAE\), but typically requires a policy\-scale critic with substantial memory and computation budgets\[[13](https://arxiv.org/html/2608.19842#bib.bib5),[21](https://arxiv.org/html/2608.19842#bib.bib6)\]\. GRPO avoids this critic by normalizing rewards across multiple sampled for the same task\[[17](https://arxiv.org/html/2608.19842#bib.bib1)\]\. Nevertheless, recent studies expose three limitations\. First, when all rollouts in a group receive identical or nearly identical rewards—an increasingly common event under sparse feedback and long\-horizon failures—the normalized advantages vanish, causing zero\-gradient or advantage\-collapse regions\[[29](https://arxiv.org/html/2608.19842#bib.bib7)\]\. Second, assigning a trajectory\-level group advantage uniformly to all generated tokens provides weak temporal credit assignment and cannot generalize value estimates across states, prompts, or policy iterations; learned critics can identify low\-value repetitive or erroneous prefixes that group\-relative estimators miss\[[30](https://arxiv.org/html/2608.19842#bib.bib8),[31](https://arxiv.org/html/2608.19842#bib.bib9),[7](https://arxiv.org/html/2608.19842#bib.bib3)\]\. Third, group\-relative estimation entails an unfavorable compute–quality trade\-off: small groups yield noisy baselines, whereas larger groups improve stability only by multiplying rollout cost and imposing synchronization barriers in long, variable\-length agent trajectories\[[3](https://arxiv.org/html/2608.19842#bib.bib10),[6](https://arxiv.org/html/2608.19842#bib.bib11)\]\.

Several work attempts to mitigate these problems\. DAPO improves group\-relative learning through dynamic sampling, asymmetric clipping, and token\-level policy losses, but remains dependent on multiple rollouts and group statistics\[[29](https://arxiv.org/html/2608.19842#bib.bib7)\]\. ReMax and RLOO eliminate the critic using greedy or leave\-one\-out baselines, while VinePPO estimates intermediate values through Monte Carlo continuations; these methods reduce training\-state memory but retain coarse credit assignment or introduce additional generation cost\[[12](https://arxiv.org/html/2608.19842#bib.bib12),[2](https://arxiv.org/html/2608.19842#bib.bib13),[9](https://arxiv.org/html/2608.19842#bib.bib14)\]\. Conversely, Open\-Reasoner\-Zero, VC\-PPO, and VAPO restore explicit value models and demonstrate that a well\-trained critic can substantially improve long\-horizon reasoning, particularly through critic pretraining and improved GAE\[[7](https://arxiv.org/html/2608.19842#bib.bib3),[30](https://arxiv.org/html/2608.19842#bib.bib8),[31](https://arxiv.org/html/2608.19842#bib.bib9)\]\. However, these methods maintain a separate, often policy\-scale value network\. Hydra\-PPO partially shares a frozen backbone, but still requires role\-specific adapters and heads, and its fully shared variant reveals potential policy–value interference\[[15](https://arxiv.org/html/2608.19842#bib.bib15)\]\. POISE reuses actor hidden states through a lightweight probe but requires cross\-rollout estimation\[[3](https://arxiv.org/html/2608.19842#bib.bib10)\], whereas SAO enables asynchronous single\-rollout training while relying on a separately pretrained critic with more frequent value updates\[[6](https://arxiv.org/html/2608.19842#bib.bib11)\]\. Thus, efficient single\-rollout learning with explicit temporal value estimation remains unresolved\.

To address this, we propose Single\-Rollout Autoregressive Policy Optimization \(SAPO\), a lightweight RL framework that preserves value\-based learning without maintaining a separate critic or relying on multiple rollouts\. SAPO builds on a simple observation: the information flow required by actor–critic learning naturally follows the causal order of language generation\. A value estimate should summarize the state before an action is taken, whereas evaluating that action may additionally condition on what has been generated\. This alignment allows policy generation and value estimation to be expressed at appropriate positions within one autoregressive stream, with causal masking enforcing their distinct conditioning contexts\. Consequently, the same backbone can generate actions and learn generalized value estimates while reusing its parameters and intermediate computation\. SAPO thus reconciles two competing goals in existing approaches: it retains the explicit value generalization and temporal credit assignment of actor–critic algorithms, yet approaches the memory and computational efficiency sought by critic\-free, group\-relative optimization\. Its central contribution is therefore not weight sharing alone, but a correspondence between autoregressive modeling and actor–critic learning that enables efficient value\-based optimization within a single causal model\.

We evaluate SAPO on ALFWorld and WebShop using Qwen2\.5\-1\.5B and Qwen2\.5\-7B backbone models\. SAPO trains stably from one rollout per prompt and improves task success over conventional PPO and GRPO by\+15\.1and\+12\.1percentage points, respectively\. It simultaneously eliminates the memory footprint of a separate critic and reduces per\-iteration runtime by33\.2%relative to PPO\. Our contributions are threefold\.

- •We introduce a single\-rollout autoregressive actor–critic architecture that jointly represents and trains policy and value functions through causal\-boundary readouts\.
- •We develop a trajectory\-level advantage estimator that combinesλ\\lambda\-returns with batch normalization, enabling explicit turn\-level credit assignment without group\-relative sampling\.
- •We demonstrate on long\-horizon agentic tasks that this unified framework achieves a favorable combination of performance, training stability, memory efficiency, and runtime efficiency\.

## 2Related Work

Agentic RL extends reinforcement learning with verifiable rewards from single\-turn response optimization to multi\-turn decision making, enabling LLM agents to learn from the consequences of actions taken in external environments\. A prominent line of work builds on critic\-free, group\-relative optimization\. GRPO\[[17](https://arxiv.org/html/2608.19842#bib.bib1)\]replaces a learned value baseline with within\-prompt outcome comparison, a formulation used by DeepSeek\-R1\[[5](https://arxiv.org/html/2608.19842#bib.bib2)\]and refined by DAPO\[[29](https://arxiv.org/html/2608.19842#bib.bib7)\]with dynamic sampling and asymmetric clipping, GSPO\[[33](https://arxiv.org/html/2608.19842#bib.bib17)\]with sequence\-level importance ratios, and GVPO\[[32](https://arxiv.org/html/2608.19842#bib.bib18)\]through variance\-aware gradient weighting\. This paradigm has been extended to interactive settings: Search\-R1\[[8](https://arxiv.org/html/2608.19842#bib.bib4)\]and DeepResearcher\[[34](https://arxiv.org/html/2608.19842#bib.bib19)\]optimize iterative retrieval, ToRL\[[11](https://arxiv.org/html/2608.19842#bib.bib20)\]and WebSailor\[[10](https://arxiv.org/html/2608.19842#bib.bib21)\]train tool use and web navigation, and RAGEN\[[25](https://arxiv.org/html/2608.19842#bib.bib22)\]characterizes optimization instabilities in multi\-turn training\. Long\-horizon tasks have also motivated more structured credit mechanisms\. ARChER\[[35](https://arxiv.org/html/2608.19842#bib.bib23)\]and HiPER\[[14](https://arxiv.org/html/2608.19842#bib.bib24)\]decompose planning and execution across temporal scales, AgentGym\-RL\[[26](https://arxiv.org/html/2608.19842#bib.bib25)\]progressively increases the interaction horizon, and IGPO\[[23](https://arxiv.org/html/2608.19842#bib.bib26)\]derives dense intrinsic rewards from information gain during search\.

Efficiency\-oriented LLM RL has developed along critic\-free advantage estimation, parameter sharing between policy and value functions, and systems\-level execution\. ReMax\[[12](https://arxiv.org/html/2608.19842#bib.bib12)\]and RLOO\[[2](https://arxiv.org/html/2608.19842#bib.bib13)\]replace learned values with greedy or leave\-one\-out baselines, while GRPO\[[17](https://arxiv.org/html/2608.19842#bib.bib1)\]normalizes rewards within each prompt group\. Eliminating the value network reduces parameters and optimizer state, but these estimators require additional decoded trajectories and provide no value generalization across states\. VinePPO\[[9](https://arxiv.org/html/2608.19842#bib.bib14)\]obtains intermediate value estimates from branched Monte Carlo continuations, likewise exchanging critic training for increased rollout computation\. Parameter\-sharing approaches retain explicit value learning while reducing model duplication\. Hydra\-PPO\[[15](https://arxiv.org/html/2608.19842#bib.bib15)\]places policy and value adapters over a frozen backbone; J\-Hydra\[[15](https://arxiv.org/html/2608.19842#bib.bib15)\]further shares the trainable adapter, with potential interference between the two objectives\. POISE\[[3](https://arxiv.org/html/2608.19842#bib.bib10)\]predicts returns from actor hidden states using a lightweight probe, but relies on cross\-rollout construction to decouple value estimation from the evaluated trajectory\. SAO\[[6](https://arxiv.org/html/2608.19842#bib.bib11)\]addresses rollout underutilization through asynchronous single\-rollout collection, while retaining a separately pretrained critic with more frequent value updates\. HybridFlow\[[18](https://arxiv.org/html/2608.19842#bib.bib27)\]instead improves distributed execution through model placement, resharding, and scheduling, independently of the advantage estimator\. SAPO is distinguished by jointly representing the policy, state value, and action value in one causal autoregressive model, thereby eliminating both group sampling and the separate critic backbone while retaining explicit temporal credit assignment\.

## 3Preliminaries

#### Multi\-Turn Agentic RL\.

We consider RL for LLM\-based agents that interact with an environment over multiple turns, where the interaction process is formulated as a finite\-horizon Markov Decision Process \(MDP\)\. Given a task instance𝒙∈p⁡\(X\)\\boldsymbol\{x\}\\in p\(X\), at each turnt=1,2,…,Tt=1,2,\\ldots,T, an agent policyπθ\\pi\_\{\\theta\}observes a state𝒔t∈𝒮\\boldsymbol\{s\}\_\{t\}\\in\\mathcal\{S\}and generates a textual action𝒂t∈𝒜\\boldsymbol\{a\}\_\{t\}\\in\\mathcal\{A\}, and then transitions to the next state𝒔t\+1∈𝒮\\boldsymbol\{s\}\_\{t\+1\}\\in\\mathcal\{S\}while yielding a scalar rewardrt∈ℝr\_\{t\}\\in\\mathbb\{R\}\. The interaction unfolds as a trajectory𝝉=\{\(𝒔1,𝒂1,r1\),\(𝒔2,𝒂2,r2\),…,\(𝒔T,𝒂T,rT\)\}\\boldsymbol\{\\tau\}=\\\{\(\\boldsymbol\{s\}\_\{1\},\\boldsymbol\{a\}\_\{1\},r\_\{1\}\),\(\\boldsymbol\{s\}\_\{2\},\\boldsymbol\{a\}\_\{2\},r\_\{2\}\),\.\.\.,\(\\boldsymbol\{s\}\_\{T\},\\boldsymbol\{a\}\_\{T\},r\_\{T\}\)\\\}, whereTTdenotes the trajectory length in interaction turns\. In practical tasks such as ALFWorld, the agent generates a response as its action for each observation\. This response is structured within<think\></think\>and<action\></action\>tags, where the former contains the reasoning process, while the latter specifies the actual action executed in the environment\. Notably, for most task settings, reward signals are sparse and delayed, e\.g\., the environment provides an outcome rewardR⁡\(𝝉\)R\(\\boldsymbol\{\\tau\}\)only after the trajectory terminates\.

### 3\.1Group\-relative Policy Optimization

Recent agentic RL methods for LLMs commonly adopt a group\-relative policy optimization paradigm\. Given a task instance𝒙\\boldsymbol\{x\}, the old policyπθold\\pi\_\{\\theta\_\{\\rm old\}\}samples a group ofNNcandidate trajectories𝒢x=\{𝝉1,𝝉2,…,𝝉N\}\\mathcal\{G\}\_\{x\}=\\\{\\boldsymbol\{\\tau\}\_\{1\},\\boldsymbol\{\\tau\}\_\{2\},\\ldots,\\boldsymbol\{\\tau\}\_\{N\}\\\}, where each trajectory corresponds to one complete rollout\. Each trajectory𝝉i\\boldsymbol\{\\tau\}\_\{i\}receives a scalar rewardR⁡\(𝝉i\)R\(\\boldsymbol\{\\tau\}\_\{i\}\)that reflects the overall quality or success of the generated outcome\. Instead of learning an advantage functionA⁡\(𝒔t,𝒂t\)A\(\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{a\}\_\{t\}\)with critic networks in PPO\[[16](https://arxiv.org/html/2608.19842#bib.bib16)\], group\-based RL computes the advantage using only statistics within the sampled group:

A⁡\(𝝉i\)=𝙶𝚛𝚘𝚞𝚙𝙽𝚘𝚛𝚖𝚊𝚕𝚒𝚣𝚊𝚝𝚒𝚘𝚗⁡\(\{R⁡\(𝝉i\)\}i=1N\)\.A\(\\boldsymbol\{\\tau\}\_\{i\}\)=\\mathtt\{GroupNormalization\}\\left\(\\\{R\(\\boldsymbol\{\\tau\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}\\right\)\.In GRPO\[[17](https://arxiv.org/html/2608.19842#bib.bib1)\], the advantage is evaluated by normalizing each trajectory reward with the mean and variance of group rewards\(\{R⁡\(𝝉i\)\}i=1N\)\(\\\{R\(\\boldsymbol\{\\tau\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}\)\. This sampling\-based estimator reduces the memory and computational overhead introduced by the critic architecture in conventional PPO\.

### 3\.2Proximal Policy Optimization

Proximal Policy Optimization \(PPO\) is an actor\-critic RL algorithm previously used for LLM post\-training\. Given prompts sampled from a dataset, the policy generates response trajectories and receives rewards from task\-specific environments or learned reward models\. PPO updates the policy by maximizing a clipped surrogate objective,

ℒPPO=𝔼t​\[min⁡\(ρt​A^t,clip⁡\(ρt,1−ϵ,1\+ϵ\)​A^t\)\],\\mathcal\{L\}\_\{\\mathrm\{PPO\}\}=\\mathbb\{E\}\_\{t\}\\\!\\left\[\\min\\\!\\left\(\\rho\_\{t\}\\hat\{A\}\_\{t\},\\,\\operatorname\{clip\}\(\\rho\_\{t\},1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{t\}\\right\)\\right\],
whereρt=πθ​\(at∣st\)/πθold​\(at∣st\)\\rho\_\{t\}=\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)is the likelihood ratio andA^t\\hat\{A\}\_\{t\}is an advantage estimate\. The clipping operation constrains abrupt policy changes and improves optimization stability\. A learned criticVϕ​\(st\)V\_\{\\phi\}\(s\_\{t\}\)predicts expected returns, enabling advantage estimation—commonly through generalized advantage estimation—and propagating delayed rewards to earlier decisions\. In LLMs, states correspond to token prefixes or interaction histories, while actions are generated tokens\. Although effective for long\-horizon credit assignment, PPO typically requires separate policy, critic, reference, and rollout models, resulting in substantial memory and computational overhead\.

## 4Method

SAPO is a single\-rollout actor–critic method in which one causal language model represents the policy, the state value, and the action value\. As shown in Figure[1](https://arxiv.org/html/2608.19842#S4.F1), the design follows the temporal structure of an agent turn: the model first summarizes the current interaction state, then generates a textual action, and finally evaluates that action before observing its environmental consequence\. This order matches the conditioning structure required by actor–critic learning and allows all three quantities to be extracted from one autoregressive sequence\.

![Refer to caption](https://arxiv.org/html/2608.19842v1/sapo_fw.png)Figure 1:Framework of SAPO\.We begin with introducing a two\-token value basis that reads bounded scalar values from the language\-model logits at two causal boundaries \(Section[4\.1](https://arxiv.org/html/2608.19842#S4.SS1)\)\. Second, we construct turn\-level advantages and value targets by traversing each sampled trajectory backward \(Section[4\.2](https://arxiv.org/html/2608.19842#S4.SS2)\)\. Third, we jointly optimize the token\-level policy objective and the turn\-level value objectives in one actor update \(Section[4\.3](https://arxiv.org/html/2608.19842#S4.SS3)\)\. Algorithm[1](https://arxiv.org/html/2608.19842#alg1)summarizes the complete procedure\.

### 4\.1Autoregressive Actor–Critic Representation

#### Causal factorization of an agent turn\.

At turntt, let𝒄ts\\boldsymbol\{c\}^\{s\}\_\{t\}denote the tokenized context representing state𝒔t\\boldsymbol\{s\}\_\{t\}, and let𝒂t=\(at,1,…,at,Mt\)\\boldsymbol\{a\}\_\{t\}=\(a\_\{t,1\},\\ldots,a\_\{t,M\_\{t\}\}\)be the generated response\. We write𝒄ts​a=\[𝒄ts;𝒂t\]\\boldsymbol\{c\}^\{sa\}\_\{t\}=\[\\boldsymbol\{c\}^\{s\}\_\{t\};\\boldsymbol\{a\}\_\{t\}\]for their concatenation\. A valid state value may depend on𝒄ts\\boldsymbol\{c\}^\{s\}\_\{t\}but not on the yet\-unseen action, whereas an action value may additionally depend on the complete𝒂t\\boldsymbol\{a\}\_\{t\}but not on the subsequent reward or observation\. These dependencies occur naturally at the two boundaries of causal decoding: immediately before the first response token and immediately after the last response token\. SAPO readsV⁡\(𝒔t\)V\(\\boldsymbol\{s\}\_\{t\}\)andQ⁡\(𝒔t,𝒂t\)Q\(\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{a\}\_\{t\}\)at these respective boundaries while generating𝒂t\\boldsymbol\{a\}\_\{t\}between them\. The causal mask enforces the desired information separation without separate encoders or manually detached context representations\.

#### A shared two\-token value basis\.

Let𝒛θ​\(𝒄\)∈ℝ\|𝒲\|\\boldsymbol\{z\}\_\{\\theta\}\(\\boldsymbol\{c\}\)\\in\\mathbb\{R\}^\{\|\\mathcal\{W\}\|\}be the next\-token logits produced by the causal language model after context𝒄\\boldsymbol\{c\}, where𝒲\\mathcal\{W\}is its vocabulary\. We reserve two existing vocabulary entries,w\+w^\{\+\}andw−w^\{\-\}, exclusively as a value basis\. For any causal context, their relative evidence defines a normalized scalar readout:

pθ​\(𝒄\)=clip​\(\(zθ,w\+​\(𝒄\)−zθ,w−​\(𝒄\)τv\),−1,1\)p\_\{\\theta\}\(\\boldsymbol\{c\}\)=\\text\{clip\}\\left\(\\\!\\left\(\\frac\{z\_\{\\theta,w^\{\+\}\}\(\\boldsymbol\{c\}\)\-z\_\{\\theta,w^\{\-\}\}\(\\boldsymbol\{c\}\)\}\{\\tau\_\{v\}\}\\right\),\-1,1\\right\)\(1\)whereτv\>0\\tau\_\{v\}\>0is a value temperature\. For tasks whose discounted returns lie in\[−Rmax,Rmax\]\[\-R\_\{\\max\},R\_\{\\max\}\], SAPO parameterizes:

Vθ​\(𝒔t\)=Rmax​pθ​\(𝒄ts\),Qθ​\(𝒔t,𝒂t\)=Rmax​pθ​\(𝒄ts​a\)\.V\_\{\\theta\}\(\\boldsymbol\{s\}\_\{t\}\)=R\_\{\\max\}p\_\{\\theta\}\(\\boldsymbol\{c\}^\{s\}\_\{t\}\),\\qquad Q\_\{\\theta\}\(\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{a\}\_\{t\}\)=R\_\{\\max\}p\_\{\\theta\}\(\\boldsymbol\{c\}^\{sa\}\_\{t\}\)\.\(2\)The logit difference makes the readout invariant to a common shift of the two logits, while the clipping supplies a learning range matched to the return scale\. Importantly,w\+w^\{\+\}andw−w^\{\-\}are readouts rather than generated reasoning tokens: neither token is sampled, appended to the context, or revealed to the environment\.

#### Separation from the action distribution\.

The reserved entries must not become artificial actions\. We therefore define the policy vocabulary as𝒲act=𝒲∖\{w\+,w−\}\\mathcal\{W\}\_\{\\mathrm\{act\}\}=\\mathcal\{W\}\\setminus\\\{w^\{\+\},w^\{\-\}\\\}and normalize action probabilities only over this set:

πθ​\(a∣𝒄\)=exp⁡zθ,a​\(𝒄\)∑w∈𝒲actexp⁡zθ,w​\(𝒄\),a∈𝒲act\.\\pi\_\{\\theta\}\(a\\mid\\boldsymbol\{c\}\)=\\frac\{\\exp z\_\{\\theta,a\}\(\\boldsymbol\{c\}\)\}\{\\sum\_\{w\\in\\mathcal\{W\}\_\{\\mathrm\{act\}\}\}\\exp z\_\{\\theta,w\}\(\\boldsymbol\{c\}\)\},\\qquad a\\in\\mathcal\{W\}\_\{\\mathrm\{act\}\}\.\(3\)The same restriction is applied during rollout, policy\-log\-probability evaluation, and entropy computation\. Thus, policy learning never rewards or suppresses the value basis through the action softmax, while the raw logits of the two reserved entries remain available to the value objectives\. Policy and value learning share the transformer and language\-model head, but retain distinct semantics and supervision\.

#### Single\-stream evaluation\.

For every sampled response, PPO already evaluates the actor on\[𝒄ts;𝒂t\]\[\\boldsymbol\{c\}^\{s\}\_\{t\};\\boldsymbol\{a\}\_\{t\}\]to obtain old token log probabilities\. SAPO extends this actor evaluation by gathering the two reserved logits at the final state\-context position and the final valid action position\. The former yieldsVθold​\(𝒔t\)V\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\boldsymbol\{s\}\_\{t\}\)and the latter yieldsQθold​\(𝒔t,𝒂t\)Q\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{a\}\_\{t\}\); no second model or repeated encoding of the state–action prefix is required\. During optimization, the same actor forward simultaneously produces current policy log probabilities, state values, and action values\. Hence, the precise computational benefit of SAPO is the removal of the independent critic forward/backward path and its training state, rather than an assumption that value estimates are obtained without any actor evaluation\.

### 4\.2Single\-Rollout Trajectory Advantage Estimation

Group\-relative algorithms estimate advantages by comparing several trajectories sampled for the same task\. SAPO instead samples one trajectory per task and transfers delayed feedback across its interaction turns using the learned state value\. All targets are computed from rewards observed in the environment and values produced by the frozen rollout policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}; current value predictions are never used to construct their own targets\.

#### Trajectory\-wise generalized advantage estimation\.

For a trajectory𝝉=\{\(𝒔t,𝒂t,rt,dt\)\}t=1T\\boldsymbol\{\\tau\}=\\\{\(\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{a\}\_\{t\},r\_\{t\},d\_\{t\}\)\\\}\_\{t=1\}^\{T\}, letdtd\_\{t\}indicate termination after turnttand letVtold=Vθold​\(𝒔t\)V^\{\\mathrm\{old\}\}\_\{t\}=V\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\boldsymbol\{s\}\_\{t\}\)\. We compute the temporal\-difference residual and the generalized advantage recursively as:

δt\\displaystyle\\delta\_\{t\}=rt\+γ⁡\(1−dt\)​Vt\+1old−Vtold,\\displaystyle=r\_\{t\}\+\\gamma\(1\-d\_\{t\}\)V^\{\\mathrm\{old\}\}\_\{t\+1\}\-V^\{\\mathrm\{old\}\}\_\{t\},\(4\)AtGAE\\displaystyle A^\{\\mathrm\{GAE\}\}\_\{t\}=δt\+γ​λ​\(1−dt\)​At\+1GAE,\\displaystyle=\\delta\_\{t\}\+\\gamma\\lambda\(1\-d\_\{t\}\)A^\{\\mathrm\{GAE\}\}\_\{t\+1\},\(5\)withAT\+1GAE=0A^\{\\mathrm\{GAE\}\}\_\{T\+1\}=0\. A trajectory truncated at the environment horizon is treated as a finite\-horizon terminal trajectory, so its final bootstrap value is zero\. This recursion gives different turns different learning signals even when the environment supplies only a terminal outcome, whileλ\\lambdacontrols the usual bias–variance trade\-off\.

The same backward pass provides complementary targets for the two causal value boundaries\. The state value is trained toward theλ\\lambda\-return, whereas the action value is trained toward a on\-policy SARSA target:

ytV=Vtold\+AtGAE,ytQ=rt\+γ⁡\(1−dt\)​Qt\+1old\.y^\{V\}\_\{t\}=V^\{\\mathrm\{old\}\}\_\{t\}\+A^\{\\mathrm\{GAE\}\}\_\{t\},\\qquad y^\{Q\}\_\{t\}=r\_\{t\}\+\\gamma\(1\-d\_\{t\}\)Q^\{\\mathrm\{old\}\}\_\{t\+1\}\.\(6\)The first target propagates long\-range outcome information to the pre\-action state boundary\. The second grounds the post\-action boundary in the immediate consequence of the selected response\. We deliberately construct the policy advantage from GAE rather than from the difference between simultaneously learnedQQandVVestimates: this prevents early action\-value miscalibration from directly perturbing the policy while still using theQQobjective to train an action\-conditioned value representation\.

#### Batch\-normalized turn advantages\.

Before assigning advantages to response tokens, we normalize them over the valid turns in the current training batch\. If an environment exposes an invalid\-action indicatormtinvm\_\{t\}^\{\\mathrm\{inv\}\}, we first formA~t=AtGAE−cinv​mtinv\\widetilde\{A\}\_\{t\}=A^\{\\mathrm\{GAE\}\}\_\{t\}\-c\_\{\\mathrm\{inv\}\}m\_\{t\}^\{\\mathrm\{inv\}\}; otherwisecinv=0c\_\{\\mathrm\{inv\}\}=0\. With batch statisticsμℬ\\mu\_\{\\mathcal\{B\}\}andσℬ\\sigma\_\{\\mathcal\{B\}\}, the policy advantage is

A^t=A~t−μℬσℬ\+ϵadv\.\\widehat\{A\}\_\{t\}=\\frac\{\\widetilde\{A\}\_\{t\}\-\\mu\_\{\\mathcal\{B\}\}\}\{\\sigma\_\{\\mathcal\{B\}\}\+\\epsilon\_\{\\mathrm\{adv\}\}\}\.\(7\)Only after this turn\-level normalization do we broadcastA^t\\widehat\{A\}\_\{t\}to all valid tokens in𝒂t\\boldsymbol\{a\}\_\{t\}\. Normalizing before broadcasting prevents long responses from receiving disproportionate weight in the advantage statistics\. Unlike group normalization, Equation[7](https://arxiv.org/html/2608.19842#S4.E7)compares learning signals across independent turns and tasks; it therefore does not require multiple synchronized rollouts of the same prompt\.

### 4\.3Joint Policy and Value Optimization

#### Token\-level policy objective\.

Although the environment acts at the turn level, each textual action contains multiple autoregressive decisions\. Letℓt,jold\\ell^\{\\mathrm\{old\}\}\_\{t,j\}andℓt,j​\(θ\)\\ell\_\{t,j\}\(\\theta\)be the old and current log probabilities of tokenat,ja\_\{t,j\}under the restricted action distribution in Equation[3](https://arxiv.org/html/2608.19842#S4.E3)\. The importance ratio is

ρt,j​\(θ\)=exp⁡\(ℓt,j​\(θ\)−ℓt,jold\)\.\\rho\_\{t,j\}\(\\theta\)=\\exp\\\!\\left\(\\ell\_\{t,j\}\(\\theta\)\-\\ell^\{\\mathrm\{old\}\}\_\{t,j\}\\right\)\.\(8\)Every valid response token in turnttsharesA^t\\widehat\{A\}\_\{t\}, yielding the clipped policy loss

ℒpol​\(θ\)=−𝔼t,j​\[min⁡\(ρt,j​\(θ\)​A^t,clip⁡\(ρt,j​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\],\\mathcal\{L\}\_\{\\mathrm\{pol\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{t,j\}\\\!\\left\[\\min\\\!\\left\(\\rho\_\{t,j\}\(\\theta\)\\widehat\{A\}\_\{t\},\\operatorname\{clip\}\(\\rho\_\{t,j\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\widehat\{A\}\_\{t\}\\right\)\\right\],\(9\)where the expectation includes only valid action tokens\. This preserves PPO’s token\-level trust\-region surrogate while replacing both the separate critic and group\-relative advantage estimator\.

#### Clipped value objectives\.

We optimize the value readouts in their normalized probability space\. DefineptV=Vθ​\(𝒔t\)/Rmaxp^\{V\}\_\{t\}=V\_\{\\theta\}\(\\boldsymbol\{s\}\_\{t\}\)/R\_\{\\max\}andptQ=Qθ​\(𝒔t,𝒂t\)/Rmaxp^\{Q\}\_\{t\}=Q\_\{\\theta\}\(\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{a\}\_\{t\}\)/R\_\{\\max\}, with frozen rollout\-time predictionsptV,oldp^\{V,\\mathrm\{old\}\}\_\{t\}andptQ,oldp^\{Q,\\mathrm\{old\}\}\_\{t\}\. Accordingly, the normalized targets arey¯tV=ytV/Rmax\\bar\{y\}^\{V\}\_\{t\}=y^\{V\}\_\{t\}/R\_\{\\max\}andy¯tQ=ytQ/Rmax\\bar\{y\}^\{Q\}\_\{t\}=y^\{Q\}\_\{t\}/R\_\{\\max\}with normalized range\[−1,1\]\[\-1,1\]\. ForX∈\{V,Q\}X\\in\\\{V,Q\\\}, let

ptX,clip=ptX,old\+clip⁡\(ptX−ptX,old,−ϵX,ϵX\)\.p^\{X,\\mathrm\{clip\}\}\_\{t\}=p^\{X,\\mathrm\{old\}\}\_\{t\}\+\\operatorname\{clip\}\\\!\\left\(p^\{X\}\_\{t\}\-p^\{X,\\mathrm\{old\}\}\_\{t\},\-\\epsilon\_\{X\},\\epsilon\_\{X\}\\right\)\.\(10\)We use the PPO\-style clipped regression loss

ℒX​\(θ\)=12​𝔼t​\[max⁡\(\(ptX−y¯tX\)2,\(ptX,clip−y¯tX\)2\)\],X∈\{V,Q\}\.\\mathcal\{L\}\_\{X\}\(\\theta\)=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{t\}\\\!\\left\[\\max\\\!\\left\(\(p^\{X\}\_\{t\}\-\\bar\{y\}^\{X\}\_\{t\}\)^\{2\},\(p^\{X,\\mathrm\{clip\}\}\_\{t\}\-\\bar\{y\}^\{X\}\_\{t\}\)^\{2\}\\right\)\\right\],\\qquad X\\in\\\{V,Q\\\}\.\(11\)Both losses are averaged over valid turns, not over response tokens\. This gives each environment decision equal weight regardless of response length\.

#### Unified objective\.

The final loss combines policy optimization, the two causal value objectives, reference\-policy regularization, and entropy regularization:

ℒS​A​P​O​\(θ\)=ℒpol\+cV​ℒV\+cQ​ℒQ\+β​ℒKL−cH​ℋ​\(πθ\)\.\\mathcal\{L\}\_\{SAPO\{\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{pol\}\}\+c\_\{V\}\\mathcal\{L\}\_\{V\}\+c\_\{Q\}\\mathcal\{L\}\_\{Q\}\+\\beta\\mathcal\{L\}\_\{\\mathrm\{KL\}\}\-c\_\{H\}\\mathcal\{H\}\(\\pi\_\{\\theta\}\)\.\(12\)The policy, state\-value, and action\-value terms use separate masks but update the same transformer and language\-model head in one backward pass\. CoefficientscVc\_\{V\}andcQc\_\{Q\}control interference between generation and value learning, while the KL and entropy terms regularize policy drift and exploration\. This objective retains distinct supervision for the three roles without allocating role\-specific model parameters\.

Algorithm 1Single\-Rollout Autoregressive Policy Optimization0:Policy

πθ\\pi\_\{\\theta\}, reference policy

πref\\pi\_\{\\mathrm\{ref\}\}, task batch

ℬ\\mathcal\{B\},

γ\\gamma,

λ\\lambda
1:foreach training iterationdo

2:Set

θold←θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\thetaand sample one trajectory per task with

πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}
3:Evaluate sampled state–action sequences with the old actor to obtain token log probabilities,

VoldV^\{\\mathrm\{old\}\}, and

QoldQ^\{\\mathrm\{old\}\}
4:Traverse each trajectory backward using Equations[4](https://arxiv.org/html/2608.19842#S4.E4)–[6](https://arxiv.org/html/2608.19842#S4.E6)

5:Normalize turn\-level policy advantages using Equation[7](https://arxiv.org/html/2608.19842#S4.E7)and broadcast them to valid action tokens

6:Run one joint actor forward on the sampled sequences to obtain current policy,

VV, and

QQpredictions

7:Update

θ\\thetaby minimizing Equation[12](https://arxiv.org/html/2608.19842#S4.E12)

8:endfor

In implementation, trajectory identifiers and turn indices are retained until after the backward target computation, ensuring that batching, duplication, or shuffling cannot break temporal adjacency\. Reward is used only as a target after environment execution and is never included in the action\-value input\. Together with the causal boundary construction, this preserves the semantics of both state and action values throughout single\-stream training\.

TypeMethodALFWorldWebShopPickLookCleanHeatCoolPick2AllScoreSucc\.Closed\-Source ModelPromptingGPT\-4o75\.360\.831\.256\.721\.649\.848\.031\.823\.7PromptingGemini\-2\.5\-Pro92\.863\.362\.169\.026\.658\.760\.342\.535\.9Qwen2\.5\-1\.5B\-InstructPromptingReAct17\.420\.515\.76\.27\.72\.012\.840\.111\.3PromptingReflexion35\.322\.221\.713\.619\.43\.721\.855\.821\.9RL TrainingRLOO88\.3±3\.052\.8±8\.671\.0±5\.962\.8±8\.766\.4±5\.556\.9±4\.769\.7±2\.573\.9±5\.652\.1±6\.7RL TrainingEMPG85\.533\.578\.976\.274\.789\.173\.780\.460\.8RL TrainingGiGPOw/ std94\.4±5\.967\.5±4\.694\.8±3\.894\.4±7\.879\.8±4\.776\.4±5\.486\.7±1\.783\.1±1\.665\.0±3\.2RL TrainingGiGPOw/o std96\.0±1\.476\.5±3\.991\.8±5\.591\.3±6\.371\.7±8\.479\.5±7\.786\.1±4\.783\.5±1\.867\.4±4\.5RL TrainingPPO \(with critic\)64\.8±3\.540\.5±6\.957\.1±4\.960\.6±6\.646\.4±4\.047\.4±1\.954\.4±3\.173\.8±3\.051\.5±2\.9RL TrainingGRPO85\.3±1\.553\.7±8\.084\.5±6\.878\.2±7\.959\.7±5\.053\.5±5\.672\.8±3\.675\.8±3\.556\.8±3\.8RL TrainingSAPO92\.0±2\.976\.9±6\.3100\.0±0\.0100\.0±0\.082\.8±4\.782\.4±5\.090\.1±2\.382\.21\.463\.71\.6Qwen2\.5\-7B\-InstructPromptingReAct48\.535\.434\.313\.218\.217\.631\.246\.219\.5PromptingReflexion62\.041\.644\.930\.936\.323\.842\.758\.128\.8RL TrainingRLOO87\.6±4\.378\.2±8\.387\.3±5\.881\.3±7\.671\.9±5\.248\.9±8\.475\.5±4\.680\.3±3\.265\.7±4\.0RL TrainingEMPG92\.975\.274\.886\.373\.765\.378\.581\.069\.3RL TrainingGiGPOw/ std97\.7±1\.682\.7±7\.998\.8±1\.683\.7±7\.289\.3±8\.279\.2±6\.690\.8±1\.384\.4±2\.972\.8±3\.2RL TrainingGiGPOw/o std91\.8±5\.488\.6±6\.395\.9±3\.290\.2±2\.686\.5±5\.585\.2±7\.590\.2±2\.386\.2±2\.675\.2±3\.8RL TrainingPPO \(with critic\)92\.3±4\.064\.0±8\.492\.5±2\.489\.5±7\.080\.3±2\.068\.8±8\.380\.4±2\.781\.4±3\.168\.7±5\.1RL TrainingGRPO90\.8±5\.166\.1±6\.789\.3±5\.474\.7±6\.972\.5±5\.464\.7±7\.377\.6±5\.279\.3±2\.866\.1±3\.7RL TrainingSAPO99\.0±1\.482\.3±2\.1100\.0±0\.097\.9±4\.779\.7±3\.991\.7±1\.694\.0±1\.788\.6±1\.882\.4±2\.0

Table 1:Evaluation results on ALFWorld and WebShop\. For each RL training method, we report the mean and standard deviation over three random seeds\. The ALFWorld contains six categories: Pick & Place \(Pick\), Examine in Light \(Look\), Clean & Place \(Clean\), Heat & Place \(Heat\), Cool & Place \(Cool\), and Pick Two & Place \(Pick2\)\. Most entries in this table are reported by Feng et al\.\[[4](https://arxiv.org/html/2608.19842#bib.bib35)\]\. Notably, the baseline GiGPOw/o stdreplaces the group\-relative normalization std with one\.

## 5Experiments

We evaluate SAPO across a range of multi\-turn environments\. Specifically, our experiments aim to answer four research questions: \(1\)How does SAPO perform compared with the core PPO and GRPO baselines?\(2\)How sensitive is SAPO to its components, and how much does each component contribute?\(3\)How does SAPO compare with a comparable PPO baseline in terms of resource overhead?\(4\)Can SAPO effectively maintain stable performance over continuous training iterations over long\-horizon tasks?

### 5\.1Experimental Setup

#### Benchmarks and Baselines\.

We train and evaluate SAPO on two challenging multi\-turn ALFWorld\[[20](https://arxiv.org/html/2608.19842#bib.bib28)\]and WebShop\[[27](https://arxiv.org/html/2608.19842#bib.bib29)\]benchmarks\. ALFWorld is an embodied household environment for evaluating long\-horizon textual reasoning and decision\-making\. In each episode, the agent receives a concrete task instruction sampled from3,8273\{,\}827tasks from six categories\. WebShop is an interactive web\-shopping environment containing nearly1\.11\.1million products and12,00012\{,\}000user instructions\. We compare SAPO against three categories of competitive baselines: \(1\) Proprietary models: GPT\-4o\[[1](https://arxiv.org/html/2608.19842#bib.bib30)\]and Gemini\-2\.5\-Pro\[[22](https://arxiv.org/html/2608.19842#bib.bib31)\]; \(2\) training\-free prompting agents: ReAct\[[28](https://arxiv.org/html/2608.19842#bib.bib33)\]and Reflexion\[[19](https://arxiv.org/html/2608.19842#bib.bib32)\]; and \(3\) RL\-based methods: PPO\[[16](https://arxiv.org/html/2608.19842#bib.bib16)\], RLOO\[[2](https://arxiv.org/html/2608.19842#bib.bib13)\], GRPO\[[17](https://arxiv.org/html/2608.19842#bib.bib1)\], EMPG\[[24](https://arxiv.org/html/2608.19842#bib.bib34)\]and GiGPO\[[4](https://arxiv.org/html/2608.19842#bib.bib35)\]\.

#### Implementation details\.

We employ Qwen2\.5\-1\.5B\-Instruct and Qwen2\.5\-7B\-Instruct as the base models for training experiments\. To ensure fair comparisons, the hyperparameter settings of SAPO follow the existing RL framework\[[4](https://arxiv.org/html/2608.19842#bib.bib35)\]in each benchmark\. Specifically, we use a learning rate of1×10−61\\times 10^\{\-6\}, and a KL\-penalty coefficient of0\.010\.01\. The maximum number of interaction turns is set to5050for ALFWorld and1515for WebShop, and we train for150150steps overall\. All our experiments were run on 4 NVIDIA H200s and 8 NVIDIA A40s\. The technical supplement provides additional implementation details\.

### 5\.2Experiment Results

Figure 2:Per\-iteration runtime breakdown of PPO and SAPO on ALFWorld using Qwen2\.5\-1\.5B\. The vertical axis is shown on a logarithmic scale to accommodate the large variation in module runtimes\.N/Aindicates that SAPO does not require the corresponding value\-model or critic\-update module\. Overall, SAPO reduces the measured per\-iteration runtime from 451\.2 s to 301\.4 s, corresponding to a 33\.2% reduction compared with PPO\.As shown in Table[1](https://arxiv.org/html/2608.19842#S4.T1), across ALFWorld and WebShop with Qwen2\.5\-1\.5B/7B, SAPO consistently outperforms standard PPO and GRPO at both model scales, while matching or exceeding recent improved variants overall, with particularly clear gains at Qwen2\.5\-7B\. For example, with Qwen2\.5\-1\.5B, SAPO achieves90\.1%90\.1\\%aggregate success on ALFWorld, improving over PPO and GRPO by35\.735\.7and17\.317\.3percentage points, respectively, and reaches perfect performance on the Clean and Heat categories\. On WebShop, it improves PPO by8\.48\.4points in score and12\.212\.2points in success rate\. The highlighted advantage persists at 7B, i\.e\., SAPO obtains94\.0%94\.0\\%ALFWorld success and an88\.688\.6WebShop score with82\.4%82\.4\\%success, outperforming both PPO and GRPO as well as the recent variants on all three aggregate metrics\. Although SAPO is not uniformly best on every individual category, its aggregate performance is consistently best\.

Overall, the results indicate that SAPO’s gains are not confined to a specific model capacity or environment\. By integrating value estimation and policy optimization within a single\-rollout autoregressive process, SAPO retains effective temporal credit assignment without a separate critic or group\-based sampling\. Its strong performance using sparse outcome rewards further suggests that the architectural alignment between autoregressive modeling and actor–critic learning can reduce dependence on task\-specific dense reward engineering\.

### 5\.3Per\-iteration Runtime Comparison

Figure[2](https://arxiv.org/html/2608.19842#S5.F2)compares the per\-iteration runtime breakdown of PPO and SAPO on ALFWorld using Qwen2\.5\-1\.5B\. SAPO reduces the total measured runtime from 451\.2 s to 301\.4 s, yielding a 33\.2% reduction over PPO\. The largest absolute saving comes from trajectory generation, whose runtime decreases from 306\.4 s to 221\.4 s\. More importantly, SAPO eliminates the separate value inference and critic optimization stages, which together account for 61\.4 s per iteration in PPO\. In contrast, the remaining actor\-side costs are comparable: old\-policy log\-probability computation takes 14\.3 s versus 14\.2 s, reference\-model evaluation takes 13\.3 s versus 12\.3 s, and actor updates take 54\.2 s versus 52\.6 s for PPO and SAPO, respectively\. These results indicate that integrating value estimation into the shared autoregressive model introduces little additional actor\-side overhead\. Overall, SAPO improves runtime efficiency structurally by removing the independent critic pathway while retaining explicit value learning and temporal credit assignment\.

## 6Conclusion

We introduced SAPO, a single\-rollout autoregressive actor–critic framework for RL of LLM agents\. SAPO exploits causal boundaries within one language model to represent the policy, state value, and action value without a separate critic network or group\-relative sampling\. The unified objective jointly optimizes policy and value learning through a shared backbone while preserving their distinct conditioning and supervision\. Experiments on ALFWorld and WebShop with Qwen2\.5\-1\.5B and 7B demonstrate stable training and task performance, outperforming PPO and GRPO while using only one rollout per task\. SAPO also removes the memory cost of a policy\-scale critic and reduces measured per\-iteration runtime by 33\.2% relative to PPO\. These results show that explicit value learning need not require duplicated models or costly sampling, offering a path toward efficient long\-horizon agentic RL\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.Cited by:[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[2\]A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. Hooker\(2024\)Back to basics: revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 12248–12267\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.662),[Link](https://aclanthology.org/2024.acl-long.662/)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p3.1),[§2](https://arxiv.org/html/2608.19842#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[3\]Y\. Choi, J\. Lim, W\. Ahn, M\. Oh, J\. Shim, and Y\. Jo\(2026\)Your language model is its own critic: reinforcement learning with value estimation from actor’s internal states\.External Links:2605\.07579,[Document](https://dx.doi.org/10.48550/arXiv.2605.07579),[Link](https://arxiv.org/abs/2605.07579)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p2.1),[§1](https://arxiv.org/html/2608.19842#S1.p3.1),[§2](https://arxiv.org/html/2608.19842#S2.p2.1)\.
- \[4\]L\. Feng, Z\. Xue, T\. Liu, and B\. An\(2026\)Group\-in\-group policy optimization for llm agent training\.Vol\.38\.Cited by:[Table 1](https://arxiv.org/html/2608.19842#S4.T1),[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px2.p1.1)\.
- \[5\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645,pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),[Link](https://doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p1.1),[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[6\]Z\. Hou, Y\. Li, J\. Tang, and Y\. Dong\(2026\)Single\-rollout asynchronous optimization for agentic reinforcement learning\.External Links:2607\.07508,[Document](https://dx.doi.org/10.48550/arXiv.2607.07508),[Link](https://arxiv.org/abs/2607.07508)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p2.1),[§1](https://arxiv.org/html/2608.19842#S1.p3.1),[§2](https://arxiv.org/html/2608.19842#S2.p2.1)\.
- \[7\]J\. Hu, Y\. Zhang, Q\. Han, D\. Jiang, X\. Zhang, and H\. Shum\(2025\)Open\-Reasoner\-Zero: an open source approach to scaling up reinforcement learning on the base model\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 162239–162262\.External Links:[Document](https://dx.doi.org/10.52202/085713-5418),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/ed873d79e7c268c020c4b4db13a2812a-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p1.1),[§1](https://arxiv.org/html/2608.19842#S1.p2.1),[§1](https://arxiv.org/html/2608.19842#S1.p3.1)\.
- \[8\]B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han\(2025\)Search\-R1: training LLMs to reason and leverage search engines with reinforcement learning\.External Links:2503\.09516,[Document](https://dx.doi.org/10.48550/arXiv.2503.09516),[Link](https://arxiv.org/abs/2503.09516)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p1.1),[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[9\]A\. Kazemnejad, M\. Aghajohari, E\. Portelance, A\. Sordoni, S\. Reddy, A\. Courville, and N\. Le Roux\(2024\)VinePPO: unlocking RL potential for LLM reasoning through refined credit assignment\.External Links:2410\.01679,[Document](https://dx.doi.org/10.48550/arXiv.2410.01679),[Link](https://arxiv.org/abs/2410.01679)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p3.1),[§2](https://arxiv.org/html/2608.19842#S2.p2.1)\.
- \[10\]K\. Li, Z\. Zhang, H\. Yin, L\. Zhang, L\. Ou, J\. Wu, W\. Yin, B\. Li, Z\. Tao, X\. Wang,et al\.\(2025\)WebSailor: navigating super\-human reasoning for web agent\.Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[11\]X\. Li, H\. Zou, and P\. Liu\(2025\)ToRL: scaling tool\-integrated reinforcement learning\.Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[12\]Z\. Li, T\. Xu, Y\. Zhang, Z\. Lin, Y\. Yu, R\. Sun, and Z\. Luo\(2024\)ReMax: a simple, effective, and efficient reinforcement learning method for aligning large language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 29128–29163\.External Links:[Link](https://proceedings.mlr.press/v235/li24cd.html)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p3.1),[§2](https://arxiv.org/html/2608.19842#S2.p2.1)\.
- \[13\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.External Links:[Document](https://dx.doi.org/10.52202/068431-2011),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract.html)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p2.1)\.
- \[14\]J\. Peng, Y\. Liu, R\. Zhou, C\. Fleming, Z\. Wang, A\. Garcia, and M\. Hong\(2026\)HiPER: hierarchical plan–execute reinforcement learning for multi\-turn LLM agents\.InForty\-third International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[15\]M\. Santacroce, Y\. Lu, H\. Yu, Y\. Li, and Y\. Shen\(2023\)Efficient RLHF: reducing the memory usage of PPO\.External Links:2309\.00754,[Document](https://dx.doi.org/10.48550/arXiv.2309.00754),[Link](https://arxiv.org/abs/2309.00754)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p3.1),[§2](https://arxiv.org/html/2608.19842#S2.p2.1)\.
- \[16\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.External Links:1707\.06347,[Document](https://dx.doi.org/10.48550/arXiv.1707.06347),[Link](https://arxiv.org/abs/1707.06347)Cited by:[§3\.1](https://arxiv.org/html/2608.19842#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[17\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300),[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p1.1),[§1](https://arxiv.org/html/2608.19842#S1.p2.1),[§2](https://arxiv.org/html/2608.19842#S2.p1.1),[§2](https://arxiv.org/html/2608.19842#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.19842#S3.SS1.p1.2),[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[18\]G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu\(2025\)HybridFlow: a flexible and efficient RLHF framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p2.1)\.
- \[19\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.Vol\.36\.Cited by:[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[20\]M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht\(2020\)Alfworld: aligning text and embodied environments for interactive learning\.Cited by:[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[21\]N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano\(2020\)Learning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 3008–3021\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p2.1)\.
- \[22\]G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.Cited by:[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[23\]G\. Wang, S\. Dai, G\. Ye, Z\. Gan, W\. Yao, Y\. Deng, X\. Wu, and Z\. Ying\(2026\)Information gain\-based policy optimization: a simple and effective approach for multi\-turn search agents\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[24\]J\. Wang, J\. Liu, Y\. Fu, Y\. Li, X\. Wang, Y\. Lin, Y\. Yue, L\. Zhang, Y\. Wang, and W\. KE\(2026\)Harnessing uncertainty: entropy\-modulated policy gradients for long\-horizon LLM agents\.InForty\-third International Conference on Machine Learning,Cited by:[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[25\]Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu,et al\.\(2025\)RAGEN: understanding self\-evolution in LLM agents via multi\-turn reinforcement learning\.Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[26\]Z\. Xi, J\. Huang, C\. Liao, B\. Huang, J\. Liu, H\. Guo, Y\. Yang, R\. Zheng, J\. Ye, J\. Zhang,et al\.\(2026\)AgentGym\-RL: an open\-source framework to train LLM agents for long\-horizon decision making via multi\-turn reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[27\]S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan\(2022\)Webshop: towards scalable real\-world web interaction with grounded language agents\.Vol\.35\.Cited by:[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[28\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2022\)React: synergizing reasoning and acting in language models\.Cited by:[§5\.1](https://arxiv.org/html/2608.19842#S5.SS1.SSS0.Px1.p1.1)\.
- \[29\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.External Links:2503\.14476,[Document](https://dx.doi.org/10.48550/arXiv.2503.14476),[Link](https://arxiv.org/abs/2503.14476)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p2.1),[§1](https://arxiv.org/html/2608.19842#S1.p3.1),[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[30\]Y\. Yuan, Y\. Yue, R\. Zhu, T\. Fan, and L\. Yan\(2025\)What’s behind PPO’s collapse in long\-CoT? value optimization holds the secret\.External Links:2503\.01491,[Document](https://dx.doi.org/10.48550/arXiv.2503.01491),[Link](https://arxiv.org/abs/2503.01491)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p2.1),[§1](https://arxiv.org/html/2608.19842#S1.p3.1)\.
- \[31\]Y\. Yue, Y\. Yuan, Q\. Yu, X\. Zuo, R\. Zhu, W\. Xu, J\. Chen, C\. Wang, T\. Fan, Z\. Du, X\. Wei, X\. Yu, G\. Liu, J\. Liu, L\. Liu, H\. Lin, Z\. Lin, B\. Ma, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, R\. Zhang, X\. Liu, M\. Wang, Y\. Wu, and L\. Yan\(2025\)VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks\.External Links:2504\.05118,[Document](https://dx.doi.org/10.48550/arXiv.2504.05118),[Link](https://arxiv.org/abs/2504.05118)Cited by:[§1](https://arxiv.org/html/2608.19842#S1.p2.1),[§1](https://arxiv.org/html/2608.19842#S1.p3.1)\.
- \[32\]K\. Zhang, Y\. Hong, J\. Bao, H\. Jiang, Y\. Song, H\. Dingqian, and H\. Xiong\(2025\)GVPO: group variance policy optimization for large language model post\-training\.External Links:2504\.19599,[Document](https://dx.doi.org/10.48550/arXiv.2504.19599),[Link](https://arxiv.org/abs/2504.19599)Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[33\]C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.\(2025\)Group sequence policy optimization\.Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[34\]Y\. Zheng, D\. Fu, X\. Hu, X\. Cai, L\. Ye, P\. Lu, and P\. Liu\(2025\)DeepResearcher: scaling deep research via reinforcement learning in real\-world environments\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 414–431\.Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.
- \[35\]Y\. Zhou, A\. Zanette, J\. Pan, S\. Levine, and A\. Kumar\(2024\)ARChER: training language model agents via hierarchical multi\-turn reinforcement learning\.Cited by:[§2](https://arxiv.org/html/2608.19842#S2.p1.1)\.