PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs

# PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
Source: [https://arxiv.org/html/2608.17289](https://arxiv.org/html/2608.17289)
###### Abstract

Group\-relative policy optimization has emerged as a key paradigm for training agentic large language models \(LLMs\) on multi\-turn interactive tasks\. However, most existing variants fail to distinguish advantages among successful trajectories even when these trajectories differ substantially in their interaction efficiency\. For instance, circuitous successes are often assigned the identical outcome reward, causing advantage collapse and severe performance bottlenecks\. To this end, we proposeGroup Planning\-aware Policy Optimization\(PlanPO\), a simple yet effective RL method for learning generalizable planning abilities beyond task\-specific high\-quality behavior patterns\. Specifically, PlanPO introduces coarse\-to\-fine advantage signals, which capture the relative differences intrajectory\-levellengths andturn\-levelresponse lengthsconditioned on successful trajectoriessampled for the same task\. Within the group\-relative optimization structure, this enables agents to actively learn generalizable and deliberate behaviors spanning interaction planning and textual generation from high\-quality rollouts, without degenerating into vanilla length minimization\. Experimentally, PlanPO improves over GRPO by27\.2%on average across the challenging multi\-turn benchmarks ALFWorld, WebShop, and SciWorld, outperforming recent powerful baselines while incurring negligible additional training cost\.

## Introduction

Large Language Models \(LLMs\) have demonstrated remarkable progress across a wide range of complex multi\-turn tasks, including information retrieval\([17](https://arxiv.org/html/2608.17289#bib.bib9);[8](https://arxiv.org/html/2608.17289#bib.bib11)\), web navigation\([25](https://arxiv.org/html/2608.17289#bib.bib14)\)\([45](https://arxiv.org/html/2608.17289#bib.bib15)\), code generation\([48](https://arxiv.org/html/2608.17289#bib.bib8);[7](https://arxiv.org/html/2608.17289#bib.bib5)\), and embodied interaction\([14](https://arxiv.org/html/2608.17289#bib.bib13);[49](https://arxiv.org/html/2608.17289#bib.bib12)\)\. Recent work has increasingly explored agentic reinforcement learning \(RL\) with verifiable outcome rewards\([52](https://arxiv.org/html/2608.17289#bib.bib6);[7](https://arxiv.org/html/2608.17289#bib.bib5);[15](https://arxiv.org/html/2608.17289#bib.bib10)\), particularly Group Relative Policy Optimization \(GRPO\)\([27](https://arxiv.org/html/2608.17289#bib.bib43)\), to fine\-tune open\-source LLMs such as Qwen2\.5\([39](https://arxiv.org/html/2608.17289#bib.bib2)\), thereby improving the capabilities of LLM agents in multi\-turn tasks\.

However, a central challenge in on\-policy agentic RL lies inenriching the reward signals of rollout trajectories while improving data utilization\([44](https://arxiv.org/html/2608.17289#bib.bib7);[50](https://arxiv.org/html/2608.17289#bib.bib1);[34](https://arxiv.org/html/2608.17289#bib.bib3);[24](https://arxiv.org/html/2608.17289#bib.bib4)\)\. Many early studies addressed this issue by introducing value models, such as critics\([26](https://arxiv.org/html/2608.17289#bib.bib22);[6](https://arxiv.org/html/2608.17289#bib.bib19)\)and process reward models \(PRMs\)\([3](https://arxiv.org/html/2608.17289#bib.bib18)\), to evaluate turn\-level behaviors\. Yet these models may introduce estimation or proxy bias, and incur substantial memory costs\([3](https://arxiv.org/html/2608.17289#bib.bib18)\)\. Recent group\-optimization approaches instead mine informative rollouts to craft turn\-level reward signals and achieve discriminative advantages\. For example, HiPER\([24](https://arxiv.org/html/2608.17289#bib.bib4)\)computes the multiple returns of trajectory segments via sub\-task decomposition, while GiGPO\([9](https://arxiv.org/html/2608.17289#bib.bib20)\)constructs step advantage signals over the action space by identifying repeated anchor states\. Other methods, such as R3L\([29](https://arxiv.org/html/2608.17289#bib.bib23)\)and GVPO\([47](https://arxiv.org/html/2608.17289#bib.bib47)\), establish turn\-level credit from failure reflection and diverse code execution feedback\. However, most methods rely on labor\-intensive manual design and empirical heuristics, which limits their generality across task settings\. As a result, advantage collapse within successful rollout groups remains difficult to mitigate in a broadly applicable way\([51](https://arxiv.org/html/2608.17289#bib.bib24)\)\.

This motivates the question of how to enrich rollout\-driven training signals for long\-horizon tasks in a simple, effective, and more task\-general manner\.We begin by revisiting a naturally available yet underutilized signal in group rollouts, i\.e\., the length profiles across both turn\-level interaction trajectories and token\-level generated responses\. Figure[1](https://arxiv.org/html/2608.17289#Sx1.F1)provides an abstract illustration of this intuition\. Specifically, our key observation is that many inefficiencies in agentic RL manifest as excessive interaction or generation length\. In multi\-turn interactions, agents may hesitate between states, repeatedly visit similar observations, or enter dead ends before eventually completing the task\. A similar issue arises in token\-level textual responses\. Given the same question or turn\-level observation, sampled responses may produce correct actions while still containing unnecessarily verbose, convoluted, or even logically flawed reasoning traces\. Nevertheless, both inefficient turns and reasoning tokens can still share identical success rewards just like the superior solutions\. Crucially, treating such heterogeneous successes as equally preferable weakens distinguishable signals and the underlying abilities, while allowing noisy rollouts to degrade training quality and impose substantial performance bottlenecks\.

To address this, we presentGroup Planning\-aware Policy Optimization \(PlanPO\), an effective group\-based RL method for learning generalizable planning abilities beyond specific planful behaviors\. Specifically, within the set of successful trajectories sampled in same task, PlanPO normalizes outcome rewards by turn\-level trajectory lengths and token\-level response lengths, constructing coarse\-to\-fine dense reward signals\. Rather than directly summing these dense rewards, PlanPO then computes their relative advantages separately and combines them through a weighted formulation, preserving discriminative information at different granularities\([20](https://arxiv.org/html/2608.17289#bib.bib25)\)\. Additionally, unlike generic length\-based reward shaping\([23](https://arxiv.org/html/2608.17289#bib.bib26);[21](https://arxiv.org/html/2608.17289#bib.bib27)\), PlanPO performs multiscale length normalization only conditioned on successes with group\-relative structure\. Thus, successful completion is a prerequisite for group length normalization\. Empirically, we show that these success\-conditioned relative advantages help agents acquire planning behaviors that generalize beyond specific high\-quality trajectories, rather than merely imitating task\-specific success patterns\.

![Refer to caption](https://arxiv.org/html/2608.17289v1/figure/intro.png)Figure 1:Left:Successful rollouts are not equally informative\. Rollouts that reach the same task goal, can differ substantially in length, directness, and reasoning quality, while the most optimal strategies may reveal more sophisticated and generalizable capabilities\.Right:Averaged normalization performance comparison across ALFWorld, WebShop, and SciWorld environments using the Qwen2\.5\-1\.5b model\.We evaluate PlanPO on three challenging multi\-turn benchmarks, ALFWorld\([31](https://arxiv.org/html/2608.17289#bib.bib44)\), WebShop\([40](https://arxiv.org/html/2608.17289#bib.bib28)\), and SciWorld\([36](https://arxiv.org/html/2608.17289#bib.bib45)\), using Qwen2\.5\-1\.5B\-Instruct and Qwen2\.5\-7B\-Instruct\. PlanPO consistently outperforms recent strong baselines while incurring negligible additional training cost, specifically improving over GRPO by27\.2%on average and notably achieving a24\.3%gain on out\-of\-distribution tasks in ALFWorld\. Extensive ablations and analyses further validate its effectiveness and generalizability\. Our contributions are threefold\.

- •We identify successful\-rollout heterogeneity as a critical bottleneck in group\-relative optimization, where redundant turns and reasoning can mask rollout quality and then limit policy learning\.
- •We propose PlanPO, which constructs success\-conditioned length\-normalized advantages, encouraging agents to learning planning\-aware abilities beyond specific behavioral patterns\.
- •We show that PlanPO consistently outperforms recent strong baselines across ALFWorld, WebShop, and SciWorld, while incurring negligible additional training cost\.

## Preliminaries

#### Multi\-Turn Agentic RL\.

We consider RL for LLM\-based agents that interact with an environment over multiple turns, where the interaction process is formulated as a finite\-horizon Markov Decision Process \(MDP\)\. Given a task instance𝒙∈p⁡\(X\)\\boldsymbol\{x\}\\in p\(X\), at each turnt=1,2,…,Tt=1,2,\\ldots,T, an agent policyπθ\\pi\_\{\\theta\}observes a state𝒔t∈𝒮\\boldsymbol\{s\}\_\{t\}\\in\\mathcal\{S\}and generates a textual action𝒂t∈𝒜\\boldsymbol\{a\}\_\{t\}\\in\\mathcal\{A\}, and then transitions to the next state𝒔t\+1∈𝒮\\boldsymbol\{s\}\_\{t\+1\}\\in\\mathcal\{S\}while yielding a scalar rewardrt∈ℝr\_\{t\}\\in\\mathbb\{R\}\. The interaction unfolds as a trajectory𝝉=\{\(𝒔1,𝒂1,r1\),\(𝒔2,𝒂2,r2\),…,\(𝒔T,𝒂T,rT\)\}\\boldsymbol\{\\tau\}=\\\{\(\\boldsymbol\{s\}\_\{1\},\\boldsymbol\{a\}\_\{1\},r\_\{1\}\),\(\\boldsymbol\{s\}\_\{2\},\\boldsymbol\{a\}\_\{2\},r\_\{2\}\),\.\.\.,\(\\boldsymbol\{s\}\_\{T\},\\boldsymbol\{a\}\_\{T\},r\_\{T\}\)\\\}, whereTTdenotes the trajectory length in interaction turns\. In practical tasks such as ALFWorld, the agent generates a response as its action for each observation\. This response is structured within<think\></think\>and<action\></action\>tags, where the former contains the reasoning process, while the latter specifies the actual action executed in the environment\. Notably, for most task settings, reward signals are sparse and delayed, e\.g\., the environment provides an outcome rewardR⁡\(𝝉\)R\(\\boldsymbol\{\\tau\}\)only after the trajectory terminates\.

### Group\-relative Policy Optimization

Recent agentic RL methods for LLMs commonly adopt a group\-relative policy optimization paradigm\. Given a task instance𝒙\\boldsymbol\{x\}, the old policyπθold\\pi\_\{\\theta\_\{\\rm old\}\}samples a group ofNNcandidate trajectories𝒢x=\{𝝉1,𝝉2,…,𝝉N\}\\mathcal\{G\}\_\{x\}=\\\{\\boldsymbol\{\\tau\}\_\{1\},\\boldsymbol\{\\tau\}\_\{2\},\\ldots,\\boldsymbol\{\\tau\}\_\{N\}\\\}, where each trajectory corresponds to one complete rollout\. Each trajectory𝝉i\\boldsymbol\{\\tau\}\_\{i\}receives a scalar rewardR⁡\(𝝉i\)R\(\\boldsymbol\{\\tau\}\_\{i\}\)that reflects the overall quality or success of the generated outcome\. Instead of learning an advantage functionA⁡\(𝒔t,𝒂t\)A\(\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{a\}\_\{t\}\)with critic networks in PPO\([26](https://arxiv.org/html/2608.17289#bib.bib22)\), group\-based RL computes the advantage using only statistics within the sampled group:

A⁡\(𝝉i\)=𝙶𝚛𝚘𝚞𝚙𝙽𝚘𝚛𝚖𝚊𝚕𝚒𝚣𝚊𝚝𝚒𝚘𝚗⁡\(\{R⁡\(𝝉i\)\}i=1N\)\.A\(\\boldsymbol\{\\tau\}\_\{i\}\)=\\mathtt\{GroupNormalization\}\\left\(\\\{R\(\\boldsymbol\{\\tau\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}\\right\)\.In GRPO\([27](https://arxiv.org/html/2608.17289#bib.bib43)\), the advantage is evaluated by normalizing each trajectory reward with the mean and variance of group rewards\(\{R⁡\(𝝉i\)\}i=1N\)\(\\\{R\(\\boldsymbol\{\\tau\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}\)\. This sampling\-based estimator reduces the memory and computational overhead introduced by the critic architecture in conventional PPO\.

![Refer to caption](https://arxiv.org/html/2608.17289v1/figure/fw.png)Figure 2:Overview of PlanPO\.Left: Rollouts sampled for the same task can reach the same outcome through interaction paths and textual responses of markedly different quality\.Middle: PlanPO converts the outcome into success\-conditioned trajectory\- and turn\-level scores using trajectory and response lengths, respectively\.Right: The two scores are normalized separately within the same\-task group and combined into a coarse\-to\-fine advantage for policy optimization\.

## Group Planning\-Aware Policy Optimization for Multi\-turn Agentic LLMs

We proposegroup\-relative Planning\-aware Policy Optimization \(PlanPO\), a simple and effective group\-relative RL method for learning high\-level planning strategies beyond task\-specific high\-quality rollouts\. We begin with our motivation, followed by the introduction of the coarse\-to\-fine advantage design, and conclude with the policy optimization loss and theoretical analysis\.

### Motivation

As illustrated in the right of Figure[2](https://arxiv.org/html/2608.17289#Sx2.F2), given the same task, some rollouts reach the goal through short and coherent interaction paths, whereas others involve redundant state transitions or even enter loops and dead ends\. At the turn level, verbose reasoning can also introduce inconsistencies or misleading intermediate claims\. For example, as illustrated, given a task or turn observation, although the final sampled actions "<action\>take potato 1 from countertop 1</action\>" are all correct, the thought process "I have already taken the potato \.\.\." treats it as already happened\. Such cases are common in the rollouts yet introduce hallucinated or logically inconsistent reasoning traces\.

How should we quantify the quality of successful rollouts?Existing solutions may introduce post\-hoc reflection\([29](https://arxiv.org/html/2608.17289#bib.bib23)\)or agentic verifier models\([46](https://arxiv.org/html/2608.17289#bib.bib29)\), but these signals are often costly and heuristic\. The aforementioned observations motivate us to leverage multiscale rollout length as an initial signal\. However, unconditional length\-based reward shaping would degenerate into meaningless length minimization, even distorting the representation space of LLMs\. Our goal is instead to comparatively learn planful behaviors within successful rollouts, while promoting task\-general planning capabilities rather than fitting task\-specific patterns\. This objective naturally aligns with group\-relative policy optimization\. Below, we progressively construct the above conditional advantage signals\.

### Trajectory Length\-Normalized Advantage

To achieve this, we first instantiate the success\-conditioned length signal at the trajectory \(or episode\) level\. For successful rollouts with the same outcome reward, trajectory length serves as a coarse proxy for planning efficiency across environment interactions\. We therefore normalize the outcome reward by trajectory length only for successful rollouts, and compute the conditional group\-relative advantage\.

Formally, given a task instance𝒙\\boldsymbol\{x\}, the policyπθold\\pi\_\{\\theta\_\{\\rm old\}\}samplesNNtrajectories\{𝝉1,𝝉2,…,𝝉N\}\\\{\\boldsymbol\{\\tau\}\_\{1\},\\boldsymbol\{\\tau\}\_\{2\},\\ldots,\\boldsymbol\{\\tau\}\_\{N\}\\\}start from the identical and initial state𝒔0\\boldsymbol\{s\}\_\{0\}\. Each rollout trajectory is represented as𝝉i=\{\(𝒔i,1,𝒂i,1,ri,1\),\(𝒔i,2,𝒂i,2,ri,2\),…,\(𝒔i,Ti,𝒂i,Ti,ri,Ti\)\}\\boldsymbol\{\\tau\}\_\{i\}=\\\{\(\\boldsymbol\{s\}\_\{i,1\},\\boldsymbol\{a\}\_\{i,1\},r\_\{i,1\}\),\(\\boldsymbol\{s\}\_\{i,2\},\\boldsymbol\{a\}\_\{i,2\},r\_\{i,2\}\),\.\.\.,\(\\boldsymbol\{s\}\_\{i,T\_\{i\}\},\\boldsymbol\{a\}\_\{i,T\_\{i\}\},r\_\{i,T\_\{i\}\}\)\\\}, whereTiT\_\{i\}represents the number of turns inii\-th trajectory\. In our task setting, each trajectory receives only a terminal outcome rewardR⁡\(𝝉i\)=10R\(\\boldsymbol\{\\tau\}\_\{i\}\)=10when the task goal is reached\. We then denote the trajectory\-level group of sampled trajectories and rewards as:

𝒢𝒙E=\{\(𝝉1,R⁡\(𝝉1\)\),\(𝝉2,R⁡\(𝝉2\)\),…,\(𝝉N,R⁡\(𝝉N\)\)\},\\mathcal\{G\}\_\{\\boldsymbol\{x\}\}^\{\\mathrm\{E\}\}=\\left\\\{\\bigl\(\\boldsymbol\{\\tau\}\_\{1\},R\(\\boldsymbol\{\\tau\}\_\{1\}\)\\bigr\),\\bigl\(\\boldsymbol\{\\tau\}\_\{2\},R\(\\boldsymbol\{\\tau\}\_\{2\}\)\\bigr\),\\ldots,\\bigl\(\\boldsymbol\{\\tau\}\_\{N\},R\(\\boldsymbol\{\\tau\}\_\{N\}\)\\bigr\)\\right\\\},\(1\)where the superscriptE\\mathrm\{E\}denotes the episode, i\.e\., trajectory level\. Let𝒢𝒙U⊆𝒢𝒙E\\mathcal\{G\}\_\{\\boldsymbol\{x\}\}^\{\\mathrm\{U\}\}\\subseteq\\mathcal\{G\}\_\{\\boldsymbol\{x\}\}^\{\\mathrm\{E\}\}denote the subset of successful trajectories, and let𝟙\[𝝉i∈𝒢𝒙U\]\\mathds\{1\}\[\\boldsymbol\{\\tau\}\_\{i\}\\in\\mathcal\{G\}\_\{\\boldsymbol\{x\}\}^\{\\mathrm\{U\}\}\]indicate whether𝝉i\\boldsymbol\{\\tau\}\_\{i\}succeeds\. For each trajectory, we define its length\-normalized episode reward as,

RE\(𝝉i\)=𝟙\[𝝉i∈𝒢𝒙U\]R\(𝝉i\)/Ti,R^\{\\mathrm\{E\}\}\(\\boldsymbol\{\\tau\}\_\{i\}\)=\\mathds\{1\}\[\\boldsymbol\{\\tau\}\_\{i\}\\in\\mathcal\{G\}\_\{\\boldsymbol\{x\}\}^\{\\mathrm\{U\}\}\]\\,R\(\\boldsymbol\{\\tau\}\_\{i\}\)/T\_\{i\},\(2\)whereTiT\_\{i\}is the number of valid interaction turns in𝝉i\\boldsymbol\{\\tau\}\_\{i\}\. The corresponding trajectory\-level advantageAEA^\{\\mathrm\{E\}\}is computed by group\-relative normalization:

AE​\(𝝉i\)=RE​\(𝝉i\)−mean⁡\(\{RE​\(𝝉j\)\}j=1N\)Fnorm​\(\{RE​\(𝝉j\)\}j=1N\)\.A^\{\\mathrm\{E\}\}\(\\boldsymbol\{\\tau\}\_\{i\}\)=\\frac\{R^\{\\mathrm\{E\}\}\(\\boldsymbol\{\\tau\}\_\{i\}\)\-\\operatorname\{mean\}\\left\(\\left\\\{R^\{\\mathrm\{E\}\}\(\\boldsymbol\{\\tau\}\_\{j\}\)\\right\\\}\_\{j=1\}^\{N\}\\right\)\}\{F\_\{\\mathrm\{norm\}\}\\left\(\\left\\\{R^\{\\mathrm\{E\}\}\(\\boldsymbol\{\\tau\}\_\{j\}\)\\right\\\}\_\{j=1\}^\{N\}\\right\)\}\.\(3\)Here,Fnorm​\(⋅\)F\_\{\\mathrm\{norm\}\}\(\\cdot\)is the normalization factor, instantiated as eitherstd⁡\(⋅\)\+ϵ\\operatorname\{std\}\(\\cdot\)\+\\epsilonor11\([9](https://arxiv.org/html/2608.17289#bib.bib20)\), with the former usually used by default, whereϵ\\epsilonis a small constant for numerical stability\([27](https://arxiv.org/html/2608.17289#bib.bib43)\)\. The trajectory\-levelAE​\(𝝉i\)A^\{\\mathrm\{E\}\}\(\\boldsymbol\{\\tau\}\_\{i\}\)is broadcast to each turn in theii\-th trajectory, providing a trajectory\-wide discrimination signal among successful rollout trajectories that complete the same task but differ in process quality or long\-term planning\.

### Response Length\-Normalized Advantage

While the above advantage provides learning signals for discriminating trajectories, each response turn within a trajectory still lacks fine\-grained credit\. Next, we compute group\-relative advantages only among successful responses, so that the agent learns that planful responses are desirable only when the corresponding actions remain correct\. Additionally, we flatten all successful responses generated for the same task into one group for advantage computation\.

We then employ a similar group relative advantage structure to achieve the above idea\. Formally, for thett\-th interaction turn in trajectory𝝉i\\boldsymbol\{\\tau\}\_\{i\}, we collect all active responses into a turn\-level group,

𝒢𝒙,tS=\{\(𝒂i,t,R\(𝝉i\)\)∣𝝉i∈𝒢𝒙E,t≤Ti\},\\mathcal\{G\}\_\{\\boldsymbol\{x\},t\}^\{\\mathrm\{S\}\}=\\left\\\{\\bigl\(\\boldsymbol\{a\}\_\{i,t\},R\(\\boldsymbol\{\\tau\}\_\{i\}\)\\bigr\)\\mid\\boldsymbol\{\\tau\}\_\{i\}\\in\\mathcal\{G\}\_\{\\boldsymbol\{x\}\}^\{\\mathrm\{E\}\},\\ t\\leq T\_\{i\}\\right\\\},\(4\)whereS\\mathrm\{S\}denotes the step, i\.e\., turn level, andLi,t=\|𝒂i,t\|L\_\{i,t\}=\|\\boldsymbol\{a\}\_\{i,t\}\|is the token length of the response generated by trajectory𝝉i\\boldsymbol\{\\tau\}\_\{i\}at turntt\. Similar to the trajectory\-level case, we define the response length\-normalized reward as,

RS\(𝒂i,t\)=𝟙\[𝝉i∈𝒢𝒙U\]R\(𝝉i\)/Li,t\.R^\{\\mathrm\{S\}\}\(\\boldsymbol\{a\}\_\{i,t\}\)=\\mathds\{1\}\[\\boldsymbol\{\\tau\}\_\{i\}\\in\\mathcal\{G\}\_\{\\boldsymbol\{x\}\}^\{\\mathrm\{U\}\}\]R\(\\boldsymbol\{\\tau\}\_\{i\}\)/L\_\{i,t\}\.\(5\)The turn\-level relative advantage is then computed within the active response group:

AS​\(𝒂i,t\)=RS​\(𝒂i,t\)−mean⁡\(\{RS​\(𝒂j,t\)∣𝒂j,t∈𝒢𝒙,tS\}\)Fnorm​\(\{RS​\(𝒂j,t\)∣𝒂j,t∈𝒢𝒙,tS\}\)\.A^\{\\mathrm\{S\}\}\(\\boldsymbol\{a\}\_\{i,t\}\)=\\frac\{R^\{\\mathrm\{S\}\}\(\\boldsymbol\{a\}\_\{i,t\}\)\-\\operatorname\{mean\}\\left\(\\left\\\{R^\{\\mathrm\{S\}\}\(\\boldsymbol\{a\}\_\{j,t\}\)\\mid\\boldsymbol\{a\}\_\{j,t\}\\in\\mathcal\{G\}\_\{\\boldsymbol\{x\},t\}^\{S\}\\right\\\}\\right\)\}\{F\_\{\\mathrm\{norm\}\}\\left\(\\left\\\{R^\{\\mathrm\{S\}\}\(\\boldsymbol\{a\}\_\{j,t\}\)\\mid\\boldsymbol\{a\}\_\{j,t\}\\in\\mathcal\{G\}\_\{\\boldsymbol\{x\},t\}^\{S\}\\right\\\}\\right\)\}\.\(6\)Here, the resulting turn advantageAS​\(𝒂i,t\)A^\{\\mathrm\{S\}\}\(\\boldsymbol\{a\}\_\{i,t\}\)is assigned to all tokens in the response𝒂i,t\\boldsymbol\{a\}\_\{i,t\}, which provides a fine\-grained supervision signal for successful responses\.

### Coarse\-to\-Fine Group Policy Optimization

We integrate the trajectory\-level and turn\-level advantages into a multiscale group\-relative advantage for policy optimization\. For the response𝒂i,t\\boldsymbol\{a\}\_\{i,t\}in trajectory𝝉i\\boldsymbol\{\\tau\}\_\{i\}, the final PlanPO advantage is defined as,

APlanPO​\(𝒂i,t\)=AE​\(𝝉i\)\+α⁡\(k\)​AS​\(𝒂i,t\),A^\{\\mathrm\{PlanPO\}\{\}\}\(\\boldsymbol\{a\}\_\{i,t\}\)=A^\{\\mathrm\{E\}\}\(\\boldsymbol\{\\tau\}\_\{i\}\)\+\\alpha\(k\)A^\{\\mathrm\{S\}\}\(\\boldsymbol\{a\}\_\{i,t\}\),\(7\)with​α​\(k\)=𝙻𝚒𝚗𝚎𝚊𝚛𝙳𝚎𝚌𝚊𝚢⁡\(k,αinit,αfinal\),\\mathrm\{with\}~~\\alpha\(k\)=\\mathtt\{LinearDecay\}\(k;\\alpha\_\{\\rm init\},\\alpha\_\{\\rm final\}\),\(8\)whereα⁡\(k\)\\alpha\(k\)denotes the decay weight with training stepkkfor balancing the two level signals\. We set the turn\-level coefficient smaller than the trajectory\-level coefficient, i\.e\.,0≤αfinal≤α⁡\(k\)<αinit≤10\\leq\\alpha\_\{\\rm final\}\\leq\\alpha\(k\)<\\alpha\_\{\\rm init\}\\leq 1\. This design keeps trajectory\-level planning quality as the dominant signal and uses response length signals only as a refinement\. Although the response constraint is applied only to successful trajectories, an overly largeα⁡\(k\)\\alpha\(k\)experimentally over\-penalize response length, thereby hurting task performance\. Please see the ablation study for details\. Therefore, we gradually decayα⁡\(k\)\\alpha\(k\)during training to reduce the strength of response\-length normalization as the policy becomes more capable\.

Finally, PlanPO optimizes the policy with the same clipped group\-relative objective backbone as GRPO\-style methods:

𝒥PlanPO​\(θ\)\\displaystyle\\mathcal\{J\}^\{\\mathrm\{PlanPO\}\{\}\}\(\\theta\)=𝔼𝒙∼p⁡\(X\),\{𝝉i\}i=1N∼πθold\[1∑i=1NTi∑i=1N∑t=1Tiℓi,t\\displaystyle=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}\\boldsymbol\{x\}\\sim p\(X\),\\\\ \\\{\\boldsymbol\{\\tau\}\_\{i\}\\\}\_\{i=1\}^\{N\}\\sim\\pi\_\{\\theta\_\{\\rm old\}\}\\end\{subarray\}\}\\Bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{N\}T\_\{i\}\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\ell\_\{i,t\}\(9\)−β𝔻KL\(πθ\(⋅∣𝒙\)∥πref\(⋅∣𝒙\)\)\],\\displaystyle\-\\beta\\mathbb\{D\}\_\{\\mathrm\{KL\}\}\\bigl\(\\pi\_\{\\theta\}\(\\cdot\\mid\\boldsymbol\{x\}\)\\\|\\pi\_\{\\rm ref\}\(\\cdot\\mid\\boldsymbol\{x\}\)\\bigr\)\\Bigg\],ℓi,t=min⁡\(CLOSE\\displaystyle\\ell\_\{i,t\}=\\min\\Big\(ρθ​\(𝒂i,t\)​APlanPO​\(𝒂i,t\),\\displaystyle\\rho\_\{\\theta\}\(\\boldsymbol\{a\}\_\{i,t\}\)A^\{\\mathrm\{PlanPO\}\{\}\}\(\\boldsymbol\{a\}\_\{i,t\}\),\(10\)OPENclip⁡\(ρθ​\(𝒂i,t\),1−ϵ,1\+ϵ\)​APlanPO​\(𝒂i,t\)\)\.\\displaystyle\\operatorname\{clip\}\(\\rho\_\{\\theta\}\(\\boldsymbol\{a\}\_\{i,t\}\),1\-\\epsilon,1\+\\epsilon\)A^\{\\mathrm\{PlanPO\}\{\}\}\(\\boldsymbol\{a\}\_\{i,t\}\)\\Big\)\.
Here,ρθ​\(𝒂i,t\)=πθ​\(𝒂i,t∣𝒔i,t,𝒙\)/πθold​\(𝒂i,t∣𝒔i,t,𝒙\)\\rho\_\{\\theta\}\(\\boldsymbol\{a\}\_\{i,t\}\)=\\pi\_\{\\theta\}\(\\boldsymbol\{a\}\_\{i,t\}\\mid\\boldsymbol\{s\}\_\{i,t\},\\boldsymbol\{x\}\)/\\pi\_\{\\theta\_\{\\rm old\}\}\(\\boldsymbol\{a\}\_\{i,t\}\\mid\\boldsymbol\{s\}\_\{i,t\},\\boldsymbol\{x\}\)is the importance sampling ratio,ϵ\\epsilonis the clipping coefficient, andβ\\betacontrols the strength of KL regularization with respect to the reference policyπref\\pi\_\{\\rm ref\}\.

TypeMethodALFWorldWebShopPickLookCleanHeatCoolPick2AllScoreSucc\.Closed\-Source ModelPromptingGPT\-4o75\.360\.831\.256\.721\.649\.848\.031\.823\.7PromptingGemini\-2\.5\-Pro92\.863\.362\.169\.026\.658\.760\.342\.535\.9Qwen2\.5\-1\.5B\-InstructPromptingReAct17\.420\.515\.76\.27\.72\.012\.840\.111\.3PromptingReflexion35\.322\.221\.713\.619\.43\.721\.855\.821\.9RL TrainingPPO \(with critic\)64\.8±3\.540\.5±6\.957\.1±4\.960\.6±6\.646\.4±4\.047\.4±1\.954\.4±3\.173\.8±3\.051\.5±2\.9RL TrainingRLOO88\.3±3\.052\.8±8\.671\.0±5\.962\.8±8\.766\.4±5\.556\.9±4\.769\.7±2\.573\.9±5\.652\.1±6\.7RL TrainingGRPO85\.3±1\.553\.7±8\.084\.5±6\.878\.2±7\.959\.7±5\.053\.5±5\.672\.8±3\.675\.8±3\.556\.8±3\.8RL TrainingEMPG85\.533\.578\.976\.274\.789\.173\.780\.460\.8RL TrainingGiGPOw/ std94\.4±5\.967\.5±4\.694\.8±3\.894\.4±7\.879\.8±4\.776\.4±5\.486\.7±1\.783\.1±1\.665\.0±3\.2RL TrainingGiGPOw/o std96\.0±1\.476\.5±3\.991\.8±5\.591\.3±6\.371\.7±8\.479\.5±7\.786\.1±4\.783\.5±1\.867\.4±4\.5RL TrainingPlanPO98\.2±1\.185\.1±4\.694\.6±4\.793\.8±6\.082\.4±6\.783\.7±8\.091\.3±4\.186\.8±1\.577\.2±5\.6Qwen2\.5\-7B\-InstructPromptingReAct48\.535\.434\.313\.218\.217\.631\.246\.219\.5PromptingReflexion62\.041\.644\.930\.936\.323\.842\.758\.128\.8RL TrainingPPO \(with critic\)92\.3±4\.064\.0±8\.492\.5±2\.489\.5±7\.080\.3±2\.068\.8±8\.380\.4±2\.781\.4±3\.168\.7±5\.1RL TrainingRLOO87\.6±4\.378\.2±8\.387\.3±5\.881\.3±7\.671\.9±5\.248\.9±8\.475\.5±4\.680\.3±3\.265\.7±4\.0RL TrainingGRPO90\.8±5\.166\.1±6\.789\.3±5\.474\.7±6\.972\.5±5\.464\.7±7\.377\.6±5\.279\.3±2\.866\.1±3\.7RL TrainingEMPG92\.975\.274\.886\.373\.765\.378\.581\.069\.3RL TrainingGiGPOw/ std97\.7±1\.682\.7±7\.998\.8±1\.683\.7±7\.289\.3±8\.279\.2±6\.690\.8±1\.384\.4±2\.972\.8±3\.2RL TrainingGiGPOw/o std91\.8±5\.488\.6±6\.395\.9±3\.290\.2±2\.686\.5±5\.585\.2±7\.590\.2±2\.386\.2±2\.675\.2±3\.8RL TrainingPlanPO100\.0±0\.092\.1±6\.396\.8±2\.599\.0±1\.488\.3±6\.889\.7±7\.894\.4±2\.188\.5±3\.280\.6±4\.7

Table 1:Evaluation results on ALFWorld and WebShop\. For each RL training method, we report the mean and variance over three random seeds\. The ALFWorld contains six categories: Pick & Place \(Pick\), Examine in Light \(Look\), Clean & Place \(Clean\), Heat & Place \(Heat\), Cool & Place \(Cool\), and Pick Two & Place \(Pick2\)\. Most entries in this table are reported by Feng et al\.\([9](https://arxiv.org/html/2608.17289#bib.bib20)\)\. Notably, the baseline GiGPOw/ stddenotesFnorm=stdF\_\{\\text\{norm\}\}=\\text\{std\}, and GiGPOw/o stddenotesFnorm=1F\_\{\\text\{norm\}\}=1\.Table 2:Evaluation results on SciWorld over three random seeds\. The task Test\-Cond\. means test conductivity and Chem\-Mix means chemistry mix\. Baseline entries are reported by Zhang et al\.\([46](https://arxiv.org/html/2608.17289#bib.bib29)\)\.![Refer to caption](https://arxiv.org/html/2608.17289v1/figure/abla.png)Figure 3:Left two panels:Ablation study on the weight scheduleα⁡\(k\)\\alpha\(k\)withαinit=2​αfinal\\alpha\_\{\\mathrm\{init\}\}=2\\alpha\_\{\\mathrm\{final\}\}in PlanPO\.Third panel:Component ablation\. We compared the ablation results ofPlanPO with default decay,trajectory\-level advantageAEA^\{E\}only,turn\-level advantageASA^\{S\}only, andPlanPO with constantα=0\.1\\alpha=0\.1\.Right panel:Runtime breakdown of each PlanPO iteration on four NVIDIA A40 GPUs\. Compared to the GRPO, the additional computation introduced by PlanPO lies in the reward and advantage stages, which is negligible\. Conversely, PlanPO can reduce overall task runtime by 12\.5%###### Theorem 1\(Bias–variance trade\-off in PlanPO\)\.

Consider a fixed responseai,ta\_\{i,t\}at optimization stepkk, with the randomness induced by repeated same\-task group sampling\. LetAE,⋆=𝔼𝒢​\[AE\]A^\{E,\\star\}=\\mathbb\{E\}\_\{\\mathcal\{G\}\}\[A^\{E\}\]andAS,⋆=𝔼𝒢​\[AS\]A^\{S,\\star\}=\\mathbb\{E\}\_\{\\mathcal\{G\}\}\[A^\{S\}\]denote the expected trajectory\-level signal and turn\-level refinement, respectively\. Assume that their centered estimation errors have variancesvEv\_\{E\}andvSv\_\{S\}and non\-negative covariancecc\. Relative to the full\-refinement targetA⋆=AE,⋆\+AS,⋆A^\{\\star\}=A^\{E,\\star\}\+A^\{S,\\star\}, the PlanPO estimatorAα=AE\+α​ASA\_\{\\alpha\}=A^\{E\}\+\\alpha A^\{S\}, whereα∈\[0,1\]\\alpha\\in\[0,1\], satisfies

Bias2⁡\(Aα\)\\displaystyle\\operatorname\{Bias\}^\{2\}\(A\_\{\\alpha\}\)=\(1−α\)2​\(AS,⋆\)2,\\displaystyle=\(1\-\\alpha\)^\{2\}\(A^\{S,\\star\}\)^\{2\},Var⁡\(Aα\)\\displaystyle\\operatorname\{Var\}\(A\_\{\\alpha\}\)=vE\+α2​vS\+2​α​c\.\\displaystyle=v\_\{E\}\+\\alpha^\{2\}v\_\{S\}\+2\\alpha c\.Consequently, increasingα\\alphadecreases the squared bias while increasing the variance, with:

0\\displaystyle 0≤Bias2⁡\(Aα\)≤\(AS,⋆\)2,\\displaystyle\\leq\\operatorname\{Bias\}^\{2\}\(A\_\{\\alpha\}\)\\leq\(A^\{S,\\star\}\)^\{2\},vE\\displaystyle v\_\{E\}≤Var⁡\(Aα\)≤\(vE\+vS\)2\.\\displaystyle\\leq\\operatorname\{Var\}\(A\_\{\\alpha\}\)\\leq\\left\(\\sqrt\{v\_\{E\}\}\+\\sqrt\{v\_\{S\}\}\\right\)^\{2\}\.

This theorem shows thatα⁡\(k\)\\alpha\(k\)relates to PlanPO’s bias–variance trade\-off\. A largerα⁡\(k\)\\alpha\(k\)preserves more turn\-level refinement but raises sampling variance, whereas a smallerα⁡\(k\)\\alpha\(k\)reduces variance at the cost of greater shrinkage bias\. The technical supplement provides detailed proof\.

## Experiments

We evaluate PlanPO across a range of multi\-turn environments\. Specifically, the experiments aim to answer three questions: \(1\)How well does PlanPO perform overall across these challenging environments?\(2\)How does PlanPO achieve its performance gains?\(3\)Does PlanPO acquire the capabilities of task generalization and planning awareness rather than overfitting to task\-specific success patterns?

### Experimental Setup

#### Benchmarks\.

We train and evaluate PlanPO on three challenging multi\-turn benchmarks: ALFWorld\([31](https://arxiv.org/html/2608.17289#bib.bib44)\), WebShop\([40](https://arxiv.org/html/2608.17289#bib.bib28)\), and SciWorld\([36](https://arxiv.org/html/2608.17289#bib.bib45)\)\. ALFWorld is an embodied household environment for evaluating long\-horizon textual reasoning and decision\-making\. In each episode, the agent receives a concrete task instruction sampled from3,8273\{,\}827tasks from six categories\. WebShop is an interactive web\-shopping environment containing nearly1\.11\.1million products and12,00012\{,\}000user instructions\. SciWorld is designed for scientific tasks and provides APIs through which agents can manipulate scientific instruments and conduct experiments\. The technical supplement provides further descriptions of the benchmarks\.

#### Baselines\.

For ALFWorld and WebShop, we compare PlanPO against three categories of competitive baselines: \(1\) Proprietary models: GPT\-4o\([1](https://arxiv.org/html/2608.17289#bib.bib31)\)and Gemini\-2\.5\-Pro\([32](https://arxiv.org/html/2608.17289#bib.bib32)\); \(2\) training\-free prompting agents: ReAct\([41](https://arxiv.org/html/2608.17289#bib.bib16)\)and Reflexion\([30](https://arxiv.org/html/2608.17289#bib.bib33)\); and \(3\) RL\-based methods: PPO\([26](https://arxiv.org/html/2608.17289#bib.bib22)\), RLOO\([2](https://arxiv.org/html/2608.17289#bib.bib34)\), GRPO\([27](https://arxiv.org/html/2608.17289#bib.bib43)\), EMPG\([35](https://arxiv.org/html/2608.17289#bib.bib21)\), and GiGPO\([9](https://arxiv.org/html/2608.17289#bib.bib20)\), which cover representative actor–critic and group\-relative optimization approaches\. For SciWorld, we consider \(1\) training\-free agents based on the OpenAI\([1](https://arxiv.org/html/2608.17289#bib.bib31)\), Gemini\([32](https://arxiv.org/html/2608.17289#bib.bib32)\), Qwen\([38](https://arxiv.org/html/2608.17289#bib.bib35)\), Llama\([10](https://arxiv.org/html/2608.17289#bib.bib38)\), and DeepSeek\([19](https://arxiv.org/html/2608.17289#bib.bib36);[11](https://arxiv.org/html/2608.17289#bib.bib37)\)model families, spanning both proprietary models and open\-source models of different scales; and \(2\) RL\-based methods: AgentGym\-RL\([37](https://arxiv.org/html/2608.17289#bib.bib46)\), a GRPO framework for training across multiple environments, and ScalingInter\([37](https://arxiv.org/html/2608.17289#bib.bib46)\), a group\-based RL method that reports strong performance on SciWorld\.

#### Implementation details\.

We employ Qwen2\.5\-1\.5B\-Instruct and Qwen2\.5\-7B\-Instruct as the base models for training experiments\. To ensure fair comparisons, the hyperparameter settings of PlanPO follow the existing RL framework\([9](https://arxiv.org/html/2608.17289#bib.bib20);[37](https://arxiv.org/html/2608.17289#bib.bib46)\)in each benchmark\. Specifically, we use a group size ofN=8N=8, a learning rate of1×10−61\\times 10^\{\-6\}, and a KL\-penalty coefficient of1×10−31\\times 10^\{\-3\}for SciWorld and0\.010\.01for the other environments\. The maximum number of interaction turns is set to5050for ALFWorld,1515for WebShop, and2020for SciWorld\. We train for150150steps on ALFWorld and WebShop and for200200steps on SciWorld\. On ALFWorld and WebShop, we set the turn\-level weightαinit=2​αfinal=0\.1\\alpha\_\{\\mathrm\{init\}\}=2\\alpha\_\{\\mathrm\{final\}\}=0\.1in PlanPO\. For the exploratory SciWorld, we reduce it to0\.050\.05without further tuning to improve training stability\. Finally, all our experiments were run on 6 NVIDIA H200s and 8 NVIDIA A40s\. The technical supplement provides additional implementation details\.

### Experiment Results

#### Performance on ALFWorld and WebShop\.

As shown in Table[1](https://arxiv.org/html/2608.17289#Sx3.T1), PlanPO substantially outperforms advanced closed\-source models on both benchmarks\. For the open\-source Qwen2\.5\-1\.5B\-Instruct and Qwen2\.5\-7B\-Instruct models, RL methods are significantly stronger than prompting\-based methods\. Nevertheless, PlanPO also outperforms existing powerful baselines, e\.g\., GiGPO and EMPG\. For example, on the 1\.5B model, PlanPO achieves an overall success rate of91\.3%91\.3\\%across the six ALFWorld categories, improving over GRPO by\+18\.5\+18\.5points, while requiring only several regularization operations with negligible computational cost\.

#### Performance on SciWorld\.

In Table[2](https://arxiv.org/html/2608.17289#Sx3.T2), PlanPO achieves consistently the best overall scores on complex and exploratory scientific scenarios, notably improving the GRPO\-based AgentGym\-RL from50\.550\.5to68\.4668\.46\. However, we observe that RL training methods exhibit a consistent failure pattern in the Chem\-Mix domain, which may be due to the model’s limited understanding and exploration ability required for rigorous scientific analysis\([37](https://arxiv.org/html/2608.17289#bib.bib46)\)\.

### In\-depth Analysis

#### Ablation Study\.

To verify how PlanPO achieves performance gains, we performed two ablation studies on the scheduling coefficientα⁡\(k\)\\alpha\(k\)and each component\. As shown in left Figure[3](https://arxiv.org/html/2608.17289#Sx3.F3), we compared a series of initialαinit=\{0\.5,0\.2,0\.1,0\.05,0\}\\alpha\_\{\\mathrm\{init\}\}=\\\{0\.5,0\.2,0\.1,0\.05,0\\\}\. We observed thatαinit=0\.1\\alpha\_\{\\mathrm\{init\}\}=0\.1achieved the best performance, while both smaller values \(e\.g\.,αinit=0\\alpha\_\{\\mathrm\{init\}\}=0\) and excessively large values \(e\.g\.,0\.50\.5\) degraded performance\. This matches the theoretical bias–variance trade\-off, i\.e\., the turn\-level signal is useful as a refinement, but should not dominate the trajectory\-level planning signal\. In the third panel of Figure[3](https://arxiv.org/html/2608.17289#Sx3.F3), we observe that ablating any level signal significantly degrades PlanPO’s performance, especially withoutAEA^\{E\}\. Regarding turn\-level advantages, PlanPO with coefficient decay settings consistently outperforms those using constant coefficients\. In summary, its performance gains primarily stem from theAEA^\{E\}, while a properly weightedASA^\{S\}further enables optimal performance\.

Table 3:Generalization evaluation with three seeds in ALFWorld using Qwen2\.5\-1\.5B\-Instruct\. Some entries are reported by He et al\.\([12](https://arxiv.org/html/2608.17289#bib.bib30)\)\. The In\-Success reports the success rate on the in\-distribution tasks, while the Out\-Success reports the success rate on out\-of\-distribution tasks\.
#### Generalization Verification\.

The gains of PlanPO remain strong when the evaluation tasks differ from the training distribution\. As shown in Table[3](https://arxiv.org/html/2608.17289#Sx4.T3), PlanPO achieves87\.1%87\.1\\%success on out\-of\-distribution ALFWorld tasks, improving over GRPO by\+17\.0\+17\.0points and over GiGPO by\+4\.7\+4\.7points\. The gap between in\-distribution and out\-of\-distribution performance is also modest, decreasing from91\.3%91\.3\\%to87\.1%87\.1\\%\. This suggests that group\-based PlanPO does not merely memorize some better planning templates, but learns planning\-aware abilities that generalize to unseen task configurations\.

![Refer to caption](https://arxiv.org/html/2608.17289v1/figure/len_norm_ana.png)Figure 4:Length\-normalized Comparison Analysis with different reward shaping settings in ALFWorld\.Figure 5:Mean Length Comparison in ALFWorld tasks\.
#### Length\-Normalized Method Comparison\.

Figure[4](https://arxiv.org/html/2608.17289#Sx4.F4)further isolates the role of success\-conditioned length normalization\. Directly applying unconditional length shaping in PPO quickly collapses to nearly zero success, showing that shorter generations or trajectories are not intrinsically better\. Success conditioning alone is also insufficient, as PPO with success\-conditioned length shaping remains close to standard PPO\. By contrast, PlanPO steadily improves after training begins and reaches around0\.550\.55success near the end, clearly above GRPO and PPO variants\. These results indicate that the key benefit comes from coupling length\-induced signals with group\-relative policy optimization, which turns heterogeneous successful rollouts into useful supervision\. Note that the reward shaping operation and outcome reward sizeR⁡\(τ\)=10R\(\\tau\)=10involved are consistent with PlanPO settings, except for the condition control and basic algorithm\.

#### Planning\-Aware Strategies\.

We further examine whether PlanPO induces planning\-aware behavior beyond improving final task success\. As shown in Figure[5](https://arxiv.org/html/2608.17289#Sx4.F5), PlanPO consistently produces shorter trajectories than GRPO during training, reducing the mean number of interaction turns to13\.813\.8compared with26\.126\.1for GRPO\. This suggests that PlanPO learns to reach task goals through more direct interaction paths, avoiding redundant transitions, repeated trials, and unnecessary detours\. At the response level, PlanPO also yields more compact generations, with the mean response length decreasing to56\.356\.3tokens, while GRPO remains substantially more verbose at95\.195\.1tokens\. Importantly, this behavior does not arise from naive length minimization: as shown in Figure[4](https://arxiv.org/html/2608.17289#Sx4.F4), unconditional length shaping hurts task success, whereas PlanPO applies length normalization only under success\-conditioned group\-relative comparisons\. These results indicate that PlanPO encourages planning\-aware strategies by preferring successful rollouts that are both interaction\-efficient and response\-concise, thereby turning heterogeneity among successful trajectories into a useful signal for learning generalizable agentic planning abilities\.

## Related Work

RL has become a central recipe for improving LLM reasoning and interactive agents\([28](https://arxiv.org/html/2608.17289#bib.bib17);[27](https://arxiv.org/html/2608.17289#bib.bib43);[43](https://arxiv.org/html/2608.17289#bib.bib49);[33](https://arxiv.org/html/2608.17289#bib.bib54)\)\. PPO\-based RLHF\([26](https://arxiv.org/html/2608.17289#bib.bib22)\)trains a value model to reduce variance, while lighter critic\-free alternatives such as ReMax\([16](https://arxiv.org/html/2608.17289#bib.bib48)\), RLOO\([2](https://arxiv.org/html/2608.17289#bib.bib34)\), and GRPO\([27](https://arxiv.org/html/2608.17289#bib.bib43)\)replace the learned critic with sampled baselines or group\-relative rewards, improving scalability for LLMs\. Recent GRPO variants further refine this objective\([21](https://arxiv.org/html/2608.17289#bib.bib27);[5](https://arxiv.org/html/2608.17289#bib.bib39);[54](https://arxiv.org/html/2608.17289#bib.bib40);[18](https://arxiv.org/html/2608.17289#bib.bib41)\)\. DAPO improves stability with decoupled clipping, dynamic sampling, and overlong\-response shaping\([42](https://arxiv.org/html/2608.17289#bib.bib50)\), Dr\. GRPO analyzes length and reward\-variance normalization biases\([22](https://arxiv.org/html/2608.17289#bib.bib51)\), and GMPO stabilizes updates by changing the aggregation of token\-level rewards\([53](https://arxiv.org/html/2608.17289#bib.bib52)\)\. In parallel, agentic RL methods extend outcome\-reward optimization to long\-horizon interaction, including WebSailor\([13](https://arxiv.org/html/2608.17289#bib.bib42)\), GiGPO\([9](https://arxiv.org/html/2608.17289#bib.bib20)\), HGPO\([12](https://arxiv.org/html/2608.17289#bib.bib30)\), and A2TGPO\([4](https://arxiv.org/html/2608.17289#bib.bib53)\), which study credit assignment through episode\-, state\-, history\-, or turn\-grouped advantages\. Different from these methods, our work focuses on planning\-aware credit assignment under successful\-rollout heterogeneity\.

## Conclusion

This paper addresses successful\-rollout heterogeneity in group\-relative RL for multi\-turn agentic LLMs\. We propose PlanPO, which constructs success\-conditioned length\-induced advantages from trajectory\-level interaction and token\-level response generation\. The experiments across multiple benchmarks demonstrate that PlanPO consistently outperforms existing methods and achieves the best task performance with stronger generalization, planful Strategies, and negligible overhead\. Our results suggest that successful\-rollout heterogeneity provides a simple and scalable signal for learning planning\-aware and generalizable behaviors\.

## References

- Achiamet al\.\(2023\)J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Ahmadianet al\.\(2024\)A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. HookerBack to basics: revisiting reinforce\-style optimization for learning from human feedback in llms\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12248–12267\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1),[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Chaeet al\.\(2026\)H\. Chae, S\. Kim, J\. Cho, S\. Kim, S\. Moon, G\. Hwangbo, D\. Lim, M\. Kim, Y\. Hwang, M\. Gwak,et al\.Web\-shepherd: advancing prms for reinforcing web agents\.Vol\.38\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Chenet al\.\(2026\)D\. Chen, Z\. Zong, Z\. Ma, L\. Luo, Y\. Li, C\. Li, P\. Chen, and J\. JiangA2TGPO: agentic turn\-group policy optimization with adaptive turn\-level clipping\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Cuiet al\.\(2025\)G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen,et al\.The entropy mechanism of reinforcement learning for reasoning language models\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Daiet al\.\(2025\)R\. Dai, L\. Song, H\. Liu, Z\. Liang, D\. Yu, H\. Mi, Z\. Tu, R\. Liu, T\. Zheng, H\. Zhu,et al\.Cde: curiosity\-driven exploration for efficient reinforcement learning in large language models\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Daiet al\.\(2026\)S\. Dai, C\. Sun, H\. Wu, H\. Zheng, T\. Ji, J\. Yan, Y\. Wu, D\. Zhang, X\. Wang, and X\. LiGroup verification\-based policy optimization for interactive coding agents\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Eugeneet al\.\(2026\)J\. Y\. Eugene, X\. Zhang, Y\. Xia, T\. Ge, X\. Wang, F\. Kartik, V\. Suryanarayanan, C\. Yang, A\. Jiang, J\. Ding,et al\.FormAct: agentic source editing for rich\-format document generation\.InForty\-third International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Fenget al\.\(2026\)L\. Feng, Z\. Xue, T\. Liu, and B\. AnGroup\-in\-group policy optimization for llm agent training\.Vol\.38\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1),[Trajectory Length\-Normalized Advantage](https://arxiv.org/html/2608.17289#Sx3.SSx2.p2.4),[Table 1](https://arxiv.org/html/2608.17289#Sx3.T1),[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1),[Implementation details\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px3.p1.1),[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Heet al\.\(2026\)S\. He, L\. Feng, Q\. Wei, X\. Cheng, L\. Feng, and B\. AnHierarchy\-of\-groups policy optimization for long\-horizon agentic tasks\.Cited by:[Table 3](https://arxiv.org/html/2608.17289#Sx4.T3),[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Liet al\.\(2025\)K\. Li, Z\. Zhang, H\. Yin, L\. Zhang, L\. Ou, J\. Wu, W\. Yin, B\. Li, Z\. Tao, X\. Wang,et al\.Websailor: navigating super\-human reasoning for web agent\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Liet al\.\(2024\)M\. Li, S\. Zhao, Q\. Wang, K\. Wang, Y\. Zhou, S\. Srivastava, C\. Gokmen, T\. Lee, L\. E\. Li, R\. Zhang,et al\.Embodied agent interface: benchmarking llms for embodied decision making\.Vol\.37\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Liet al\.\(2026\)Y\. Li, C\. Zhang, R\. Lv, A\. Liu, K\. Deng, Y\. Zhang, J\. Liu, and B\. ZhouRelook: vision\-grounded rl with a multimodal llm critic for agentic web coding\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 25471–25485\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Liet al\.\(2023\)Z\. Li, T\. Xu, Y\. Zhang, Z\. Lin, Y\. Yu, R\. Sun, and Z\. LuoRemax: a simple, effective, and efficient reinforcement learning method for aligning large language models\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Liaoet al\.\(2025\)C\. C\. Liao, D\. Liao, and S\. S\. GadirajuAgentmaster: a multi\-agent conversational framework using a2a and mcp protocols for multimodal information retrieval and analysis\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 52–72\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Linet al\.\(2026\)Z\. Lin, M\. Lin, Y\. Xie, and R\. JiCppo: accelerating the training of group relative policy optimization\-based reasoning models\.Advances in Neural Information Processing Systems38,pp\. 61043–61068\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)S\. Liu, X\. Dong, X\. Lu, S\. Diao, P\. Belcak, M\. Liu, M\. Chen, H\. Yin, Y\. F\. Wang, K\. Cheng,et al\.Gdpo: group reward\-decoupled normalization policy optimization for multi\-reward rl optimization\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p4.1)\.
- Liuet al\.\(2025a\)W\. Liu, R\. Zhou, Y\. Deng, Y\. Huang, J\. Liu, Y\. Deng, Y\. Zhang, and J\. HeLearn to reason efficiently with adaptive length\-based reward shaping\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p4.1),[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Liuet al\.\(2025b\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Pardoet al\.\(2018\)F\. Pardo, A\. Tavakoli, V\. Levdik, and P\. KormushevTime limits in reinforcement learning\.InInternational Conference on Machine Learning,pp\. 4045–4054\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p4.1)\.
- Penget al\.\(2026\)J\. Peng, Y\. Liu, R\. Zhou, C\. Fleming, Z\. Wang, A\. Garcia, and M\. HongHiPER: hierarchical plan–execute rl for multi\-turn llm agents\.InForty\-third International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Rawleset al\.\(2025\)C\. Rawles, S\. Clinckemaillie, Y\. Chang, J\. Waltz, G\. Lau, M\. Fair, A\. Li, W\. Bishop, W\. Li, F\. Campbell\-Ajala,et al\.Androidworld: a dynamic benchmarking environment for autonomous agents\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 406–441\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1),[Group\-relative Policy Optimization](https://arxiv.org/html/2608.17289#Sx2.SSx1.p1.1),[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1),[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1),[Group\-relative Policy Optimization](https://arxiv.org/html/2608.17289#Sx2.SSx1.p1.2),[Trajectory Length\-Normalized Advantage](https://arxiv.org/html/2608.17289#Sx3.SSx2.p2.4),[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1),[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Shiet al\.\(2026\)W\. Shi, Y\. Chen, Z\. Li, X\. Pan, Y\. Sun, J\. Xu, X\. Zhou, and Y\. LiR3L: reflect\-then\-retry reinforcement learning with language\-guided exploration, pivotal credit, and positive amplification\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1),[Motivation](https://arxiv.org/html/2608.17289#Sx3.SSx1.p2.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Vol\.36\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Shridharet al\.\(2020\)M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. HausknechtAlfworld: aligning text and embodied environments for interactive learning\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p5.1),[Benchmarks\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Teamet al\.\(2023\)G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.Gemini: a family of highly capable multimodal models\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Teamet al\.\(2026\)K\. Team, T\. Bai, Y\. Bai, Y\. Bao, M\. C\., J\. Cai, X\. Cai, P\. Cao, Y\. Cao, Z\. Chai, Y\. Charles, H\. S\. Che, G\. Chen, G\. Chen, G\. Chen,et al\.Kimi k3: open frontier intelligence\.External Links:2607\.24653Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Wanget al\.\(2026a\)G\. Wang, S\. Dai, G\. Ye, Z\. Gan, W\. Yao, Y\. Deng, X\. Wu, and Z\. YingInformation gain\-based policy optimization: a simple and effective approach for multi\-turn search agents\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Wanget al\.\(2026b\)J\. Wang, J\. Liu, Y\. Fu, Y\. Li, X\. Wang, Y\. Lin, Y\. Yue, L\. Zhang, Y\. Wang, and W\. KEHarnessing uncertainty: entropy\-modulated policy gradients for long\-horizon LLM agents\.InForty\-third International Conference on Machine Learning,Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Wanget al\.\(2022\)R\. Wang, P\. Jansen, M\. Côté, and P\. AmmanabroluScienceworld: is your agent smarter than a 5th grader?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 11279–11298\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p5.1),[Benchmarks\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Xiet al\.\(2026\)Z\. Xi, J\. Huang, C\. Liao, B\. Huang, J\. Liu, H\. Guo, yajie yang, R\. Zheng, J\. Ye, J\. Zhang, W\. Chen, W\. He, Y\. Ding, G\. Li, Z\. Chen, Z\. Du, X\. Yao, Y\. Xu, J\. Chen, T\. Gui, Z\. Wu, Q\. Zhang, X\. Huang, and Y\. JiangAgentGym\-rl: an open\-source framework to train LLM agents for long\-horizon decision making via multi\-turn RL\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1),[Implementation details\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px3.p1.1),[Performance on SciWorld\.](https://arxiv.org/html/2608.17289#Sx4.SSx2.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Yaoet al\.\(2022a\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebshop: towards scalable real\-world web interaction with grounded language agents\.Vol\.35\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p5.1),[Benchmarks\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2022b\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReact: synergizing reasoning and acting in language models\.Cited by:[Baselines\.](https://arxiv.org/html/2608.17289#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Yuet al\.\(2026\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.Vol\.38\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Zenget al\.\(2026\)A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie,et al\.Glm\-5: from vibe coding to agentic engineering\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Zhanget al\.\(2025a\)G\. Zhang, H\. Geng, X\. Yu, Z\. Yin, Z\. Zhang, Z\. Tan, H\. Zhou, Z\. Li, X\. Xue, Y\. Li,et al\.The landscape of agentic reinforcement learning for llms: a survey\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Zhanget al\.\(2026a\)J\. Zhang, K\. Chen, Z\. Lu, E\. Zhou, Q\. Yu, and J\. ZhangPrune4web: dom tree pruning programming for web agent\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 34710–34718\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Zhanget al\.\(2026b\)J\. Zhang, Z\. Fu, Z\. Xi, W\. Jing, M\. Chai, W\. He, G\. Zhang, C\. Fan, C\. An, W\. Chen,et al\.AgentV\-rl: scaling reward modeling with agentic verifier\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 23078–23100\.Cited by:[Motivation](https://arxiv.org/html/2608.17289#Sx3.SSx1.p2.1),[Table 2](https://arxiv.org/html/2608.17289#Sx3.T2)\.
- Zhanget al\.\(2026c\)K\. Zhang, Y\. Hong, J\. Bao, H\. Jiang, Y\. Song, H\. Dingqian, and H\. XiongGvpo: group variance policy optimization for large language model post\-training\.Vol\.38\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Zhanget al\.\(2024\)K\. Zhang, J\. Li, G\. Li, X\. Shi, and Z\. JinCodeagent: enhancing code generation with tool\-integrated agent systems for real\-world repo\-level coding challenges\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13643–13658\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Zhanget al\.\(2026d\)W\. Zhang, M\. Wang, G\. Liu, H\. Xu, Y\. Jiang, Y\. Shen, G\. Hou, Z\. Zheng, H\. Zhang, X\. Li,et al\.Embodied\-reasoner: synergizing visual search, reasoning, and action for embodied interactive tasks\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 41178–41207\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Zhanget al\.\(2025b\)X\. Zhang, R\. Li, Z\. Zhou, L\. Li, Y\. Qin, K\. Li, X\. Sun, X\. Tan, C\. Qu, and Y\. QiCount counts: motivating exploration in llm reasoning with count\-based intrinsic rewards\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Zhaoet al\.\(2026\)H\. Zhao, S\. Zhou, Y\. Zhang, S\. S\. Yau, W\. Zhang, L\. Tian, T\. Zhu, Y\. Huang, Y\. Zeng, J\. Gu,et al\.AEM: adaptive entropy modulation for multi\-turn agentic reinforcement learning\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p2.1)\.
- Zhaoet al\.\(2025a\)X\. Zhao, Z\. Kang, A\. Feng, S\. Levine, and D\. SongLearning to reason without external rewards\.Cited by:[Introduction](https://arxiv.org/html/2608.17289#Sx1.p1.1)\.
- Zhaoet al\.\(2025b\)Y\. Zhao, Y\. Liu, J\. Liu, J\. Chen, X\. Wu, Y\. Hao, T\. Lv, S\. Huang, L\. Cui, Q\. Ye,et al\.Geometric\-mean policy optimization\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.Group sequence policy optimization\.Cited by:[Related Work](https://arxiv.org/html/2608.17289#Sx5.p1.1)\.