Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

# Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
Source: [https://arxiv.org/html/2607.04242](https://arxiv.org/html/2607.04242)
###### Abstract

Group\-based reinforcement learning \(RL\) has become an effective paradigm for improving large language model agents on long\-horizon interactive tasks\. To obtain finer\-grained policy updates than trajectory\-level optimization, recent work has moved toward step\-level group\-based RL, where intermediate steps are grouped and compared within a rollout batch\. However, step\-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer\-comparison signal\. In this paper, we proposeProGPO\(Progress\- and Reliability\-Oriented Group Policy Optimization\), a learned\-critic\-free method for context\-consistent step\-level learning\. ProGPO keeps exact\-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout\-based state potentials\. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse\-variance fusion across history depths\. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2\.5\-1\.5B\-Instruct\. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2\.5\-3B\-Instruct experiments further test the scalability of the proposed method\.

## 1Introduction

Large language models \(LLMs\) are increasingly deployed as agents that perceive, reason, and act in external environments\(Brownet al\.,[2020](https://arxiv.org/html/2607.04242#bib.bib3); Yaoet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib1); Liuet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib2); Wanget al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib10)\)\. Representative applications include web navigation\(Yaoet al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib5); Denget al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib7); Zhouet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib8); Zhenget al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib6)\), embodied household tasks\(Shridharet al\.,[2021](https://arxiv.org/html/2607.04242#bib.bib4); Ahnet al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib11); Driesset al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib12)\), and tool\-augmented reasoning\(Schicket al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib33); Qinet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib34); Jinet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib32)\)\. Unlike single\-turn generation, these tasks require long\-horizon planning, recovery from earlier mistakes, and credit assignment under sparse delayed rewards\.

Reinforcement learning \(RL\) has therefore become a key post\-training paradigm for improving model behavior, from human\-preference optimization\(Christianoet al\.,[2017](https://arxiv.org/html/2607.04242#bib.bib17); Ouyanget al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib16); Baiet al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib18); Rafailovet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib19)\)to reasoning\-oriented RL\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib20)\)\. In particular, group\-based methods estimate advantages from multiple sampled responses or trajectories for the same task, avoiding a learned critic while retaining scalable policy optimization\. GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib22)\)and variants such as DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib23)\)compare trajectory\-level outcomes within a group, while RLOO\(Koolet al\.,[2019](https://arxiv.org/html/2607.04242#bib.bib21)\)uses leave\-one\-out baselines\. However, these methods are primarily designed for single\-turn tasks such as mathematical reasoning and code generation, where each response receives a complete outcome and step\-level credit is less ambiguous\.

In multi\-turn agentic tasks, direct trajectory\-wise optimization becomes inefficient\. Approaches such as RAGEN\(Wanget al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib31)\)and Search\-R1\(Jinet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib32)\)concatenate the full interaction history into a single sequence, causing the effective context length to grow rapidly with the number of turns\. To avoid this context explosion, recent step\-level methods such as GiGPO\(Fenget al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib35)\)group repeated intermediate states and compute within\-group advantages, enabling finer\-grained updates without per\-step extra rollouts\.

Nevertheless, step\-level grouping introduces a new comparability requirement\. If steps are grouped only by the current observation, then actions taken under different histories, goals, or available affordances may be compared in the same group, producing biased action advantages\. A natural remedy is*context consistency*: compare only steps that share the same historical prefix\. Under this stricter grouping, however, the batch becomes fragmented\. Many steps become*singletons*or low\-contrast groups, leaving no usable peer\-comparison signal\. Figure[2](https://arxiv.org/html/2607.04242#S1.F2)illustrates this phenomenon in ProGPO training traces: 33\.7% of WebShop steps and 44\.5% of ALFWorld steps have zero prefix\-consistent peer\-comparison advantage before transition credit is applied, while small value groups are common throughout training\.

![Refer to caption](https://arxiv.org/html/2607.04242v1/figures/progpo_motivation.png)Figure 1:Motivation for ProGPO\. Left: rollout trajectories\. Middle: step\-level grouping by the current state can mix different histories\. Right: context\-consistent grouping creates smaller peer groups\.To address this sparse\-signal regime, we proposeProGPO\(Progress\- and Reliability\-Oriented Group Policy Optimization\), a step\-level learning framework that adds progress\-sensitive credit to context\-consistent training without training a parametric critic or collecting additional rollouts\. ProGPO separates action comparison from progress estimation: prefix\-consistent groups provide action\-relative advantages, and rollout\-derived state potentials define transition creditV​\(st\+1\)−V​\(st\)V\(s\_\{t\+1\}\)\-V\(s\_\{t\}\)\. Thus, action comparison remains exact, while state progression can still be learned from batch\-level rollout statistics\.

Estimating these state potentials is itself a reliability problem\. Shallow groups contain more samples but blur history\-dependent value; deeper prefixes are more specific but noisier\. ProGPO therefore treats value lookup as a multi\-resolution estimation problem and fuses eligible depths with inverse\-variance weights based on sample size and return variance\. Semantic expansion further stabilizes potential estimation, while action comparison remains exact\. This design keeps the method free of a trained value network while supplying progress credit for steps that peer comparison cannot score\.

We evaluate ProGPO on WebShop and ALFWorld using Qwen2\.5\-1\.5B\-Instruct\. ProGPO improves the best matched ALFWorld baseline from 87\.8% to 90\.1% success and increases WebShop success from 67\.6% to 71\.5%\. Additional Qwen2\.5\-3B\-Instruct ALFWorld runs show the same late\-stage advantage over GiGPO and HGPO in validation success and test score\.

The main contribution of this work is a credit assignment framework that keeps the fairness of history\-consistent comparison while recovering denser learning signals from the same rollouts\. Rather than relaxing the grouping rule and risking unfair action comparisons, ProGPO keeps action comparison prefix\-consistent and adds a separate transition\-credit signal from rollout\-derived state potentials\. We further show how to estimate these potentials without training a parametric critic: semantic expansion supplies nearby evidence when exact groups are small, and inverse\-variance multi\-resolution fusion lets better\-supported history depths carry more weight\.

![Refer to caption](https://arxiv.org/html/2607.04242v1/x1.png)Figure 2:Sparse\-signal diagnostic\. \(a\) WebShop exact state groups\. \(b\) Training\-trace shares for small value groups, zero peer signal, and residual zero\-signal steps\.
## 2Related Work

Reinforcement learning for LLMs\.RL has become a central ingredient in aligning and improving LLMs beyond supervised fine\-tuning\. PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.04242#bib.bib24)\)with a learned value function was standard in early RLHF pipelines\(Christianoet al\.,[2017](https://arxiv.org/html/2607.04242#bib.bib17); Ouyanget al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib16); Baiet al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib18)\), while DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib19)\)showed that some preference\-learning objectives can avoid explicit online RL\. Recent reasoning models again rely heavily on RL\-style post\-training\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib20)\)\. GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib22)\)removes the critic by comparing multiple sampled responses for the same prompt and computing group\-relative advantages\. Later refinements include DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib23)\)and RLOO\(Koolet al\.,[2019](https://arxiv.org/html/2607.04242#bib.bib21)\)\. These methods inherit the policy\-gradient lineage of REINFORCE, actor\-critic learning, and advantage\-weighted updates\(Williams,[1992](https://arxiv.org/html/2607.04242#bib.bib25); Suttonet al\.,[1999](https://arxiv.org/html/2607.04242#bib.bib26); Konda and Tsitsiklis,[1999](https://arxiv.org/html/2607.04242#bib.bib27); Penget al\.,[2019](https://arxiv.org/html/2607.04242#bib.bib28)\), but are particularly effective on single\-turn tasks such as mathematical reasoning and code generation, where one response constitutes the entire interaction\. Step\-level supervision has also been studied through learned process verifiers\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.04242#bib.bib29); Lightmanet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib30)\); ProGPO instead derives step signals from rollout groups without human process labels or an additional verifier\.

Multi\-turn LLM agents\.Deploying LLMs as interactive agents in sequential environments\(Yaoet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib1); Shinnet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib37)\)raises challenges absent from single\-turn settings, including long\-horizon credit assignment, context management across turns, and non\-stationary interaction dynamics\. Early prompt\-based approaches such as ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib1)\)and Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib37)\)rely on in\-context learning without parameter updates, but their performance often saturates on harder tasks\. The benchmark landscape has expanded from simulated shopping and household environments\(Yaoet al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib5); Shridharet al\.,[2021](https://arxiv.org/html/2607.04242#bib.bib4)\)to realistic web, GUI, and tool\-use settings\(Denget al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib7); Zhouet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib8); Kohet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib9); Qinet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib34)\)\. Related embodied and digital\-agent systems ground LMs in robot or interface actions\(Ahnet al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib11); Driesset al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib12); Brohanet al\.,[2023](https://arxiv.org/html/2607.04242#bib.bib13); Furutaet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib14); Baiet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib15)\)\. More recent work applies RL to adapt agent policies through interaction, with substantial gains on web navigation, embodied tasks, and tool\-augmented search\(Jinet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib32); Wanget al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib10)\)\.

RL for long\-horizon agents\.Applying group\-based RL to multi\-turn agents requires finer credit assignment across steps\. Trajectory\-wise approaches\(Wanget al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib31); Jinet al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib32)\)concatenate the full history into a single sequence but suffer from context explosion as episodes lengthen\. GiGPO\(Fenget al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib35)\)instead constructs step\-level groups from repeated anchor states, while HGPO\(Heet al\.,[2026](https://arxiv.org/html/2607.04242#bib.bib36)\)improves fairness through hierarchical historical\-context matching\. ProGPO derives progress credit from rollout statistics without fitting a separate value network\.

## 3Preliminaries

### 3\.1Problem Definition

We consider long\-horizon agentic tasks where an LLM agentπθ\\pi\_\{\\theta\}interacts with an environment overTTsteps to accomplish a goal described by a natural language instruction𝒙\\boldsymbol\{x\}\. At each steptt, the agent observes state𝒔t∈𝒮\\boldsymbol\{s\}\_\{t\}\\in\\mathcal\{S\}\(comprising the environment observation and a bounded history window\) and produces an action𝒂t∈𝒱∗\\boldsymbol\{a\}\_\{t\}\\in\\mathcal\{V\}^\{\*\}as a variable\-length token sequence\. The environment transitions to a new state𝒔t\+1\\boldsymbol\{s\}\_\{t\+1\}based on the action\. A sparse scalar rewardr∈ℝr\\in\\mathbb\{R\}is provided only upon episode termination, indicating task success or failure\. We denote a full trajectory asτ=\{\(𝒔1,𝒂1\),…,\(𝒔T,𝒂T\),r\}\\tau=\\\{\(\\boldsymbol\{s\}\_\{1\},\\boldsymbol\{a\}\_\{1\}\),\\ldots,\(\\boldsymbol\{s\}\_\{T\},\\boldsymbol\{a\}\_\{T\}\),r\\\}, and define the discounted step return asRt=γT−t​rR\_\{t\}=\\gamma^\{T\-t\}r, whereγ∈\(0,1\]\\gamma\\in\(0,1\]controls the temporal discount\. The objective is to maximize expected return𝔼τ∼πθ​\[r\]\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\[r\]\.

### 3\.2Group\-Based RL for Agents

Group\-based methods estimate advantages without a learned critic by sampling multiple rollouts for the same task and comparing their outcomes\. GivenNNtrajectories\{τ\(1\),…,τ\(N\)\}\\\{\\tau^\{\(1\)\},\\ldots,\\tau^\{\(N\)\}\\\}for task𝒙\\boldsymbol\{x\}, the trajectory\-level group advantage \(as in GRPO\) is:

Atraj​\(τ\(i\)\)=r\(i\)−μσ,μ=1N​∑j=1Nr\(j\),σ=std​\(\{r\(j\)\}j=1N\)\.A^\{\\text\{traj\}\}\(\\tau^\{\(i\)\}\)=\\frac\{r^\{\(i\)\}\-\\mu\}\{\\sigma\},\\quad\\mu=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}r^\{\(j\)\},\\quad\\sigma=\\text\{std\}\(\\\{r^\{\(j\)\}\\\}\_\{j=1\}^\{N\}\)\.\(1\)This advantage is shared across all tokens in the trajectory, providing a coarse learning signal that does not distinguish between steps that contributed to success and those that did not\.

### 3\.3Trajectory\-Level vs\. Step\-Level Advantage

To achieve finer credit assignment, stepwise frameworks\(Fenget al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib35); Heet al\.,[2026](https://arxiv.org/html/2607.04242#bib.bib36)\)decompose trajectories into individual steps and estimate advantages at the step level\. Under this framework, the policy is conditioned on a bounded context:πθ​\(𝒂t\|𝒔t,𝒉t−K:t\)\\pi\_\{\\theta\}\(\\boldsymbol\{a\}\_\{t\}\|\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{h\}\_\{t\-K:t\}\), where𝒉t−K:t\\boldsymbol\{h\}\_\{t\-K:t\}represents a history window of lengthK≪TK\\ll T\. Steps are grouped by their observation context, and within\-group comparison yields step\-level advantages\. For stepiiwith contexts¯i\\bar\{s\}\_\{i\}, letGs¯i=\{j:s¯j=s¯i\}G\_\{\\bar\{s\}\_\{i\}\}=\\\{j:\\bar\{s\}\_\{j\}=\\bar\{s\}\_\{i\}\\\}denote the set of steps sharing the same context\. Following common notation for step\-level group advantages, we use the superscriptSSto denote “step”\. The grouped step\-level advantage is:

AS​\(s¯i\)=Ri−1\|Gs¯i\|​∑j∈Gs¯iRjσGs¯i\+ϵ,A^\{S\}\(\\bar\{s\}\_\{i\}\)=\\frac\{R\_\{i\}\-\\frac\{1\}\{\|G\_\{\\bar\{s\}\_\{i\}\}\|\}\\sum\_\{j\\in G\_\{\\bar\{s\}\_\{i\}\}\}R\_\{j\}\}\{\\sigma\_\{G\_\{\\bar\{s\}\_\{i\}\}\}\+\\epsilon\},\(2\)whereσGs¯i\\sigma\_\{G\_\{\\bar\{s\}\_\{i\}\}\}is the within\-group standard deviation\. This estimator isolates the effect of action choice by controlling for state, providing much richer feedback than trajectory\-level advantages\. However, it requires\|Gs¯i\|≥2\|G\_\{\\bar\{s\}\_\{i\}\}\|\\geq 2and non\-identical returns within the group; when a step’s context is unique within the batch \(a*singleton*\) or all peers have the same return, no peer comparison is possible and the step receives zero learning signal\.

## 4Proposed Method

### 4\.1Limitations of Existing Stepwise Group\-Based RL

We first analyze why stepwise group\-based RL loses signal after enforcing context consistency\.

The step\-level advantage in Eq\. \([2](https://arxiv.org/html/2607.04242#S3.E2)\) requires at least two comparable steps with non\-identical outcomes\. In diverse environments with large state spaces, exact matching often produces*singleton groups*, where no peer exists, or low\-contrast groups whose returns are identical\. In ProGPO training traces, zero prefix\-consistent peer\-comparison advantage occurs for 33\.7% of WebShop steps and 44\.5% of ALFWorld steps before transition credit is applied\. Pure peer comparison therefore underuses many steps, including decisions that may affect later task progress\.

Relaxing the grouping criterion can reintroduce comparisons across different goals, constraints, or available affordances\. ProGPO keeps action comparison exact and introduces progress credit through state\-potential estimation\.

### 4\.2ProGPO: Progress\- and Reliability\-Oriented Group Policy Optimization

We now describe ProGPO, a learned\-critic\-free credit assignment framework for context\-consistent step\-level RL \(Figure[3](https://arxiv.org/html/2607.04242#S4.F3)\)\. The method keeps exact\-prefix grouping for action\-relative comparison\. The remaining steps are handled through non\-parametric state potentials estimated from rollout outcomes, with semantic expansion and reliability\-aware multi\-resolution fusion used to make these estimates less brittle\. Thus ProGPO still performs value estimation, but it does so through rollout\-batch statistics rather than a learned value model\.

![Refer to caption](https://arxiv.org/html/2607.04242v1/figures/progpo_method.png)Figure 3:Overview of ProGPO\. State potentials are estimated from rollout outcomes, expanded with nearby states when needed, and fused across history depths before transition credit is computed\.#### 4\.2\.1Prefix\-Consistent Step Comparison

At steptt, let𝒐t\\boldsymbol\{o\}\_\{t\}denote the current environment observation and let𝒑t\(k\)=\(𝒐t−k\+1,…,𝒐t\)\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}=\(\\boldsymbol\{o\}\_\{t\-k\+1\},\\ldots,\\boldsymbol\{o\}\_\{t\}\)denote an exact observation prefix of depthkk, wherek=1k=1uses only the current observation and largerkkincludes a longer history\. Following the principle of context\-consistent comparison\(Heet al\.,[2026](https://arxiv.org/html/2607.04242#bib.bib36)\), ProGPO forms exact\-prefix groups:

G\(k\)​\(𝒑t\(k\)\)=\{j:𝒑j\(k\)=𝒑t\(k\)\}\.G^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)=\\\{j:\\boldsymbol\{p\}^\{\(k\)\}\_\{j\}=\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\\\}\.\(3\)For each valid group, ProGPO computes a group\-relative advantage:

At\(k\)=Rt−μ​\(G\(k\)​\(𝒑t\(k\)\)\)σ​\(G\(k\)​\(𝒑t\(k\)\)\)\+ϵ\.A^\{\(k\)\}\_\{t\}=\\frac\{R\_\{t\}\-\\mu\(G^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)\)\}\{\\sigma\(G^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)\)\+\\epsilon\}\.\(4\)Groups with a single member or zero return variance contribute no action\-comparison signal\. The hierarchical advantage aggregates all valid depths:

Athier=∑k∈𝒦t\(k\+1\)αh​At\(k\)∑k∈𝒦t\(k\+1\)αh,A^\{\\mathrm\{hier\}\}\_\{t\}=\\frac\{\\sum\_\{k\\in\\mathcal\{K\}\_\{t\}\}\(k\+1\)^\{\\alpha\_\{h\}\}A^\{\(k\)\}\_\{t\}\}\{\\sum\_\{k\\in\\mathcal\{K\}\_\{t\}\}\(k\+1\)^\{\\alpha\_\{h\}\}\},\(5\)where𝒦t\\mathcal\{K\}\_\{t\}is the set of depths that produce a valid group\-relative estimate\. This estimator preserves prefix consistency while using all valid history depths\.

#### 4\.2\.2State Transition Scoring

To score singleton and low\-contrast steps, ProGPO introduces a non\-parametric state potential and derives credit from state transitions\. The prefix hierarchy supplies multiple estimates ofV^​\(s\)\\hat\{V\}\(s\)\. Short prefixes often have enough samples to be stable, but they can blur history\-dependent value\. Long prefixes match the interaction context more closely, but the corresponding groups are often small\. Selecting one depth globally is therefore brittle\.

Depth\-specific state potentials\.For each eligible prefix groupG\(k\)​\(𝒑t\(k\)\)G^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)in a training batch, ProGPO estimates a depth\-specific state potential as the mean return of steps sharing that prefix:

V^\(k\)​\(𝒑t\(k\)\)=1\|G\(k\)​\(𝒑t\(k\)\)\|​∑j∈G\(k\)​\(𝒑t\(k\)\)Rj\.\\hat\{V\}^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)=\\frac\{1\}\{\|G^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)\|\}\\sum\_\{j\\in G^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)\}R\_\{j\}\.\(6\)This estimate is computed directly from rollout\-batch statistics\.

Reliability\-aware multi\-resolution fusion\.ProGPO treats the hierarchy as a family of value estimators rather than selecting one depth\. For the current state lookup, letnkn\_\{k\}andσk2\\sigma\_\{k\}^\{2\}be the sample count and empirical return variance of the eligible group at depthkk\. A depth participates only whennkn\_\{k\}exceeds a small minimum threshold\. ProGPO assigns the precision\-style reliability weight

wk=nkσk2\+ϵvar,w\_\{k\}=\\frac\{n\_\{k\}\}\{\\sigma\_\{k\}^\{2\}\+\\epsilon\_\{\\mathrm\{var\}\}\},\(7\)whereϵvar\>0\\epsilon\_\{\\mathrm\{var\}\}\>0is a small variance floor that prevents unstable weights when empirical return variance is near zero\. ProGPO then computes the fused potential

V^mr​\(st\)=∑k∈ℳtwk​V^\(k\)​\(𝒑t\(k\)\)∑k∈ℳtwk,\\hat\{V\}^\{\\mathrm\{mr\}\}\(s\_\{t\}\)=\\frac\{\\sum\_\{k\\in\\mathcal\{M\}\_\{t\}\}w\_\{k\}\\hat\{V\}^\{\(k\)\}\(\\boldsymbol\{p\}^\{\(k\)\}\_\{t\}\)\}\{\\sum\_\{k\\in\\mathcal\{M\}\_\{t\}\}w\_\{k\}\},\(8\)whereℳt\\mathcal\{M\}\_\{t\}contains the eligible depths for the state lookup\. Equation \([8](https://arxiv.org/html/2607.04242#S4.E8)\) gives larger influence to well\-sampled, low\-variance estimates\.

Unlike the depth\-only weighting in Eq\. \([5](https://arxiv.org/html/2607.04242#S4.E5)\), this fusion is data\-dependent: two groups at the same depth can receive different weights when their sample counts or return variance differ\.

Transition credit\.Given the state potentials, we derive a directional learning signal for each step from the state transition:

Attrans=V^mr​\(st\+1\)−V^mr​\(st\),A^\{\\text\{trans\}\}\_\{t\}=\\hat\{V\}^\{\\mathrm\{mr\}\}\(s\_\{t\+1\}\)\-\\hat\{V\}^\{\\mathrm\{mr\}\}\(s\_\{t\}\),\(9\)which measures whether the action at stepttmoves the agent toward a higher\-potential state\. This yields a progress signal without training a parametric critic\.

Semantic similarity expansion for potential stabilization\.When thek=1k=1observation group is small, ProGPO expands the value lookup with semantically similar states\. In our implementation, the state embedding is the last\-layer policy hidden state collected during policy evaluation at the action boundary\. For a group with\|Gs¯\|<m\|G\_\{\\bar\{s\}\}\|<m, we retrieve the top\-KsemK\_\{\\mathrm\{sem\}\}nearest states and compute a weighted potential:

V^sem​\(s¯\)=∑j∈Gs¯R​\(s¯j\)\+∑ℓ=1Ksemωℓ⋅R​\(s¯qℓ\)\|Gs¯\|\+∑ℓ=1Ksemωℓ,\\hat\{V\}^\{\\text\{sem\}\}\(\\bar\{s\}\)=\\frac\{\\sum\_\{j\\in G\_\{\\bar\{s\}\}\}R\(\\bar\{s\}\_\{j\}\)\+\\sum\_\{\\ell=1\}^\{K\_\{\\mathrm\{sem\}\}\}\\omega\_\{\\ell\}\\cdot R\(\\bar\{s\}\_\{q\_\{\\ell\}\}\)\}\{\|G\_\{\\bar\{s\}\}\|\+\\sum\_\{\\ell=1\}^\{K\_\{\\mathrm\{sem\}\}\}\\omega\_\{\\ell\}\},\(10\)whereqℓq\_\{\\ell\}indexes theℓ\\ell\-th nearest neighbor by cosine similarity of hidden state embeddings, andωℓ=softmax​\(sim​\(𝒉s¯,𝒉qℓ\)/τ\)\\omega\_\{\\ell\}=\\text\{softmax\}\(\\text\{sim\}\(\\boldsymbol\{h\}\_\{\\bar\{s\}\},\\boldsymbol\{h\}\_\{q\_\{\\ell\}\}\)/\\tau\)are similarity\-based weights\.

Use in state\-potential estimation\.ProGPO uses semantic expansion only for state potentials; action advantages remain tied to exact\-prefix comparison\. This keeps action comparison prefix\-consistent, although the final advantage can still be influenced by the expanded potential through transition credit\.

#### 4\.2\.3Complementary Credit Composition

ProGPO combines prefix\-consistent peer comparison with transition credit\. We setAthier=0A^\{\\mathrm\{hier\}\}\_\{t\}=0when no valid exact\-prefix comparison is available\. Letctc\_\{t\}indicate whether peer comparison provides a nonzero signal, and letηt\\eta\_\{t\}denote the residual gate applied to transition credit:

ct\\displaystyle c\_\{t\}=𝟏​\{\|Athier\|\>ϵ\},ηt=1−\(1−λfb\)​ct,\\displaystyle=\\mathbf\{1\}\\\{\|A^\{\\mathrm\{hier\}\}\_\{t\}\|\>\\epsilon\\\},\\qquad\\eta\_\{t\}=1\-\(1\-\\lambda\_\{\\mathrm\{fb\}\}\)c\_\{t\},\(11\)AtProGPO\\displaystyle A^\{\\mathrm\{ProGPO\}\}\_\{t\}=Athier\+λtr​ηt​Attrans\.\\displaystyle=A^\{\\mathrm\{hier\}\}\_\{t\}\+\\lambda\_\{\\mathrm\{tr\}\}\\eta\_\{t\}A^\{\\mathrm\{trans\}\}\_\{t\}\.\(12\)Here𝟏​\{⋅\}\\mathbf\{1\}\\\{\\cdot\\\}is the indicator function,λtr\\lambda\_\{\\mathrm\{tr\}\}scales transition credit, andλfb\\lambda\_\{\\mathrm\{fb\}\}controls how much transition credit remains when peer comparison is already active\.

### 4\.3The Policy Optimization Objective

ProGPO optimizes the standard clipped surrogate:

𝒥ProGPO\(θ\)=𝔼\[1N​T∑i=1N∑t=1Tmin\(\\displaystyle\\mathcal\{J\}\_\{\\text\{ProGPO\}\}\(\\theta\)=\\mathbb\{E\}\\Bigg\[\\frac\{1\}\{NT\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{T\}\\min\\Big\(ρθ​\(𝒂t\(i\)\)​AtProGPO,\(i\),\\displaystyle\\rho\_\{\\theta\}\(\\boldsymbol\{a\}\_\{t\}^\{\(i\)\}\)A\_\{t\}^\{\\text\{ProGPO\},\(i\)\},clip\(ρθ,1±ϵ\)AtProGPO,\(i\)\)\]−λ𝔻KL\[πθ∥πref\],\\displaystyle\\text\{clip\}\(\\rho\_\{\\theta\},1\\pm\\epsilon\)A\_\{t\}^\{\\text\{ProGPO\},\(i\)\}\\Big\)\\Bigg\]\-\\lambda\\mathbb\{D\}\_\{\\text\{KL\}\}\[\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\],\(13\)whereρθ=πθ​\(𝒂t\|𝒔t,𝒙\)/πold​\(𝒂t\|𝒔t,𝒙\)\\rho\_\{\\theta\}=\\pi\_\{\\theta\}\(\\boldsymbol\{a\}\_\{t\}\|\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{x\}\)/\\pi\_\{\\text\{old\}\}\(\\boldsymbol\{a\}\_\{t\}\|\\boldsymbol\{s\}\_\{t\},\\boldsymbol\{x\}\)is the importance sampling ratio andλ\\lambdacontrols the KL penalty\.

### 4\.4Analysis

Special cases\.Several familiar variants appear as ablations of ProGPO\. Settingλtr=0\\lambda\_\{\\mathrm\{tr\}\}=0removes transition credit, using onlyk=1k=1reduces the grouping to current\-observation matching, disabling multi\-resolution fusion gives a single\-depth potential estimator, and settingm=0m=0removes semantic expansion\.

Computational overhead\.ProGPO computes group statistics and state potentials from the existing rollout batch\. Multi\-resolution fusion adds per\-group variance estimates, and semantic expansion reuses hidden states from policy evaluation\. No additional rollouts, parametric critic networks, or reward models are required\.

## 5Experiments

### 5\.1Experiment Setup

Agentic benchmarks\.We evaluate on WebShop\(Yaoet al\.,[2022](https://arxiv.org/html/2607.04242#bib.bib5)\)and ALFWorld\(Shridharet al\.,[2021](https://arxiv.org/html/2607.04242#bib.bib4)\), two standard long\-horizon agent benchmarks\. WebShop requires agents to search for and purchase products matching natural\-language instructions, with episodes lasting up to 30 steps\. ALFWorld covers embodied household tasks such as finding objects, heating food, and cleaning items, with episodes lasting up to 50 steps\.

Models\.The main matched validation results use Qwen2\.5\-1\.5B\-Instruct\. To test whether the training trend persists at a larger policy scale, we additionally run ALFWorld training\-side comparisons with Qwen2\.5\-3B\-Instruct\.

Compared methods\.For the 1\.5B validation setting, we compare matched runs of GRPO, GiGPO, HGPO, and ProGPO, with three validation seeds for each method at step 160\. The 3B ALFWorld scaling study compares GiGPO, HGPO, and ProGPO using training success, validation success, and validation test score\. For broader context, Table[1](https://arxiv.org/html/2607.04242#S5.T1)also includes aggregate numbers for prompting, PPO, and RLOO from prior reports; conclusions about relative improvement are drawn from the matched local reruns\.

Reference methods\.GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.04242#bib.bib22)\)provides trajectory\-level group\-relative advantages\. GiGPO\(Fenget al\.,[2025](https://arxiv.org/html/2607.04242#bib.bib35)\)and HGPO\(Heet al\.,[2026](https://arxiv.org/html/2607.04242#bib.bib36)\)are step\-level group\-based methods for long\-horizon agents\. PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.04242#bib.bib24)\)and RLOO\(Koolet al\.,[2019](https://arxiv.org/html/2607.04242#bib.bib21)\)are included through cited aggregate results\.

Training details\.The matched 1\.5B runs use batch size 16, rollout group size 8, learning rate1×10−61\\times 10^\{\-6\}, KL coefficient0\.010\.01, and vLLM rollout generation\. WebShop episodes last up to 30 steps and ALFWorld episodes last up to 50 steps\. ProGPO uses history lengthKh=2K\_\{h\}=2, length\-weight exponentαh=1\.0\\alpha\_\{h\}=1\.0, minimum value group sizem=3m=3, semantic neighborsKsem=5K\_\{\\mathrm\{sem\}\}=5, transition scaleλtr=0\.5\\lambda\_\{\\mathrm\{tr\}\}=0\.5, fallback scaleλfb=0\.3\\lambda\_\{\\mathrm\{fb\}\}=0\.3, inverse\-variance fusion with minimum multi\-resolution group size 2, and variance floor10−310^\{\-3\}\.

### 5\.2Main Results

Table 1:Performance on ALFWorld and WebShop\. Prompting, PPO, and RLOO numbers are cited from prior reports; GRPO, GiGPO, HGPO, and ProGPO are local Qwen2\.5\-1\.5B\-Instruct reruns\. Parentheses denote standard deviations over three seeds when available\.TypeMethodALFWorldWebShopPickCleanCoolLookHeatPick2AllScoreSucc\.*Prompt\-based methods*Prompt\.GPT\-4o75\.360\.831\.256\.721\.649\.848\.031\.823\.7Prompt\.Gemini\-2\.5\-Pro92\.863\.362\.169\.026\.658\.760\.342\.535\.9Prompt\.Qwen2\.55\.95\.53\.39\.74\.20\.04\.123\.15\.2Prompt\.ReAct17\.420\.515\.76\.27\.72\.012\.840\.111\.3Prompt\.Reflexion35\.322\.221\.713\.619\.43\.721\.855\.821\.9*RL\-based methods*RLPPO64\.8\(3\.5\)40\.5\(6\.9\)57\.1\(4\.9\)60\.6\(6\.6\)46\.4\(4\.0\)47\.4\(1\.9\)54\.4\(3\.1\)73\.8\(3\.0\)51\.5\(2\.9\)RLRLOO88\.3\(3\.0\)52\.8\(8\.6\)71\.0\(5\.9\)62\.8\(8\.7\)66\.4\(5\.5\)56\.9\(4\.7\)69\.7\(2\.5\)73\.9\(5\.6\)52\.1\(6\.7\)RLGRPO68\.5\(8\.5\)79\.1\(2\.0\)80\.5\(2\.9\)83\.4\(7\.5\)80\.2\(4\.2\)60\.1\(4\.0\)75\.8\(1\.0\)81\.9\(1\.2\)56\.8\(2\.3\)RLGiGPO73\.9\(7\.6\)85\.6\(3\.0\)88\.8\(3\.4\)99\.6\(0\.7\)83\.1\(2\.1\)84\.9\(3\.7\)85\.5\(1\.3\)85\.8\(1\.5\)67\.6\(2\.2\)RLHGPO78\.3\(7\.9\)83\.6\(2\.5\)89\.2\(2\.8\)97\.2\(1\.3\)86\.5\(1\.9\)99\.1\(1\.5\)87\.8\(1\.3\)83\.2\(1\.6\)65\.6\(2\.5\)RLProGPO79\.3\(5\.3\)86\.9\(2\.5\)98\.0\(1\.7\)100\.0\(0\.0\)94\.9\(0\.8\)84\.6\(1\.1\)90\.1\(1\.1\)85\.4\(1\.5\)71\.5\(3\.0\)Table[1](https://arxiv.org/html/2607.04242#S5.T1)reports prompt\-based and RL\-based baselines in one comparison\. Since the prompting, PPO, and RLOO entries are taken from prior reports, we use them as reference points rather than matched\-protocol comparisons\. Among our local reruns, ProGPO improves ALFWorld overall success from 75\.8% for GRPO, 85\.5% for GiGPO, and 87\.8% for HGPO to 90\.1%\. It also leads the Pick, Clean, Cool, Look, and Heat diagnostics, while HGPO remains higher on the ALFWorld two\-object task\. On WebShop, ProGPO obtains the highest success rate at 71\.5%, while GiGPO is slightly higher on task score\.

![Refer to caption](https://arxiv.org/html/2607.04242v1/x2.png)Figure 4:Training curves for ProGPO \(red\), HGPO \(orange\), GiGPO \(green\), and GRPO \(blue\)\. Lighter curves show raw values; darker curves use EMA smoothing with decayα=0\.95\\alpha=0\.95\.Beyond aggregate success, Table[1](https://arxiv.org/html/2607.04242#S5.T1)reports the full ALFWorld task taxonomy\. ProGPO is higher on Pick, Clean, Cool, Look, Heat, and All, while HGPO remains higher on Pick2\. On WebShop, ProGPO has the highest success rate and GiGPO is marginally higher on task score, suggesting that ProGPO improves terminal completion more clearly than partial task progress in this setting\. The larger\-scale ALFWorld runs show a similar trend: at the last common non\-empty checkpoint \(step 155\), ProGPO reaches 93\.0% training success, 94\.5% validation success, and 7\.57 validation test score, compared with 89\.1%, 93\.0%, and 6\.99 for GiGPO, and 86\.7%, 89\.1%, and 5\.84 for HGPO\.

### 5\.3Mechanism Diagnostics

![Refer to caption](https://arxiv.org/html/2607.04242v1/figures/progpo_mechanism_160_updates.png)Figure 5:Mechanism diagnostics over 160 training updates\. Transition\-active denotes the union of transition\-only and peer\-and\-transition steps\. Curves are EMA\-smoothed; faint traces show raw transition\-only shares\.Table 2:Mechanism diagnostics averaged over training traces\. Peer\-zero is measured before transition credit; still\-zero is measured after credit composition\.Figure[5](https://arxiv.org/html/2607.04242#S5.F5)and Table[2](https://arxiv.org/html/2607.04242#S5.T2)show that transition credit and peer comparison cover different parts of training\. Transition\-only signal appears on 11\.4% of steps in both environments, but transition credit is active more broadly because it also appears on steps that already have peer\-comparison signal\. We call this union*transition\-active*; it averages 43\.0% of WebShop steps and 63\.3% of ALFWorld steps in Table[2](https://arxiv.org/html/2607.04242#S5.T2)\. Transition credit is inactive when the current or next state has no reliable potential lookup, when the step has no next state, or whenV^​\(st\+1\)−V^​\(st\)\\hat\{V\}\(s\_\{t\+1\}\)\-\\hat\{V\}\(s\_\{t\}\)is numerically zero\. ProGPO therefore complements prefix\-consistent peer comparison rather than converting every peer\-zero step into a nonzero update\. Multi\-resolution fusion is used in most value lookups, with higher\-depth estimates active in 82\.5% of WebShop lookups and 73\.3% of ALFWorld lookups\.

### 5\.4Transition\-Credit Ablations

We ablate the components used to estimate transition credit on ALFWorld\. These runs are training\-side diagnostics at the final training update, intended to isolate the estimator components rather than replace the matched validation comparison in Table[1](https://arxiv.org/html/2607.04242#S5.T1)\. Since semantic expansion and reliability\-aware fusion both operate through the state\-potential estimator, Table[3](https://arxiv.org/html/2607.04242#S5.T3)compares plain transition credit, each stabilization component alone, and the full ProGPO estimator\.

Table 3:ALFWorld transition\-credit ablations measured on training rollouts at step 160\. Success\-rate columns are percentages\.The ablation results indicate that transition credit depends strongly on the quality of the state\-potential estimate\. Plain transition credit is weak on the two\-object task\. Adding semantic expansion improves training success from 87\.5% to 92\.2%, while reliability\-aware fusion alone improves it to 90\.6%\. Combining both components gives the best result, reaching 96\.9% success and the highest training score\. These results support the design in Section[4\.2\.2](https://arxiv.org/html/2607.04242#S4.SS2.SSS2), where semantic expansion and multi\-resolution fusion stabilize the potential estimates used by transition credit\.

## 6Limitations and Future Work

ProGPO still depends on useful state recurrence inside a rollout batch\. Transition credit can fill part of the gap left by prefix\-consistent peer comparison, but it does not turn every step into a supervised update; in our diagnostics, a residual set of steps remains zero after credit composition\. This is expected when both the current and next states have nearly identical estimated potentials, or when neither state has enough reliable evidence in the batch\.

The non\-parametric value table also inherits the quality of the state representation\. Exact\-prefix matching is conservative for action comparison, but state\-potential estimation can still be noisy when observations are mostly unique or when semantically retrieved neighbors correspond to superficially similar but behaviorally different states\. The current semantic expansion uses policy hidden states as a practical similarity signal; it does not by itself prove that nearby states have similar expected return\. A learned state abstraction, better calibrated uncertainty estimates, or task\-specific filters for semantic expansion may reduce this failure mode\.

Transition credit should also be interpreted as a batch\-estimated progress signal rather than a causal effect estimator\. Changes inV^​\(st\+1\)−V^​\(st\)\\hat\{V\}\(s\_\{t\+1\}\)\-\\hat\{V\}\(s\_\{t\}\)can reflect environment stochasticity, task difficulty, or representation error in addition to the chosen action\. Systematic sweeps over rollout group size and method\-specific hyperparameters such asKhK\_\{h\},mm,KsemK\_\{\\mathrm\{sem\}\},λtr\\lambda\_\{\\mathrm\{tr\}\},λfb\\lambda\_\{\\mathrm\{fb\}\}, and the variance floor are left for future work\.

Finally, the experiments focus on ALFWorld and WebShop with Qwen2\.5 policies up to 3B parameters\. These environments expose long\-horizon interaction and sparse rewards, but they do not cover all agent settings\. Evaluating ProGPO on more open\-ended web tasks, larger models, and multimodal agents would give a clearer picture of when progress\-based credit is most useful\.

## 7Conclusion

This paper studied credit sparsity in context\-consistent step\-level group RL\. Exact context matching yields fairer action comparisons, but also leaves many steps without peer\-comparison signal\. ProGPO addresses this limitation with rollout\-derived state potentials, transition credit, and inverse\-variance multi\-resolution fusion across history depths\. Matched WebShop and ALFWorld traces show that progress\- and reliability\-oriented group policy optimization improves context\-consistent training without additional reward models or parametric value networks\.

## References

- M\. Ahn, A\. Brohan, N\. Brown, Y\. Chebotar, O\. Cortes, B\. David, C\. Finn, C\. Fu, K\. Gopalakrishnan, K\. Hausman,et al\.\(2022\)Do as i can, not as i say: grounding language in robotic affordances\.arXiv preprint arXiv:2204\.01691\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- H\. Bai, Y\. Zhou, J\. Pan, M\. Cemri, A\. Suhr, S\. Levine, and A\. Kumar \(2024\)DigiRL: training in\-the\-wild device\-control agents with autonomous reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- A\. Brohan, N\. Brown, J\. Carbajal, Y\. Chebotar, J\. Dabis, C\. Finn, K\. Gopalakrishnan, K\. Hausman, A\. Herzog, J\. Hsu,et al\.\(2023\)RT\-2: vision\-language\-action models transfer web knowledge to robotic control\.InConference on Robot Learning,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma,et al\.\(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su \(2023\)Mind2Web: towards a generalist agent for the web\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- D\. Driess, F\. Xia, M\. S\. M\. Sajjadi, C\. Lynch, A\. Chowdhery, B\. Ichter, A\. Wahid, J\. Tompson, Q\. Vuong, T\. Yu,et al\.\(2023\)PaLM\-e: an embodied multimodal language model\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- L\. Feng, Z\. Xue, T\. Liu, and B\. An \(2025\)Group\-in\-group policy optimization for llm agent training\.arXiv preprint arXiv:2505\.10978\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p3.1),[§2](https://arxiv.org/html/2607.04242#S2.p3.1),[§3\.3](https://arxiv.org/html/2607.04242#S3.SS3.p1.7),[§5\.1](https://arxiv.org/html/2607.04242#S5.SS1.p4.1)\.
- H\. Furuta, K\. Lee, O\. Nachum, Y\. Matsuo, A\. Faust, S\. S\. Gu, and I\. Gur \(2024\)Multimodal web navigation with instruction\-finetuned foundation models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- S\. He, L\. Feng, Q\. Wei, X\. Cheng, L\. Feng, and B\. An \(2026\)Hierarchy\-of\-groups policy optimization for long\-horizon agentic tasks\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p3.1),[§3\.3](https://arxiv.org/html/2607.04242#S3.SS3.p1.7),[§4\.2\.1](https://arxiv.org/html/2607.04242#S4.SS2.SSS1.p1.6),[§5\.1](https://arxiv.org/html/2607.04242#S5.SS1.p4.1)\.
- B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§1](https://arxiv.org/html/2607.04242#S1.p3.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p3.1)\.
- J\. Y\. Koh, R\. Lo, L\. Jang, V\. Duvvur, M\. C\. Lim, P\. Huang, G\. Neubig, S\. Zhou, R\. Salakhutdinov, and D\. Fried \(2024\)VisualWebArena: evaluating multimodal agents on realistic visual web tasks\.InAnnual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- V\. R\. Konda and J\. N\. Tsitsiklis \(1999\)Actor\-critic algorithms\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- W\. Kool, H\. van Hoof, and M\. Welling \(2019\)Buy 4 reinforce samples, get a baseline for free\.InDeep Reinforcement Learning Meets Structured Prediction Workshop at ICLR,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.04242#S5.SS1.p4.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2024\)AgentBench: evaluating llms as agents\.arXiv preprint arXiv:2308\.03688\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- X\. B\. Peng, A\. Kumar, G\. Zhang, and S\. Levine \(2019\)Advantage\-weighted regression: simple and scalable off\-policy reinforcement learning\.arXiv preprint arXiv:1910\.00177\.Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.04242#S5.SS1.p4.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.04242#S5.SS1.p4.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2021\)ALFWorld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1),[§5\.1](https://arxiv.org/html/2607.04242#S5.SS1.p1.1)\.
- R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour \(1999\)Policy gradient methods for reinforcement learning with function approximation\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu,et al\.\(2025\)RAGEN: understanding self\-evolution in llm agents via multi\-turn reinforcement learning\.arXiv preprint arXiv:2504\.20073\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p3.1),[§2](https://arxiv.org/html/2607.04242#S2.p3.1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8,pp\. 229–256\.Cited by:[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan \(2022\)WebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1),[§5\.1](https://arxiv.org/html/2607.04242#S5.SS1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2025\)DAPO: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p2.1),[§2](https://arxiv.org/html/2607.04242#S2.p1.1)\.
- B\. Zheng, B\. Gou, J\. Kil, H\. Sun, and Y\. Su \(2024\)GPT\-4v\(ision\) is a generalist web agent, if grounded\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024\)WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.04242#S1.p1.1),[§2](https://arxiv.org/html/2607.04242#S2.p2.1)\.