Retrospective Progress-Aware Self-Refinement for LLM Agent Training

# Retrospective Progress-Aware Self-Refinement for LLM Agent Training
Source: [https://arxiv.org/html/2606.14302](https://arxiv.org/html/2606.14302)
Xinbei Ma1, Congmin Zheng1, Jiyang Qiu1, Jiale Hong1, Yao Yao1, Xiangmou Qu2, Jiaxin Yin2, Xingyu Lou2,†, Jun Wang2,†, Weiwen Liu1, Weinan Zhang1, Zhuosheng Zhang1,†, Hai Zhao1 1Shanghai Jiao Tong University,2OPPO Research Institute \{sjtumaxb, zhangzs\}@sjtu\.edu\.cn,zhaohai@cs\.sjtu\.edu\.cn louxingyu@oppo\.com,junwang\.lu@gmail\.com

###### Abstract

LLM\-based agents trained with reinforcement learning optimize step\-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders long\-horizon scaling\. A pilot study reveals that online progress prompting hurts performance while retrospective demonstrations help, yet this capability cannot emerge from outcome\-reward training alone\. We presentRePro,RetrospectiveProgress\-Aware Training, a framework that trains agents to self\-generate progress signals via aforward\-then\-reflectrollout paradigm: the agent executes actions online, then retrospectively reassesses its step\-wise progress given the completed trajectory and known outcome\. RePro initializes with aRetrospection Warmupthat teaches reflection format from minimal external demonstrations, then further trains throughRePro\-POwith a composite reward that produces self\-generated signals without continuous external supervision\. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family’s performance, with up to12%12\\%absolute success rate gains\.

Retrospective Progress\-Aware Self\-Refinement for LLM Agent Training

Xinbei Ma1††thanks:Work done during internship at OPPO\., Congmin Zheng1, Jiyang Qiu1, Jiale Hong1, Yao Yao1,Xiangmou Qu2, Jiaxin Yin2, Xingyu Lou2,†, Jun Wang2,†,Weiwen Liu1, Weinan Zhang1, Zhuosheng Zhang1,†, Hai Zhao1††thanks:Corresponding authors\.1Shanghai Jiao Tong University,2OPPO Research Institute\{sjtumaxb, zhangzs\}@sjtu\.edu\.cn,zhaohai@cs\.sjtu\.edu\.cnlouxingyu@oppo\.com,junwang\.lu@gmail\.com,

## 1Introduction

LLM\-based agents have significantly expanded their capabilities on long\-horizon tasks, tackling complex real\-world interactions across domains such as computer navigationYaoet al\.\([2022](https://arxiv.org/html/2606.14302#bib.bib46)\); Xieet al\.\([2024](https://arxiv.org/html/2606.14302#bib.bib54)\); Liuet al\.\([2026](https://arxiv.org/html/2606.14302#bib.bib50)\), embodied planning\(Shridharet al\.,[2020](https://arxiv.org/html/2606.14302#bib.bib47); Zhanget al\.,[2025c](https://arxiv.org/html/2606.14302#bib.bib51); Wanget al\.,[2026b](https://arxiv.org/html/2606.14302#bib.bib52)\), and games and daily tasksMerrillet al\.\([2026](https://arxiv.org/html/2606.14302#bib.bib48)\); Yeet al\.\([2026](https://arxiv.org/html/2606.14302#bib.bib53)\)\. Reinforcement learning\-based approaches enable agents to explore solutions and optimize their actions toward verifiable rewards, resulting in substantial performance gainsDeepSeek \([2024](https://arxiv.org/html/2606.14302#bib.bib9)\); Fenget al\.\([2026](https://arxiv.org/html/2606.14302#bib.bib10)\); Donget al\.\([2025b](https://arxiv.org/html/2606.14302#bib.bib11)\)\. A common major training objective is to optimize the agent to select the best action at each step\. However, long\-horizon tasks require agents to go beyond step\-level decision\-making by maintainingmetacognitive awarenessof their task progress, serving as a continuous sense of what has been accomplished, what remains, and whether the current trajectory is on trackLinet al\.\([2026](https://arxiv.org/html/2606.14302#bib.bib55)\); Liet al\.\([2025b](https://arxiv.org/html/2606.14302#bib.bib56)\); Hanet al\.\([2025](https://arxiv.org/html/2606.14302#bib.bib57)\)\. In this sense, progress awareness offers an auxiliary signal that can help agents allocate effort appropriately across a multi\-step episode\.

Motivated by this, we conduct a pilot study to investigate the potential of progress awareness and the difficulty of eliciting it\. This online prompting substantially reduces task success, 8\.6% on average, suggesting that forced self\-assessment introduces noisy signals rather than useful guidance\. In contrast, when the agent is provided with*retrospective*progress demonstrations constructed from completed trajectories and their outcomes, the average success rate increases 7\.9%\. This improvement indicates that progress information can be beneficial when it is grounded in reliable trajectory\-level evidence\. This asymmetry reveals that progress awareness is beneficial yet cannot be reliably elicited by prompting alone, motivating a dedicated training approach that learns progress assessment from retrospective trajectory outcomes\.

This paper proposesRePro\(RetrospectiveProgress\-Aware Training\), a two\-phase framework that learns progress awareness from retrospective trajectory outcomes and uses it as an auxiliary signal for agent training\. RePro augments trajectories with aforward\-then\-reflectparadigm: the agent first*forward*to execute the task while generating online progress estimates; once completing the task and knowing the outcome, the agent retrospectively reassesses its step\-wise progress anchored by the final result\. Based on this, RePro first applies a stage ofRetrospection Warmup: a small set of external\-LLM demonstrations teaches the agent the forward\-then\-reflect format for retrospective reflection\. After that,RePro\-POleverages the retrospective progress signal by a composite reward, comprising retrospective progress shaping, online\-retrospective alignment, and format regularization, producing per\-step training signals that complement the sparse outcome reward\.

We evaluate RePro across LLMs of varying sizes on WebShop, ALFWorld, and Sokoban\. On WebShop, RePro improves the absolute task success rate over baseline training by\+8\.98%\+8\.98\\%,\+11\.57%\+11\.57\\%, and\+5\.82%\+5\.82\\%across different model sizes\. It also yields consistent gains on ALFWorld and Sokoban, and outperforms all ablation variants\. Progress quality analysis further shows that RePro agents develop metacognitive awareness: their progress estimates discriminate successful from failed trajectories significantly better than baselines\. Our contributions are summarized in threefold:

1. 1\.Pilot study: We reveal a clear asymmetry between online progress prompting and retrospective demonstrations, showing that progress awareness requires dedicated training\.
2. 2\.RePro: We propose a forward\-then\-reflect framework with Retrospection Warmup and RePro\-PO, enabling agents to learn self\-generated progress metacognition\.
3. 3\.Empirical validation: RePro achieves consistent performance gains across three benchmarks and three model scales, supported by progress\-quality analysis\.

## 2Related Work

### 2\.1LLM Agent Training

LLM\-based agent training has long drawn inspiration from reinforcement learning, showing that agent behavior can be improved by reusing past interaction trajectories as training signals\(Xionget al\.,[2024](https://arxiv.org/html/2606.14302#bib.bib60); Songet al\.,[2024](https://arxiv.org/html/2606.14302#bib.bib59); Caoet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib58)\)\. Becoming a dominant post\-training method, recent critic\-free policy optimization methods improve efficiency by estimating advantages from rollout groups, trajectory\-level comparisons, or improved sampling strategies\(Guoet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib8); DeepSeek,[2024](https://arxiv.org/html/2606.14302#bib.bib9); Fenget al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib10); Donget al\.,[2025b](https://arxiv.org/html/2606.14302#bib.bib11),[a](https://arxiv.org/html/2606.14302#bib.bib12); Zhenget al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib13)\), and have been applied to tool\-use and multi\-environment agent training\(Qianet al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib15); Wanget al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib16); Xiet al\.,[2026a](https://arxiv.org/html/2606.14302#bib.bib17)\)\.

Reward design remains a central challenge in long\-horizon agent RL\. Outcome\-based methods suffer from sparse terminal feedback\(Guoet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib8); DeepSeek,[2024](https://arxiv.org/html/2606.14302#bib.bib9)\); process\-based methods introduce intermediate reward models or subgoals\(Lightmanet al\.,[2024](https://arxiv.org/html/2606.14302#bib.bib19); Zouet al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib20); Xiet al\.,[2026b](https://arxiv.org/html/2606.14302#bib.bib18); Zhenget al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib21); Wanget al\.,[2026c](https://arxiv.org/html/2606.14302#bib.bib22)\), but require costly annotation or environment\-specific design; and step\-wise rewards derived from state changes depend on environment\-specific signals\(Luet al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib23)\)\. Progress\-based methods address this issue by modeling task completion progress, either through heuristic estimates or stronger\-LLM supervision\(Zhanget al\.,[2025a](https://arxiv.org/html/2606.14302#bib.bib25),[2026b](https://arxiv.org/html/2606.14302#bib.bib24); Chaiet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib26)\)\. Differently, our method uses minimal external supervision and learns progress awareness from the agent’s own completed trajectory outcomes, internalizing progress estimation into the agent itself without relying on an additional reward model\.

### 2\.2Self\-Improving LLMs via Reflection

Inference\-time methods improve agent behavior without parameter updates by turning past experience into reusable guidance\. Failed episodes are converted into verbal feedback or episodic memory\(Shinnet al\.,[2023](https://arxiv.org/html/2606.14302#bib.bib28); Quet al\.,[2024](https://arxiv.org/html/2606.14302#bib.bib27)\), while others ground reflection in execution states or evolving contextual playbooks\(Kimet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib29); Guoet al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib31); Caiet al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib32); Zhanget al\.,[2025b](https://arxiv.org/html/2606.14302#bib.bib41)\)\. Recent work further extends reflection to uncertainty\-aware test\-time adaptation, reusable experiential heuristics, and pre\-execution plan critique\(Acikgozet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib40); Geet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib38); Allardet al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib39); Wanget al\.,[2026a](https://arxiv.org/html/2606.14302#bib.bib43)\)\. These methods can elicit reflective behaviors already present in the base model, but do not directly train new reflection capabilities into the agent\.

Training\-time methods instead aim to internalize reflection through learning\. They improve meta\-introspection or reward useful self\-reflective tokens\(Liet al\.,[2025a](https://arxiv.org/html/2606.14302#bib.bib42); Bensalet al\.,[2025](https://arxiv.org/html/2606.14302#bib.bib37)\), support staged self\-correction and consistency\-driven training\([Kumaret al\.,](https://arxiv.org/html/2606.14302#bib.bib33); Zhanget al\.,[2026a](https://arxiv.org/html/2606.14302#bib.bib34)\), and enable self\-evolution\(Ouet al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib35)\)\. Our work follows this line in internalizing self\-improvement signals into the agent itself\. However, rather than correcting individual actions, reasoning steps, or plans, our method introduces a retrospective progress\-aware reflection, where the agent uses completed trajectory outcomes to retroactively assess task\-level progress at intermediate steps\. This yields self\-generated progress targets that train the agent to trackwhere it standsin a long\-horizon task, without relying on an additional reward model\.

## 3Pilot Study

Agentic tasks require multi\-turn interaction, where an LLM agent maintains global awareness over a sequence of actions and observations\. We investigate two questions: \(i\) Can progress awareness improve task success? and \(ii\) can agents produce reliable progress estimates? We conduct a controlled diagnostic experiment on WebShopYaoet al\.\([2022](https://arxiv.org/html/2606.14302#bib.bib46)\)using DeepSeek\-V4DeepSeek\-AI \([2026](https://arxiv.org/html/2606.14302#bib.bib62)\)and GPT\-5\.1 as agent backbones\.

##### Setup\.

The baseline agent interacts with the environment conditioned on the full interaction history\. We evaluate two progress\-aware variants\.

\(i\) Online prompting: the agent is prompted to verbalize a numeric progress estimate \(0\-100%\) before each action during execution, without access to the trajectory outcome\.

\(ii\) Retrospective demonstration: we collect completed trajectories from a 100\-sample training set and retrospectively annotate step\-wise progress assessments using the known trajectory outcome\. Each demonstration includes action\-observation history and the current progress\. At test time, we randomly sample three such demonstrations as in\-context examples\. At each step, the agent first generates its own progress assessment conditioned on these demonstrations\. The demonstrations are then removed, and the agent predicts the next action with the generated progress assessment\.

We further include a dummy random progress estimation to test whether improvements come from meaningful progress signals rather than added context\.

![Refer to caption](https://arxiv.org/html/2606.14302v1/x1.png)Figure 1:Pilot study results on WebShop\. Online progress prompting hurts performance, while retrospective demonstrations for progress improve it\.
##### Findings\.

Figure[1](https://arxiv.org/html/2606.14302#S3.F1)reveals that online and retrospective progress awareness have opposite effects\. Online progress prompting consistently degrades performance \(DS\-v4: 35\.9%→\\to33\.6%; GPT\-5\.1: 44\.1%→\\to29\.3%;−\-8\.6% on average\), and random progress performs similarly poorly \(31\.2% and 32\.4%\), confirming that the degradation stems from unreliable online prediction disrupting decision\-making rather than the progress format itself\. By contrast, retrospective demonstrations substantially improve success rate on both models \(DS\-v4: 35\.9%→\\to44\.1%,\+\+8\.2%; GPT\-5\.1: 44\.1%→\\to51\.6%,\+\+7\.5%;\+\+7\.9% on average\)\. Random demonstrations yield only marginal gains \(DS\-v4:\+\+2\.0%; GPT\-5\.1:\+\+4\.3%\), suggesting that the improvement comes from meaningful progress signals rather than additional context alone\. This asymmetry, that progress is informative in hindsight but harmful when predicted online, indicates that progress awareness cannot be reliably elicited by prompting alone and motivates a dedicated training approach to learn progress assessment from retrospective trajectory outcomes\.

![Refer to caption](https://arxiv.org/html/2606.14302v1/x2.png)Figure 2:Overview ofRePro, our retrospective progress\-aware training framework\. \(A\) Duringforward\-then\-reflect, the agent executes actions with online progress estimates, then retrospectively reassesses step\-wise progress after observing the final outcome\. \(B\)Retrospection Warmupuses demonstrations to initialize retrospective reflection behavior\. \(C\)Policy Updateintegrates retrospective progress into RL through progress shaping, online\-retrospective alignment, and format regularization rewards, enabling long\-horizon agent training\.

## 4Retrospective Progress\-Aware Training

The pilot study shows that progress awareness improves task completion but cannot be reliably generated during execution\. To address this, we proposeRePro\(RetrospectiveProgress\-Aware Training\), a two\-stage framework that first learns retrospective self\-assessment through supervised warmup, then refines it via reinforcement learning with task\-outcome feedback\. RePro consists of aRetrospection Warmupstage \(§[4\.3](https://arxiv.org/html/2606.14302#S4.SS3)\) and a progress\-aware policy optimization stage,RePro\-PO\(§[4\.4](https://arxiv.org/html/2606.14302#S4.SS4)\)\.

### 4\.1Formulation

Given a natural language goalgg, an LLM\-based agent interacts with an environment to complete the task\. This interaction is modeled as a Partially Observable Markov Decision Process \(POMDP\)\(Kaelblinget al\.,[1998](https://arxiv.org/html/2606.14302#bib.bib63)\), defined by the tuple\(𝒮,𝒜,𝒪,𝒫,ℛ\)\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{O\},\\mathcal\{P\},\\mathcal\{R\}\), where𝒮\\mathcal\{S\},𝒜\\mathcal\{A\},𝒪\\mathcal\{O\},𝒫\\mathcal\{P\}, andℛ\\mathcal\{R\}denote the state space, action space, observation space, transition kernel, and reward function\.

At each steptt, the agent receives a partial observationot∈𝒪o\_\{t\}\\in\\mathcal\{O\}of the underlying statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}, and samples an action from its policy conditioned on the interaction historyht=\(g,o1,a1,…,ot\)h\_\{t\}=\(g,o\_\{1\},a\_\{1\},\\dots,o\_\{t\}\):

at∼πθ\(⋅∣ht\)\.a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)\.\(1\)The environment transitions according to𝒫​\(st\+1∣st,at\)\\mathcal\{P\}\(s\_\{t\+1\}\\mid s\_\{t\},a\_\{t\}\), producing observationot\+1o\_\{t\+1\}\. This sequential interaction forms a trajectory

τ=\{g,\[\(ot,at\)\]t=1T\},\\tau=\\\{g,\\,\[\(o\_\{t\},a\_\{t\}\)\]\_\{t=1\}^\{T\}\\\},\(2\)which terminates upon an environment signal or agent decision at stepTT, yielding a task rewardr=ℛ​\(τ\)r=\\mathcal\{R\}\(\\tau\)\. The policyπθ\\pi\_\{\\theta\}is optimized via RL to maximize the expected task reward\.

### 4\.2Forward\-then\-Reflect

The working pattern of the agent is redefined as forward\-then\-reflect\.

##### Forward Execution\.

At each steptt, the agent generates a progress assessment and an action conditioned on the interaction history:

\(p~t,at\)∼πθ\(⋅∣ht\),\\begin\{split\}\(\\tilde\{p\}\_\{t\},a\_\{t\}\)\\sim\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\),\\end\{split\}\(3\)wherep~t∈\[0,100\]\\tilde\{p\}\_\{t\}\\in\[0,100\]is the online self\-assessment of completion percentage,ata\_\{t\}is the action to execute\. The environment executesata\_\{t\}, returns observationot\+1o\_\{t\+1\}\. The trajectory continues until the terminal stepTT, returning task rewardrr\.

##### Retrospective Reflection\.

Following each trajectory, the agent is prompted with a retrospective reflection prompt \(Appendix[A](https://arxiv.org/html/2606.14302#A1)\) to re\-assess progress at each step given the complete trajectoryτ\\tauand outcomerr\. We denote this prompted mode asπθretro\\pi\_\{\\theta\}^\{\\text\{retro\}\}:

\(p1,…,pT\)∼πθretro\(⋅∣τ,r\),\\begin\{split\}\(p\_\{1\},\\dots,p\_\{T\}\)\\sim\\pi\_\{\\theta\}^\{\\text\{retro\}\}\(\\cdot\\mid\\tau,r\),\\end\{split\}\(4\)wherept∈\[0,100\]p\_\{t\}\\in\[0,100\]is the agent’s retrospective self\-assessed progress of task completion at steptt\. The trajectory is then re\-organized by replacing each online estimatep~t\\tilde\{p\}\_\{t\}with the corresponding retrospective estimateptp\_\{t\}:

τretro=\{g,\[\(ot,at,pt\)\]t=1T\}\.\\tau\_\{\\text\{retro\}\}=\\\{g,\[\(o\_\{t\},a\_\{t\},p\_\{t\}\)\]\_\{t=1\}^\{T\}\\\}\.\(5\)The agent generates retrospective progress estimatesp1,…,pTp\_\{1\},\\dots,p\_\{T\}conditioning on the complete trajectory and outcome\. Unlike online predictions made under uncertainty, retrospective reflections are anchored by the known outcome\. If the task succeeded, the agent knows the final step represents 100% completion, providing a supervisory signal for the last retrospective value\. The trajectory lengthTTis known, enabling the agent to calibrate intermediate values accordingly\.

### 4\.3Retrospection Warmup

Since retrospection is not directly rewarded in agentic environments and remains difficult for untrained agents to perform, we first warm up the reflection module to provide a better initialization\. Given completed trajectories with a successful outcome, we employ an external modelπdemo\\pi\_\{\\text\{demo\}\}\(DeepSeek\-V4\) to generate demonstrations of retrospective reflection, yielding the warmup dataset:

\(p1,…,pT\)∼πdemoretro\(⋅∣τ,r\),τ∈𝒟r=1,\\displaystyle\(p\_\{1\},\\dots,p\_\{T\}\)\\sim\\pi\_\{\\text\{demo\}\}^\{\\text\{retro\}\}\(\\cdot\\mid\\tau,r\),\\quad\\tau\\in\\mathcal\{D\}\_\{r=1\},\(6\)𝒟warmup=\{g,\[\(ot,at,pt\)\]t=1T\}\.\\displaystyle\\mathcal\{D\}\_\{\\text\{warmup\}\}=\\\{g,\[\(o\_\{t\},a\_\{t\},p\_\{t\}\)\]\_\{t=1\}^\{T\}\\\}\.\(7\)The agent is fine\-tuned on these demonstrations to jointly predict actions and retrospective progress assessments, learning the reflection format\.

ℒSFT\\displaystyle\\mathcal\{L\}\_\{\\text\{SFT\}\}=−𝔼τ∈𝒟warmup​\[ℒfwd\+ℒretro\],\\displaystyle=\-\\mathbb\{E\}\_\{\\tau\\in\\mathcal\{D\}\_\{\\text\{warmup\}\}\}\\big\[\\mathcal\{L\}\_\{\\text\{fwd\}\}\+\\mathcal\{L\}\_\{\\text\{retro\}\}\\big\],ℒfwd\\displaystyle\\mathcal\{L\}\_\{\\text\{fwd\}\}=∑tlog⁡πθ​\(p~t,at∣ht\),\\displaystyle=\\sum\_\{t\}\\log\\pi\_\{\\theta\}\(\\tilde\{p\}\_\{t\},a\_\{t\}\\mid h\_\{t\}\),\(8\)ℒretro\\displaystyle\\mathcal\{L\}\_\{\\text\{retro\}\}=log⁡πθretro​\(p1,…,pT∣τ,r\)\.\\displaystyle=\\log\\pi\_\{\\theta\}^\{\\text\{retro\}\}\(p\_\{1\},\\dots,p\_\{T\}\\mid\\tau,r\)\.

### 4\.4RePro\-PO: Policy Optimization with Retrospective Progress

The reward integrates the outcome task performance with progress\-specific objectives\.

R​\(τ\)=∑t=1Trt\+wb⋅Rformat\.R\(\\tau\)=\\sum\_\{t=1\}^\{T\}r\_\{t\}\+w\_\{b\}\\cdot R\_\{\\text\{format\}\}\.\(9\)
For each step, progress reflection provides intermediate signals beyond sparse outcome reward:

rt=renv​\(t\)\+rp​\(t\)\+ralign​\(t\)\+rformat​\(t\)r\_\{t\}=r\_\{\\text\{env\}\}\(t\)\+r\_\{p\}\(t\)\+r\_\{\\text\{align\}\}\(t\)\+r\_\{\\text\{format\}\}\(t\)\(10\)whererenvr\_\{\\text\{env\}\}is the environment reward \(sparse, typically nonzero only at the terminal step\),rpr\_\{p\}shapes step\-wise progress differences,ralignr\_\{\\text\{align\}\}drives online predictions toward retrospective quality, andrformatr\_\{\\text\{format\}\}is based on format validity\.

rp​\(t\)=β⋅\(pt\+1−pt\),ralign​\(t\)=1T​α⋅max⁡\(0,1−\|pt−p~t\|\)\.\\begin\{split\}r\_\{p\}\(t\)&=\\beta\\cdot\(p\_\{t\+1\}\-p\_\{t\}\),\\\\ r\_\{\\text\{align\}\}\(t\)&=\\frac\{1\}\{T\}\\alpha\\cdot\\max\(0,1\-\|p\_\{t\}\-\\tilde\{p\}\_\{t\}\|\)\.\\end\{split\}\(11\)
RformatR\_\{\\text\{format\}\}rewards format compliance and penalizes progress boundary violations \(p1≠0p\_\{1\}\\neq 0orpT≠1p\_\{T\}\\neq 1for successful trajectories\):

Rformat=±0\.1​based on format validity,−wp​\(\|p0−0\|\+𝟙​\[success\]⋅\|pT−1\|\)\.\\begin\{split\}R\_\{\\text\{format\}\}&=\\pm 0\.1\\text\{ based on format validity\},\\\\ &\-w\_\{p\}\(\|p\_\{0\}\-0\|\+\\mathbb\{1\}\[\\text\{success\}\]\\cdot\|p\_\{T\}\-1\|\)\.\\end\{split\}\(12\)
Following GiGPO\(Fenget al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib10)\), we compute hierarchical advantages at two granularities\. The episode advantageAepisode​\(τ\)A^\{\\text\{episode\}\}\(\\tau\)normalizes the total returnR​\(τ\)R\(\\tau\)against trajectories rolled out from the same task\. The step advantageAtstepA^\{\\text\{step\}\}\_\{t\}compares discounted returnsGtG\_\{t\}among actions taken from the same anchor state, i\.e\., identical observationsok=oto\_\{k\}=o\_\{t\}that naturally recur across trajectories, where

Gt=∑k=0T−tγk​rt\+k=rt\+γ​Gt\+1\.G\_\{t\}=\\sum\_\{k=0\}^\{T\-t\}\\gamma^\{k\}r\_\{t\+k\}=r\_\{t\}\+\\gamma G\_\{t\+1\}\.\(13\)Sincertr\_\{t\}incorporates progress shaping,GtG\_\{t\}encodeswhenprogress increases occur, enabling fine\-grained credit assignment\. The final advantage combines both levels and is optimized with a clipped objective \(ρt​\(θ\)=πθ​\(at∣ot\)πold​\(at∣ot\)\\rho\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\}\)\}\{\\pi\_\{\\text\{old\}\}\(a\_\{t\}\\mid o\_\{t\}\)\}\):

At=R​\(τ\)−mean​\(\{R​\(τj\)\}j=1N\)Fnorm​\(\{R​\(τj\)\}j=1N\)⏟Aepisode​\(τ\)\+ω⋅Gt−mean​\(\{Gk∣ok=ot\}\)Fnorm​\(\{Gk∣ok=ot\}\)⏟Atstep\\displaystyle A\_\{t\}=\\underbrace\{\\frac\{R\(\\tau\)\-\\text\{mean\}\\bigl\(\\\{R\(\\tau\_\{j\}\)\\\}\_\{j=1\}^\{N\}\\bigr\)\}\{F\_\{\\text\{norm\}\}\\bigl\(\\\{R\(\\tau\_\{j\}\)\\\}\_\{j=1\}^\{N\}\\bigr\)\}\}\_\{A^\{\\text\{episode\}\}\(\\tau\)\}\+\\omega\\cdot\\underbrace\{\\frac\{G\_\{t\}\-\\text\{mean\}\\bigl\(\\\{G\_\{k\}\\mid o\_\{k\}=o\_\{t\}\\\}\\bigr\)\}\{F\_\{\\text\{norm\}\}\\bigl\(\\\{G\_\{k\}\\mid o\_\{k\}=o\_\{t\}\\\}\\bigr\)\}\}\_\{A^\{\\text\{step\}\}\_\{t\}\}

\(14\)ℒPG​\(θ\)=−𝔼τ,t​\[min⁡\(ρt​\(θ\)​At,clip​\(ρt​\(θ\),1−ϵ,1\+ϵ\)​At\)\]\.\\displaystyle\\mathcal\{L\}\_\{\\text\{PG\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\\tau,t\}\\big\[\\min\\big\(\\rho\_\{t\}\(\\theta\)A\_\{t\},\\;\\text\{clip\}\(\\rho\_\{t\}\(\\theta\),1\{\-\}\\epsilon,1\{\+\}\\epsilon\)A\_\{t\}\\big\)\\big\]\.

\(15\)

## 5Experiments

This section presents empirical results of training\-based Retrospective Progress Reflection across agentic benchmarks and foundation models, together with key empirical observations\.

### 5\.1Setup

##### Benchmarks

Three widely used agentic environments are adopted as our benchmarks\. \(1\)WebShopis an online shopping simulation where the agent must purchase a specific product matching user requirements in a simulated e\-commerce website through text actions likesearchandclick\(Yaoet al\.,[2022](https://arxiv.org/html/2606.14302#bib.bib46)\)\. Each episode is limited to 15 steps\. \(2\)ALFWorldis a text\-based household task environment where agents navigate rooms and manipulate objects to achieve goals \(e\.g\., “put a clean apple in the refrigerator”\)\(Shridharet al\.,[2020](https://arxiv.org/html/2606.14302#bib.bib47)\)\. Actions include navigation and object interactions likego toandtake/put\. Episodes are limited to 50 steps\. \(3\)Sokoban\.A puzzle environment where the agent pushes several boxes to designated target positions, given a puzzle map\(Schrader,[2018](https://arxiv.org/html/2606.14302#bib.bib61)\)\. The action space includes moving in four directions and pushing boxes when adjacent\.

The trajectory rewardRt​a​s​kR\_\{task\}of WebShop and ALFWorld is a final\-step outcome\-based reward without signals for intermediate steps\. Differently, Sokoban has hybrid rewards, combining outcome reward with a sparse intermediate reward for each box’s success\.Metricsof task Success Rate \(SR\) and Score from the environment are reported on 256 test samples\.

##### Baselines

Baseline methods for comparison are twofold\.\(1\) Prompting methods\.Following our pilot study, we evaluate prompting methods that elicit online and retrospective progress estimates during inference, with random progress included as an ablation to control for the added prompt structure\.\(2\) Training methods\.We use standard GRPODeepSeek \([2024](https://arxiv.org/html/2606.14302#bib.bib9)\)and GiGPOZhenget al\.\([2025](https://arxiv.org/html/2606.14302#bib.bib13)\)as our primary RL training baselines\. We further include a progress\-aware prompting baseline, denoted asMeta Prompt, where the agent is explicitly prompted to predict progress at each step during training, but receives no progress\-related reward\. This baseline tests whether progress assessment can emerge from naive prompting alone under training\. SinceMeta Promptuses the same progress\-aware prompting format as our method, we use it as the reference baseline for computingΔ\\Deltaacross methods\. On the basis ofMeta Prompt, two vanilla implementations of progress augmentation are evaluated\.L1denotes only adding format penalty in the reward function to theMeta Prompt, whileL2denotes directly usingp^t\\hat\{p\}\_\{t\}as the step rewardrtr\_\{t\}in Equation[13](https://arxiv.org/html/2606.14302#S4.E13)\.

Qwen2\.5\-1\.5BQwen2\.5\-3BQwen2\.5\-7BMethodSRΔ\\DeltaScoreΔ\\DeltaSRΔ\\DeltaScoreΔ\\DeltaSRΔ\\DeltaScoreΔ\\DeltaPrompting MethodsBase Prompt0\.4–5\.0–12\.5–36\.5–14\.1–50\.8–Meta Prompt for Progress1\.2\+0\.83\.8\-1\.216\.4\+3\.941\.5\+5\.010\.9\-3\.241\.4\-9\.4Random Progress0\.8\+0\.413\.2\+8\.216\.4\+3\.952\.4\+15\.910\.2\-3\.944\.4\-6\.4Retro Progress5\.1\+4\.720\.7\+15\.710\.6\-1\.944\.5\+8\.023\.8\+9\.751\.5\+0\.7Random Retro Progress2\.0\+1\.618\.5\+13\.516\.0\+3\.548\.4\+11\.912\.5\-1\.646\.5\-4\.3Training MethodsGRPOBest66\.41\-7\.0381\.22\-8\.6050\.00\-21\.8871\.02\-13\.3973\.29\-5\.6282\.18\-3\.75Avg62\.77\-4\.3877\.26\-8\.0144\.53\-23\.6363\.90\-20\.4864\.02\-10\.6376\.46\-8\.63GiGPOBest67\.97\-5\.4781\.35\-8\.4771\.88\+0\.0083\.94\-0\.4774\.22\-4\.6983\.98\-1\.95Avg59\.84\-7\.3177\.68\-7\.5963\.57\-4\.5981\.49\-2\.8971\.76\-2\.8983\.77\-1\.32Best73\.44–89\.82–71\.88–84\.41–78\.91–85\.93–Meta PromptAvg67\.15–85\.27–68\.16–84\.38–74\.65–85\.09–L1Best68\.75\-4\.6984\.51\-5\.3176\.56\+4\.6888\.53\+4\.1280\.86\+1\.9591\.74\+5\.81Avg63\.20\-3\.9582\.17\-3\.1071\.68\+3\.5286\.21\+1\.8375\.62\+0\.9787\.81\+2\.72L2Best75\.00\+1\.5688\.01\-1\.8176\.56\+4\.6888\.60\+4\.1975\.39\-3\.5287\.54\+1\.61Avg70\.47\+3\.3285\.67\+0\.4070\.86\+2\.7086\.49\+2\.1170\.59\-4\.0683\.45\-1\.64RePro \(Ours\)Best81\.64\+8\.2091\.72\+1\.9083\.59\+11\.7192\.09\+7\.6884\.38\+5\.4791\.18\+5\.25ReProAvg76\.13\+8\.9887\.56\+2\.2979\.73\+11\.5790\.75\+6\.3780\.47\+5\.8289\.13\+4\.04

Table 1:Main results on WebShop across three model sizes\. SR: success rate \(%\)\. For trained methods, we report both the best score \(Best\) and convergence window average \(Avg\)\.Δ\\Delta: improvement over the corresponding baseline\.Qwen2\.5\-1\.5B on ALFWorldQwen2\.5\-7B on ALFWorldQwen3\-4B on SokobanMethodSRΔ\\DeltaScoreΔ\\DeltaSRΔ\\DeltaScoreΔ\\DeltaSRΔ\\DeltaScoreΔ\\DeltaTraining MethodsGiGPOBest89\.84\+2\.345\.74\+0\.1095\.31\+0\.007\.82\-0\.1586\.72\+1\.566\.21\+0\.27Avg84\.38\+5\.905\.49\+1\.0290\.08\-1\.296\.34\-0\.1576\.09\+2\.034\.21\+0\.29Best87\.50–5\.64–95\.31–7\.97–85\.16–5\.94–Meta PromptAvg78\.48–4\.47–91\.37–6\.49–74\.06–3\.91–L1Best95\.70\+8\.207\.88\+2\.2499\.61\+4\.309\.78\+1\.8189\.84\+4\.686\.53\+0\.60Avg88\.05\+9\.575\.81\+1\.3497\.81\+6\.448\.83\+2\.3479\.22\+5\.164\.56\+0\.64L2Best74\.22\-13\.284\.43\-1\.2198\.83\+3\.529\.29\+1\.3288\.28\+3\.126\.44\+0\.50Avg65\.62\-12\.863\.63\-0\.8496\.60\+5\.238\.30\+1\.8180\.08\+6\.024\.79\+0\.88RePro \(Ours\)Best99\.22\+11\.729\.42\+3\.78100\.00\+4\.6910\.26\+2\.2988\.28\+3\.126\.61\+0\.67ReProAvg96\.02\+17\.547\.94\+3\.4797\.54\+6\.178\.90\+2\.4181\.64\+7\.585\.10\+1\.18

Table 2:Main results on ALFWorld and Sokoban across three model sizes\. SR: success rate \(%\)\. We report both the best score \(Best\) and convergence window average \(Avg\)\.Δ\\Delta: improvement over the corresponding baseline\.Qwen2\.5\-1\.5BQwen2\.5\-3BQwen2\.5\-7BMethodSRΔ\\DeltaScoreΔ\\DeltaSRΔ\\DeltaScoreΔ\\DeltaSRΔ\\DeltaScoreΔ\\DeltaRePro \(Ours\)RePro81\.25–91\.80–83\.98–93\.67–84\.77–94\.10–Ablationsw/ 7B Warmup data83\.59\+2\.3493\.14\+1\.3482\.81\-1\.1791\.28\-2\.3984\.77\+0\.0094\.10\+0\.00Warmup \+ GiGPO75\.00\-6\.2588\.44\-3\.3678\.91\-5\.0787\.94\-5\.7379\.69\-5\.0888\.38\-5\.72Warmup Only0\.00\-81\.250\.00\-91\.8028\.52\-55\.4632\.49\-61\.1861\.72\-23\.0571\.94\-22\.16w/o WarmupNANANANANANANANA80\.08\-4\.6989\.96\-4\.14VariantsL1\+ Reshaping72\.27\-8\.9887\.43\-4\.3772\.66\-11\.3284\.82\-8\.8580\.08\-4\.6987\.54\-6\.56L1\+ Reweighting71\.48\-9\.7783\.55\-8\.2575\.00\-8\.9887\.62\-6\.0577\.34\-7\.4388\.98\-5\.12L1\+ Grouping75\.78\-5\.4789\.63\-2\.1773\.83\-10\.1586\.32\-7\.3578\.13\-6\.6489\.23\-4\.87

Table 3:Results for ablation study and variants of our method, using the Best numbers\.

### 5\.2Main Results

##### Progress awareness does not naturally emerge from prompting or outcome\-oriented training\.

As shown in Table[1](https://arxiv.org/html/2606.14302#S5.T1), explicitly prompting models to track progress fails to improve performance over the base prompt, and often degrades it\. Retrospective prompting without training similarly yields limited gains, indicating that progress awareness cannot be reliably induced through prompting alone, especially for foundation open\-source LLMs\. Compared with GRPO and GiGPO,Meta Promptinduces longer reasoning traces and yields performance improvements in most settings\. This suggests that progress\-aware prompting provides a richer context for training, with additional tokens not necessarily becoming a burden\. We further observe that outcome\-oriented RL with progress prompt and simple leverage \(L1/L2\) produces inconsistent improvements across settings, indicating crafting prompt and shaping rewards alone are insufficient for learning stable progress awareness\.

##### RePro enables effective progress\-aware agent training\.

Table[1](https://arxiv.org/html/2606.14302#S5.T1)shows that RePro consistently outperforms all baselines on WebShop across 1\.5B, 3B, and 7B models, achieving absolute success rate gains of \+8\.98%, \+11\.57%, and \+5\.82% over theMeta Promptbaseline, respectively\. Compared to progress\-related variants, RePro further improves performance, demonstrating the effectiveness of retrospective progress modeling beyond vanilla reward engineering\. Notably, RePro achieves strong performance across diverse environments, including ALFWorld and Sokoban \(Table[2](https://arxiv.org/html/2606.14302#S5.T2)\)\.

##### RePro generalizes across model scales and task environments\.

RePro exhibits consistent gains across both model scales and environments\. On WebShop, improvements remain stable from 1\.5B to 7B models, suggesting that progress\-aware training complements model scaling rather than depending on a specific model capacity\. Across environments, RePro generalizes to both ALFWorld household planning and Sokoban spatial reasoning, supporting the hypothesis that retrospective progress awareness provides a general mechanism for long\-horizon agent learning\. On ALFWorld, RePro improves over theMeta Promptbaseline by \+11\.72 SR for Qwen2\.5\-1\.5B and \+4\.69 SR for Qwen2\.5\-7B\. On Sokoban, it also yields gains of \+3\.12 SR for Qwen3\-4B\. The improvement on Sokoban is relatively smaller, which may be because its reward already provides partial process\-oriented feedback\.

## 6Analysis

### 6\.1Ablations and Variants

We evaluate our full approach with ablations, including \(1\) Warmup Only: Performance after only warmup by supervised fine\-tuning on retrospective\-labeled data without subsequent RL training\. \(2\) Warmup \+ GiGPO: Two\-phase training with SFT warmup followed by standard GiGPO without retrospective progress mechanism\. \(3\) w/o Warmup: Directly initialize RL from base model with retrospective mechanism enabled\. The following are variants that leverage progress assessments in different ways\.

\(1\) The Role of SFT Warmup\.Warmup Onlyachieves substantially lower performance than RePro \(crashing on 1\.5B and 3B, \-23\.05% on 7B\), despite achieving nearly perfect format accuracy\. This suggests that supervised warmup teaches the*format*of retrospective reflection but does not yield effective progress awareness without task\-reward grounding\. Removing warmup destabilizes training for 1\.5B and 3B models, indicating that warmup is critical for establishing the forward\-then\-reflect behavior\. Using stronger warmup demonstrations further improves smaller models\.

\(2\) The Role of Retrospective Reflection\.Comparisons againstWarmup \+ GiGPOandL1 \+ Reshapingshow that retrospective reflection contributes beyond format initialization or reward shaping alone\. The benefit remains consistent across model scales, indicating that retrospective progress modeling scales effectively with model capacity\. Further, retrospective progress deltasΔ​p\\Delta pexhibit positive correlation with step\-level advantages in 91\.5% of training rollouts, suggesting that progress reflection provides meaningful credit assignment signals during policy optimization\.

Qwen2\.5\-1\.5BQwen2\.5\-3BQwen2\.5\-7BMetricMethodSuccFailedΔ\\DeltaSuccFailedΔ\\DeltaSuccFailedΔ\\DeltaPrimary MetricsFormatBaseline99\.7199\.78\-0\.0799\.5999\.41\+0\.1899\.2097\.57\+1\.63Ours100\.00100\.00\+0\.00100\.00100\.00\+0\.00100\.00100\.00\+0\.00IntermDisc↑\\uparrowBaseline1\.861\.11−\-1\.07Ours6\.3731\.6211\.14Analytical MetricsTemporal CorrBaseline96\.8997\.02\-0\.1321\.6218\.69\+2\.9348\.7446\.13\+2\.61Ours78\.1475\.64\+2\.5094\.7160\.72\+33\.9996\.5279\.24\+17\.28MonotonicityBaseline99\.99100\.00\-0\.0199\.3499\.32\+0\.0299\.9899\.94\+0\.04Ours99\.6099\.26\+0\.34100\.0092\.76\+7\.2499\.9995\.72\+4\.27Failure PatternsDeclineBaseline0\.040\.00\+0\.042\.723\.01\-0\.290\.150\.42\-0\.27Ours1\.282\.88\-1\.600\.0330\.58\-30\.550\.0220\.07\-20\.05PlateauBaseline0\.470\.57\-0\.1078\.4982\.18\-3\.6999\.9899\.86\+0\.12Ours27\.0833\.03\-5\.951\.3712\.77\-11\.400\.456\.54\-6\.09Table 4:Progress prediction metrics across model sizes withMeta promptas the baseline\.IntermDisc\(Intermediate Discrimination\) is the primary metric for outcome\-based tasks\.
### 6\.2Progress Predictions

#### 6\.2\.1Progress Metrics

Progress predictions lack objective ground truth in outcome\-based agentic tasks, making direct accuracy evaluation infeasible\. We therefore evaluate them through behavioral properties and outcome alignment\. Specifically, we measure whether progress values can be stably extracted \(Format Rate\) and whether they distinguish successful from failed trajectories \(Intermediate Discrimination\)\. For further analysis, we also report descriptive indicators of progress dynamics, includingTemporal Correlation,Monotonicity,Decline Rate, andPlateau Rate\. Among these metrics,Intermediate Discriminationserves as the primary metric, as it directly reflects whether progress estimates align with final task outcomes\. Detailed metric definitions are provided in Appendix[B](https://arxiv.org/html/2606.14302#A2)\.

#### 6\.2\.2Findings

Table[4](https://arxiv.org/html/2606.14302#S6.T4)presents progress prediction analysis across different model sizes, comparing our RePro against the baseline\. We focus analysis on retrospective progress, which serves as the supervision signal in our framework\.

##### Our method improves the progress prediction compared to the baseline\.

The critical metric,Intermediate Discrimination \(IntermDisc\), reveals stark differences\. Baseline discrimination ranges from−1\.07\-1\.07to1\.861\.86\(1\.5B\)\. RePro achieves substantially higher discrimination:6\.376\.37\(1\.5B\),31\.6231\.62\(3B\), and11\.1411\.14\(7B\)\. The 3B model shows particularly strong outcome awareness, with successful trajectories averaging 31 percentage points higher final progress than failures\.

##### The task achievement signal is observable\.

The auxiliary metrics show that task achievement signals differ systematically\. The analytical metrics further show more coherent progress dynamics on successful trajectories\. For our method, successful trajectories consistently obtain higher temporal correlation than failed ones, with gaps of \+2\.50, \+33\.99, and \+17\.28, and higher monotonicity with gaps of \+0\.34, \+7\.24, and \+4\.27\. Failure\-pattern metrics provide complementary evidence: baseline predictions rarely decline, but often collapse into plateau behavior, especially on 3B and 7B, where failed trajectories have plateau rates of 82\.18 and 99\.86\. In contrast, RePro yields non\-trivial decline rates on failed trajectories, 2\.88/30\.58/20\.07, while reducing plateau rates on 3B and 7B to 12\.77 and 6\.54\. These results indicate that the learned progress signal retains meaningful variation and can reflect setbacks even when the task fails\.

## 7Conclusion

This paper studies progress awareness as a metacognitive signal for long\-horizon LLM\-based agents\. We show that while progress estimates can help agent execution when grounded in retrospective trajectory evidence, they do not reliably emerge from naive prompting or standard outcome\-reward training\. Motivated by this asymmetry, we proposeRePro, a retrospective progress\-aware training framework that learns progress signals from completed trajectories and uses them as auxiliary supervision for agent training\. Through a forward\-then\-reflect paradigm, Retrospection Warmup, and RePro\-PO, RePro enables agents to incorporate progress\-aware feedback without relying on continuous external supervision or an additional reward model\. Experiments show consistent improvements across model scales and environments, with further analyses indicating improved progress estimation quality\. Ultimately, these findings demonstrate the potential of progress awareness not only for execution, but also for fostering internal cognition within long\-horizon tasks, opening promising avenues for future research in autonomous self\-reflective agents\.

## Limitations

RePro currently represents progress awareness through verbalized progress assessments\. While this makes the signal interpretable and easy to supervise, other forms of progress representation, such as latent states or hidden\-vector supervision, remain unexplored\. Our evaluation covers representative long\-horizon environments, including WebShop, ALFWorld, and Sokoban, but resource and infrastructure constraints prevent us from scaling to broader and more complex agent settings such as GUI automation or open\-ended embodied interaction\. Finally, progress awareness lacks naturally available ground\-truth labels\. RePro derives progress signals retrospectively from completed trajectories and outcomes, but future work could use human annotations or strong\-LLM evaluations as pseudo\-gold references to better calibrate progress estimation\.

## References

- E\. C\. Acikgoz, C\. Qian, H\. Ji, D\. Hakkani\-Tür, and G\. Tur \(2025\)Self\-improving llm agents at test\-time\.arXiv preprint arXiv:2510\.07841\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- M\. Allard, A\. Teinturier, V\. Xing, and G\. Viaud \(2026\)Experiential reflective learning for self\-improving llm agents\.arXiv preprint arXiv:2603\.24639\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- S\. Bensal, U\. Jamil, C\. Bryant, M\. Russak, K\. Kamble, D\. Mozolevskyi, M\. Ali, and W\. AlShikh \(2025\)Reflect, retry, reward: self\-improving llms via reinforcement learning\.arXiv preprint arXiv:2505\.24726\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p2.1)\.
- Y\. Cai, Y\. Hao, J\. Zhou, H\. Yan, Z\. Lei, R\. Zheng, Z\. Han, Y\. Yang, J\. Li, Q\. Pan, T\. Huai, Q\. Chen, K\. Chen, B\. Zhang, X\. Qiu, and L\. He \(2026\)A benchmark for self\-evolving agents via experience\-driven lifelong learning\.External Links:[Link](https://openreview.net/forum?id=vznmtmUPmA)Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- Z\. Cao, R\. Wang, Y\. Yang, X\. Ma, X\. Zhu, B\. Zheng, and H\. Zhao \(2025\)Pgpo: enhancing agent reasoning via pseudocode\-style planning guided preference optimization\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 14966–14985\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- H\. Chai, Z\. Cao, M\. Ran, Y\. Yang, J\. Lin, X\. Peng, H\. Wang, R\. Ding, Z\. Wan, M\. Wen,et al\.\(2025\)PARL\-mt: learning to call functions in multi\-turn conversation with progress awareness\.arXiv preprint arXiv:2509\.23206\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§3](https://arxiv.org/html/2606.14302#S3.p1.1)\.
- DeepSeek \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.Vol\.abs/2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1),[§5\.1](https://arxiv.org/html/2606.14302#S5.SS1.SSS0.Px2.p1.3)\.
- G\. Dong, L\. Bao, Z\. Wang, K\. Zhao, X\. Li, J\. Jin, J\. Yang, H\. Mao, F\. Zhang, K\. Gai, G\. Zhou, Y\. Zhu, J\. Wen, and Z\. Dou \(2025a\)Agentic entropy\-balanced policy optimization\.CoRRabs/2510\.14545\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.14545),[Document](https://dx.doi.org/10.48550/ARXIV.2510.14545),2510\.14545Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- G\. Dong, H\. Mao, K\. Ma, L\. Bao, Y\. Chen, Z\. Wang, Z\. Chen, J\. Du, H\. Wang, F\. Zhang, G\. Zhou, Y\. Zhu, J\. Wen, and Z\. Dou \(2025b\)Agentic reinforced policy optimization\.CoRRabs/2507\.19849\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.19849),[Document](https://dx.doi.org/10.48550/ARXIV.2507.19849),2507\.19849Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- L\. Feng, Z\. Xue, T\. Liu, and B\. An \(2026\)Group\-in\-group policy optimization for LLM agent training\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=QXEhBMNrCW)Cited by:[Appendix A](https://arxiv.org/html/2606.14302#A1.p3.1),[§1](https://arxiv.org/html/2606.14302#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1),[§4\.4](https://arxiv.org/html/2606.14302#S4.SS4.p4.5)\.
- Y\. Ge, S\. Romeo, J\. Cai, M\. Sunkara, and Y\. Zhang \(2025\)Samule: self\-learning agents enhanced by multi\-level reflection\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 16602–16621\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- D\. Guo, T\. Zhou, D\. Liu, C\. Qian, Q\. Ren, S\. Shao, Z\. Fan, Y\. R\. Fung, K\. Wang, L\. Zhang, and J\. Shao \(2026\)Towards self\-evolving agent benchmarks : validatable agent trajectory via test\-time exploration\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2H03gm4Rq6)Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- T\. Han, Z\. Wang, C\. Fang, S\. Zhao, S\. Ma, and Z\. Chen \(2025\)Token\-budget\-aware LLM reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 24842–24855\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1274/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1274),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- L\. P\. Kaelbling, M\. L\. Littman, and A\. R\. Cassandra \(1998\)Planning and acting in partially observable stochastic domains\.Artificial intelligence101\(1\-2\),pp\. 99–134\.Cited by:[§4\.1](https://arxiv.org/html/2606.14302#S4.SS1.p1.7)\.
- J\. Kim, S\. Rhee, M\. Kim, D\. Kim, S\. Lee, Y\. Sung, and K\. Jung \(2025\)Reflact: world\-grounded decision making in llm agents via goal\-state reflection\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 33421–33453\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- \[18\]A\. Kumar, V\. Zhuang, R\. Agarwal, Y\. Su, J\. D\. Co\-Reyes, A\. Singh, K\. Baumli, S\. Iqbal, C\. Bishop, R\. Roelofs,et al\.Training language models to self\-correct via reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p2.1)\.
- J\. Li, X\. Dong, Y\. Liu, Z\. Yang, Q\. Wang, X\. Wang, S\. Zhu, Z\. Jia, and Z\. Zheng \(2025a\)Reflectevo: improving meta introspection of small llms by learning self\-reflection\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 16948–16966\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p2.1)\.
- Z\. Li, Q\. Dong, J\. Ma, D\. Zhang, K\. Jia, and Z\. Sui \(2025b\)SelfBudgeter: adaptive token allocation for efficient LLM reasoning\.External Links:[Link](https://openreview.net/forum?id=e7EBzbi8Qd)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=v8L0pN6EOi)Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- J\. Lin, X\. Zeng, J\. Zhu, S\. Wang, J\. Shun, J\. Wu, and D\. Zhou \(2026\)Plan and budget: effective and efficient test\-time scaling on reasoning large language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ctspw4CqbS)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- Z\. Liu, J\. Xie, Z\. Ding, Z\. Li, B\. Yang, Z\. Wu, X\. Wang, Q\. Sun, S\. Liu, W\. Wang, S\. Ye, Q\. Li, Z\. Tian, G\. Luo, X\. Yue, B\. Qi, K\. Chen, B\. Zhou, Y\. Qiao, Q\. Chen, and W\. Wang \(2026\)ScaleCUA: scaling open\-source computer use agents with cross\-platform data\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yBFUqdJFZn)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- Z\. Lu, Y\. Chai, Y\. Guo, X\. Yin, L\. Liu, H\. Wang, H\. Xiao, S\. Ren, P\. Zhao, G\. Liu,et al\.\(2026\)Ui\-r1: enhancing efficient action prediction of gui agents by reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 17608–17616\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu, Z\. Chen, Y\. Liu, R\. Zhang, L\. L\. Chen, A\. Kashyap, J\. Uslu, J\. Li, J\. Wu, M\. Yan, S\. Bian, V\. Sharma, K\. Sun, S\. Dillmann, A\. Anand, A\. Lanpouthakoun, B\. Koopah, C\. Hu, E\. K\. Guha, G\. H\. S\. Dreiman, J\. Zhu, K\. Krauth, L\. Zhong, N\. Muennighoff, R\. K\. Amanfu, S\. Tan, S\. Pimpalgaonkar, T\. Aggarwal, X\. Lin, X\. Lan, X\. Zhao, Y\. Liang, Y\. Wang, Z\. Wang, C\. Zhou, D\. Heineman, H\. Liu, H\. Trivedi, J\. Yang, J\. Lin, M\. Shetty, M\. Yang, N\. Omi, N\. Raoof, S\. Li, T\. Y\. Zhuo, W\. Lin, Y\. Dai, Y\. Wang, W\. Chai, S\. Zhou, D\. Wahdany, Z\. She, J\. Hu, Z\. Dong, Y\. Zhu, S\. Cui, A\. Saiyed, A\. Kolbeinsson, C\. M\. Rytting, R\. Marten, Y\. Wang, J\. Jitsev, A\. Dimakis, A\. Konwinski, and L\. Schmidt \(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=a7Qa4CcHak)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- W\. Ou, Y\. Zheng, S\. Sun, W\. Zhang, B\. Dong, H\. Zhu, R\. Huang, G\. Yu, P\. Yan, and Y\. Qiao \(2026\)Serl: self\-examining reinforcement learning on open\-domain\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 32619–32627\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p2.1)\.
- C\. Qian, E\. C\. Acikgoz, Q\. He, H\. WANG, X\. Chen, D\. Hakkani\-Tür, G\. Tur, and H\. Ji \(2026\)ToolRL: reward is all tool learning needs\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=eOLdGbXT6t)Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- Y\. Qu, T\. Zhang, N\. Garg, and A\. Kumar \(2024\)Recursive introspection: teaching language model agents how to self\-improve\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=DRC9pZwBwR)Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- M\. B\. Schrader \(2018\)Gym\-sokoban\.GitHub\.Note:[https://github\.com/mpSchrader/gym\-sokoban](https://github.com/mpSchrader/gym-sokoban)Cited by:[§5\.1](https://arxiv.org/html/2606.14302#S5.SS1.SSS0.Px1.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[Appendix A](https://arxiv.org/html/2606.14302#A1.p3.2)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. R\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=vAElhFcKW6)Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2020\)Alfworld: aligning text and embodied environments for interactive learning\.arXiv preprint arXiv:2010\.03768\.Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1),[§5\.1](https://arxiv.org/html/2606.14302#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Song, D\. Yin, X\. Yue, J\. Huang, S\. Li, and B\. Y\. Lin \(2024\)Trial and error: exploration\-based trajectory optimization of llm agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7584–7600\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- H\. Wang, Y\. Cao, L\. Lin, and J\. Chen \(2026a\)PreFlect: from retrospective to prospective reflection in large language model agents\.arXiv preprint arXiv:2602\.07187\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- Q\. Wang, W\. Huang, Y\. Zhou, H\. Yin, T\. Bao, J\. Lyu, W\. Liu, R\. Zhang, J\. Wu, L\. Fei\-Fei, and M\. Li \(2026b\)ENACT: evaluating embodied cognition with world modeling of egocentric interaction\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Patx6MRipw)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- T\. Wang, S\. Gooding, F\. Hartmann, O\. Riva, and E\. Grefenstette \(2026c\)A subgoal\-driven framework for improving long\-horizon llm agents\.arXiv preprint arXiv:2603\.19685\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu, E\. Gottlieb, Y\. Lu, K\. Cho, J\. Wu, L\. Fei\-Fei, L\. Wang, Y\. Choi, and M\. Li \(2025\)RAGEN: understanding self\-evolution in llm agents via multi\-turn reinforcement learning\.External Links:2504\.20073,[Link](https://arxiv.org/abs/2504.20073)Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- Z\. Xi, J\. Huang, C\. Liao, B\. Huang, J\. Liu, H\. Guo, yajie yang, R\. Zheng, J\. Ye, J\. Zhang, W\. Chen, W\. He, Y\. Ding, G\. Li, Z\. Chen, Z\. Du, X\. Yao, Y\. Xu, J\. Chen, T\. Gui, Z\. Wu, Q\. Zhang, X\. Huang, and Y\. Jiang \(2026a\)AgentGym\-RL: an open\-source framework to train LLM agents for long\-horizon decision making via multi\-turn RL\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZgCCDwcGwn)Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- Z\. Xi, C\. Liao, G\. Li, Z\. Zhang, W\. Chen, B\. Wang, S\. Jin, Y\. Zhou, J\. Guan, W\. Wu,et al\.\(2026b\)Agentprm: process reward models for llm agents via step\-wise promise and progress\.InProceedings of the ACM Web Conference 2026,pp\. 4184–4195\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.\(2024\)Osworld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- W\. Xiong, Y\. Song, X\. Zhao, W\. Wu, X\. Wang, K\. Wang, C\. Li, W\. Peng, and S\. Li \(2024\)Watch every step\! llm agent learning via iterative step\-level process refinement\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 1556–1572\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan \(2022\)Webshop: towards scalable real\-world web interaction with grounded language agents\.Advances in Neural Information Processing Systems35,pp\. 20744–20757\.Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1),[§3](https://arxiv.org/html/2606.14302#S3.p1.1),[§5\.1](https://arxiv.org/html/2606.14302#S5.SS1.SSS0.Px1.p1.1)\.
- B\. Ye, R\. Li, Q\. Yang, Y\. Liu, L\. Yao, H\. Lv, Z\. Xie, C\. An, L\. Li, L\. Kong, Q\. Liu, Z\. Sui, and T\. Yang \(2026\)Claw\-eval: towards trustworthy evaluation of autonomous agents\.External Links:2604\.06132,[Link](https://arxiv.org/abs/2604.06132)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- D\. Zhang, S\. Zhang, Z\. Yang, Z\. Zhu, Z\. Zhao, R\. Cao, L\. Chen, and K\. Yu \(2025a\)Progrm: build better gui agents with progress rewards\.arXiv preprint arXiv:2505\.18121\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- K\. Zhang, Q\. YAO, S\. Liu, Y\. Wang, B\. Lai, J\. Ye, M\. Song, and D\. Tao \(2026a\)Consistent paths lead to truth: self\-rewarding reinforcement learning for LLM reasoning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=ckW70ls93V)Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p2.1)\.
- Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li,et al\.\(2025b\)Agentic context engineering: evolving contexts for self\-improving language models\.arXiv preprint arXiv:2510\.04618\.Cited by:[§2\.2](https://arxiv.org/html/2606.14302#S2.SS2.p1.1)\.
- S\. Zhang, X\. Ma, Y\. Xu, Z\. Cao, P\. Lu, H\. Yuan, T\. Shen, Z\. Zhang, H\. Zhao, and M\. Yang \(2025c\)ParaCook: on time\-efficient planning for multi\-agent systems\.External Links:2510\.11608,[Link](https://arxiv.org/abs/2510.11608)Cited by:[§1](https://arxiv.org/html/2606.14302#S1.p1.1)\.
- X\. Zhang, N\. Gao, X\. Jiang, Y\. Chen, Y\. Pan, M\. Zhang, and Y\. Deng \(2026b\)Progress reward model for reinforcement learning via large language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=TJhHb6CscW)Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.\(2025\)Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2606.14302#S5.SS1.SSS0.Px2.p1.3)\.
- C\. Zheng, X\. Mo, X\. Ma, Q\. Lin, Y\. Zhao, J\. Zhu, X\. Lou, J\. Wang, Z\. Wang, W\. Liu,et al\.\(2026\)Adaptive milestone reward for gui agents\.arXiv preprint arXiv:2602\.11524\.Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.
- J\. Zou, L\. Yang, J\. Gu, J\. Qiu, K\. Shen, J\. He, and M\. Wang \(2026\)ReasonFlux\-PRM: trajectory\-aware PRMs for long chain\-of\-thought reasoning in LLMs\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=f3sZjkQbv2)Cited by:[§2\.1](https://arxiv.org/html/2606.14302#S2.SS1.p2.1)\.

## Appendix AImplementation

Retrospective Progress Reflection PromptYou have just completed a task on an online shopping website\. Now review the trajectory with the benefit of hindsight\.Your trajectory \(4 steps\): Step 1: action="search\[wireless mouse under 20 dollars\]" your\_estimate=10% Step 2: action="click\[item \- Logitech wireless mouse\]" your\_estimate=35% Step 3: action="click\[add to cart\]" your\_estimate=70% Step 4: action="click\[buy now\]" your\_estimate=90% Final outcome: SUCCEEDED \(score=1\.00\) Re\-assess what fraction of the task was genuinely completed at each step\. Guidelines: \- Step 1 should be near 0 \(task just started\)\. \- If SUCCEEDED, the last step should be near 100\. \- Values should generally be non\-decreasing\. First reason briefly in <think\>\.\.\.</think\>, then output exactly 4 integers \(0\-\-100\) in <hindsight\_progress\>v1,v2,v3,v4 </hindsight\_progress\>\. Example: <think\>step 1 started, step 2 found item, step 3 bought it\.</think\> <hindsight\_progress\>0,60,100 </hindsight\_progress\>

\(1\) SFT Warmup \(Phase 1\) We construct hindsight\-labeled data as described in §[4\.3](https://arxiv.org/html/2606.14302#S4.SS3):

- •Data source: Successful trajectories \(score≥0\.9\\geq 0\.9for WebShop, success=True for ALFWorld\) from pre\-trained GiGPO checkpoints
- •Hindsight labeling: DeepSeek\-v4 API \(temperature=0\) generates hindsight progress sequences
- •Training: Standard SFT with learning rate5×10−65\\times 10^\{\-6\}, batch size 8, 3 epochs\.

\(2\)RL Training \(Phase 2\)\.Following GiGPO\(Fenget al\.,[2026](https://arxiv.org/html/2606.14302#bib.bib10)\)with modifications for hindsight progress reflection:

- •Rollout: 8 trajectories per batch, with a maximum horizon of 15 steps for WebShop\.
- •PPO hyperparameters:ϵclip=0\.2\\epsilon\_\{\\text\{clip\}\}=0\.2,γ=0\.95\\gamma=0\.95, and KL coefficientβKL=0\.01\\beta\_\{\\text\{KL\}\}=0\.01\.
- •Learning rate:1×10−61\\times 10^\{\-6\}\.
- •Training steps: 150 iterations\.
- •Progress reward weights:β=0\.1\\beta=0\.1for hindsight progress shaping,α=0\.2\\alpha=0\.2for online\-retrospective alignment,αcorr=0\.3\\alpha\_\{\\mathrm\{corr\}\}=0\.3for episode\-level correlation, and±0\.1\\pm 0\.1for format reward/penalty\.

All RL experiments use full\-parameter fine\-tuning with the same configuration as SFT\. We train on PPU using the veRL framework\(Shenget al\.,[2024](https://arxiv.org/html/2606.14302#bib.bib64)\)\.

## Appendix BProgress Metrics

Progress predictions lack objective ground truth in outcome\-based tasks, precluding direct accuracy assessment\. We instead evaluate predictions through behavioral properties and outcome alignment, progressptp\_\{t\}\(generated after observing outcomes\)\. All metrics are computed separately for successful \(rT≥9\.0r\_\{T\}\\geq 9\.0\) and failed \(rT<1\.0r\_\{T\}<1\.0\) trajectories where applicable\.

##### Primary Metrics\.

Format Rate \(Format\)\.Percentage of steps with valid progress extraction:

Format=1\|ℰ\|​∑e∈ℰ\|\{t:pte≠null\}\|Te\\text\{Format\}=\\frac\{1\}\{\|\\mathcal\{E\}\|\}\\sum\_\{e\\in\\mathcal\{E\}\}\\frac\{\|\\\{t:p\_\{t\}^\{e\}\\neq\\text\{null\}\\\}\|\}\{T\_\{e\}\}\(16\)whereℰ\\mathcal\{E\}denotes the set of episodes andTeT\_\{e\}is the length of episodeee\. Measures the stability of progress output format across training\.

Intermediate Discrimination \(IntermDisc\)\.Gap between mean final progress of successful versus failed trajectories:

IntermDisc=𝔼​\[pT∣rT≥9\.0\]−𝔼​\[pT∣rT<1\.0\]\\text\{IntermDisc\}=\\mathbb\{E\}\[p\_\{T\}\\mid r\_\{T\}\\geq 9\.0\]\-\\mathbb\{E\}\[p\_\{T\}\\mid r\_\{T\}<1\.0\]\(17\)This is the most critical metric for outcome\-based tasks, measuring whether agents distinguish task outcomes through progress predictions\.

##### Analytical Metrics\.

Temporal Correlation \(Corr\)\.Pearson correlation between progress and relative step position:

Corr=corr​\(pt,tT\)\\text\{Corr\}=\\text\{corr\}\\left\(p\_\{t\},\\frac\{t\}\{T\}\\right\)\(18\)For successful trajectories, progress should increase with task advancement, yielding a positive correlation \(expected\>0\.7\>0\.7\)\. Failed trajectories may exhibit weaker correlation due to plateau patterns\.

Monotonicity Rate \(Mono\)\.Percentage of non\-decreasing transitions \(evaluated on successful trajectories only\):

Mono=1T−1​∑t=1T−1𝟙​\[pt\+1≥pt\]\\text\{Mono\}=\\frac\{1\}\{T\-1\}\\sum\_\{t=1\}^\{T\-1\}\\mathbb\{1\}\[p\_\{t\+1\}\\geq p\_\{t\}\]\(19\)Successful task execution should exhibit monotonically increasing progress\. Expected value exceeds 85% for coherent predictions\. Not evaluated on failed trajectories, where plateau or decline may reflect accurate failure awareness\.

Decline Rate \(Decline\)\.Percentage of trajectories with substantial backward progress \(descriptive, computed on failed trajectories\):

Decline=1\|ℰfail\|∑e∈ℰfail𝟙\[∃t:pt\+1e−pte<−10\]\\text\{Decline\}=\\frac\{1\}\{\|\\mathcal\{E\}\_\{\\text\{fail\}\}\|\}\\sum\_\{e\\in\\mathcal\{E\}\_\{\\text\{fail\}\}\}\\mathbb\{1\}\[\\exists t:p\_\{t\+1\}^\{e\}\-p\_\{t\}^\{e\}<\-10\]\(20\)Measures the frequency of progress regression\. While undesirable in successful trajectories \(hence evaluated via Mono\), decline in failed trajectories may indicate accurate recognition of setbacks\.

Plateau Rate \(Plateau\)\.Percentage of trajectories with small overall progress range \(descriptive, computed on failed trajectories\):

Plateau=1\|ℰfail\|​∑e∈ℰfail𝟙​\[maxt⁡pte−mint⁡pte<20\]\\text\{Plateau\}=\\frac\{1\}\{\|\\mathcal\{E\}\_\{\\text\{fail\}\}\|\}\\sum\_\{e\\in\\mathcal\{E\}\_\{\\text\{fail\}\}\}\\mathbb\{1\}\[\\max\_\{t\}p\_\{t\}^\{e\}\-\\min\_\{t\}p\_\{t\}^\{e\}<20\]\(21\)Identifies trajectories where progress remains stagnant\. Distinct from Collapse \(10% threshold\), Plateau uses a 20% threshold to capture trajectories showing minimal advancement without complete degeneration\.

All metrics are computed separately for online progressptp\_\{t\}, retrospective progressptp\_\{t\}, and across success/failure partitions, yielding up to four values per metric for comprehensive characterization\.

## Appendix CThe Use of Large Language Models

LLMs were used solely in an auxiliary capacity, primarily for linguistic refinement \(e\.g\., grammar correction, improved clarity, and removal of non\-academic expressions\)\. Importantly, LLMs werenotinvolved in generating research ideas, designing experiments, or conducting literature reviews\. All conceptual contributions, experimental designs, and methodological decisions were entirely conceived and executed by the authors\.