以退为进：面向长时程智能体的回溯探索与演进

# Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agents
Source: [https://arxiv.org/html/2610.13058](https://arxiv.org/html/2610.13058)
Ziang Ye††thanks:Work done during internship at Meituan\.Affiliation:University of Science and Technology of China Shanghai Innovation InstituteEmail:[yza03@mail\.ustc\.edu\.cn](mailto:[email protected])Lei ChenEmail:[fulifeng93@gmail\.com](mailto:[email protected])Zhengzhou CaiEmail:[caoyixin2011@gmail\.com](mailto:[email protected])Yu WangZhiyuan YaoZihan WangXierui Song††thanks:Corresponding authorsFuli FengYixin Cao22footnotemark:2Affiliation:Facebook Zhejiang University Tsinghua University Meituan Fudan University

###### Abstract

Large language model agents increasingly operate in long\-horizon environments where early mistakes can trigger cascading failures and trap agents in dead\-end trajectories\. Agents must therefore be able to proactively recover from such states and turn their exploratory experience into guidance for subsequent decisions\. We introduceReTRAIL, a framework that enables agents to rewind the environment to earlier states while preserving valid progress and carrying forward knowledge learned from abandoned exploration\. The environment state moves backward while the agent’s knowledge continues to move forward\. ReTRAIL trains a shared policy with two complementary signals: State\-Anchored GRPO compares alternative continuations from revisited states to improve execution, while a branch\-advancement reward guides reflection through the progress of subsequent exploration\. Experiments across four interactive benchmark MineSweeper, Sokoban, ALFWorld and ScienceWorld show that simply adding a rewind primitive provides limited benefit and can even degrade performance\. In contrast, ReTRAIL enables agents to rewind from unproductive states and use insights from abandoned branches to guide subsequent exploration\. These results demonstrate the value of coupling environment rewinding with experience reuse within an ongoing task for long\-horizon task solving\. The code is available at[https://github\.com/whi497/ReTRAIL](https://github.com/whi497/ReTRAIL)\.

## 1Introduction

Large language model \(LLM\) agents are rapidly evolving from conversational systems into autonomous decision\-making agents capable of solving long\-horizon tasks through extended interactions, diverse tool use, and complex state transitions\([Jimenez et al\., 2024](https://arxiv.org/html/2610.13058#bib.bib33);[Sun et al\., 2026b](https://arxiv.org/html/2610.13058#bib.bib23);[He et al\., 2026b](https://arxiv.org/html/2610.13058#bib.bib56)\)\. As task complexity increases, agents inevitably encounter unfamiliar constraints and make early mistakes that may alter the environment, trigger cascading failures, and accumulate irrelevant information into the context\([Zhou et al\., 2024](https://arxiv.org/html/2610.13058#bib.bib4);[Ye et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib20)\)\. Recovering from such failures requires agents to revisit earlier viable environment states without discarding either valid progress or useful experience\. Crucially, returning to an earlier state should not mean returning to the same decisions: agents must distill what they have learned from abandoned exploration and use it to guide a more informed continuation\.

Prior work has sought to guide LLMs to recover from errors and learn from unsuccessful history through reflection, repeated trials, and backtracking\. Interleaving reasoning with acting lets an agent revise its plan at each step, but the revision stays local and the execution keeps moving forward\([Yao et al\., 2023](https://arxiv.org/html/2610.13058#bib.bib27);[Choi et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib5)\)\. Cross\-episode adaptation uses accumulated experience to guide new attempts\([Shinn et al\., 2023](https://arxiv.org/html/2610.13058#bib.bib9);[Ouyang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib10);[Jiang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib11)\), but applies it after an episode terminates and execution restarts from the initial state, requiring earlier progress to be repeated\. Backtracking and trajectory\-revision methods enable agents to reconsider earlier decisions through tree search, prompted controllers, or supervised correction traces\([Yuan et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib7);[Li et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib94);[Zhang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib8)\)\. A central challenge is to train agents to proactively recover within an ongoing task and use the experience gained from abandoned branches to improve subsequent exploration\. This calls for learning signals that improve both execution after a rewind and the experience carried forward\.

Our key observation is that environment states can be rewound while the knowledge gained from exploration should be accumulated\. An unsuccessful branch may reveal hidden constraints, ineffective actions, or promising alternatives, allowing the agent to revisit an earlier state with a better\-informed context\. We introduceReTRAIL, a framework that exposes an environment\-levelrewindinterface to the agent, allowing it to restore earlier environment states\. Building on this interface, we proposeRetrospective Exploration: the agent proactively leaves an unproductive branch, rewinds the environment to an earlier state, distills useful information from the abandoned branch into a compact reflection, and resumes execution with the preserved execution prefix and accumulated experience\. To jointly develop the agent’s ability to rewind and distill useful experience from abandoned branches, we useState\-Anchored GRPOto improve execution by comparing alternative continuations from the same environment state, and abranch\-advancement rewardto train reflection based on the relative progress of subsequent exploration\.

We evaluate ReTRAIL across four diverse interactive benchmarks: MineSweeper\([Li et al\., 2024](https://arxiv.org/html/2610.13058#bib.bib2)\), Sokoban\([Racanière et al\., 2017](https://arxiv.org/html/2610.13058#bib.bib19)\), ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2610.13058#bib.bib17)\), and ScienceWorld\([Wang et al\., 2022](https://arxiv.org/html/2610.13058#bib.bib18)\)\. ReTRAIL achieves the highest success rates among the evaluated methods when rewind is enabled\. Our experiments reveal that simply providing existing agents with a rewind operation does not consistently improve performance and can even degrade it\. In contrast, ReTRAIL benefits from proactive rewinding across all four benchmarks and achieves further gains through reflection\-guided retries\. On MineSweeper, success increases from 58\.8% without rewind to 63\.5% with proactive rewinding, and reaches 95\.3% with up to three reflection\-guided attempts, compared with 70\.8% for the strongest baseline under the same evaluation protocol\. Further analysis shows that retry gains are larger on longer tasks and when abandoned branches reveal information useful for subsequent decisions\. These findings highlight the value of learning to rewind and carry forward useful experience, enabling unsuccessful exploration to inform continued progress within an ongoing long\-horizon task\.

Our main contributions are summarized as follows:

- •We introduceReTRAIL, a framework forRetrospective Explorationthrough an environment\-level rewind interface\. Agents proactively revisit earlier states while preserving valid progress and carrying forward experience distilled from abandoned branches to guide subsequent exploration\.
- •We develop two complementary learning signals for a shared execution and reflection policy\.State\-Anchored GRPOcompares alternative continuations from the same environment state to improve execution, while abranch\-advancement rewardtrains reflection based on the relative progress of subsequent exploration\.
- •We evaluate ReTRAIL across four interactive benchmarks, demonstrating improved success through proactive rewinding and reflection\-guided retries\. Further experiments show larger gains on longer and harder tasks, highlighting the value of retaining experience to guide subsequent exploration\.

## 2Related Work

### 2\.1Long\-Horizon Execution and Recovery

Sustaining reliable execution over long interaction sequences motivates research on planning, context management, and error recovery in LLM agents\([Yao et al\., 2023](https://arxiv.org/html/2610.13058#bib.bib27);[Zhang et al\., 2023](https://arxiv.org/html/2610.13058#bib.bib44);[Yuan et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib7)\)\. Search\-based methods explore alternative action sequences within context\([Zhou et al\., 2024](https://arxiv.org/html/2610.13058#bib.bib4);[Choi et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib5)\)\. Agent\-R learns decision correction from revision traces\([Yuan et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib7)\)\. However, revising decisions through context alone does not reverse changes to the environment, which can leave agents trapped in unproductive states\. WebRollback uses action, critique, and rollback modules to select recovery points and carry forward experience summaries, with prompting and supervised fine\-tuning from demonstrations\([Zhang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib8)\)\. GA\-Rollback uses a prompted assistant to detect and localize errors during inference, restores earlier environment states through reset\-and\-replay, and retains error feedback to guide subsequent actions\([Li et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib94)\)\. ReTRAIL take a step forward to train a shared execution and reflection policy that couples environment\-level rewind with reinforcement learning, enabling more diverse exploration and more effective interaction budgets using\.

### 2\.2Test\-Time Adaptation and Self\-Evolving Agents

Adapting from interaction experience enables LLM agents to improve their behavior in unfamiliar tasks and environments\. Reflection and memory mechanisms distill past trajectories into feedback and reusable strategies that guide subsequent decisions\([Shinn et al\., 2023](https://arxiv.org/html/2610.13058#bib.bib9);[Ouyang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib10)\)\. Beyond memory accumulation, EvoTest and Meta\-TTL adapt prompts, memory, and tool\-use routines from episode feedback\([He et al\., 2026c](https://arxiv.org/html/2610.13058#bib.bib13);[Lou et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib14)\)\. Recent work further uses reinforcement learning to optimize how agents explore and adapt from experience\. LaMer and MR\-Search train exploration and post\-episode reflection through meta\-reinforcement learning, using discounted outcome reward to provide feedback for adaptation\([Jiang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib11);[Xiao et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib12)\)\. Learning to Self\-Evolve rewards context revisions according to downstream performance improvements\([Chen et al\., 2026b](https://arxiv.org/html/2610.13058#bib.bib15)\)\. ReTRAIL brings experience\-informed adaptation into recovery within an ongoing task\. By revisiting earlier environment states while preserving valid progress and distilled branch experience, agents can revise their exploration before the current attempt terminates, with subsequent branch progress providing a learning signal for reflection\.

## 3Retrospective Exploration

We formalizeRetrospective Exploration, in which an agent rewinds to an earlier environment state while preserving the execution prefix and retaining experience from abandoned branches\. We first introduce agent interaction and multi\-episode adaptation, then describe how an environment\-levelrewindinterface enables experience\-informed recovery within an ongoing task\.

![Refer to caption](https://arxiv.org/html/2610.13058v1/flow_retrail_styled.png)Figure 1:Retrospective exploration with retained experience\.Multi\-episode adaptation \(top\) restarts the task froms0s\_\{0\}after an unsuccessful attempt, carrying retained experience into a new episode\. In contrast, ReTRAIL \(bottom\) proactively abandons an unproductive branch, distills its experience, rewinds to an earlier state, and resumes with the retained execution prefix and accumulated experience\. Recovery can occur before terminal failure\.##### Agent Interaction and Multi\-Episode Adaptation\.

Given a taskuu, an LLM agent starts from environment states0s\_\{0\}\. At steptt, it observesoto\_\{t\}and selects an actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}based on the interaction historyht=\(o0,a0,…,at−1,ot\)h\_\{t\}=\(o\_\{0\},a\_\{0\},\\ldots,a\_\{t\-1\},o\_\{t\}\)\. The environment transitions tost\+1s\_\{t\+1\}, returning rewardrtr\_\{t\}and observationot\+1o\_\{t\+1\}\. An episode yields a trajectoryτ=\(s0,a0,r0,…,aT−1,rT−1,sT\)\\tau=\(s\_\{0\},a\_\{0\},r\_\{0\},\\ldots,a\_\{T\-1\},r\_\{T\-1\},s\_\{T\}\), ending upon task completion, environment termination, or exhaustion of its interaction budget\.

Multi\-episode adaptation groups successive attempts into a trial𝒯=\(τ\(0\),…,τ\(N−1\)\)\\mathcal\{T\}=\(\\tau^\{\(0\)\},\\ldots,\\tau^\{\(N\-1\)\}\)\([Jiang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib11)\)\. Each episode restarts froms0s\_\{0\}, while the policy conditions on experienceℰ\(n\)\\mathcal\{E\}^\{\(n\)\}retained from preceding episodes:

at\(n\)∼πθ\(⋅∣u,ht\(n\),ℰ\(n\)\),ℰ\(0\)=∅\.a\_\{t\}^\{\(n\)\}\\sim\\pi\_\{\\theta\}\\bigl\(\\cdot\\mid u,h\_\{t\}^\{\(n\)\},\\mathcal\{E\}^\{\(n\)\}\\bigr\),\\qquad\\mathcal\{E\}^\{\(0\)\}=\\varnothing\.\(1\)The trial ends upon success or exhaustion of the attempt budget\. Experience thus accumulates across episodes, but each restart requires rebuilding previously achieved environment progress\.

##### Retrospective Exploration\.

ReTRAIL provides an environment\-levelrewindinterface that allows the agent to restore a previously visited state during an ongoing attempt\. When the current branch is unlikely to yield further progress, the agent can proactively rewind, preserve the execution prefix, and use experience distilled from the abandoned branch to guide a new continuation\. We call this process*retrospective exploration*\.

Execute\.A forward segmentσm\\sigma\_\{m\}is the contiguous interaction sequence from the initial state or a restored state to the next recovery boundary or trial termination\. Rewinding ends the current branch and starts a new segment from the selected earlier state, while accumulated experience remains available\. We index these segments bymmand condition execution on the interaction historyhth\_\{t\}and accumulated branch experienceℰm=\(e0,…,em−1\)\\mathcal\{E\}\_\{m\}=\(e\_\{0\},\\ldots,e\_\{m\-1\}\), whereeie\_\{i\}is the reflection distilled at recoveryiiandℰ0=∅\\mathcal\{E\}\_\{0\}=\\varnothing:

at∼πθ\(⋅∣u,ht,ℰm\)\.a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid u,h\_\{t\},\\mathcal\{E\}\_\{m\}\)\.\(2\)Detect recovery\.Atskms\_\{k\_\{m\}\}, recovery may be initiated proactively once the agent judges that the current branch is unlikely to yield further progress, or triggered by a failure signal\. The agent selects an earlier statesjms\_\{j\_\{m\}\}, partitioning the current execution path into a retained prefix and an abandoned suffix:

ρm\\displaystyle\\rho\_\{m\}=rewind\(skm→sjm\),jm<km,\\displaystyle=\\operatorname\{rewind\}\\left\(s\_\{k\_\{m\}\}\\rightarrow s\_\{j\_\{m\}\}\\right\),\\qquad j\_\{m\}<k\_\{m\},\(3\)τ0:km\\displaystyle\\tau\_\{0:k\_\{m\}\}=τ0:jm⏟retained prefix⊕βm⏟abandoned suffix,βm=τjm:km\.\\displaystyle=\\underbrace\{\\tau\_\{0:j\_\{m\}\}\}\_\{\\text\{retained prefix\}\}\\oplus\\underbrace\{\\beta\_\{m\}\}\_\{\\text\{abandoned suffix\}\},\\qquad\\beta\_\{m\}=\\tau\_\{j\_\{m\}:k\_\{m\}\}\.Here,ρm\\rho\_\{m\}denotes the rewind decision, before restoration is executed\. The indicesjmj\_\{m\}andkmk\_\{m\}refer to positions on the current execution path\. The operator⊕\\oplusdenotes concatenation, with a shared boundary state included only once when joining trajectory segments\.

Reflect\.Before discardingβm\\beta\_\{m\}, the agent distills its useful information into a compact reflection:

em∼πθ\(⋅∣u,hjm,βm,ℰm\)\.e\_\{m\}\\sim\\pi\_\{\\theta\}\\left\(\\cdot\\mid u,h\_\{j\_\{m\}\},\\beta\_\{m\},\\mathcal\{E\}\_\{m\}\\right\)\.\(4\)
The reflection may capture discovered constraints, failure causes, actions to avoid, or alternative directions\.

Restore and resume\.Return tosjms\_\{j\_\{m\}\}, truncate history tohjmh\_\{j\_\{m\}\}, and retain experience:

ℰm\+1=ℰm⊕em\.\\mathcal\{E\}\_\{m\+1\}=\\mathcal\{E\}\_\{m\}\\oplus e\_\{m\}\.\(5\)
\(skm,hkm,ℰm\)⟶\(sjm,hjm,ℰm\+1\)\.\\big\(s\_\{k\_\{m\}\},h\_\{k\_\{m\}\},\\mathcal\{E\}\_\{m\}\\big\)\\;\\longrightarrow\\;\\big\(s\_\{j\_\{m\}\},h\_\{j\_\{m\}\},\\mathcal\{E\}\_\{m\+1\}\\big\)\.\(6\)

Execution resumes with the retained prefix and updated experience, allowing subsequent decisions to use a better\-informed context from the same environment state\. A complete trial of retrospective exploration is written as

Γ=\(σ0,ρ0,e0,σ1,ρ1,e1,…,σM\),\\Gamma=\\big\(\\sigma\_\{0\},\\rho\_\{0\},e\_\{0\},\\sigma\_\{1\},\\rho\_\{1\},e\_\{1\},\\ldots,\\sigma\_\{M\}\\big\),\(7\)whereσm\\sigma\_\{m\}denotes a forward trajectory segment ending at a recovery boundary or trial termination\. The first segment starts froms0s\_\{0\}; each subsequent segment resumes from the selected earlier state and conditions on the updated experience\. The abandoned suffixβm\\beta\_\{m\}is determined by the selected rewind point and may span portions of multiple earlier segments\.

## 4Learning from Retrospective Exploration

Retrospective exploration allows agents to revisit earlier states and continue with accumulated experience\. Learning to use this capability effectively requires improving both execution decisions and the experience distilled from abandoned branches\. It therefore yields two complementary learning signals\. First, alternative continuations from the same environment state and retained prefix can be compared to identify more productive execution paths\. Second, the progress of a resumed continuation can be compared with that of the abandoned suffix to assess the usefulness of the experience carried forward\.

We use these two signals to jointly train a shared execution and reflection policy\.State\-Anchored GRPOimplements the first comparison by assigning credit to continuations from the same state and retained prefix\. Abranch\-advancement rewardimplements the second by measuring the progress of the resumed continuation relative to the abandoned suffix\. Figure[2](https://arxiv.org/html/2610.13058#S4.F2)illustrates these two learning signals\.

![Refer to caption](https://arxiv.org/html/2610.13058v1/method.png)Figure 2:Learning from retrospective exploration\.The top panel collects retrospective trials for one task\. Left: State\-Anchored GRPO combines task\-wide endpoint progress with return comparisons among continuations sharing an environment state and retained prefix, producingAcontA^\{\\mathrm\{cont\}\}for execution tokens including rewind\. Right: the verified\-progress difference between a matched abandoned/resumed pair producesArefA^\{\\mathrm\{ref\}\}for the reflection tokens that generateeme\_\{m\}\. The separate masked objectives jointly update one shared policy\.### 4\.1State\-Anchored GRPO over Revisited States

A final task outcome gives limited guidance about which decisions were productive\. Local comparisons, as used in GiGPO\([Feng et al\., 2025a](https://arxiv.org/html/2610.13058#bib.bib1)\), provide more focused feedback\. Retrospective exploration creates the comparison groups needed for execution credit: after returning to an earlier point, the agent can generate alternative continuations from the same environment state while retaining the same execution prefix\.

##### Constructing continuation groups\.

For each taskuu, we sample a group of retrospective trials\{Γi\}i=1B\\\{\\Gamma\_\{i\}\\\}\_\{i=1\}^\{B\}and collect their forward segments in𝒮⁡\(u\)\\mathcal\{S\}\(u\)\. Each recorded state visit defines a continuationξ\\xiending with its segment at success, failure, rewind, or budget exhaustion\. A model\-initiated rewind is included as the terminal execution decision when present\.

Letsξs\_\{\\xi\}andhξh\_\{\\xi\}denote the starting environment state and retained history ofξ\\xi\. We group continuations by the same task, starting state, and retained history:

ℋu\(s,h\)=\{ξ∈𝒞\(u\):sξ=s,hξ=h\},\\mathcal\{H\}\_\{u\}\(s,h\)=\\left\\\{\\xi\\in\\mathcal\{C\}\(u\):s\_\{\\xi\}=s,\\ h\_\{\\xi\}=h\\right\\\},\(8\)where𝒞⁡\(u\)\\mathcal\{C\}\(u\)is the collection of continuations extracted from the sampled trials\. Groups include earlier segment suffixes passing throughssand new continuations after rewind\. Samples retain their original generation history and experience; the accumulated experience may differ within a group\.

##### Assigning execution credit\.

Each continuation has a discounted returnGξG\_\{\\xi\}from its starting state to the segment endpoint and a task\-progress scoreSξS\_\{\\xi\}at that endpoint, shared by continuations from the same segment\. Its advantage combines task\-wide and state\-relative feedback:

Aξcont=Norm𝒮⁡\(u\)⁡\(Sξ\)⏟Aξmacro\+λstate​Normℋu​\(sξ,hξ\)⁡\(Gξ\)⏟Aξstate,A\_\{\\xi\}^\{\\mathrm\{cont\}\}=\\underbrace\{\\operatorname\{Norm\}\_\{\\mathcal\{S\}\(u\)\}\(S\_\{\\xi\}\)\}\_\{A\_\{\\xi\}^\{\\mathrm\{macro\}\}\}\+\\lambda\_\{\\mathrm\{state\}\}\\underbrace\{\\operatorname\{Norm\}\_\{\\mathcal\{H\}\_\{u\}\(s\_\{\\xi\},h\_\{\\xi\}\)\}\(G\_\{\\xi\}\)\}\_\{A\_\{\\xi\}^\{\\mathrm\{state\}\}\},\(9\)whereNorm\\operatorname\{Norm\}subtracts the corresponding group mean and divides by its standard deviation plusϵ\\epsilon\. The macro term normalizes endpoint progress over𝒮⁡\(u\)\\mathcal\{S\}\(u\), counting each segment once, to preserve the task\-wide objective\. The state\-relative term favors higher\-return continuations from the same state and prefix\. For singleton groups, this term is zero and the macro signal remains\.

We applyAξcontA\_\{\\xi\}^\{\\mathrm\{cont\}\}to the execution tokens inξ\\xi, including any terminal model\-initiated rewind decision\. The retained prefix serves as input context and is excluded from the continuation loss\. Execution returns end at rewind; subsequent advancement supervises the reflection response\. Appendix[D\.5](https://arxiv.org/html/2610.13058#A4.SS5)details the construction, and Appendix[D\.6](https://arxiv.org/html/2610.13058#A4.SS6)illustrates this credit boundary\.

### 4\.2Learning Reflection from Branch Advancement

Reflection is generated from an abandoned branch, but its value is realized only through the continuation that follows recovery\. The discarded suffix may contain useful information about invalid actions, hidden constraints, or promising alternatives; however, the final trial outcome is too coarse and too delayed to determine whether this information improved the resumed exploration\. A successful trial may succeed despite an uninformative reflection, while a useful reflection may produce measurable progress before task completion\. We therefore evaluate reflection by comparing the progress of the post\-recovery continuation with that of the abandoned branch from the same restored state\.

At recoverymm, the agent distills the abandoned suffixβm\\beta\_\{m\}into experienceeme\_\{m\}and resumes from statesjms\_\{j\_\{m\}\}\. The abandoned suffix and the new forward segmentσm\+1\\sigma\_\{m\+1\}share this starting state, while the latter conditions on the updated experienceℰm\+1\\mathcal\{E\}\_\{m\+1\}\. Here,βm\\beta\_\{m\}is the suffix discarded by the rewind and may contain portions of multiple earlier segments; it need not coincide with a single continuation used for execution grouping\.

Using an environment\-verifiable progress measureVV, we evaluate the two paths:

vm−=V⁡\(βm∣sjm\),vm\+=V⁡\(σm\+1∣sjm\),v^\{\-\}\_\{m\}=V\\\!\\left\(\\beta\_\{m\}\\mid s\_\{j\_\{m\}\}\\right\),\\qquad v^\{\+\}\_\{m\}=V\\\!\\left\(\\sigma\_\{m\+1\}\\mid s\_\{j\_\{m\}\}\\right\),\(10\)whereVVmeasures task progress, such as completed subgoals, for exploration starting fromsjms\_\{j\_\{m\}\}\. The branch\-advancement reward uses their difference as a proxy for experience utility; positive values indicate greater progress after recovery:

rmref=vm\+−vm−\.r\_\{m\}^\{\\mathrm\{ref\}\}=v^\{\+\}\_\{m\}\-v^\{\-\}\_\{m\}\.\(11\)
##### Joint policy optimization\.

We jointly optimize execution and reflection through

𝒥⁡\(θ\)=𝒥GRPOexec​\(θ,Acont\)\+λref​𝒥GRPOref​\(θ,Aref\),\\mathcal\{J\}\(\\theta\)=\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{exec\}\}\\left\(\\theta;A^\{\\mathrm\{cont\}\}\\right\)\+\\lambda\_\{\\mathrm\{ref\}\}\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{ref\}\}\\left\(\\theta;A^\{\\mathrm\{ref\}\}\\right\),\(12\)whereArefA^\{\\mathrm\{ref\}\}is obtained by group\-normalizing the branch\-advancement rewardsrmrefr\_\{m\}^\{\\mathrm\{ref\}\}, andλref\\lambda\_\{\\mathrm\{ref\}\}controls the relative weight of reflection training\. The two objectives updateπθ\\pi\_\{\\theta\}through execution tokens \(including model\-initiated rewind\) and reflection tokens, respectively\. Appendix[D\.5](https://arxiv.org/html/2610.13058#A4.SS5)gives sample weighting and GRPO details\.

## 5Experiments

Our experiments ask whether ReTRAIL improves recovery\-enabled success while retaining competitive forward execution \(Section[5\.2](https://arxiv.org/html/2610.13058#S5.SS2)\), which rollout and learning components support these gains \(Section[6](https://arxiv.org/html/2610.13058#S6)\), and how performance varies with instance difficulty, task length, and the number of recovery attempts \(Sections[5\.3](https://arxiv.org/html/2610.13058#S5.SS3)and[B\.1](https://arxiv.org/html/2610.13058#A2.SS1)\)\.

### 5\.1Experimental Setup

##### Environments\.

The reported results cover four interactive benchmarks with complementary failure structures\.MineSweeper\([Li et al\., 2024](https://arxiv.org/html/2610.13058#bib.bib2)\)requires logical deduction under partial observability; unsuccessful exploration may reveal board constraints\.Sokoban\([Racanière et al\., 2017](https://arxiv.org/html/2610.13058#bib.bib19)\)tests planning with irreversible forward transitions, such as pushing a box into a dead corner\.ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2610.13058#bib.bib17)\)evaluates multi\-step instruction following, whileScienceWorld\([Wang et al\., 2022](https://arxiv.org/html/2610.13058#bib.bib18)\)involves procedural reasoning with long dependency chains\. All use text interaction and the state\-restoration interface in Section[3](https://arxiv.org/html/2610.13058#S3)\. Dataset partitions are listed in Table[5](https://arxiv.org/html/2610.13058#A3.T5)\.

##### Baselines\.

We compare with zero\-shot prompting, ReAct\([Yao et al\., 2023](https://arxiv.org/html/2610.13058#bib.bib27)\), Reflexion\([Shinn et al\., 2023](https://arxiv.org/html/2610.13058#bib.bib9)\), adapted GA\-Rollback\([Li et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib94)\), and trained baselines GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.13058#bib.bib78)\), RLOO\([Ahmadian et al\., 2024](https://arxiv.org/html/2610.13058#bib.bib22)\), GiGPO\([Feng et al\., 2025a](https://arxiv.org/html/2610.13058#bib.bib1)\), BEACON\([Wang et al\., 2026c](https://arxiv.org/html/2610.13058#bib.bib3)\), and LaMer\([Jiang et al\., 2026](https://arxiv.org/html/2610.13058#bib.bib11)\)\. GiGPO uses local group comparisons; LaMer learns across episodes\. Recovery\-enabled evaluations share a rewind interface with corresponding execution and reflection prompts\.

##### Evaluation protocol and metrics\.

Pass@1 \(P@1\)measures single\-pass execution without rewind\.Trial@1 \(T@1\)permits model\-initiated rewind but terminates at the first irreversible failure; one macro\-attempt may therefore explore multiple branches\.Δpro=T​@​1−P​@​1\\Delta\_\{\\mathrm\{pro\}\}=\\mathrm\{T@1\}\-\\mathrm\{P@1\}measures the net gain from enabling proactive recovery\.Trial@3 \(T@3\)permits up to three macro\-attempts, with reflection\-guided recovery after irreversible failure\. We denoteΔref=T​@​3−T​@​1\\Delta\_\{\\mathrm\{ref\}\}=\\mathrm\{T@3\}\-\\mathrm\{T@1\}as the*retry gain*: it measures the combined benefit of additional attempts, recovery, and retained experience\. The configured step limit for all settings is 60\. Details are provided in Appendix[C\.4](https://arxiv.org/html/2610.13058#A3.SS4)\.

##### Training details\.

We use Qwen3\-4B\([Yang et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib61)\)as the base model for all methods\. ReTRAIL samples groups of trials using retrospective exploration and is optimized using our State\-Anchored GRPO objective \(Section[4\.1](https://arxiv.org/html/2610.13058#S4.SS1)\)\. For the standard RL baselines, we adjust the rollout group size to match the sampling budget of ReTRAIL at each gradient update\. We otherwise keep the learning rate, batch size, prompt template, and task splits identical across methods\. The continuation construction and execution objective are detailed in Appendix[D\.5](https://arxiv.org/html/2610.13058#A4.SS5)\. Appendix[C\.5](https://arxiv.org/html/2610.13058#A3.SS5)reports equal\-benchmark aggregation\. Reported results are means over three evaluation seeds\.

### 5\.2Main Results

Table 1:Performance comparison on four long\-horizon interactive benchmarks\.P@1evaluates forward execution without rewind,T@1enables proactive rewind within one trial, andT@3permits up to three macro attempts with reflection\-guided recovery\. Every setting shares a 60\-call budget per trial, including execution and reflection\. Colored annotations show absolute changes relative to P@1; bold values indicate the best result in each column\. All values are success rates \(%\)\.##### ReTRAIL achieves stronger performance\.

ReTRAIL achieves the highest T@1 and T@3 on all four benchmarks \(Table[1](https://arxiv.org/html/2610.13058#S5.T1)\), outperforming the strongest baseline in T@3 by24\.524\.5percentage points on MineSweeper and10\.210\.2on ScienceWorld\. Its forward\-only P@1 remains competitive, slightly exceeding the strongest baseline on MineSweeper, ALFWorld, and ScienceWorld, while trailing GiGPO by2\.32\.3points on Sokoban\. Together, these results show that ReTRAIL’s advantage becomes more pronounced when recovery is enabled, while its forward\-solving ability remains competitive\.

##### ReTRAIL makes effective use of test\-time recovery\.

ReTRAIL benefits from recovery both within and across macro\-attempts\. Within a single attempt, proactive recovery yields positive gains on every benchmark, ranging from1\.31\.3percentage points on ALFWorld to10\.910\.9on Sokoban, whereas enabling rewind for baseline policies produces mixed changes\. Across attempts, ReTRAIL further improves with reflection\-guided recovery and retained experience\. On MineSweeper, success rises from63\.563\.5at T@1 to95\.395\.3at T@3, a gain of31\.831\.8points, compared with9\.19\.1for GiGPO and13\.613\.6for LaMer\. These results demonstrate ReTRAIL’s ability to adapt through recovery, benefiting from both within\-attempt course correction and continued exploration after failure\.

##### ReTRAIL improves exploration coverage and execution efficiency\.

Figure[3](https://arxiv.org/html/2610.13058#S5.F3)compares the coverage of distinct environment states and the number of actions taken in successful trials\. Relative to LaMer, ReTRAIL increases the median number of distinct states visited from11\.011\.0to14\.014\.0on MineSweeper and from8\.08\.0to11\.511\.5on Sokoban\. Meanwhile, the median action count among successful trials decreases from53\.053\.0to40\.040\.0and from48\.548\.5to37\.037\.0, respectively\. Together, these findings suggest that recovering early from unproductive branches helps ReTRAIL explore more broadly while completing tasks more efficiently\.

Figure 3:Exploration diversity and efficiency of LaMer and ReTRAIL on MineSweeper and Sokoban\. \(a\) States visited, where higher values indicate broader exploration; \(b\) Actions to success, where lower values indicate more efficient solving\. ReTRAIL visits more states while requiring fewer actions\.

### 5\.3Scaling Instance Difficulty and Recovery Attempts

We next examine performance as instance difficulty and inference\-time computation increase\.

Figure 4:Scaling instance difficulty and recovery attempts\. \(a\) T@3 success with additional mines or boxes; these settings vary instance difficulty\. \(b\) Success with one to four permitted macro\-attempts\. ReTRAIL’s endpoint value is annotated in each panel\. Y\-axis ranges differ by subplot and are explicitly labeled\. Higher is better\.##### Scaling to harder instances\.

We increase the number of mines in MineSweeper and boxes in Sokoban to construct harder variants\. Figure[4](https://arxiv.org/html/2610.13058#S5.F4)\(a\) shows that success decreases for all plotted methods, while ReTRAIL retains an advantage at every displayed setting\. At the hardest setting, ReTRAIL reaches63\.963\.9on MineSweeper, compared with45\.345\.3for the strongest baseline, and55\.255\.2on Sokoban, compared with45\.545\.5, yielding margins of18\.618\.6and9\.79\.7percentage points\. These results show that ReTRAIL maintains its advantage on harder instances\. The analysis by shortest\-solution length is reported separately in Figure[5](https://arxiv.org/html/2610.13058#A2.F5)\.

##### Scaling the number of attempts\.

Allowing up to four macro\-attempts provides additional opportunities for recovery with accumulated experience\. ReTRAIL achieves the highest success rate throughout Figure[4](https://arxiv.org/html/2610.13058#S5.F4)\(b\)\. On MineSweeper, success increases from63\.563\.5at T@1 to97\.897\.8at T@4, a gain of34\.334\.3percentage points, compared with10\.810\.8for GiGPO and16\.316\.3for LaMer\. On Sokoban, ReTRAIL reaches95\.195\.1at T@4 and remains highest at each attempt budget\. Thus, the learned execution and reflection policy continues to benefit from additional recovery opportunities\.

## 6Analysis

Table 2:Ablation of retrospective exploration\. Success rate \(%\)\.
Table 3:Ablation of the learning objective\. Success rate \(%\)\.
##### Ablation\.

Tables[3](https://arxiv.org/html/2610.13058#S6.T3)and[3](https://arxiv.org/html/2610.13058#S6.T3)evaluate the roles of retrospective exploration and its learning objectives\. Removing model\-initiated rewind lowers T@1 from63\.563\.5to58\.158\.1on MineSweeper and from87\.587\.5to72\.372\.3on Sokoban, showing the benefit of proactive recovery\. Replacing distilled reflection with the raw abandoned suffix has a smaller effect on T@1, but lowers T@3 to69\.769\.7and87\.287\.2, respectively\. This suggests that distilling branch experience provides more useful guidance for subsequent attempts than retaining raw history alone\.

We next ablate the learning objective\. Replacing the branch\-advancement rewardrmref=vm\+−vm−r\_\{m\}^\{\\mathrm\{ref\}\}=v\_\{m\}^\{\+\}\-v\_\{m\}^\{\-\}with post\-recovery progressvm\+v\_\{m\}^\{\+\}alone lowers T@3 from95\.395\.3to80\.280\.2on MineSweeper and from93\.793\.7to84\.984\.9on Sokoban, alongside smaller declines in T@1\. These results support using subsequent branch progress to guide reflection\. We also replace State\-Anchored GRPO with GiGPO within the same retrospective\-exploration framework; this variant is distinct from the standalone GiGPO baseline in Table[1](https://arxiv.org/html/2610.13058#S5.T1)\. T@1/T@3 falls from63\.5/95\.363\.5/95\.3to57\.6/90\.157\.6/90\.1on MineSweeper and from87\.5/93\.787\.5/93\.7to84\.2/90\.384\.2/90\.3on Sokoban\. Together, these results support the proposed execution and reflection objectives, with the GiGPO comparison evaluating the credit\-assignment designs as implemented\.

##### Recovery Gains across Task Horizons\.

We stratify tasks by reference\-solution length to examine how ReTRAIL’s recovery gains vary with task horizon \(Appendix[B\.1](https://arxiv.org/html/2610.13058#A2.SS1)\)\. From short to long tasks, its reflection\-guided retry gain increases from4\.54\.5to19\.019\.0percentage points, while its proactive gain rises from1\.31\.3to7\.87\.8points\. Although LaMer achieves a slightly larger retry gain on short tasks, ReTRAIL surpasses it by5\.85\.8points on long tasks\. These results suggest that retrospective exploration is particularly beneficial for long\-horizon execution\.

## 7Conclusion

Learning to recover from unproductive exploration is crucial for autonomous agents to make sustained progress on long\-horizon tasks\. We introducedReTRAIL, a framework that combines execution recovery with experiential learning through retrospective exploration\. With an environment\-levelrewindinterface, agents can revisit earlier states and explore alternative continuations while retaining experience from abandoned branches\. ReTRAIL trains a shared execution and reflection policy under this protocol, using state\-anchored comparisons to improve execution and branch\-advancement feedback to guide reflection\. Across four interactive benchmarks, ReTRAIL consistently outperforms both methods without rewind and baselines equipped with the same interface, with ablations supporting the complementary roles of recovery, retained experience, and progress\-based feedback\. More broadly, these findings highlight retrospective exploration as a promising foundation for autonomous agents that can recover from setbacks, adapt through accumulated experience, and sustain progress toward long\-horizon goals\.

## AI Use Statement

In preparing this manuscript, we used generative AI tools for language polishing and literature search and survey\. Language assistance focused on improving grammar, sentence flow, clarity, and readability while preserving the intended technical meaning\. Literature assistance supported the identification of relevant prior work and the organization and summarization of existing research\. The authors remain responsible for reviewing all AI\-assisted material, checking literature summaries and citations against the original sources, and ensuring that the manuscript accurately represents the research\. We take full responsibility for the final content of this work, including all text, claims, and artifacts produced with the assistance of generative AI\.

## Ethics Statement

This work studies recovery and learning in autonomous agents through experiments on four controlled benchmark environments: MineSweeper, Sokoban, ALFWorld, and ScienceWorld\. The framework aims to improve agents’ ability to recover from unproductive exploration, but state restoration in these environments should not be interpreted as reversing the consequences of actions in real\-world systems\. Deployment beyond these benchmarks should therefore include appropriate authorization, safeguards for external actions, and human oversight where actions may have consequential or irreversible effects\. Retained interaction histories and reflection memories should be handled with appropriate privacy and access controls, and any reuse or release of data, models, and software should respect their applicable licenses and terms of use\.

## Reproducibility Statement

To support reproducibility, we specify the retrospective\-exploration framework and learning objectives in Sections[3](https://arxiv.org/html/2610.13058#S3)and[4](https://arxiv.org/html/2610.13058#S4)\. Appendix[D](https://arxiv.org/html/2610.13058#A4)details environment restoration, recovery and context management, and provides pseudocode for rollout collection and policy updates\. Section[5\.1](https://arxiv.org/html/2610.13058#S5.SS1)and Appendix[C](https://arxiv.org/html/2610.13058#A3)document the benchmarks and data partitions, baseline configurations, training hyperparameters, reward and progress signals, evaluation protocols, model\-call accounting, and aggregation across benchmarks and seeds\. Appendix[E](https://arxiv.org/html/2610.13058#A5)provides execution and reflection prompt templates and examples of observation formatting for the four benchmark environments\.

## References

- Ahmadianet al\.\(2024\)A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. HookerBack to basics: revisiting reinforce style optimization for learning from human feedback in llms\.External Links:2402\.14740,[Link](https://arxiv.org/abs/2402.14740)Cited by:[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Anthropic \(2025\)AnthropicIntroducing claude opus 4\.5\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-5](https://www.anthropic.com/news/claude-opus-4-5)Accessed: 2026\-04\-29
- Bandiet al\.\(2026\)C\. Bandi, B\. Hertzberg, G\. Boo, T\. Polakam, J\. Da, S\. Hassaan, M\. Sharma, A\. Park, E\. Hernandez, D\. Rambado, I\. Salazar, R\. Cruz, C\. Rane, B\. Levin, B\. Kenstler, and B\. LiuMCP\-atlas: a large\-scale benchmark for tool\-use competency with real mcp servers\.External Links:2602\.00933,[Link](https://arxiv.org/abs/2602.00933)
- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.External Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)
- Bhaskaret al\.\(2025\)A\. Bhaskar, X\. Ye, and D\. ChenLanguage models that think, chat better\.CoRRabs/2509\.20357\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InNeurIPS,
- Chenet al\.\(2026a\)A\. Chen, Z\. Liu, J\. Zhang, A\. Prabhakar, Z\. Liu, S\. Heinecke, S\. Savarese, V\. Zhong, and C\. XiongTest\-time adaptation for llm agents via environment interaction\.External Links:2511\.04847,[Link](https://arxiv.org/abs/2511.04847)
- Chenet al\.\(2024\)M\. Chen, Y\. Li, Y\. Yang, S\. Yu, B\. Lin, and X\. HeAutoManual: generating instruction manuals by LLM agents via interactive environmental learning\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Pwl9n4zlf5)
- Chenet al\.\(2026b\)X\. Chen, C\. Xu, Y\. Wang, B\. Liu, Z\. Yao, and Y\. HeLearning to self\-evolve\.arXiv preprint arXiv:2603\.18620\.External Links:[Link](https://arxiv.org/abs/2603.18620)Cited by:[§2\.2](https://arxiv.org/html/2610.13058#S2.SS2.p1.1)\.
- Chenet al\.\(2026c\)Y\. Chen, Z\. Cai, X\. Ji, W\. Zhao, A\. Zhang, X\. Wang, and T\. ChuaUnderstanding multilingualism in mixture\-of\-experts llms: routing mechanism, expert specialization, and layerwise steering\.External Links:2601\.14050,[Link](https://arxiv.org/abs/2601.14050)
- Chenet al\.\(2026d\)Y\. Chen, Y\. Wang, Y\. Zhang, Z\. Ye, Z\. Cai, Y\. Shi, Q\. Gu, H\. Su, X\. Cai, X\. Wang, A\. Zhang, and T\. ChuaLearning to self\-verify makes language models better reasoners\.External Links:2602\.07594,[Link](https://arxiv.org/abs/2602.07594)
- Chenet al\.\(2026e\)Y\. Chen, Y\. Zhang, Z\. Cai, Y\. Shi, Z\. Yao, C\. Cui, J\. Zheng, Y\. Huo, X\. Su, Q\. Gu, X\. Cai, X\. Wang, A\. Zhang, and T\. ChuaVitaBench 2\.0: evaluating personalized and proactive agents in long\-term user interactions\.External Links:2605\.27141,[Link](https://arxiv.org/abs/2605.27141)
- Choiet al\.\(2026\)J\. Choi, H\. Kim, H\. Ong, Y\. Yoon, M\. Jang, D\. Kim, and J\. KimReAcTree: hierarchical LLM agent trees with control flow for long\-horizon task planning\.InInternational Conference on Autonomous Agents and Multiagent Systems,External Links:[Link](https://arxiv.org/abs/2511.02424)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.13058#S2.SS1.p1.1)\.
- de Lamo Castrilloet al\.\(2025\)V\. de Lamo Castrillo, H\. K\. Gidey, A\. Lenz, and A\. KnollFundamentals of building autonomous llm agents\.External Links:2510\.09244,[Link](https://arxiv.org/abs/2510.09244)
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Lu, C\. Zhao, C\. Deng, C\. Xu, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, E\. Li, F\. Zhou, F\. Lin, F\. Dai, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Li, H\. Liang, H\. Wei, H\. Zhang, H\. Luo, H\. Ji, H\. Ding, H\. Tang, H\. Cao, H\. Gao, H\. Qu, H\. Zeng, J\. Huang, J\. Li, J\. Xu, J\. Hu, J\. Chen, J\. Xiang, J\. Yuan, J\. Cheng, J\. Zhu, J\. Ran, J\. Jiang, J\. Qiu, J\. Li, J\. Song, K\. Dong, K\. Gao, K\. Guan, K\. Huang, K\. Zhou, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Wang, L\. Zhao, L\. Yin, L\. Guo, L\. Luo, L\. Ma, L\. Wang, L\. Zhang, M\. S\. Di, M\. Y\. Xu, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, P\. Huang, P\. Cong, P\. Wang, Q\. Wang, Q\. Zhu, Q\. Li, Q\. Chen, Q\. Du, R\. Xu, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Yin, R\. Xu, R\. Shen, R\. Zhang, S\. H\. Liu, S\. Lu, S\. Zhou, S\. Chen, S\. Cai, S\. Chen, S\. Hu, S\. Liu, S\. Hu, S\. Ma, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. Zhou, T\. Ni, T\. Yun, T\. Pei, T\. Ye, T\. Yue, W\. Zeng, W\. Liu, W\. Liang, W\. Pang, W\. Luo, W\. Gao, W\. Zhang, X\. Gao, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Chen, X\. Zhang, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yu, X\. Li, X\. Yang, X\. Li, X\. Chen, X\. Su, X\. Pan, X\. Lin, X\. Fu, Y\. Q\. Wang, Y\. Zhang, Y\. Xu, Y\. Ma, Y\. Li, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Qian, Y\. Yu, Y\. Zhang, Y\. Ding, Y\. Shi, Y\. Xiong, Y\. He, Y\. Zhou, Y\. Zhong, Y\. Piao, Y\. Wang, Y\. Chen, Y\. Tan, Y\. Wei, Y\. Ma, Y\. Liu, Y\. Yang, Y\. Guo, Y\. Wu, Y\. Wu, Y\. Cheng, Y\. Ou, Y\. Xu, Y\. Wang, Y\. Gong, Y\. Wu, Y\. Zou, Y\. Li, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Z\. F\. Wu, Z\. Z\. Ren, Z\. Zhao, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Huang, Z\. Wu, Z\. Li, Z\. Zhang, Z\. Xu, Z\. Wang, Z\. Gu, Z\. Zhu, Z\. Li, Z\. Zhang, Z\. Xie, Z\. Gao, Z\. Pan, Z\. Yao, B\. Feng, H\. Li, J\. L\. Cai, J\. Ni, L\. Xu, M\. Li, N\. Tian, R\. J\. Chen, R\. L\. Jin, S\. S\. Li, S\. Zhou, T\. Sun, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Song, X\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Z\. Huang, Z\. Xu, Z\. Zhang, D\. Ji, J\. Liang, J\. Guo, J\. Chen, L\. Xia, M\. Wang, M\. Li, P\. Zhang, R\. Chen, S\. Sun, S\. Wu, S\. Ye, T\. Wang, W\. L\. Xiao, W\. An, X\. Wang, X\. Sun, X\. Wang, Y\. Tang, Y\. Zha, Z\. Zhang, Z\. Ju, Z\. Zhang, and Z\. QuDeepSeek\-v3\.2: pushing the frontier of open large language models\.Vol\.abs/2512\.02556\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.CoRRabs/2501\.12948\.
- Fenget al\.\(2025a\)L\. Feng, Z\. Xue, T\. Liu, and B\. AnGroup\-in\-group policy optimization for LLM agent training\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/420c9f777c0b4f78d515e53cf74d58b2-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2610.13058#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Fenget al\.\(2025b\)S\. Feng, G\. Fang, X\. Ma, and X\. WangEfficient reasoning models: A survey\.Trans\. Mach\. Learn\. Res\.2025\.
- Google \(2025\)GoogleA new era of intelligence with gemini 3\.External Links:[Link](https://blog.google/products-and-platforms/products/gemini/gemini-3/#note-from-ceo)
- Gunjalet al\.\(2025\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, B\. Liu, and S\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.CoRRabs/2507\.17746\.
- Guret al\.\(2024\)I\. Gur, H\. Furuta, A\. V\. Huang, M\. Safdari, Y\. Matsuo, D\. Eck, and A\. FaustA real\-world webagent with planning, long context understanding, and program synthesis\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=9JQtrumvg8)
- Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. SunOlympiadBench: A challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InACL \(1\),pp\. 3828–3850\.
- Heet al\.\(2026a\)S\. He, L\. Feng, Q\. Wei, X\. Cheng, L\. Feng, and B\. AnHierarchy\-of\-groups policy optimization for long\-horizon agentic tasks\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=T8Dev99qnz)
- Heet al\.\(2026b\)W\. He, Y\. Sun, H\. Hao, X\. Hao, Z\. Xia, Q\. GU, H\. Su, and X\. CaiVitaBench: benchmarking LLM agents with versatile interactive tasks in real\-world applications\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rtcX9qOBaz)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p1.1)\.
- Heet al\.\(2026c\)Y\. He, J\. Liu, Y\. Liu, Y\. Li, T\. Cao, Z\. Hu, X\. Xu, and B\. HooiEvoTest: evolutionary test\-time learning for self\-improving agentic systems\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2510.13220)Cited by:[§2\.2](https://arxiv.org/html/2610.13058#S2.SS2.p1.1)\.
- Huanget al\.\(2024\)X\. Huang, W\. Liu, X\. Chen, X\. Wang, D\. Lian, Y\. Wang, R\. Tang, and E\. ChenWESE: weak exploration to strong exploitation for llm agents\.External Links:2404\.07456,[Link](https://arxiv.org/abs/2404.07456)
- Jiet al\.\(2026\)X\. Ji, Y\. Chen, Z\. Cai, X\. Wang, A\. Zhang, and T\. ChuaTiny brains, giant impact: uncovering the keystone neurons of llm with just a few prompts\.External Links:2605\.24846,[Link](https://arxiv.org/abs/2605.24846)
- Jianget al\.\(2025\)D\. Jiang, Y\. Lu, Z\. Li, Z\. Lyu, P\. Nie, H\. Wang, A\. Su, H\. Chen, K\. Zou, C\. Du,et al\.VerlTool: towards holistic agentic reinforcement learning with tool use\.arXiv preprint arXiv:2509\.01055\.
- Jianget al\.\(2026\)Y\. Jiang, L\. Jiang, D\. Teney, M\. Moor, and M\. BrbićMeta\-RL induces exploration in language agents\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2512.16848)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.13058#S2.SS2.p1.1),[§3](https://arxiv.org/html/2610.13058#S3.SS0.SSS0.Px1.p2.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p1.1)\.
- Jinet al\.\(2025\)B\. Jin, H\. Zeng, Z\. Yue, D\. Wang, H\. Zamani, and J\. HanSearch\-r1: training llms to reason and leverage search engines with reinforcement learning\.CoRRabs/2503\.09516\.
- Liet al\.\(2025\)X\. Li, K\. Chen, Y\. Long, X\. Bai, Y\. Xu, and M\. ZhangGenerator\-assistant stepwise rollback framework for large language model agent\.External Links:2503\.02519,[Link](https://arxiv.org/abs/2503.02519)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.13058#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)Y\. Li, H\. Wang, and C\. ZhangAssessing logical puzzle solving in large language models: insights from a minesweeper case study\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 59–81\.External Links:[Link](http://dx.doi.org/10.18653/v1/2024.naacl-long.4),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.4)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p4.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px1.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InICLR,
- Liuet al\.\(2024\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. TangAgentBench: evaluating LLMs as agents\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=zAdUB0aCTQ)
- Liuet al\.\(2026\)Y\. Liu, Z\. Ye, Y\. Sun, M\. Zhu, J\. Xiao, Z\. Han, Q\. GU, X\. Cai, and L\. ZhangMAP: a map\-then\-act paradigm for long\-horizon interactive agent reasoning\.External Links:2605\.13037,[Link](https://arxiv.org/abs/2605.13037)
- Louet al\.\(2026\)Z\. Lou, H\. Chen, Y\. Li, Q\. Wang, and B\. HooiLearning to learn\-at\-test\-time: language agents with learnable adaptation policies\.arXiv preprint arXiv:2604\.00830\.External Links:[Link](https://arxiv.org/abs/2604.00830)Cited by:[§2\.2](https://arxiv.org/html/2610.13058#S2.SS2.p1.1)\.
- Luoet al\.\(2025\)Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. ZhangAn empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.External Links:2308\.08747,[Link](https://arxiv.org/abs/2308.08747)
- Maiet al\.\(2025\)S\. Mai, Y\. Zhai, Z\. Chen, C\. Chen, A\. Zou, S\. Tao, Z\. Liu, and B\. DingCuES: a curiosity\-driven and environment\-grounded synthesis framework for agentic rl\.External Links:2512\.01311,[Document](https://dx.doi.org/10.48550/arXiv.2512.01311),[Link](https://arxiv.org/abs/2512.01311)
- MiniMax \(2025\)MiniMaxMiniMax m2 and agent: ingenious in simplicity\.External Links:[Link](https://www.minimax.io/news/minimax-m2)
- OpenAI \(2023\)OpenAIGPT\-4 technical report\.CoRRabs/2303\.08774\.
- OpenAI \(2025\)OpenAIIntroducing gpt\-5\.2\.External Links:[Link](https://openai.com/index/introducing-gpt-5-2/)
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InNeurIPS,
- Ouyanget al\.\(2026\)S\. Ouyang, J\. Yan, I\. Hsu, Y\. Chen, K\. Jiang, Z\. Wang, R\. Han, L\. T\. Le, S\. Daruki, X\. Tang, V\. Tirumalashetty, G\. Lee, M\. Rofouei, H\. Lin, J\. Han, C\. Lee, and T\. PfisterReasoningBank: scaling agent self\-evolving with reasoning memory\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2509.25140)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.13058#S2.SS2.p1.1)\.
- Pahujaet al\.\(2025\)V\. Pahuja, Y\. Lu, C\. Rosset, B\. Gou, A\. Mitra, S\. Whitehead, Y\. Su, and A\. H\. AwadallahExplorer: scaling exploration\-driven web trajectory synthesis for multimodal web agents\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 6300–6323\.External Links:[Link](https://aclanthology.org/2025.findings-acl.326/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.326),ISBN 979\-8\-89176\-256\-5
- Patilet al\.\(2024\)S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. GonzalezGorilla: large language model connected with massive APIs\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=tBRNC6YemY)
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, dahai li, Z\. Liu, and M\. SunToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dHng2O0Jjr)
- Racanièreet al\.\(2017\)S\. Racanière, T\. Weber, D\. P\. Reichert, L\. Buesing, A\. Guez, D\. Rezende, A\. P\. Badia, O\. Vinyals, N\. Heess, Y\. Li, R\. Pascanu, P\. Battaglia, D\. Hassabis, D\. Silver, and D\. WierstraImagination\-augmented agents for deep reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p4.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px1.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.CoRRabs/1707\.06347\.
- Shaoet al\.\(2025\)Z\. Shao, Y\. Luo, C\. Lu, Z\. Z\. Ren, J\. Hu, T\. Ye, Z\. Gou, S\. Ma, and X\. ZhangDeepSeekMath\-v2: towards self\-verifiable mathematical reasoning\.CoRRabs/2511\.22570\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.CoRRabs/2402\.03300\.Cited by:[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: A flexible and efficient RLHF framework\.InEuroSys,pp\. 1279–1297\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.13058#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Shridharet al\.\(2021\)M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. HausknechtALFWorld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p4.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px1.p1.1)\.
- Songet al\.\(2026\)X\. Song, H\. Chang, G\. Dong, Y\. Zhu, J\. Wen, and Z\. DouEnvScaler: scaling tool\-interactive environments for llm agent via programmatic synthesis\.External Links:2601\.05808,[Link](https://arxiv.org/abs/2601.05808)
- SUet al\.\(2025\)H\. SU, R\. Sun, J\. Yoon, P\. Yin, T\. Yu, and S\. O\. ArikLearn\-by\-interact: a data\-centric framework for self\-adaptive agents in realistic environments\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3UKOzGWCVY)
- Suet al\.\(2025\)Y\. Su, D\. Yu, L\. Song, J\. Li, H\. Mi, Z\. Tu, M\. Zhang, and D\. YuCrossing the reward bridge: expanding RL with verifiable rewards across diverse domains\.CoRRabs/2503\.23829\.
- Suiet al\.\(2025\)Y\. Sui, Y\. Chuang, G\. Wang, J\. Zhang, T\. Zhang, J\. Yuan, H\. Liu, A\. Wen, S\. Zhong, N\. Zou, H\. Chen, and X\. HuStop overthinking: A survey on efficient reasoning for large language models\.Trans\. Mach\. Learn\. Res\.2025\.
- Sunet al\.\(2026a\)W\. Sun, M\. Lu, Z\. Ling, K\. Liu, X\. Yao, Y\. Yang, and J\. ChenScaling long\-horizon agent via context folding\.InProceedings of the 43rd International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=lNRgWoGfYg)
- Sunet al\.\(2026b\)Y\. Sun, X\. Han, W\. Zhang, Y\. Pang, T\. Wang, Y\. Cao, Y\. Huang, C\. Duroiu, H\. Zhang, J\. Lin, W\. Zhang, T\. Zeng, Y\. Yan, B\. Liu, H\. Wen, M\. Xu, X\. Liu, Z\. Chen, W\. Shi, A\. Dsouza, V\. S\. Chen, P\. Bryant, C\. Boettiger, Y\. Rangan, B\. Rothenberg, K\. Steinfeld, A\. Rao, T\. Schneider, G\. Yannakakis, L\. Zanna, K\. Ozbay, I\. Sim, T\. Zohdi, G\. E\. Karniadakis, J\. Gallant, T\. Head\-Gordon, Y\. Li, W\. Deng, T\. Sun, H\. Wang, Z\. Wang, J\. Xu, C\. Y\. Liu, Y\. Cheng, R\. Hu, A\. Bacho, S\. Cao, Z\. Qin, Y\. Chen, H\. Fan, H\. Liu, L\. Zeng, S\. M\. Bharadwaj, L\. Gong, Y\. Yang, M\. Song, R\. Wang, Z\. Zhang, H\. Bao, S\. Lu, J\. Tu, Z\. Wang, Z\. Zhang, Z\. Chen, Y\. Jiang, Z\. Li, B\. Lyu, C\. Ma, P\. Xu, B\. Zhang, S\. Gu, H\. Hua, H\. Li, W\. Liao, C\. Liu, J\. Peng, H\. Sun, Z\. Xu, B\. Chen, J\. Cheng, Y\. Jiang, K\. Kuang, Y\. Li, Y\. Pan, Z\. Rao, A\. Schubert, Y\. Shen, V\. Siu, X\. Sun, K\. Zhang, X\. Zhang, Y\. Zhu, I\. S\. Chandok, L\. Ding, J\. Fan, A\. Glover, J\. Hu, Y\. Hu, W\. Huang, Z\. Jiang, H\. Jin, L\. Kim, M\. Liu, Y\. Liu, A\. Rafiei, X\. Shen, K\. Sun, S\. Sun, T\. Sun, E\. Wang, Y\. Wang, H\. Xing, S\. Xu, Y\. Xu, Z\. Xu, Z\. Yan, B\. Yuan, R\. Zhang, Y\. Zhang, Z\. Zhao, Liana, S\. B\. Antu, H\. Bai, C\. Bosio, J\. Cavanagh, P\. Cavazos\-Rehg, T\. Chen, X\. Chen, Y\. Chen, C\. Zhu, C\. Dai, S\. D\. Castro, Y\. Deng, K\. Dhole, J\. Ding, C\. Du, Z\. Du, H\. Fan, R\. Fan, H\. Fu, S\. Gu, Y\. Gu, C\. Guo, B\. Huang, B\. Huang, R\. Jaiswal, Z\. Jiang, R\. Jin, E\. Kasson, X\. Lan, J\. Lee, D\. Lei, C\. Li, D\. Li, H\. Li, H\. Li, J\. Li, X\. Li, Y\. Li, Y\. Li, Y\. Li, Z\. Li, W\. Liang, L\. Liao, K\. Q\. Lin, A\. Z\. Liu, C\. Liu, J\. Liu, K\. Liu, X\. Liu, P\. Lu, W\. Lv, Y\. Lyu, Q\. Mang, K\. Montgomery, Y\. Nie, R\. Ning, J\. Overwiening, X\. Pan, L\. Paraboschi, C\. F\. Park, J\. Purnomo, S\. Rajwal, S\. Rankin, B\. Ren, Y\. Rong, H\. Shang, V\. Shaw, F\. Shen, J\. Shen, M\. Shi, S\. Qiu, H\. Yao, T\. Shi, J\. So, V\. Susoy, H\. Szlyk, H\. Wang, J\. Wang, W\. Wang, X\. Wang, Z\. Wang, D\. Wong, A\. Wu, D\. Wu, F\. Wu, M\. ”\. Wu, Y\. Wu, Y\. Wu, Y\. Wu, Q\. Wuwu, W\. Xiao, Y\. Xiong, F\. Xu, R\. Xu, M\. Yan, B\. Yang, J\. Yang, S\. Yang, X\. Yang, Y\. Yang, H\. Ye, X\. Yu, Z\. Yu, C\. Zhang, C\. Zhang, H\. Zhang, H\. Zhang, J\. Zhang, K\. Zhang, S\. Zhang, W\. Zhang, W\. Zhang, Y\. Zhang, Y\. Zhang, B\. Zhao, Q\. Zhao, Y\. Zhao, Y\. Zheng, L\. Zhou, T\. Zhou, S\. Zhu, S\. Zhu, Y\. Zhu, Y\. Zhu, J\. Zuo, C\. Cai, H\. Casademunt, W\. Chen, C\. Cheng, N\. Deng, R\. Fu, T\. Fu, Y\. Han, H\. Ren, Z\. He, Q\. Jin, L\. Li, Y\. Li, S\. Liu, L\. Lu, L\. Zhou, S\. Mukherjee, Y\. Ouyang, Y\. Ren, D\. Shi, H\. Wu, Z\. Wu, H\. Yao, Z\. Yi, J\. Yu, R\. Zhan, H\. Zhou, B\. Zhu, J\. Zhu, A\. Yuille, Y\. Liu, R\. A\. Poldrack, J\. Li, Z\. Li, M\. Tao, J\. Huang, W\. Shi, C\. Spanos, L\. Sun, C\. Wang, O\. Xu, Z\. Dong, H\. Gomez, A\. Caliskan, A\. Emami, H\. Hu, Z\. Li, L\. Liu, M\. Niu, Y\. Shao, J\. Sun, M\. Tolonen, T\. Wang, S\. Das, Y\. Gao, W\. Guo, E\. J\. Schneider, Z\. Lu, Y\. Ma, M\. Mueller, R\. Poovendran, S\. Sojoudi, Y\. Zhu, and D\. SongAgents’ last exam\.External Links:2606\.05405,[Link](https://arxiv.org/abs/2606.05405)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p1.1)\.
- Team \(2025\)G\. TeamGemma 3 technical report\.CoRRabs/2503\.19786\.
- Touvronet al\.\(2023\)H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. LampleLLaMA: open and efficient foundation language models\.CoRRabs/2302\.13971\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. PolosukhinAttention is all you need\.InNIPS,pp\. 5998–6008\.
- Wanget al\.\(2024\)G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. AnandkumarVoyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=ehfRiF0R3a)
- Wanget al\.\(2025a\)R\. Wang, H\. Wang, B\. Xue, J\. Pang, S\. Liu, Y\. Chen, J\. Qiu, D\. F\. Wong, H\. Ji, and K\. WongHarnessing the reasoning economy: A survey of efficient reasoning for large language models\.CoRRabs/2503\.24377\.
- Wanget al\.\(2022\)R\. Wang, P\. Jansen, M\. Côté, and P\. AmmanabroluScienceWorld: is your agent smarter than a 5th grader?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 11279–11298\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.775),[Link](https://aclanthology.org/2022.emnlp-main.775/)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p4.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026a\)Y\. Wang, Y\. Zhang, W\. Shi, Z\. Ye, Y\. Miao, Y\. Sun, Q\. Gu, X\. Cai, L\. Guo, H\. Ye, and F\. FengCAST: game solvers as turn\-level teachers for llm agents\.External Links:2607\.25308,[Link](https://arxiv.org/abs/2607.25308)Cited by:[Appendix A](https://arxiv.org/html/2610.13058#A1.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026b\)Z\. Wang, Z\. Lu, Z\. Yao, J\. Wu, J\. Wu, Z\. Cai, Y\. Sun, Z\. Ye, L\. Hao, Q\. Gu, X\. Cai, Y\. Shen, and Y\. YangAgentOPSD: recursive self\-distillation for agentic reinforcement learning\.External Links:2608\.05987,[Link](https://arxiv.org/abs/2608.05987)Cited by:[Appendix A](https://arxiv.org/html/2610.13058#A1.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025b\)Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu, E\. Gottlieb, Y\. Lu, K\. Cho, J\. Wu, L\. Fei\-Fei, L\. Wang, Y\. Choi, and M\. LiRAGEN: understanding self\-evolution in LLM agents via multi\-turn reinforcement learning\.CoRRabs/2504\.20073\.
- Wanget al\.\(2026c\)Z\. Wang, Y\. Yan, H\. Li, T\. Pan, D\. Li, R\. Zhang, W\. Lu, J\. Xiao, Y\. Zhuang, and Y\. ShenMilestone\-guided policy learning for long\-horizon language agents\.InProceedings of the 43rd International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2605.06078)Cited by:[§B\.1](https://arxiv.org/html/2610.13058#A2.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Weiet al\.\(2025\)J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. GlaeseBrowseComp: a simple yet challenging benchmark for browsing agents\.External Links:2504\.12516,[Link](https://arxiv.org/abs/2504.12516)
- Wuet al\.\(2024\)Z\. Wu, Z\. Wu, F\. Xu, Y\. Wang, Q\. Sun, C\. Jia, K\. Cheng, Z\. Ding, L\. Chen, P\. P\. Liang, and Y\. QiaoOS\-atlas: a foundation action model for generalist gui agents\.External Links:2410\.23218,[Link](https://arxiv.org/abs/2410.23218)
- Xiet al\.\(2024\)Z\. Xi, Y\. Ding, W\. Chen, B\. Hong, H\. Guo, J\. Wang, D\. Yang, C\. Liao, X\. Guo, W\. He, S\. Gao, L\. Chen, R\. Zheng, Y\. Zou, T\. Gui, Q\. Zhang, X\. Qiu, X\. Huang, Z\. Wu, and Y\. JiangAgentGym: evolving large language model\-based agents across diverse environments\.External Links:2406\.04151
- Xiet al\.\(2025\)Z\. Xi, J\. Huang, C\. Liao, B\. Huang, H\. Guo, J\. Liu, R\. Zheng, J\. Ye, J\. Zhang, W\. Chen, W\. He, Y\. Ding, G\. Li, Z\. Chen, Z\. Du, X\. Yao, Y\. Xu, J\. Chen, T\. Gui, Z\. Wu, Q\. Zhang, X\. Huang, and Y\. JiangAgentGym\-rl: training llm agents for long\-horizon decision making through multi\-turn reinforcement learning\.External Links:2509\.08755,[Link](https://arxiv.org/abs/2509.08755)
- Xiaoet al\.\(2026\)T\. Xiao, Y\. Yuan, H\. Ivison, H\. Zhu, F\. Brahman, N\. Lambert, P\. Dasigi, N\. A\. Smith, and H\. HajishirziMeta\-reinforcement learning with self\-reflection for agentic search\.arXiv preprint arXiv:2603\.11327\.External Links:[Link](https://arxiv.org/abs/2603.11327)Cited by:[§2\.2](https://arxiv.org/html/2610.13058#S2.SS2.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. YuOSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2404.07972),[Link](https://doi.org/10.48550/arXiv.2404.07972)
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.CoRRabs/2505\.09388\.Cited by:[§C\.3](https://arxiv.org/html/2610.13058#A3.SS3.p1.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px4.p1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.CoRRabs/2412\.15115\.
- Yaoet al\.\(2022\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2207\.01206,[Link](https://arxiv.org/abs/2207.01206)
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/forum?id=WE\_vluYUL-X)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.13058#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.13058#S5.SS1.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2026\)Z\. Yao, Y\. Chen, Z\. Lu, Z\. Xu, Y\. Sun, Y\. Guo, Y\. Lu, Z\. Cai, K\. Zhang, Z\. Han, Z\. Wang, Z\. Ye, Q\. Gu, X\. Cai, W\. Liu, and Y\. ShenSkillRise: agentic reinforcement learning for cross\-task skill evolution\.External Links:2607\.26784,[Link](https://arxiv.org/abs/2607.26784)
- Yeet al\.\(2026\)Z\. Ye, W\. Shi, Y\. Liu, Y\. Wang, Z\. Cai, Y\. Shi, Q\. Gu, X\. Cai, and F\. FengLook before you leap: autonomous exploration for llm agents\.External Links:2605\.16143,[Link](https://arxiv.org/abs/2605.16143)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p1.1)\.
- Yeet al\.\(2025a\)Z\. Ye, Y\. Zhang, W\. Shi, X\. You, F\. Feng, and T\. ChuaVisualTrap: a stealthy backdoor attack on gui agents via visual grounding manipulation\.External Links:2507\.06899,[Link](https://arxiv.org/abs/2507.06899)
- Yeet al\.\(2025b\)Z\. Ye, Z\. Zhang, Y\. Zhang, J\. Ma, J\. Lin, and F\. FengDisentangling reasoning tokens and boilerplate tokens for language model fine\-tuning\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 20939–20957\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1078/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1078),ISBN 979\-8\-89176\-256\-5
- Yuet al\.\(2025a\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, W\. Dai, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.CoRRabs/2503\.14476\.
- Yuet al\.\(2025b\)T\. Yu, B\. Ji, S\. Wang, S\. Yao, Z\. Wang, G\. Cui, L\. Yuan, N\. Ding, Y\. Yao, Z\. Liu, M\. Sun, and T\. ChuaRLPR: extrapolating RLVR to general domains without verifiers\.CoRRabs/2506\.18254\.
- Yuanet al\.\(2025\)S\. Yuan, Z\. Chen, Z\. Xi, J\. Ye, Z\. Du, and J\. ChenAgent\-R: training language model agents to reflect via iterative self\-training\.arXiv preprint arXiv:2501\.11425\.External Links:[Link](https://arxiv.org/abs/2501.11425)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.13058#S2.SS1.p1.1)\.
- Z\.AI \(2025\)Z\.AIGLM\-4\.7: advancing the coding capability\.External Links:[Link](https://z.ai/blog/glm-4.7)
- Zenget al\.\(2023\)A\. Zeng, M\. Liu, R\. Lu, B\. Wang, X\. Liu, Y\. Dong, and J\. TangAgentTuning: enabling generalized agent abilities for llms\.External Links:2310\.12823
- Zenget al\.\(2025\)Y\. Zeng, Y\. Huang, C\. Xu, Q\. Sun, J\. Yan, G\. Xu, T\. Yang, and F\. LianZero reinforcement learning towards general domains\.CoRRabs/2510\.25528\.
- Zhanget al\.\(2025\)H\. Zhang, X\. Liu, B\. Lv, X\. Sun, B\. Jing, I\. L\. Iong, Z\. Hou, Z\. Qi, H\. Lai, Y\. Xu, R\. Lu, H\. Wang, J\. Tang, and Y\. DongAgentRL: scaling agentic reinforcement learning with a multi\-turn, multi\-task framework\.External Links:2510\.04206,[Link](https://arxiv.org/abs/2510.04206)
- Zhanget al\.\(2024\)K\. Zhang, J\. Li, G\. Li, X\. Shi, and Z\. JinCodeAgent: enhancing code generation with tool\-integrated agent systems for real\-world repo\-level coding challenges\.External Links:2401\.07339,[Link](https://arxiv.org/abs/2401.07339)
- Zhang and Math\-AI \(2024\)Y\. Zhang and T\. Math\-AIAmerican invitational mathematics examination \(aime\) 2024\.
- Zhang and Math\-AI \(2025\)Y\. Zhang and T\. Math\-AIAmerican invitational mathematics examination \(aime\) 2025\.
- Zhanget al\.\(2026\)Z\. Zhang, T\. Fang, K\. Ma, W\. Yu, H\. Zhang, H\. Mi, and D\. YuWebRollback: enhancing web agents with explicit rollback mechanisms\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 187–197\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-short.12),[Link](https://aclanthology.org/2026.eacl-short.12/)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.13058#S2.SS1.p1.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, A\. Zhang, M\. Li, and A\. SmolaAutomatic chain of thought prompting in large language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5NTt8GFjUHkr)Cited by:[§2\.1](https://arxiv.org/html/2610.13058#S2.SS1.p1.1)\.
- Zhouet al\.\(2024\)A\. Zhou, K\. Yan, M\. Shlapentokh\-Rothman, H\. Wang, and Y\. WangLanguage agent tree search unifies reasoning, acting, and planning in language models\.InProceedings of the 41st International Conference on Machine Learning,Vol\.235,pp\. 62138–62160\.External Links:[Link](https://proceedings.mlr.press/v235/zhou24r.html)Cited by:[§1](https://arxiv.org/html/2610.13058#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.13058#S2.SS1.p1.1)\.
- Zhouet al\.\(2023\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.ArXivabs/2307\.13854\.External Links:[Link](https://api.semanticscholar.org/CorpusID:260164780)
- Zhouet al\.\(2026\)S\. Zhou, T\. Zhou, Y\. Yang, G\. Long, D\. Ye, J\. Jiang, and C\. ZhangWALL\-e: world alignment by neurosymbolic learning improves world model\-based LLM agents\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=DorAT49sxj)
- Zhouet al\.\(2025\)Y\. Zhou, S\. Levine, J\. Weston, X\. Li, and S\. SukhbaatarSelf\-challenging language model agents\.External Links:2506\.01716,[Link](https://arxiv.org/abs/2506.01716)

## Appendix ALimitations and Future Work

Our results motivate two directions for extending retrospective exploration\.

##### \(i\) Extending recovery to broader environments\.

The current framework assumes access to state restoration through snapshots or reset\-and\-replay; the latter requires deterministic transitions and resettable external state \(Appendix[D\.1](https://arxiv.org/html/2610.13058#A4.SS1)\)\. Extending recovery to environments with approximate restoration or irreversible side effects is an important next step\. Future work could explore restoration controllers and compensating actions, together with checks on the recovered state, to determine when earlier progress can be safely reused\.

##### \(ii\) Improving recovery credit and efficiency\.

ReTRAIL combines task\-wide execution feedback with branch advancement, an environment\-progress proxy for the utility of retained experience\. The current execution return ends at each rewind boundary, while subsequent advancement supervises reflection \(Appendix[D\.6](https://arxiv.org/html/2610.13058#A4.SS6)\)\. This design motivates studying credit assignment\([Wang et al\., 2026b](https://arxiv.org/html/2610.13058#bib.bib96);[Wang et al\., 2026a](https://arxiv.org/html/2610.13058#bib.bib100)\)across recovery boundaries, so that rewind timing and target selection can receive direct feedback from the resumed branch\. Accounting for restoration and reflection costs, and developing progress estimators for settings with sparse or noisy feedback, could further broaden the framework’s practical use\.

## Appendix BAdditional Results and Analysis

### B\.1Recovery Gains by Reference\-Solution Length

We group held\-out instances by reference\-solution length from the initial state\. All methods use the same groups, independent of their observed rollout lengths\.

##### Reference\-solution length and grouping\.

For Sokoban, the reference is the breadth\-first\-search solution lengthd0d\_\{0\}\(Appendix[C\.2](https://arxiv.org/html/2610.13058#A3.SS2)\); for MineSweeper, it is the minimum reveal count under correct deduction\. For ALFWorld and ScienceWorld, it is the length of the provided expert or gold action sequence\. We use the term*reference\-solution length*because expert sequences need not be shortest solutions\. On ALFWorld, we follow the length boundaries of[Wang et al\. \(2026c\)](https://arxiv.org/html/2610.13058#bib.bib3): Short \(at most 4 actions\), Medium \(5–7\), and Long \(more than 7\)\. For each remaining environment, we use the terciles of its held\-out reference lengths\. The bins therefore describe length within each environment, rather than a common absolute action count across environments\.

##### Metrics and aggregation\.

Within each length bin, we compute the proactive gainΔpro=T​@​1−P​@​1\\Delta\_\{\\mathrm\{pro\}\}=\\mathrm\{T@1\}\-\\mathrm\{P@1\}and the reflection\-guided retry gainΔref=T​@​3−T​@​1\\Delta\_\{\\mathrm\{ref\}\}=\\mathrm\{T@3\}\-\\mathrm\{T@1\}on the same instances for every method\. We then average each gain across the four environments with equal weight; values are success\-rate differences in percentage points\.

Figure 5:Recovery gains by reference\-solution length\. The upper row shows reflection\-guided retry gainΔref=T​@​3−T​@​1\\Delta\_\{\\mathrm\{ref\}\}=\\mathrm\{T@3\}\-\\mathrm\{T@1\}; the lower row shows proactive gainΔpro=T​@​1−P​@​1\\Delta\_\{\\mathrm\{pro\}\}=\\mathrm\{T@1\}\-\\mathrm\{P@1\}\. Values are equal\-weight averages across the four environments, in percentage points\. Short, Medium, and Long use the environment\-specific grouping rules above\.
##### Observed pattern\.

ReTRAIL’s retry gain increases from\+4\.5\+4\.5on short tasks to\+9\.2\+9\.2on medium and\+19\.0\+19\.0on long tasks; LaMer’s corresponding gains are\+5\.6\+5\.6,\+6\.0\+6\.0, and\+13\.2\+13\.2\. Thus, ReTRAIL’s retry\-gain advantage over LaMer is−1\.1\-1\.1,\+3\.2\+3\.2, and\+5\.8\+5\.8points in the three groups\. Its proactive gain also increases from\+1\.3\+1\.3to\+5\.3\+5\.3and\+7\.8\+7\.8\. These descriptive results show larger recovery gains for ReTRAIL on longer\-reference tasks, while LaMer has the larger retry gain in the short group\. They do not isolate length from other task properties or establish a causal effect of reflection alone\.

### B\.2Average Performance Across Benchmarks

Table[4](https://arxiv.org/html/2610.13058#A2.T4)summarizes Table[1](https://arxiv.org/html/2610.13058#S5.T1)using the equal\-benchmark rule in Appendix[C\.5](https://arxiv.org/html/2610.13058#A3.SS5)\. It is computed from the reported, rounded benchmark means, so minor rounding differences are possible\. The reported spread is across evaluation seeds, not additional runs\. The forward\-only results support competitive performance, without establishing statistically significant superiority on each benchmark\.

Table 4:Equal\-weight averages across the four benchmarks, computed from the reported benchmark means in Table[1](https://arxiv.org/html/2610.13058#S5.T1)and rounded to one decimal place\.±\\pmis the standard deviation across the three evaluation seeds\. These are aggregate point estimates, not additional runs\.

## Appendix CExperimental Setup

### C\.1Benchmarks and Data Partitions

All four benchmarks use text interaction and an environment\-restoration interface\. Table[5](https://arxiv.org/html/2610.13058#A3.T5)lists the registered task instances loaded by the training entrypoints\. The registered Sokoban variants use6×66\\times 6or7×77\\times 7rooms with two or three boxes; each has 256 training and 128 test entries\. The difficulty experiment varies the box count separately\. Dataset counts refer to task entries, not repeated rollouts or evaluation seeds\.

Table 5:Registered task\-instance partitions loaded by the training entrypoints\. Counts describe distinct dataset entries, not generated continuations or repeated evaluations\. ALFWorld’s selected test partition isvalid\_seen; its separately registeredvalid\_unseenpartition contains 134 entries\. ScienceWorld uses the L0 partition\.
### C\.2Rewards and Progress Signals

The optimizer uses three quantities: the per\-decision rewardδt\\delta\_\{t\}, the segment\-endpoint scoreSS, and the branch scoreV⁡\(ξ∣s\)V\(\\xi\\mid s\)\. Table[6](https://arxiv.org/html/2610.13058#A3.T6)gives their definitions\. If one decision executes several primitive actions,δt\\delta\_\{t\}sums their environment rewards\. A decision with no environment transition, including a rewind request, hasδt=0\\delta\_\{t\}=0\.

Table 6:Training signals by environment\.δt\\delta\_\{t\}supplies the execution return,SSthe task\-wide endpoint score, andVVthe branch score for reflection\. Derived advantages and branch\-score differences are not policy inputs\.##### MineSweeper\.

Completion and mine revelation end the attempt, with the corresponding terminal reward in Table[6](https://arxiv.org/html/2610.13058#A3.T6)applied to that action\.SSaccumulates rewards along the retained path froms0s\_\{0\}; restoring a checkpoint also restores this prefix score\.VVsums rewards only after the selected anchor, so the common prefix cancels in the branch comparison\.

##### Sokoban\.

Letd0d\_\{0\}be the shortest\-solution distance from the initial state anddendd\_\{\\mathrm\{end\}\}the remaining distance at a branch endpoint, both computed by breadth\-first search\. We useV=d0−dendV=d\_\{0\}\-d\_\{\\mathrm\{end\}\}, takingdend=0d\_\{\\mathrm\{end\}\}=0for a solved state anddend=d0d\_\{\\mathrm\{end\}\}=d\_\{0\}for a deadlocked or unsolvable state\. The new\-minus\-old branch score is thereforedend−−dend\+d\_\{\\mathrm\{end\}\}^\{\-\}\-d\_\{\\mathrm\{end\}\}^\{\+\}\. These search distances are evaluator\-only and are not shown to the policy\.

##### ALFWorld and ScienceWorld\.

ALFWorld measures the fraction of satisfied goal conditions; ScienceWorld uses the native task\-completion score\. In each environment,SSis the endpoint score andVVis the maximum score along the branch, including its starting state\. These differ when progress can be undone\. ScienceWorld’s native score is visible in its observations; the derived branch\-advancement reward is not\.

### C\.3Training Configuration and Sampling Budget

All trained methods start from Qwen3\-4B\([Yang et al\., 2025](https://arxiv.org/html/2610.13058#bib.bib61)\)\. They share the optimizer, learning\-rate schedule, task splits, and decoding settings\. Table[7](https://arxiv.org/html/2610.13058#A3.T7)lists the configuration\. Prompt serialization follows each rollout interface \(Appendix[E](https://arxiv.org/html/2610.13058#A5)\); shared task instructions do not imply identical execution and reflection conversations\.

Per task and update, baselines sample 24 trajectories, each allowing 60 model calls\. ReTRAIL samples eight trials, each allowing 60 calls and at most three macro\-attempts\. This aligns nominal attempt slots, with call ceilings of 1,440 for the baseline and 480 for ReTRAIL\. These ceilings do not measure realized calls, tokens, or compute\. Experience and advantage\-normalization groups remain within each task\.

Table 7:Training and evaluation configuration of ReTRAIL\. Optimizer settings are shared with the trained baselines\. The retrospective\-exploration and objective blocks describe ReTRAIL; a trial can contain several forward segments and should not be equated with one baseline trajectory\.
### C\.4Evaluation Protocol and Call Accounting

Table[8](https://arxiv.org/html/2610.13058#A3.T8)distinguishes the units used throughout the paper\. Rewinding restores an active path but never refunds calls already made\.

Table 8:Units used for rollout and budget accounting\.P@1 disables recovery\. T@1 allows proactive recovery within one macro\-attempt and stops at the first irreversible failure\. T@3 also permits failure\-triggered recovery, up to three macro\-attempts\. Training uses the same attempt cap\. Model\-initiated rewinds are limited to three per trial\. The attempt\-scaling experiment varies the macro\-attempt cap from one to four\.

Training and evaluation impose a global ceiling of 60 model calls per trial:p\+f≤60p\+f\\leq 60, whereppcounts execution andffreflection\. A proactive recovery uses an execution call to request rewind and a separate reflection call\. Forced recovery uses a reflection call after the preceding execution\. Recovery starts only when its attempt/rewind allowance permits it and at least two calls remain for reflection and continuation\. Reset\-and\-replay without model generation adds environment actions but no model calls\.

The prompt backends also track a branch allowance\. MineSweeper and Sokoban restore its saved consumption at the checkpoint; the supplied ALFWorld flow resets its segment\-turn counter\. These backend rules are distinct from the global call ceiling\. A forward\-segment cap counts both recovery types, whereas the macro\-attempt count increases only after failure\-triggered recovery\. Algorithm[1](https://arxiv.org/html/2610.13058#alg1)takes these rules as configuration inputs\.

### C\.5Aggregation Across Benchmarks and Seeds

Letps,e\(k\)p\_\{s,e\}^\{\(k\)\}be success on benchmarkee, under evaluation seedssand protocolk∈\{P​@​1,T​@​1,T​@​3\}k\\in\\\{\\mathrm\{P@1\},\\mathrm\{T@1\},\\mathrm\{T@3\}\\\}\. We average equally across the four benchmarks and three seeds:

Ms\(k\)=14​∑e=14ps,e\(k\),M¯\(k\)=13​∑s=13Ms\(k\)=14​∑e=14p¯e\(k\)\.M\_\{s\}^\{\(k\)\}=\\frac\{1\}\{4\}\\sum\_\{e=1\}^\{4\}p\_\{s,e\}^\{\(k\)\},\\qquad\\overline\{M\}^\{\(k\)\}=\\frac\{1\}\{3\}\\sum\_\{s=1\}^\{3\}M\_\{s\}^\{\(k\)\}=\\frac\{1\}\{4\}\\sum\_\{e=1\}^\{4\}\\overline\{p\}\_\{e\}^\{\(k\)\}\.\(13\)Each benchmark has equal weight regardless of its instance count\. This differs from pooling all test instances\. Table[4](https://arxiv.org/html/2610.13058#A2.T4)reports the resulting means together with the standard deviation across the three evaluation seeds\. Individual seed values cannot be recovered from these means\.

## Appendix DMethod Details

This section follows the data flow: restore an environment state, resume execution with branch memory, record the rollout, and construct the training objective\.

### D\.1Environment Restoration and Checkpoints

The environment harness implements rewind\. MineSweeper and Sokoban store and restore serializable state snapshots\. ALFWorld and ScienceWorld reset the task and replay the recorded actions up to the selected point, since their engines do not expose a general snapshot interface\. Replay requires deterministic transitions and resettable external state\.

The active path stores ordered checkpoints𝒦=\(C0,…,Ck\)\\mathcal\{K\}=\(C\_\{0\},\\ldots,C\_\{k\}\), each containing its state, path history, and restoration information\. A targetj<kj<kidentifies a path position: identical observations can occur at different positions\. RestoringCjC\_\{j\}truncates the active path but preserves the chronological training archiveΓ\\Gamma\. This archive stores all outputs with their original input contexts, segment boundaries, feedback, and abandoned/resumed branch pairs\. Thus an abandoned branch remains available for training after leaving the live prompt\.

### D\.2Recovery Triggers and Reflection

A model\-initiated recovery begins with an execution output requesting rewind\. A forced recovery begins when the controller detects a configured failure\. Both generate reflection before restoration, while the abandoned branch is still available\. The reflection returns branch memory in<remark\>tags\.

For MineSweeper and Sokoban, a valid model\-selected target uses a known\-target reflection prompt; forced recovery asks the model to select a checkpoint as well\. ALFWorld and ScienceWorld also request a checkpoint in their reflection responses\. Thus*forced*identifies who triggers recovery, not a fixed destination immediately before failure\. Table[9](https://arxiv.org/html/2610.13058#A4.T9)summarizes the two paths; Appendix[C\.4](https://arxiv.org/html/2610.13058#A3.SS4)specifies their budget limits\.

Table 9:Recovery triggers and model calls\.Restoration follows reflection, so the abandoned branch remains available when its memory is generated\. Checkpoint selection and fallback parsing follow the environment\-specific prompt implementation\.
### D\.3Context After Recovery

The logical historyhhis the retained execution path\. The actual inputccis the conversation produced by the environment’s message builder\. The distinction matters because the backends serialize history differently\.

MineSweeper and Sokoban save a conversation at each checkpoint\. Recovery restores this prefix, removes the abandoned continuation, and appends a user message with the new branch memory\. A previous trailing reflection at the same checkpoint is first made available to the reflection call, then replaced\. The next observation is appended as another user message\.

ScienceWorld also restores a message prefix, but includes branch memories in its subsequent observation\. ALFWorld rebuilds the resumed conversation from the system message, restored observation, admissible commands, and branch memories; it does not copy the earlier conversation prefix\. Appendix[E](https://arxiv.org/html/2610.13058#A5)gives the literal templates and formatting examples\. Training always uses the recorded inputcc, without adding later reflections to earlier samples\.

### D\.4Rollout Algorithm

Algorithm[1](https://arxiv.org/html/2610.13058#alg1)records one trial under a frozen behavior policy\. Its four blocks follow Section[3](https://arxiv.org/html/2610.13058#S3)\. The active historyhh, checkpoints𝒦\\mathcal\{K\}, and memoryℰ\\mathcal\{E\}guide execution; the archiveΓ\\Gammaretains every generation with its original context and feedback\. A segmentσ\\sigmagroups execution decisions between recovery boundaries, andn=p\+fn=p\+fcounts all model calls\.

Invalid outputs include unparseable actions and disabled or out\-of\-range rewind requests; they consume a call and receive feedback without changing the environment\. Checkpoints include states reached on failure\. Recovery requires allowance for reflection and continuation \(Appendix[C\.4](https://arxiv.org/html/2610.13058#A3.SS4)\)\. Target parsing follows Appendix[D\.2](https://arxiv.org/html/2610.13058#A4.SS2); memory and context updates follow Appendix[D\.3](https://arxiv.org/html/2610.13058#A4.SS3)\. Restoring a checkpoint never truncatesΓ\\Gammaor refunds calls\.

Algorithm 1Retrospective exploration for one task1:task

uu, frozen policy

πold\\pi\_\{\\mathrm\{old\}\}, protocol

ℬ\\mathcal\{B\}, call limit

Cmax=60C\_\{\\max\}=60
2:reset task; initialize

h,𝒦=\(C0\),ℰ=∅h,\\mathcal\{K\}=\(C\_\{0\}\),\\mathcal\{E\}=\\varnothingand protocol counters

3:

Γ←∅\\Gamma\\leftarrow\\varnothing;

σ←∅\\sigma\\leftarrow\\varnothing;

n←0n\\leftarrow 0
4:while

n<Cmaxn<C\_\{\\max\}and

ℬ\\mathcal\{B\}permits executiondo

5:1\. ExecuteEquation[2](https://arxiv.org/html/2610.13058#S3.E2)

6:

c←BuildContext​\(u,h,ℰ,𝒦\)c\\leftarrow\\textsc\{BuildContext\}\(u,h,\\mathcal\{E\},\\mathcal\{K\}\);

y∼πold\(⋅∣c\)y\\sim\\pi\_\{\\mathrm\{old\}\}\(\\cdot\\mid c\);

n←n\+1n\\leftarrow n\+1
7:record

\(c,y\)\(c,y\)in

Γ\\Gamma; append the execution decision to

σ\\sigma
8:if

yyis invalidthenrecord feedback;continue

9:2\. Detect recoveryEquation[3](https://arxiv.org/html/2610.13058#S3.E3)

10:if

yyrequests rewindthen

11:retain the requested target⊳\\trianglerightmodel\-initiated recovery

12:else

13:execute the parsed action; update

h,𝒦h,\\mathcal\{K\}; record rewards and observations in

Γ\\Gamma
14:iftask solvedthen break

15:ifno failure triggerthen continue⊳\\trianglerightotherwise, forced recovery

16:endif

17:close

σ\\sigmain

Γ\\Gamma
18:if

n\+2\>Cmaxn\+2\>C\_\{\\max\}or

ℬ\\mathcal\{B\}disallows recovery/continuationthen break

19:3\. ReflectEquation[4](https://arxiv.org/html/2610.13058#S3.E4)

20:build

crefc\_\{\\mathrm\{ref\}\}from the active branch, memory, and any requested target

21:

z∼πold\(⋅∣cref\)z\\sim\\pi\_\{\\mathrm\{old\}\}\(\\cdot\\mid c\_\{\\mathrm\{ref\}\}\);

n←n\+1n\\leftarrow n\+1; record

\(cref,z\)\(c\_\{\\mathrm\{ref\}\},z\)in

Γ\\Gamma
22:resolve

\(j,e\)\(j,e\)using any requested target,

zz, and the configured fallback

23:ifno valid target

0≤j<\|𝒦\|−10\\leq j<\|\\mathcal\{K\}\|\-1remainsthen break

24:4\. Restore and resumeEquation[6](https://arxiv.org/html/2610.13058#S3.E6)

25:record abandoned suffix

β=τj:k\\beta=\\tau\_\{j:k\}and recovery link in

Γ\\Gamma, where

k=\|𝒦\|−1k=\|\\mathcal\{K\}\|\-1
26:restore

CjC\_\{j\}; truncate

h,𝒦h,\\mathcal\{K\}to the retained prefix

27:update

ℰ\\mathcal\{E\}and resumed context with

ee; update branch/protocol counters

28:start a new empty

σ\\sigmalinked to this recovery

29:endwhile

30:close any nonempty open segment in

Γ\\Gamma;return

Γ\\Gamma

### D\.5State\-Anchored GRPO

Each execution decision is one training sample\. Its remaining segment supplies the return used to score that decision, while the loss applies only to its own generated output\.

#### D\.5\.1Decision Samples and Returns

For taskuu, collectBBtrials\{Γi\}i=1B\\\{\\Gamma\_\{i\}\\\}\_\{i=1\}^\{B\}and their forward segments𝒮⁡\(u\)\\mathcal\{S\}\(u\)\. Index decisionttin segmentσi,m\\sigma\_\{i,m\}byq=\(i,m,t\)q=\(i,m,t\), and letTi,mT\_\{i,m\}be the segment length:

ℐ\(u\)=\{\(i,m,t\):σi,m∈𝒮\(u\),0≤t<Ti,m\},ξq=σi,m\[t:Ti,m\]\.\\mathcal\{I\}\(u\)=\\\{\(i,m,t\):\\sigma\_\{i,m\}\\in\\mathcal\{S\}\(u\),\\;0\\leq t<T\_\{i,m\}\\\},\\qquad\\xi\_\{q\}=\\sigma\_\{i,m\}\[t:T\_\{i,m\}\]\.\(14\)The continuationξq\\xi\_\{q\}ends with its original segment, at success, failure, rewind, or budget exhaustion\. A proactive rewind is the segment’s final execution decision\. Record its starting statesqs\_\{q\}, logical retained historyhqh\_\{q\}, and actual input context\. Let𝒴q\\mathcal\{Y\}\_\{q\}contain only the generated reasoning and action or rewind request for decisionqq\. The indexed collection𝒞⁡\(u\)=\(ξq\)q∈ℐ⁡\(u\)\\mathcal\{C\}\(u\)=\(\\xi\_\{q\}\)\_\{q\\in\\mathcal\{I\}\(u\)\}contains return records, not extra sequences to train again\.

For example, an old segments0→s1→s2→s3s\_\{0\}\\to s\_\{1\}\\to s\_\{2\}\\to s\_\{3\}that ends in a rewind tos1s\_\{1\}supplies a continuation froms1s\_\{1\}, even though the segment began ats0s\_\{0\}\. This can be compared with the new continuation froms1s\_\{1\}\. The two decisions share a state and logical prefix, but retain their original generation contexts and potentially different branch memories\.

#### D\.5\.2Execution Advantages

The return from decisionqqand its comparison group are

Gq=∑ℓ=tTi,m−1γℓ−tδi,m,ℓ,ℋu\(s,h\)=\{q∈ℐ\(u\):sq=s,hq=h\},G\_\{q\}=\\sum\_\{\\ell=t\}^\{T\_\{i,m\}\-1\}\\gamma^\{\\ell\-t\}\\delta\_\{i,m,\\ell\},\\qquad\\mathcal\{H\}\_\{u\}\(s,h\)=\\\{q\\in\\mathcal\{I\}\(u\):s\_\{q\}=s,\\;h\_\{q\}=h\\\},\(15\)Hereδi,m,ℓ\\delta\_\{i,m,\\ell\}sums environment rewards produced by execution decisionℓ\\ell\. A rewind request has zero transition reward; reflection calls do not enter this return\. Group members share a task, environment state, and retained history\. Their accumulated experience and remaining global budget may differ, so the comparison does not hold the full policy input fixed\.

The macro signal uses endpoint progressSi,mS\_\{i,m\}, with each segment contributing once to normalization regardless of its length\. For any decision in that segment, the main text’sSξqS\_\{\\xi\_\{q\}\}equalsSi,mS\_\{i,m\}\. The combined advantage is

Aqstate\\displaystyle A\_\{q\}^\{\\mathrm\{state\}\}=Normℋu​\(sq,hq\)⁡\(Gq\),\\displaystyle=\\operatorname\{Norm\}\_\{\\mathcal\{H\}\_\{u\}\(s\_\{q\},h\_\{q\}\)\}\(G\_\{q\}\),\(16\)Aqcont\\displaystyle A\_\{q\}^\{\\mathrm\{cont\}\}=Norm𝒮⁡\(u\)⁡\(Si,m\)⏟Ai,mmacro\+λstate​Aqstate\.\\displaystyle=\\underbrace\{\\operatorname\{Norm\}\_\{\\mathcal\{S\}\(u\)\}\(S\_\{i,m\}\)\}\_\{A\_\{i,m\}^\{\\mathrm\{macro\}\}\}\+\\lambda\_\{\\mathrm\{state\}\}A\_\{q\}^\{\\mathrm\{state\}\}\.Norm\\operatorname\{Norm\}subtracts the group mean and divides by its standard deviation plusϵ\\epsilon\. Singleton and constant\-score groups have zero advantage; empty groups contribute no samples\. These rules apply separately to macro, state, and reflection groups within each task\. When a decision has no local comparison, its macro term remains available\.

All decisions in a segment share its macro advantage but have their own local advantages\. Each advantage applies only to that decision’s output\. A reward can contribute to several earlier returns without duplicating output tokens in the loss\. Repeated visits are separate generation events\.

#### D\.5\.3Reflection Rewards and Eligibility

At a recovery point, reflection is scored by the improvement of the resumed branch over the abandoned suffix:

rmref=V⁡\(σm\+1∣sjm\)−V⁡\(βm∣sjm\)\.r\_\{m\}^\{\\mathrm\{ref\}\}=V\(\\sigma\_\{m\+1\}\\mid s\_\{j\_\{m\}\}\)\-V\(\\beta\_\{m\}\\mid s\_\{j\_\{m\}\}\)\.\(17\)The evaluator computesVVusing the environment\-specific definitions in Appendix[C\.2](https://arxiv.org/html/2610.13058#A3.SS2)\. Let𝒴i,mref\\mathcal\{Y\}^\{\\mathrm\{ref\}\}\_\{i,m\}be the reflection output\. Eligible recoveries are

ℛ\(u\)=\{\(i,m\):𝒴i,mref≠∅,Ti,m\+1\>0,vi,m−,vi,m\+are defined\}\.\\mathcal\{R\}\(u\)=\\big\\\{\(i,m\):\\;\\mathcal\{Y\}^\{\\mathrm\{ref\}\}\_\{i,m\}\\neq\\varnothing,\\;T\_\{i,m\+1\}\>0,\\;v^\{\-\}\_\{i,m\},v^\{\+\}\_\{i,m\}\\text\{ are defined\}\\big\\\}\.\(18\)Herevi,m−v^\{\-\}\_\{i,m\}andvi,m\+v^\{\+\}\_\{i,m\}are the abandoned and resumed branch scores\. A missing resumed segment has length zero\. Eligibility requires an observed target, not positive advancement or eventual success\. Both recovery types are normalized together overℛ⁡\(u\)\\mathcal\{R\}\(u\)within the same task\. If no continuation is recorded, the reflection tokens receive no loss; earlier execution outputs keep their usual contribution\.

#### D\.5\.4Loss Masks and Policy Update

Execution and reflection use disjoint output masks\. A rewind request made during execution belongs to𝒴q\\mathcal\{Y\}\_\{q\}; a checkpoint choice generated within a reflection belongs to that reflection\. Prompts, observations, copied memories, and retained\-prefix tokens are masked out\.

Let𝒴ref​\(u\)\\mathcal\{Y\}\_\{\\mathrm\{ref\}\}\(u\)contain the eligible reflection tokens, with advantagesArefA^\{\\mathrm\{ref\}\}normalized overℛ⁡\(u\)\\mathcal\{R\}\(u\)\. The task objective is

𝒥u​\(θ\)\\displaystyle\\mathcal\{J\}\_\{u\}\(\\theta\)=1max⁡\{1,\|ℐ⁡\(u\)\|\}​∑q∈ℐ⁡\(u\)𝒥GRPO​\(θ,𝒴q,Aqcont\)\\displaystyle=\\frac\{1\}\{\\max\\\{1,\|\\mathcal\{I\}\(u\)\|\\\}\}\\sum\_\{q\\in\\mathcal\{I\}\(u\)\}\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\\\!\\left\(\\theta;\\mathcal\{Y\}\_\{q\},A\_\{q\}^\{\\mathrm\{cont\}\}\\right\)\(19\)\+λref​𝒥GRPO​\(θ,𝒴ref​\(u\),Aref\),\\displaystyle\+\\lambda\_\{\\mathrm\{ref\}\}\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\\\!\\left\(\\theta;\\mathcal\{Y\}\_\{\\mathrm\{ref\}\}\(u\),A^\{\\mathrm\{ref\}\}\\right\),𝒥GRPO\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}is the token\-averaged clipped policy surrogate\. The execution term first averages tokens within a decision, then averages over decisions\. An empty reflection set or a term with no contributing tokens gives zero loss\. Both current and behavior policies use each token’s original context\. The batch combines task objectives without pooling their normalization groups\.

Algorithm[2](https://arxiv.org/html/2610.13058#alg2)summarizes the update\. All rollouts are collected before optimization; the behavior policy, data, and advantages remain fixed throughout the corresponding update\.

Algorithm 2One State\-Anchored GRPO update with task\-local groups1:nonempty task batch

𝒰\\mathcal\{U\}, current parameters

θ\\theta,

BBtrials per task, weights

λstate,λref\\lambda\_\{\\mathrm\{state\}\},\\lambda\_\{\\mathrm\{ref\}\}
2:

θold←θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\theta⊳\\trianglerightfreeze the behavior policy for this batch

3:foreach task

u∈𝒰u\\in\\mathcal\{U\}do

4:sample

\{Γi\}i=1B\\\{\\Gamma\_\{i\}\\\}\_\{i=1\}^\{B\}with

πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}using Algorithm[1](https://arxiv.org/html/2610.13058#alg1)

5:collect segments

𝒮⁡\(u\)\\mathcal\{S\}\(u\), decisions

ℐ⁡\(u\)\\mathcal\{I\}\(u\), and return records

𝒞⁡\(u\)\\mathcal\{C\}\(u\)
6:retain each decision’s context, behavior log\-probabilities, and output mask

7:compute

Si,mS\_\{i,m\}once per segment and return\-to\-go

GqG\_\{q\}for each decision

8:normalize

\{Si,m\}σi,m∈𝒮⁡\(u\)\\\{S\_\{i,m\}\\\}\_\{\\sigma\_\{i,m\}\\in\\mathcal\{S\}\(u\)\}to obtain

Ai,mmacroA\_\{i,m\}^\{\\mathrm\{macro\}\}
9:group visits by

\(sq,hq\)\(s\_\{q\},h\_\{q\}\)within

uu; compute

AqstateA\_\{q\}^\{\\mathrm\{state\}\}and

AqcontA\_\{q\}^\{\\mathrm\{cont\}\}by Equation[16](https://arxiv.org/html/2610.13058#A4.E16)

10:form the eligible recovery set

ℛ⁡\(u\)\\mathcal\{R\}\(u\)in Equation[18](https://arxiv.org/html/2610.13058#A4.E18)

11:foreach

\(i,m\)∈ℛ⁡\(u\)\(i,m\)\\in\\mathcal\{R\}\(u\)do

12:

ri,mref←V⁡\(σi,m\+1∣sji,m\)−V⁡\(βi,m∣sji,m\)r\_\{i,m\}^\{\\mathrm\{ref\}\}\\leftarrow V\(\\sigma\_\{i,m\+1\}\\mid s\_\{j\_\{i,m\}\}\)\-V\(\\beta\_\{i,m\}\\mid s\_\{j\_\{i,m\}\}\)
13:endfor

14:normalize

\{ri,mref\}\(i,m\)∈ℛ⁡\(u\)\\\{r\_\{i,m\}^\{\\mathrm\{ref\}\}\\\}\_\{\(i,m\)\\in\\mathcal\{R\}\(u\)\}to obtain

Ai,mrefA\_\{i,m\}^\{\\mathrm\{ref\}\}
15:form

𝒥u​\(θ\)\\mathcal\{J\}\_\{u\}\(\\theta\)with each execution output once \(Equation[19](https://arxiv.org/html/2610.13058#A4.E19)\)

16:endfor

17:update

θ\\thetafrom task objectives; keep

θold\\theta\_\{\\mathrm\{old\}\}, data, and advantages fixed

### D\.6Credit Assignment at a Rewind: An Analytic Example

Consider actionaawith rewarddd, followed by proactive rewindρ\\rho\. Their decision indices areqa,qρq\_\{a\},q\_\{\\rho\}, and their shared macro advantage isM=Norm𝒮⁡\(u\)⁡\(Si,m\)M=\\operatorname\{Norm\}\_\{\\mathcal\{S\}\(u\)\}\(S\_\{i,m\}\)\. Rewind ends the segment with zero transition reward:

Gqa\\displaystyle G\_\{q\_\{a\}\}=d\+γ​0=d,\\displaystyle=d\+\\gamma\\,0=d,Gqρ\\displaystyle G\_\{q\_\{\\rho\}\}=0,\\displaystyle=0,\(20\)Aqacont\\displaystyle A\_\{q\_\{a\}\}^\{\\mathrm\{cont\}\}=M\+λstateNormℋu​\(sqa,hqa\)\(d\),\\displaystyle=M\+\\lambda\_\{\\mathrm\{state\}\}\\operatorname\{Norm\}\_\{\\mathcal\{H\}\_\{u\}\(s\_\{q\_\{a\}\},h\_\{q\_\{a\}\}\)\}\(d\),\\qquadAqρcont\\displaystyle A\_\{q\_\{\\rho\}\}^\{\\mathrm\{cont\}\}=M\+λstate​Normℋu​\(sqρ,hqρ\)⁡\(0\)\.\\displaystyle=M\+\\lambda\_\{\\mathrm\{state\}\}\\operatorname\{Norm\}\_\{\\mathcal\{H\}\_\{u\}\(s\_\{q\_\{\\rho\}\},h\_\{q\_\{\\rho\}\}\)\}\(0\)\.Each advantage trains its own output\. A singleton rewind group gives total advantageMM\. With group returns\{0,g\}\\\{0,g\\\},g\>0g\>0, its local contribution is−λstate\(g/2\)/\(Std\{0,g\}\+ϵ\)\-\\lambda\_\{\\mathrm\{state\}\}\(g/2\)/\(\\operatorname\{Std\}\\\{0,g\\\}\+\\epsilon\)\. The total remains positive ifMMis large enough\. Comparison with negative returns can instead give a positive local term\.

With abandoned scorebband resumed scoreb\+Δb\+\\Delta, reflection receives

ri,mref=\(b\+Δ\)−b=Δ,Ai,mref=Normℛ⁡\(u\)⁡\(Δ\)\.r\_\{i,m\}^\{\\mathrm\{ref\}\}=\(b\+\\Delta\)\-b=\\Delta,\\qquad A\_\{i,m\}^\{\\mathrm\{ref\}\}=\\operatorname\{Norm\}\_\{\\mathcal\{R\}\(u\)\}\(\\Delta\)\.\(21\)This trains the whole reflection, without separating target and memory credit\. Normalization can give positive advancement a negative advantage if other reflections improve more; a singleton group gives zero\.

Execution returns stop at rewind\. With the pre\-rewind segment and group statistics fixed, different proactive targets receive the same execution advantage regardless of later outcomes\. Post\-recovery feedback applies to the separate reflection response\.

## Appendix EPrompt Templates

This section groups the execution, reflection, and observation templates by environment\.

Reflection is a separate model call\. Appendix[D\.3](https://arxiv.org/html/2610.13058#A4.SS3)explains how each backend assembles the resumed context; Appendix[C\.4](https://arxiv.org/html/2610.13058#A3.SS4)defines the global call ceiling\. Literal references to “model action steps” below do not allow extra calls\.

The observation boxes are formatting examples, not measured trajectories\. They use checkpointC1C\_\{1\}, one action used, a global action allowance of 60, and a branch allowance of 20\. Sokoban allows three moves per turn; ScienceWorld uses score zero\. Invalid\-action feedback, prior memories, and recent\-event blocks are omitted from these examples and are added by the builders when present\.

### E\.1MineSweeper

Therefacted,undiscounted\_final, andgigpoflows share the following system and reflection templates\.

YouareanexpertagentoperatingintheMinesweepergamewithREWINDcapability\.

Youwillbegivena\{board\_size\}by\{board\_size\}board,with\{n\_mines\}hiddenmines\.

Rowsandcolumnsareindexedfrom1to\{board\_size\}\.

Thetaskisguaranteedtobesolvable\.Youmustneverconcludethatthetaskisunsolvable,hopeless,orimpossible,andyoumustnevergiveup\.Evenifsomebranchesappearunpromisingorleadtodeadends,youcanstilllearnfromthem,rewindwhennecessary,andcontinueexploring\.

Yourenvironmentstatemaymovebackwardthroughrewind,butyourknowledgemustkeepmovingforward\.Useeachfailedattempttorefineyourunderstanding,avoidrepeatingthesamemistakes,andmakebetterdecisionsinfuturebranchesuntilthetaskissolved\.

\#CellStates

\-Unopenedcells\(?\):hiddencellsthatmaycontainmines\.

\-Blankcells\(\.\):openedsafecellswithnoneighboringmines\.

\-Numberedcells\(1\-8\):openedsafecellsshowinghowmanyneighboringcellscontainmines\.

\-Minecells\(\*\):openedminecells;revealingonelosesthegame\.

\#Goal

RevealallUnopenedandnon\-minecellswithouthittinganymine\.

\#RevealRules

\-ChooseexactlyONEunopenedcell\(?\)torevealpermodelactionstep\.

\-Blankcellsauto\-cascadetorevealcontiguousblanksandborderingnumberedcells,youdidnotneedtorevealtheblankcells\.

\-Numberedcellsrevealonlythatsinglecell\.

\-Minecellsendthecurrentbranchinfailure\.

\#CheckpointsandRewindCapability

Youoperateundertwobudgets,andyoushouldmanagethemdifferently\.

Thefirstisaglobalactionbudget:\{step\_budget\}modelactionstepsfortheentireepisode\.Eachmodelactionsteprevealsexactlyonecell\.Thisisahardceilingsharedacrossthewholeepisode\.Youmayuseitforexploration,trialanderror,andultimatelysolvingthetask\.Thisbudgetisaccumulatedgloballyandisneverrefundedbyrewinding\.

Thesecondisatask\-solvebudget:\{branch\_attempt\_budget\}modelactionstepsformakingprogressalongthecurrentactivepath\.Youshouldaimtosolvethetaskwithinthisallowance\.Unliketheglobalactionbudget,thisbudgetistiedtothecheckpointpath\.WhenyourewindtocheckpointC\_j,thetask\-solvebudgetisrolledbacktotheamountalreadyconsumedatC\_j\.Inotherwords,rewindingtoC\_jrefundsthetask\-solvestepsspentafterC\_j,whileeverythingconsumeduptoC\_jremainsspent\.

Usetheglobalactionbudgetandthetask\-solvebudgettogethertobalanceexploration,learningfromfailedbranches,trialanderror,anddirectprogresstowardsolvingthetask\.

CheckpointstrackthecurrentpathasC\_0,C\_1,…,C\_k:

\-C\_0istheinitialboardstate\.

\-C\_iisthestatereachedafteriexecutedrevealactionsonthecurrentpath\.

Onceyouhaveexecutedatleastonerevealaction,youmaytravelbacktoany

earliercheckpoint:

\-‘<action\>rewindtoC\_j</action\>‘,where0<=j<thecurrentcheckpointindex\.

\-‘<action\>rewindtoj</action\>‘isequivalent\.

RewindingrestorestheenvironmenttocheckpointC\_jandrollsyourbranchattemptbudget

backtoitsvalueatC\_j,butwhatyoulearnedontheabandonedbranchisnotlost—it

persistsasmemory\.youcanrewindwhenyoufind

\*abetterplantostartatC\_j

\*recognizethecurrentbranchisstuck\(anactiontrap,aninvalidloop,adeadend,orunproductiveexploration\)

\*Youareingoodpathtosuccessbutfindinsufficientbudgettomakeprogress\(thetask\-solvebudgetisexhausted\)andwanttomovetocurrentstateinlessturnsandcontext\.

\#ResponseFormat

\-Firstreasonstep\-by\-stepaboutnumberedclues,candidatemines,andsafecells\.

\-Thenrevealoneunopened\(?\)cell\(numbercellandBlankcellsissafebutmakenoprogress\):‘<action\>\(row,col\)</action\>‘\.

\-ORrewindtoapreviouscheckpoint:‘<action\>rewindtoC\_j</action\>‘\.

Onlythefinal‘<action\>…</action\>‘tagwillbeexecuted\.

YouareanexpertMinesweeperplayerreflectingonabranchattempt\.

Board:\{board\_size\}by\{board\_size\},\{n\_mines\}mines\.Rows/columnsindexed1to\{board\_size\}\.

\#CellStates

\-Unopenedcells\(?\):hiddencellsthatmaycontainmines\.

\-Blankcells\(\.\):openedsafecellswithnoneighboringmines\.

\-Numberedcells\(1\-8\):openedsafecellsshowinghowmanyneighboringcellscontainmines\.

\-Minecells\(\*\):openedminecells;revealingonelosesthegame\.

\#RevealRules

\-ChooseexactlyONEunopenedcell\(?\)torevealpermodelactionstep\.

\-Blankcellsauto\-cascadetorevealcontiguousblanksandborderingnumberedcells,youdidnotneedtorevealtheblankcells\.

\-Numberedcellsrevealonlythatsinglecell\.

\-Minecellsendthecurrentbranchinfailure\.

\#Corerollbacksemantics

ThegamewasatstepC\_\{rewind\_from\_step\}whenyoudecidedtorewindbacktostepC\_\{rewind\_to\_step\}\.

ThustheenvstatereturnstostepC\_\{rewind\_to\_step\},Yourgoalistopushtheknowledgestateforward\.

compressthebranchattemptintoasmallbranchmemorythatrecordwhathaveyoudonethatleadtowhatresult,

whathaveyoulearned,whatyoushouldavoidandwhatyoushoulddonext\.

Donotpreservethefulltrajectory\.Preserveonlyinformationusefulforfuturedecisions

thecoretargetistohelpmoveclosertothetargetduringthenextnewattemptortryexploreotherstates\.

\#Context

\{rewind\_target\_instruction\}

\#Rewindtrigger

\{rewind\_reason\}

\#Relevantcheckpointstates

\{checkpoint\_context\}

\#StateatC\_\{rewind\_from\_step\}beforerewinding

\{current\_observation\}

\#BranchhistoryfromC\_\{rewind\_to\_step\}toC\_\{rewind\_from\_step\}

\{history\_after\_rewind\}

\#Yourtask

Reflectonwhatwentwrongandproduceaconcisebranchmemoryforthenextattemptbasedontheactionsyouhavedoneandtheenvironmentfeedback\.

Focuson:

\-revealedminecellsthatshouldnotberepeated;

\-invalid/already\-open/out\-of\-rangereveals;

\-numberedclueconstraintsandspecificcellsthatarelikelyminesorsafe;

\-whetherthebranchshouldrestartbeforeorafterausefulcluereveal;

\-aconcreteimprovedplanfromthecheckpointyoureturnto\.youcouldmovefasterasyoualreadyknowsomeofthestateandtheknowledges\.

Includeacompactbranchmemoryinside<remark\></remark\>tags\.

Usethisstructure,replacingC\_jwiththecheckpointyouchooseorconfirm:

<remark\>

\#HistoryAttemptfromC\_\{rewind\_to\_step\}toC\_\{rewind\_from\_step\}

StateatC\_\{rewind\_to\_step\}:

\{rewind\_to\_observation\}

StateatC\_\{rewind\_from\_step\}:

\{rewind\_from\_observation\}

ActionsIhavedone:…

What’sbadabouttheactions:…

Howtomoveforward:…

NextnewplanfromC\_jthatcanmovefasterandavoidthesamemistakes:…

</remark\>

CurrentobservationatcheckpointC\_1\(segmentturn2\):

\{observation\}

Globalactionbudgetused:1/60\(remaining:59\)\.

Task\-solvebudgetusedonthisbranch:1/20\(remaining:19\)\.

Currentcheckpoint:C\_1\.

Validmodelrewindtargets:C\_0\.

Chooseexactlyoneunopenedcelltorevealinsideonefinalactiontag,e\.g\.<action\>\(2,3\)</action\>,ortravelback:<action\>rewindtoC\_j</action\>\.

### E\.2Sokoban

Therefacted,undiscounted\_final, andgigpoflows share the following system and reflection templates\.

YouareanexpertSokobanagentoperatingwithREWINDcapability\.

Thetaskisguaranteedtobesolvable\.Youmustneverconcludethatthetaskisunsolvable,hopeless,orimpossible,andyoumustnevergiveup\.Evenifsomebranchesappearunpromisingorleadtodeadends,youcanstilllearnfromthem,rewindwhennecessary,andcontinueexploring\.

Yourenvironmentstatemaymovebackwardthroughrewind,butyourknowledgemustkeepmovingforward\.Useeachfailedattempttorefineyourunderstanding,avoidrepeatingthesamemistakes,andmakebetterdecisionsinfuturebranchesuntilthetaskissolved\.

\#SymbolsandTheirMeaning

\-Walls\(‘\#‘\):Theseblockmovement\.

\-Floor\(‘\_‘\):Openspaceswhereyoucanwalkandmoveboxes\.

\-Targets\(‘O‘\):Thespotswhereboxesneedtogo\.

\-Boxes\(‘X‘\):Thesearewhatyouneedtopushontothetargets\.

\-Player\(‘P‘\):That’syou\!

\-BoxonTarget\(‘√\\surd‘\):Aboxsuccessfullyplacedonatarget\.

\-PlayeronTarget\(‘S‘\):Youstandingonatarget\.

\#Goal

Pushallboxes\(‘X‘\)ontotargets\(‘O‘\)\.

\#Rules

\-Admissibleactions:\[”up”,”down”,”left”,”right”\]

\-Youcanonlypushoneboxatatime\.Youcan’tpullboxes\.

\-Youcan’twalkthroughorpushboxesintowallsorotherboxes\.

\-Avoidpushingboxesintocornersoragainstwallswheretheycan’tbemoved\.

\-Ablockedmovemayconsumebudgetwithoutimprovingthestate,soavoidrepeatingit\.

\#CheckpointsandRewindCapability

Youoperateundertwobudgets,andyoushouldmanagethemdifferently\.

Thefirstisaglobalactionbudget:\{step\_budget\}modelactionstepsfortheentireepisode\.Eachmodelactionstepmaycontainupto\{num\_actions\_per\_turn\}primitiveSokobanmoves\.Thisisahardceilingsharedacrossthewholeepisode\.Youmayuseitforexploration,trialanderror,andultimatelysolvingthetask\.Thisbudgetisaccumulatedgloballyandisneverrefundedbyrewinding\.

Thesecondisatask\-solvebudget:\{branch\_attempt\_budget\}modelactionstepsformakingprogressalongthecurrentactivepath\.Youshouldaimtosolvethetaskwithinthisallowance\.Unliketheglobalactionbudget,thisbudgetistiedtothecheckpointpath\.WhenyourewindtocheckpointC\_j,thetask\-solvebudgetisrolledbacktotheamountalreadyconsumedatC\_j\.Inotherwords,rewindingtoC\_jrefundsthetask\-solvestepsspentafterC\_j,whileeverythingconsumeduptoC\_jremainsspent\.

Usetheglobalactionbudgetandthetask\-solvebudgettogethertobalanceexploration,learningfromfailedbranches,trialanderror,anddirectprogresstowardsolvingthetask\.

CheckpointstrackyourprogressalongthecurrentpathasC\_0,C\_1,…,C\_k:

\-C\_0istheinitialstate\.

\-C\_iisthestatereachedafteriexecutedmodelactionstepsonthispath\.

Onceyouhaveexecutedatleastonemodelactionstep,youmaytravelbacktoanyearlier

checkpoint:

\-‘<action\>rewindtoC\_j</action\>‘,where0<=j<thecurrentcheckpointindex\.

\-‘<action\>rewindtoj</action\>‘isequivalent\.

RewindingrestorestheenvironmenttocheckpointC\_jandrollsyourbranchattemptbudget

backtoitsvalueatC\_j,butwhatyoulearnedontheabandonedbranchisnotlost—it

persistsasmemory\.youcanrewindwhenyoufind

\*abetterplantostartatC\_j

\*recognizethecurrentbranchisstuck\(anactiontrap,aninvalidloop,adeadend,orunproductiveexploration\)

\*Youareingoodpathtosuccessbutfindinsufficientbudgettomakeprogress\(thetask\-solvebudgetisexhausted\)andwanttomovetocurrentstateinlessturnsandcontext\(moreactionseachturn\)

\#ResponseFormat

\-Firstreasonstep\-by\-stepaboutthecurrentstateandbox/targetgeometry\.

\-Thenchooseupto\{num\_actions\_per\_turn\}movesinsideonefinalactiontag,e\.g\.

‘<action\>up,right,right</action\>‘\.

\-ORrewindtoapreviouscheckpoint:‘<action\>rewindtoC\_j</action\>‘\.

Onlythefinal‘<action\>…</action\>‘tagwillbeexecuted\.

YouareanexpertSokobanplayerindependentlyreflectingonabranchattempt\.

\#SymbolsandTheirMeaning

\-Walls\(‘\#‘\):Theseblockmovement\.

\-Floor\(‘\_‘\):Openspaceswhereyoucanwalkandmoveboxes\.

\-Targets\(‘O‘\):Thespotswhereboxesneedtogo\.

\-Boxes\(‘X‘\):Thesearewhatyouneedtopushontothetargets\.

\-Player\(‘P‘\):Theagent\.

\-BoxonTarget\(‘√\\surd‘\):Aboxsuccessfullyplacedonatarget\.

\-PlayeronTarget\(‘S‘\):Agentstandingonatarget\.

\#Rules

\-Admissibleactions:\[”up”,”down”,”left”,”right”\]

\-Youcanonlypushoneboxatatime\.Youcan’tpullboxes\.

\-Youcan’twalkthroughorpushboxesintowallsorotherboxes\.

\-Avoidpushingboxesintocornersoragainstwallswheretheycan’tbemoved\.

\-Ablockedmovemayconsumebudgetwithoutimprovingthestate,soavoidrepeatingit\.

\#Corerollbacksemantics

ThegamewasatstepC\_\{rewind\_from\_step\}whenyoudecidedtorewindbacktostepC\_\{rewind\_to\_step\}\.

ThustheenvstatereturnstostepC\_\{rewind\_to\_step\},Yourgoalistopushtheknowledgestateforward\.

compressthebranchattemptintoasmallbranchmemorythatrecordwhathaveyoudonethatleadtowhatresult,

whathaveyoulearned,whatyoushouldavoidandwhatyoushoulddonext\.

Donotpreservethefulltrajectory\.Preserveonlyinformationusefulforfuturedecisions

thecoretargetistohelpmoveclosertothetargetduringthenextnewattemptortryexploreotherstates\.

\#Context

\{rewind\_target\_context\}

\#Rewindtrigger

\{rewind\_reason\}

\#Relevantcheckpointstates

\{checkpoint\_context\}

\#StateatC\_\{rewind\_from\_step\}beforerewinding

\{current\_observation\}

\#BranchhistoryfromC\_\{rewind\_to\_step\}toC\_\{rewind\_from\_step\}

\{history\_after\_rewind\}

\#Priorreflectionoutputs

\{reflection\_history\_context\}

\#Yourtask

Reflectonthebranchattemptandproduceaconcisebranchmemoryforthenextnewattemptdirectionfromthecheckpointyoureturnto\.

Focuson:

\-Invaildactionsthatenvironmentnotacceptedorskippedbytheenvironmentorstuckbythewallsorboxes;

\-failurereasonorlikelybadpush/walksequence;

\-boxespushedintocorners,walls,deadzones,orawayfromreachabletargets;

\-blocked/no\-opmovesorinvalidloopsthatshouldnotberepeated;

\-usefulspatialfactsaboutplayeraccess,boxpositions,targetpositions,andrequiredorder;

\-thetaskisalwayssolvable,soyoushouldalwaysfindaconcreteimprovedplanfromthecheckpointyoureturnto\.

Includeacompactbranchmemoryinside<remark\></remark\>tags\.

Usethisstructure,replacingC\_jwiththeselectedcheckpoint:

<remark\>

\#HistoryAttemptfromC\_\{rewind\_to\_step\}toC\_\{rewind\_from\_step\}

StateatC\_\{rewind\_to\_step\}:

\{rewind\_target\_context\}

StateatC\_\{rewind\_from\_step\}:

\{current\_observation\}

ActionsIhavedone:…

What’sbadabouttheactions:…

Howtomoveforward:…

NextnewPlanfromC\_jthatMustbedifferentfromtheactionsIhavedone

andindependentlyandcarefullythinkaboutandforwardtowardsthegoalandtoadifferentstateortoagoodobservedstatebutinlessturnsandcontext\(moreactionseachturn\):…

</remark\>

CurrentobservationatcheckpointC\_1\(segmentturn2\):

\{observation\}

Globalactionbudgetused:1/60\(remaining:59\)\.

Task\-solvebudgetusedonthisbranch:1/20\(remaining:19\)\.

Currentcheckpoint:C\_1\.

Validmodelrewindtargets:C\_0\.

ChooseuptoTHREEmovesinsideonefinalactiontag,e\.g\.<action\>up,right</action\>,ortravelback:<action\>rewindtoC\_j</action\>\.

### E\.3ALFWorld

Reward variants wrap the same rewind\-choice flow\. The system prompt below extends the household prompt with explicit rewind instructions\. The user\-message builder adds checkpoints, budgets, admissible commands, and branch memories\. Actions and rewind requests use anactioncode block\.

YouareanexpertagentcontrollingarobotintheALFWorldhouseholdenvironmentwithREWINDcapability\.Yourtaskistocompletehouseholdtasksbynavigatingroomsandinteractingwithobjects\.

Thetaskisguaranteedtobesolvable\.Youmustneverconcludethatthetaskisunsolvable,hopeless,orimpossible,andyoumustnevergiveup\.Evenifsomebranchesappearunpromisingorleadtodeadends,youcanstilllearnfromthem,rewindwhennecessary,andcontinueexploring\.

Yourenvironmentstatemaymovebackwardthroughrewind,butyourknowledgemustkeepmovingforward\.Useeachfailedattempttorefineyourunderstanding,avoidrepeatingthesamemistakes,andmakebetterdecisionsinfuturebranchesuntilthetaskissolved\.

ActionCommands:

Foranenvironmentaction,chooseoneofthefollowing,strictlyfollowingthecommand\(argument\)format\.Torewindinstead,usethecheckpointcommanddescribedbelow\.

\#\#Navigation&Observation:

\-look:Lookaroundyourcurrentlocationtogetmoredetails\.

\-inventory:Checktheobjectyouarecurrentlyholding\(youcanonlyholdone\)\.

\-goto\(receptacle\):Movetoareceptacle\(e\.g\.,table,fridge,sink\)\.

\#\#InteractingwithReceptacles:

\-open\(receptacle\):Openareceptacle\.

\-close\(receptacle\):Closeareceptacle\.

\#\#InteractingwithObjects:

\-take\(object\)from\(receptacle\):Pickupanobjectfromareceptacle\.

\-put\(object\)in/on\(receptacle\):Placetheobjectyouareholdingintoorontoareceptacle\.

\-examine\(object\):Examineanobjectcloselytolearnitsproperties\.

\#\#ChangingObjectStates:

\-heat\(object\)with\(receptacle\):Heatanobjectwithadevice\(e\.g\.,microwave\)\.

\-cool\(object\)with\(receptacle\):Coolanobjectwithadevice\(e\.g\.,fridge\)\.

\-clean\(object\)with\(receptacle\):Cleananobjectwithadevice\(e\.g\.,sink\)\.

\-slice\(object\)with\(object\):Sliceanobjectusingasharpobject\(e\.g\.,knife\)\.

ImportantRules

1\.Youmustfirstnavigatetoalocationbeforeinteractingwithobjectsthere

2\.Youcanonlyholdoneobjectatatime

3\.Someobjectsneedtobeheated,cooled,orcleanedbeforeplacingthem

4\.Alwayscheckadmissiblecommandstoseewhatactionsarecurrentlyvalid

\#CheckpointsandRewindCapability

Youoperateundertwobudgets,bothshownineachobservationmessage\.

Thefirstisaglobalactionbudgetfortheentireepisode\.Eachmodelactionsteprequestsexactlyoneenvironmentactionoronerewind\.Thisisahardceilingsharedacrossexploration,trialanderror,andsolvingthetask\.Thisbudgetisaccumulatedgloballyandisneverrefundedbyrewinding\.

Thesecondisacurrent\-branchbudgetforthecurrentexecutionsegment\.Youshouldaimtocompletethetaskwithintheremainingallowance\.Afterarewind,anewsegmentstartsfromtherestoredcheckpointwithafreshcurrent\-branchallowance\.Theglobalactionbudgetremainsspent\.Alwaysusetheremainingbudgetsreportedinthelatestobservation\.

Usethesebudgetstogethertobalanceexploration,learningfromfailedbranches,anddirectprogresstowardsolvingthetask\.

CheckpointstrackthecurrentpathasC\_0,C\_1,…,C\_k:

\-C\_0istheinitialhouseholdstate\.

\-C\_iisthestatereachedafteriexecutedenvironmentactionsonthecurrentpath\.

Onceyouhaveexecutedatleastoneenvironmentaction,youmaytravelbacktoanearliercheckpointlistedunderValidrewindtargets:

“‘action

rewindtoC\_j

“‘

where0<=j<thecurrentcheckpointindex\.

RewindingrestorestheenvironmenttocheckpointC\_j,includingyourlocation,inventory,objectlocations,andobjectstates\.ChangesmadeafterC\_jareundone,butwhatyoulearnedontheabandonedbranchpersistsasbranchmemory\.Usethismemorytoguidefutureactions;donotassumethatanobjectremainsheld,moved,heated,cooled,orcleanedunlessthatwasalreadytrueatC\_j\.

Youcanrewindwhen:

\-Youfindabetterplanthatstartsatanearliercheckpoint\.

\-Thecurrentbranchisstuckinrepeatedinvalidactions,no\-effectactions,adeadend,orunproductiveexploration\.

\-Youchosethewrongobjectorreceptacle,orperformednavigation,open/take/put,orstate\-changestepsinanunhelpfulorder\.

\-Thecurrent\-branchbudgetisnearlyexhaustedandknowledgefromthisbranchcanhelpyoucompletethetaskmoreefficientlyfromanearliercheckpoint\.

Retainusefulprogresswhenchoosingacheckpoint\.Uselearnedobjectlocations,prerequisites,andfailed\-actionoutcomestoformanimprovedplanfromtherestoredstate\.

ResponseFormat

Thinkthroughyourapproachstepbystep,thenprovideexactlyoneenvironmentactionorrewindcommandinthefollowingformat:

<think\>

Yourreasoningandanalysishere…

</think\>

“‘action

<youractionhere\>

“‘

youmustadheretotheresponseformatstrictlyoryouractionwillnotbeexecuted:

Forexample:

<think\>

Ineedtogotothecountertoptogettheknife\.

</think\>

“‘action

gotocountertop1

“‘

Torewindinstead:

<think\>

Ichosethewrongobject\.IwillreturntoC\_2andusewhatIlearnedtochoosethecorrectobject\.

</think\>

“‘action

rewindtoC\_2

“‘

UsethisrewindexampleonlywhenC\_2islistedasavalidtarget\.

AlwaysprovideexactlyONEactioncodeblockatatime,containingeitheroneenvironmentactionoronerewindcommand\.Donotcombinetheminthesameresponse\.

YouarereflectingonafailedALFWorldbranch\.

\#Rewindtrigger

\{rewind\_reason\}

\#Currentcheckpoint

ThebranchiscurrentlyatC\_\{current\_position\}\.

\#Validrewindtargets

\{valid\_targets\}

\#Branchhistory

\{branch\_history\}

Writeacompactmemorythathelpsthenextattemptavoidthefailedbranch\.

Includewronglocations,wrongobjectchoices,missedopen/take/put/state\-changesteps,

andusefulfactslearnedfromobservations\.

Returnmemoryinside<remark\>…</remark\>,thenendwithexactlyoneactionblock:

“‘action

rewindtoC\_j

“‘

CurrentobservationatcheckpointC\_1:

\{observation\}

Admissiblecommands:

\-\{admissible\_command\}

Globalactionbudgetused:1/60\(remaining:59\)\.

Current\-branchbudgetused:1/20\(remaining:19\)\.

Validrewindtargets:C\_0\.

Chooseexactlyoneenvironmentactionin“‘action…“‘,ortravelbackwith“‘action

rewindtoC\_j

“‘\.

### E\.4ScienceWorld

Reward variants share the following system and reflection templates\. The initial user message includes the task description\. Later observations include admissible actions, budgets, branch memories, and the native score\.

YouareanexpertautonomousagentoperatingintheScienceWorldenvironment,atext\-basedvirtualworldforaccomplishingtasksfromtheelementarysciencecurriculum,withREWINDcapability\.

Youinteractbyissuingonefree\-textcommandperturn\.Eachturnyouareshownthecurrentobservationandtheadmissibleactions\(anactiontemplatelistplustheobjectsthatcanreplaceOBJ\)\.Typicalactionsinclude:lookaround,openOBJ,closeOBJ,activateOBJ,deactivateOBJ,gotoLOC,teleporttoLOC,pickupOBJ,moveOBJtoOBJ,focusonOBJ,readOBJ,wait,task\.

Importantrules:

1\.Youmustnavigatetoalocationbeforeinteractingwiththeobjectsthere\.

2\.ReplaceOBJ/LOCplaceholderswithconcreteobjectsandlocationsfromtheobservation\.

3\.‘focusonOBJ‘signalstheobjectthetaskisabout—useitdeliberately,itcanendthetask\.

\#CheckpointsandRewindCapability

Youoperateundertwobudgets\.

Thefirstisaglobalactionbudget:\{step\_budget\}modelactionstepsforthe

entireepisode\.Thisbudgetisneverrefundedbyrewinding\.

Thesecondisatask\-solvebudget:\{branch\_attempt\_budget\}modelactionsteps

forprogressalongthecurrentactivecheckpointpath\.Whenyourewindto

checkpointC\_j,thistask\-solvebudgetrollsbacktotheamountalreadyconsumed

atC\_j\.Knowledgefromabandonedbranchespersistsasmemory\.

CheckpointstrackthecurrentpathasC\_0,C\_1,…,C\_k:

\-C\_0istheinitialenvironmentstate\.

\-C\_iisthestatereachedafteriexecutedactionsonthecurrentpath\.

Onceyouhaveexecutedatleastoneaction,youmaytravelbacktoany

earliercheckpoint:

\-‘<action\>rewindtoC\_j</action\>‘,where0<=j<thecurrentcheckpointindex\.

\-‘<action\>rewindtoj</action\>‘isequivalent\.

Userewindwhenabranchreachesadeadend,becomesinconsistentwithearlier

observations,repeatsineffectiveactions,orwhenabetterplanshouldrestart

fromanearlierstate\.

\#ResponseFormat

\-Firstreasonstep\-by\-stepaboutthetask,theobservation,andtheadmissibleactions\.

\-ThengiveexactlyONEaction:‘<action\>focusondoor</action\>‘\.

\-ORrewindtoapreviouscheckpoint:‘<action\>rewindtoC\_j</action\>‘\.

Onlythefinal‘<action\>…</action\>‘tagwillbeexecuted\.

YouareanexpertScienceWorldagentreflectingonabranchattempt\.

\#Corerollbacksemantics

ThecurrentbranchisatC\_\{rewind\_from\_step\}\.Themodelmustchooseorconfirm

therewindtargetbeforetheenvironmentisrestored\.

Theknowledgestatemustmoveforward:compressthebranchattemptintoasmallbranchmemory\.

Donotpreservethefulltrajectory\.Preserveonlyinformationusefulforfuturedecisions\.

\{rewind\_target\_instruction\}

\#Rewindtrigger

\{rewind\_reason\}

\#Validrollbacktargets

\{valid\_targets\}

\#Relevantcheckpointstates

\{checkpoint\_context\}

\#StateatC\_\{rewind\_from\_step\}beforerewinding

\{current\_observation\}

\#BranchhistoryfromC\_\{history\_start\_step\}toC\_\{rewind\_from\_step\}

\{history\_after\_rewind\}

\#Priorreflectionoutputs

\{reflection\_history\_context\}

\#Yourtask

Reflectonwhatwentwrongandproduceaconcisebranchmemoryforthenextattempt\.

Focuson:

\-wronglocations,wrongobjectchoices,orineffectiveactionsthatshouldnotberepeated;

\-missednavigation,open/activate/move/focus,orstate\-changesteps;

\-usefulfactslearnedfromobservationsaboutobjectsandlocations;

\-whetherthebranchshouldrestartbeforeorafterausefulaction;

\-aconcreteimprovedplanfromthecheckpointyoureturnto\.

Includeacompactbranchmemoryinside<remark\></remark\>tags\.

Usethisstructure,replacingC\_jwiththecheckpointyouchooseorconfirm:

<remark\>

Failurereason:…

Usefulfactslearned:…

Avoidrepeating:…

NextnewplanfromC\_j:…

</remark\>

Endwithexactlyonefinalactiontagselectingthecheckpointtorestore:

<action\>rewindtoC\_j</action\>

CurrentobservationatcheckpointC\_1\(segmentturn2\):

\{observation\}

Scoresofar:0/100\.

Admissibleactions:

\{available\_actions\}

Globalactionbudgetused:1/60\(remaining:59\)\.

Task\-solvebudgetusedonthisbranch:1/20\(remaining:19\)\.

Currentcheckpoint:C\_1\.

Validmodelrewindtargets:C\_0\.

Chooseexactlyoneactioninsideonefinalactiontag,e\.g\.<action\>focusondoor</action\>,ortravelback:<action\>rewindtoC\_j</action\>\.

## Appendix FCase Studies

These cases illustrate failure\-triggered recovery in MineSweeper and proactive recovery in Sokoban\. Each trace follows the abandoned branch, the retained reflection, and the resumed actions\.

##### Logged cases and protocol\.

The cases are selected validation rollouts from archived training variants, not runs underlying the final evaluation tables\. Their prompts allow 60 action turns globally and 20 on the active path; the main protocol instead counts execution and reflection together within 60 model calls\. We therefore report action turns, primitive actions, and reflection calls separately\. Checkpoint indices can be reused after rewind\. Boxed excerpts preserve the logged output, with omissions marked\[\.\.\.\]; surrounding text is our analysis\. Outcomes were checked against rollout metrics and independent state replay\.

### F\.1MineSweeper: Recovering After a Failed Reveal

The task is to reveal 33 safe cells on a6×66\\times 6board with three mines\. Reveals at\(4,1\),\(5,1\), and\(6,1\)produce clues 2, 2, and 1\. AtC3C\_\{3\}, 32 safe cells are open\. The policy then reveals\(6,2\), hits a mine, and ends the branch\. It had acknowledged that\(5,2\)was unopened but incorrectly treated\(6,2\)as the only remaining cell\.

Figure 6:MineSweeper: restoring the board while retaining the failed reveal’s outcome\.Revealing\(6,2\)ends the branch\. The policy selectsC3C\_\{3\}, preserving all three preceding safe reveals, and carries the mine location into the resumed execution\. It then reveals\(5,2\), completing the board\. Coordinates are one\-indexed\.Observed branch and recoveryValid prefix,C0→C3C\_\{0\}\\rightarrow C\_\{3\}\(4,1\)→\\rightarrow2;\(5,1\)→\\rightarrow2;\(6,1\)→\\rightarrow1\.Erroneous reveal,C3→C4C\_\{3\}\\rightarrow C\_\{4\}\(6,2\)→\\rightarrowmine hit; the environment recordsdone=True, won=False\.RecoveryThe failure triggers reflection\. The policy selectsC3C\_\{3\}, restoring the board before the mine hit while preserving the three safe reveals\.

<remark\>

\[…\]

But\(6,2\)wasrevealedandit’samine—thatmeans\(5,2\)issafe\.

\[…\]

Revealing\(5,2\)willcompleteallreveals\.

Finalaction:<action\>\(5,2\)</action\>

\[…\]

</remark\>

<action\>rewindtoC\_3</action\>

The next execution prompt contains the entire remark\. Although restoration hides\(6,2\)again, the policy remembers the mine and chooses the other cell:

So\(6,2\)isamine\[…\]

\[…\]

Therefore,theonlycelllefttorevealis\(5,2\)\.

Revealing\(5,2\)willcompleteallsafereveals\.

Finalaction:

<action\>\(5,2\)</action\>

The final reveal completes the board\. The run uses five reveals and one reflection call, without repeating a reveal after restoration\. A single\-episode rollout on the same instance follows the same four\-action prefix and ends at the mine hit\.

##### What this case shows\.

The board returns toC3C\_\{3\}while the policy retains knowledge from the failed branch and preserves the three safe reveals\. The original error was avoidable: the visible clues already determined the mine layout\. Although parts of the reflection are redundant or inconsistent about completion, its final mine identification and recommended action agree with the successful continuation\.

### F\.2Sokoban: Leaving a Loop While Preserving a Subgoal

The task is to place two boxes on targets in a7×77\\times 7room\. AtC4C\_\{4\}, one box is on target at\(4,1\), the player is at\(4,4\), and the other box must move from\(3,4\)to\(3,2\)\. Four successiveRight, Leftcommands return the player to the same position without moving either box\. Two furtherLeftcommands also make no box progress\. AtC10C\_\{10\}, the policy requestsrewind to C\_4before any terminal failure\.

Figure 7:Sokoban: replacing an unproductive continuation while preserving a solved subgoal\.The policy returns fromC10C\_\{10\}toC4C\_\{4\}, retaining the first box on its target\. The final reflection identifies how to reach the right side of the remaining box; the next two action turns execute that route and complete the task\. Coordinates follow the zero\-indexed environment rendering\.Observed branch and model\-initiated rewindRetained prefix,C0→C4C\_\{0\}\\rightarrow C\_\{4\}Four action turns place the first box on\(4,1\)and leave the player at\(4,4\)\.Repeated loop,C4→C8C\_\{4\}\\rightarrow C\_\{8\}FourRight, Leftcommands each return to the same board state; neither box moves\.Further detour,C8→C10C\_\{8\}\\rightarrow C\_\{10\}TwoLeftcommands move the player to\(4,2\)\. Both box positions remain unchanged\.Policy decision atC10C\_\{10\}<action\>rewind to C\_4</action\>\. The restored state keeps the first box on target and the four\-turn prefix\.

After initially misreading the grid, the reflection identifies a route to the right side of the remaining box: move from\(4,4\)through\(4,5\)to\(3,5\), then push left twice:

<remark\>

\[…\]

SothecorrectplanfromC\_4must:

\-Rightto\(4,5\)

\-Upto\(3,5\)

\-Lefttopush\(3,4\)to\(3,3\)

\-Leftagaintopush\(3,3\)to\(3,2\)

\[…\]

NextnewPlanfromC\_4:Right,Up,Left,Left

</remark\>

The runner retains the last 4,800 of the remark’s 6,706 characters, prefixed by\[truncated\]\. Both quoted plan blocks survive\. The next two calls follow the plan:

<action\>right,up,left</action\>

\[…\]

<action\>left</action\>

The calls move the box through\(3,3\)onto\(3,2\), while the first box stays on target at\(4,1\)\. The run uses 13 action turns, including one rewind request, plus one reflection call and 20 primitive movement attempts\. No forced rewind occurs\.

##### What this case shows\.

The policy leaves a loop, preserves a solved subgoal, and follows the retained plan\. The pre\-rewind room was still solvable\. This case does not show an efficiency advantage: a single\-episode rollout on the same instance succeeds in three action turns\. In both cases, the logs confirm that reflection was supplied and its final recommendation followed; they do not isolate its effect from that of state restoration\.

\*