Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

# Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
Source: [https://arxiv.org/html/2607.03702](https://arxiv.org/html/2607.03702)
Weiyang Guo1Zesheng Shi1Longhui Zhang1Zeen Zhu1Min Zhang1Jing Li1 1Harbin Institute of Technology, Shenzhen, China weiyangguo2024@gmail\.comjingli\.phd@hotmail\.com

###### Abstract

Large language model \(LLM\) agents have shown strong decision\-making capabilities in long\-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals\. To address this, we propose PivoARL, a self\-feedback retry framework for experience exploitation in LLM agents\. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions\. From an information\-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state\-agnostic experience utilization\. Based on this insight, we design a pivotal\-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns\. We conduct a systematic evaluation on 4 agent tasks and 7 search\-based QA benchmarks\. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11\.5% over MetaRL\. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80% of the tasks\. On Minesweeper environment, PivoARL improves over GiGPO by more than 45% and reduces interaction turns by about 42% on average compared with full\-retry methods\. Code is available at[https://github\.com/yuki\-younai/PivoARL](https://github.com/yuki-younai/PivoARL)\.

## 1Introduction

Large language models \(LLMs\) have evolved from static question\-answering systems into versatile*agents*that perceive, reason, and act across multi\-turn interactions with external environments\(Yao et al\.,[2023](https://arxiv.org/html/2607.03702#bib.bib39); Shinn et al\.,[2023](https://arxiv.org/html/2607.03702#bib.bib25); Guo et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib5)\)\. These agents navigate simulated households\(Shridhar et al\.,[2021](https://arxiv.org/html/2607.03702#bib.bib26)\), plan multi\-step web browsing sessions\(Yao et al\.,[2022](https://arxiv.org/html/2607.03702#bib.bib38)\), code development\(Guo et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib6)\), and solve complex tasks that demand long\-horizon planning and adaptive decision\-making\. However, training efficiency remains a bottleneck due to high interaction costs and sparse rewards\(Zhang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib40)\)\. A fundamental challenge persists:how to effectively mine experiences from past trajectories to facilitate robust learning?

Despite its potential in long\-horizon agent tasks\(Wang et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib31)\), experience\-based reinforcement learning \(RL\) faces dual challenges\(Wang et al\.,[2026c](https://arxiv.org/html/2607.03702#bib.bib33)\)\. First, there is a trade\-off between precision and efficiency in experience utilization: static library\-based methods \(e\.g\., SkillsRL\(Xia et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib34)\)\) struggle to adapt to dynamic task evolution, while dynamic paradigms are often susceptible to interference when processing large\-scale experiences\. Second, credit assignment remains a major challenge in long\-horizon agent tasks\. Sparse final feedback from failed trajectories makes it difficult to localize the exact erroneous step\(Jiang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib10)\), causing negative advantages to be mistakenly assigned to correct prefix decisions\. This weakens the learning of useful experiences and may lead to training instability or even capability degradation\(Li et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib14); Shi et al\.,[2026a](https://arxiv.org/html/2607.03702#bib.bib23)\)\.

![Refer to caption](https://arxiv.org/html/2607.03702v1/x1.png)Figure 1:The Pivotal\-Aware Self\-Feedback Retry \(PivoARL\) framework\. The agent identifies pivotal errors via structured self\-reflection, retries from the pivotal state with the correct prefix reused, and updates the LLM through pivotal\-isolated cross\-episode credit assignment\.In this study, we propose PivoARL, a general reinforcement learning paradigm for agents\. This paradigm identifies the pivotal turn responsible for task failure through a structured reflection mechanism and initiates targeted local retries from that specific state, thereby maximizing the reuse of correct historical prefixes\. To optimize the learning process, we design a cross\-episode prefix credit assignment mechanism and implicit reflection returns\. These components enable the agent to deeply mine effective signals from failed trajectories and ultimately transform them into reflection\-guided, high\-quality successful experiences\. From an information\-gain perspective, we further demonstrate that pivotal retries concentrate useful experience signals near the error boundary, effectively mitigating the signal dilution commonly found in global reflections\.

Experimental results show that PivoARL brings consistent gains across four agent environments and seven search\-based QA benchmarks\. On agent tasks, PivoARL improves over GiGPO by about 10\.3%, 15\.3%, and 14\.7% on Pass@1/2/3, respectively; on search\-based QA tasks, it improves over GRPO by about 20\.5%\. Meanwhile, benefiting from the efficient reuse mechanism of pivotal retry, PivoARL reduces interaction cost by about 44% compared with MetaRL\. We further conduct detailed ablation studies to quantify the contribution of each component, and perform additional analyses to examine the test\-time and training\-time scaling behavior of PivoARL\.

Our contributions are as follows:

- •Pivotal\-Aware Retry\.We propose a self\-feedback retry mechanism that identifies the earliest erroneous turn in a failed trajectory and restarts from this pivotal state, reusing the correct prefix to reduce redundant interactions and improve exploitation\.
- •Information Gain Theory\.We analyze experience learning from an information\-gain perspective, showing that global reflections or retrieved experiences are easily diluted in long trajectories, while pivotal retry concentrates useful guidance around the critical error boundary\.
- •State\-of\-the\-Art Performance\.PivoARL improves Pass@k across four agent environments and seven search\-based QA benchmarks, achieving about 16% average relative gain over default RL baselines, while demonstrating stronger test\-time and training\-time scaling\.

## 2Preliminary

### 2\.1Notation and problem setup

We formulate LLM agent interaction as a Markov decision process\(Jiang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib10)\)ℳ=\(𝒮,𝒜,P,R\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,R\)\. Given a taskx∼Dx\\sim D, the agent operates overTTsteps: at steptt, it observes state𝒔t∈𝒮\\bm\{s\}\_\{t\}\\in\\mathcal\{S\}, generates action𝒂t∈𝒱n\\bm\{a\}\_\{t\}\\in\\mathcal\{V\}^\{n\}via policyπθ​\(𝒂t∣𝒔t\)\\pi\_\{\\theta\}\(\\bm\{a\}\_\{t\}\\mid\\bm\{s\}\_\{t\}\)\. The environment then provides a scalar rewardrt∈ℝr\_\{t\}\\in\\mathbb\{R\}, observationoto\_\{t\}and transitions to𝒔t\+1\\bm\{s\}\_\{t\+1\}\. For each trajectory𝝉=\{\(𝒔0,𝒂0,𝒓0\),…,\(𝒔T−1,𝒂T−1,𝒓T−1\)\}\\bm\{\\tau\}=\\\{\(\\bm\{s\}\_\{0\},\\bm\{a\}\_\{0\},\\bm\{r\}\_\{0\}\),\\ldots,\(\\bm\{s\}\_\{T\-1\},\\bm\{a\}\_\{T\-1\},\\bm\{r\}\_\{T\-1\}\)\\\}, we utilize the total returnR​\(τ\)=∑t=0T−1rtR\(\\tau\)=\\sum\_\{t=0\}^\{T\-1\}r\_\{t\}as a episode\-level reward\.

#### Self feedback with experience cycle\.

Beyond scalar rewards, the historical trajectory𝝉0\\bm\{\\tau\}\_\{0\}provides rich feedback at each step \(e\.g\., error messages, state descriptions\), revealing*why*the agent succeeded or failed\(Xiao et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib35)\)\. Given the interaction context𝒄=∑tT−1\{\(at,ot\)\}\\bm\{c\}=\\sum\_\{t\}^\{T\-1\}\\\{\(a\_\{t\},o\_\{t\}\)\\\}from an initial trajectory, the agent constructs an augmented context to extract the past experience information:

remark=πθ\(⋅\|x,\{𝒄\}\)\\textbf\{remark\}=\\pi\_\{\\theta\}\\big\(\\cdot\|x,\\;\\\{\\bm\{c\}\\\}\\big\)\(1\)Whereremarkrepresents the experience gained from the previous attempt, utilized in the next attempt\(Shi et al\.,[2026b](https://arxiv.org/html/2607.03702#bib.bib24)\)as a new enhanced task prompt𝒙\+\\bm\{x\}^\{\+\}\. This enables the model to leverage past experience for optimized subsequent attempts\.

### 2\.2Agentic Reinforcement Learning

Agentic Reinforcement Learning \(ARL\) typically adopts policy\-gradient methods\(Schulman et al\.,[2017](https://arxiv.org/html/2607.03702#bib.bib22)\)to optimize the agent policyπθ\\pi\_\{\\theta\}\. We formulate the agentic RL training objective as:

maxπθ⁡𝔼x∼𝒟,τ∼πθ\(⋅\|x\)​\[∑t=0T−1At\]−β​𝔻KL​\[πθ​\(⋅\)∥πref​\(⋅\)\]\\max\_\{\\pi\_\{\\theta\}\}\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\mathcal\{\\tau\}\\sim\\pi\_\{\\theta\}\(\\cdot\|x\)\}\\left\[\\sum\_\{t=0\}^\{T\-1\}A\_\{t\}\\right\]\-\\beta\\mathbb\{D\}\_\{\\text\{KL\}\}\\left\[\\pi\_\{\\theta\}\(\\cdot\)\\\|\\pi\_\{\\text\{ref\}\}\(\\cdot\)\\right\]\(2\)whereπref\\pi\_\{\\text\{ref\}\}is the reference LLM, and𝔻KL\\mathbb\{D\}\_\{\\text\{KL\}\}andβ\\betadenote the KL divergence and its coefficient, respectively\.AtA\_\{t\}is the step\-wise advantage, typically computed from both step\-level and episode\-level rewards within the group\.

## 3Method

### 3\.1Overview: The PivoARL Framework

![Refer to caption](https://arxiv.org/html/2607.03702v1/x2.png)Figure 2:Comparison between MetaRL and Pivotal\-Aware Self\-Feedback Retry RL\.The previous cross\-episode paradigm\(Jiang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib10)\)treats each retry as a full restart froms0s\_\{0\}, discarding the correct prefix of the failed trajectory\. We proposePivotal\-Aware Self\-Feedback, which reuses the correct prefix and restarts from the pivotal error\.

#### Pivotal Retry Training Framework\.

In the training of PivoARL, each trial consists ofNNepisodes sequentially generated by the agent:

𝒯=\(τ\(0\),τ\(1\),…,τ\(N−1\)\),n∈\[0,N−1\]\\mathcal\{T\}=\(\\tau^\{\(0\)\},\\tau^\{\(1\)\},\\dots,\\tau^\{\(N\-1\)\}\),\\ n\\in\[0,N\-1\]\(3\)
The rollout process terminates atnnifτn\\tau^\{n\}is successful\. Otherwise, the agent initiates a new episodeτ\(n\+1\)\\tau^\{\(n\+1\)\}\. Unlike prior work relying on past experiences for a full restart, our agent reflects to identify pivotal decision points, enabling partial trajectory reuse\.

#### Structured Reflection and Pivotal Retry\.

After a failed episodeτ\(n\)\\tau^\{\(n\)\}ofT\(n\)T^\{\(n\)\}turns, the agent generates a structured reflection containing a textual remark and a*pivotal turn index*k\(n\)∈\[0,T\(n\)−1\]k^\{\(n\)\}\\in\[0,T^\{\(n\)\}\-1\]that identifies the earliest erroneous turn\. The remark provides experience guidance injected into the original state during retry:

Δreflect\(n\)∼πθ\(⋅∣x,𝒄n\)=\(k\(n\);remark\(n\)\)\.\\Delta^\{\(n\)\}\_\{\\text\{reflect\}\}\\sim\\pi\_\{\\theta\}\\bigl\(\\cdot\\mid x,\\bm\{c\}^\{n\}\\bigr\)=\\bigl\(k^\{\(n\)\};\\text\{remark\}^\{\(n\)\}\\ \\bigr\)\.\(4\)
Given the pivotal turnk\(n\)k^\{\(n\)\}, the next episodeτ\(n\+1\)\\tau^\{\(n\+1\)\}reuses the correct prefix ofτ\(n\)\\tau^\{\(n\)\}, regenerating actions only fromk\(n\)k^\{\(n\)\}:

τ\(n\+1\)=\(\\displaystyle\\tau^\{\(n\+1\)\}=\\bigl\(s0\(n\),a0\(n\),…,sk∗\(n\)⏟reused prefix,ak∗\(n\+1\),sk∗\+1\(n\+1\),…,aT\(n\+1\)−1\(n\+1\)⏟regenerated from​k∗​\(where​k∗=k\(n\)\)\)\.\\displaystyle\\underbrace\{s\_\{0\}^\{\(n\)\},a\_\{0\}^\{\(n\)\},\\ldots,s\_\{k^\{\*\}\}^\{\(n\)\}\}\_\{\\text\{reused prefix\}\},\\underbrace\{a\_\{k^\{\*\}\}^\{\(n\+1\)\},s\_\{k^\{\*\}\+1\}^\{\(n\+1\)\},\\ldots,a\_\{T^\{\(n\+1\)\}\-1\}^\{\(n\+1\)\}\}\_\{\\text\{regenerated from \}k^\{\*\}\(\\text\{where\}\\;k^\{\*\}=k^\{\(n\)\}\)\}\\bigr\)\.\(5\)
WhereT\(n\+1\)T^\{\(n\+1\)\}is the length of the new episode\. This reduces the generation overhead for each retry fromT\(n\+1\)T^\{\(n\+1\)\}toT\(n\+1\)−k\(n\)T^\{\(n\+1\)\}\-k^\{\(n\)\}tokens, while preserving the correct decision history as context\.

### 3\.2Pivotal\-Aware Credit Assignment

![Refer to caption](https://arxiv.org/html/2607.03702v1/x3.png)Figure 3:Information gain \(IG\) analysis of experience guidance\. \(a\)Per\-turn IG density: our method concentrates IG in high\-signal turns\. \(b\)Signal Quality: Peak IG and percentage of high\-IG turns\. \(c\)Cumulative IG over turns: showing a characteristic leap at turn k for our method\.#### Credit Assignment with Pivotal Isolation\.

To propagate credit both within and across episodes, we decompose credit returnGt\(n\)G\_\{t\}^\{\(n\)\}into intra\-episode returnsgt\(n\)g\_\{t\}^\{\(n\)\}and inter\-episode returns:

gt\(n\)=∑l=tT\(n\)−1γstepl−t​rl\(n\),Gt\(n\)=gt\(n\)\+∑m=n\+1N−1γtrajm−n​g0\(m\)\.g\_\{t\}^\{\(n\)\}=\\sum\_\{l=t\}^\{T^\{\(n\)\}\-1\}\\gamma\_\{\\text\{step\}\}^\{l\-t\}\\,r\_\{l\}^\{\(n\)\},\\qquad G\_\{t\}^\{\(n\)\}=g\_\{t\}^\{\(n\)\}\+\\sum\_\{m=n\+1\}^\{N\-1\}\\gamma\_\{\\text\{traj\}\}^\{m\-n\}\\,g\_\{0\}^\{\(m\)\}\.\(6\)Whereγstep\\gamma\_\{\\text\{step\}\}andγtraj\\gamma\_\{\\text\{traj\}\}are the intra\- and inter\-episode discount factors, respectively\.

However, naively applying this in the pivotal setting lets erroneous steps \(t≥k∗\(n\)t\\geq k^\{\*\(n\)\}\) receive positive returns from the successful retry, rewarding the very mistakes that caused failure\. We introduce pivotal credit isolation to partition steps atk∗\(n\)k^\{\*\(n\)\}:

Gt\(n\)=\{gt\(n\)\+γtraj⋅G0\(n\+1\),t<k\(n\)∑l=tT\(n\)−1γstepl−t​rl\(n\),t≥k\(n\)G\_\{t\}^\{\(n\)\}=\\begin\{cases\}g\_\{t\}^\{\(n\)\}\+\\gamma\_\{\\text\{traj\}\}\\cdot G\_\{0\}^\{\(n\+1\)\},&t<k^\{\(n\)\}\\\\\[4\.0pt\] \\displaystyle\\sum\_\{l=t\}^\{T^\{\(n\)\}\-1\}\\gamma\_\{\\text\{step\}\}^\{l\-t\}\\,r\_\{l\}^\{\(n\)\},&t\\geq k^\{\(n\)\}\\end\{cases\}\(7\)
We computeGt\(n\)G\_\{t\}^\{\(n\)\}backward fromτ\(N−1\)\\tau^\{\(N\-1\)\}toτ\(0\)\\tau^\{\(0\)\}, so later retries serve as prior credit sources\. Pivotal isolation prevents this future credit from leaking into the erroneous suffixt≥k∗\(n\)t\\geq k^\{\*\(n\)\}\.

#### Reflection Reward through Implicit Returns\.

To co\-optimize reflection generation and action retries, we define the reflection rewardrreflect\(n\)r\_\{\\text\{reflect\}\}^\{\(n\)\}as the discounted return of the subsequent retry episode:

rreflect\(n\)=𝕀​\(success\)⋅∑l=k∗T′−1γstepl−k∗​rl\(n\+1\)r\_\{\\text\{reflect\}\}^\{\(n\)\}=\\mathbb\{I\}\(\\text\{success\}\)\\cdot\\sum\_\{l=k^\{\*\}\}^\{T^\{\\prime\}\-1\}\\gamma\_\{\\text\{step\}\}^\{l\-k^\{\*\}\}r\_\{l\}^\{\(n\+1\)\}\(8\)This reward incentivizes precise failure localization by coupling reflection quality with retry success and penalizing redundancy viaγstep\\gamma\_\{\\text\{step\}\}\. Moreover, the reward is contingent upon correct formatting to ensure structural validity\.

#### Optimization\.

Given the per\-action cross\-episode returnGt\(n\)G\_\{t\}^\{\(n\)\}defined above, we define the gradient objective as follows:

∇θℒRL​\(θ\)\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{RL\}\}\(\\theta\)=𝔼𝝉∼πθ\[∑n,t∇θlogπθ\(at\(n\)∣st\(n\)\)At\(n\)\+∑n∇θlogπθ\(⋅∣x,𝒄n\)Areflect\(n\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\\bm\{\\tau\}\\sim\\pi\_\{\\theta\}\}\\bigg\[\\sum\_\{n,t\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\\bigl\(a\_\{t\}^\{\(n\)\}\\mid s\_\{t\}^\{\(n\)\}\\bigr\)A\_\{t\}^\{\(n\)\}\+\\sum\_\{n\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\\bigl\(\\cdot\\mid x,\\bm\{c\}^\{n\}\\bigr\)A\_\{\\text\{reflect\}\}^\{\(n\)\}\\bigg\]\.\(9\)
WhereAt\(n\)A\_\{t\}^\{\(n\)\}is the advantage estimate fromGt\(n\)G\_\{t\}^\{\(n\)\}, andAreflect\(n\)A^\{\(n\)\}\_\{\\text\{reflect\}\}is derived via group\-wise normalization of the reflection rewardsrreflect\(n\)r^\{\(n\)\}\_\{\\text\{reflect\}\}\. The framework is compatible with widely used optimizers such as GRPO and GiGPO\(Feng et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib3)\)\. See Appendix D for the detailed advantage computation\.

Notably, pivotal retry also improves the model’s first\-attempt success ability \(See in Appendix[E\.2](https://arxiv.org/html/2607.03702#A5.SS2)\)\.

### 3\.3Concentrated Information Gain Analysis

We analyze PivoARL via an information\-gain lens\. Unlike global reflections facing dilution , PivoARL concentrates guidance at the error boundary maximizing efficiency\.

#### Per\-Turn Information Gain\.

Given a trajectoryτ\\tauand experience guidanceremark\(n\)\\textbf\{remark\}^\{\(n\)\}, we define the*per\-turn information gain*:

IG​\(e,t\)=log⁡πθ​\(at∗∣st,e\)−log⁡πθ​\(at∗∣st\)\.\\mathrm\{IG\}\(e,t\)=\\log\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\},e\)\-\\log\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\}\)\.\(10\)HighIG​\(e,t\)\\mathrm\{IG\}\(e,t\)indicates actionable guidance at turntt; near\-zero values indicate noise\.

We argue that existing experience utilization paradigms suffer from low*effective signal density*\(Wang et al\.,[2026b](https://arxiv.org/html/2607.03702#bib.bib32)\): the fraction of turns where the guidance produces non\-negligible IG\.

###### Proposition 1\(IG degradation under state\-agnostic guidance\)\.

Letρ​\(e\)=1T​\|\{t:IG​\(e,t\)\>δ\}\|\\rho\(e\)=\\frac\{1\}\{T\}\|\\\{t:\\mathrm\{IG\}\(e,t\)\>\\delta\\\}\|denote the effective signal density of guidanceee\. For state\-agnostic guidance—whether a fixed reflectionΔ\\Deltareused across all turns, or a retrieved experiencee∗e^\{\*\}from a finite memory bank—the per\-turn IG satisfiesIG​\(e,t\)=O​\(1/T\)\\mathrm\{IG\}\(e,t\)=O\(1/T\), yieldingρ​\(e\)→0\\rho\(e\)\\to 0asTTgrows\. Proof and detailed case analysis are provided in Appendix[E\.1](https://arxiv.org/html/2607.03702#A5.SS1)\.

#### Concentrated Information Gain via Pivotal Retry\.

By identifying the error boundaryk∗k^\{\*\}and applying state\-specific guidance at the pivotal turn, our mechanism concentrates information gain:

IG​\(e,t\)≈\{IGhigh,t≥k∗,0,t<k∗,\\mathrm\{IG\}\(e,t\)\\approx\\begin\{cases\}\\mathrm\{IG\}\_\{\\mathrm\{high\}\},&t\\geq k^\{\*\},\\\\ 0,&t<k^\{\*\},\\end\{cases\}\(11\)whereIGhigh≫1T​∑tIG​\(Δ,t\)\\mathrm\{IG\}\_\{\\mathrm\{high\}\}\\gg\\frac\{1\}\{T\}\\sum\_\{t\}\\mathrm\{IG\}\(\\Delta,t\)\. Although the effective signal density isρ​\(e\)=1/T\\rho\(e\)=1/T, the total IG is concentrated rather than diluted: the correct prefix requires no guidance, while the error suffix is regenerated fromk∗k^\{\*\}\. As training proceeds,k∗k^\{\*\}moves later, yielding more precise experience signals\.

## 4Experiments

### 4\.1Experimental setup

#### Environment\.

We evaluate PivoARL on four agent exploration environments \( Sokoban\(Racanière et al\.,[2017](https://arxiv.org/html/2607.03702#bib.bib21)\), Minesweeper\(Li et al\.,[2024](https://arxiv.org/html/2607.03702#bib.bib15)\), WebShop\(Yao et al\.,[2022](https://arxiv.org/html/2607.03702#bib.bib38)\), and ALFWorld\(Shridhar et al\.,[2021](https://arxiv.org/html/2607.03702#bib.bib26)\)\) and multi\-step search tasks, where we use a constructed Wiki knowledge base as the retrieval source\. And we evaluate the multi\-turn search performance on single\-hop QA datasets \(NQ\(Kwiatkowski et al\.,[2019](https://arxiv.org/html/2607.03702#bib.bib13)\), TriviaQA\(Joshi et al\.,[2017](https://arxiv.org/html/2607.03702#bib.bib12)\), and PopQA\(Mallen et al\.,[2023](https://arxiv.org/html/2607.03702#bib.bib18)\)\) and multi\-hop QA datasets \(HotpotQA\(Yang et al\.,[2018](https://arxiv.org/html/2607.03702#bib.bib37)\), 2Wiki\(Ho et al\.,[2020](https://arxiv.org/html/2607.03702#bib.bib7)\), MuSiQue\(Trivedi et al\.,[2022](https://arxiv.org/html/2607.03702#bib.bib29)\), and Bamboogle\(Press et al\.,[2023](https://arxiv.org/html/2607.03702#bib.bib20)\)\)\. Details are provided in Appendix[C](https://arxiv.org/html/2607.03702#A3)\.

#### Baselines\.

We compare PivoARL with six categories of competitive baselines\. 1\)Closed\-source LLMs, including GPT\-4o and Gemini\-2\.5\-Pro\. 2\)Prompt\-based Agentic Methods, including ReAct\(Yao et al\.,[2023](https://arxiv.org/html/2607.03702#bib.bib39)\)and Reflexion\(Shinn et al\.,[2023](https://arxiv.org/html/2607.03702#bib.bib25)\)\. 3\)RL\-based Methods, including group\-based online RL algorithms such as GRPO and GiGPO, as well as the clipping\-based RL algorithm GSPO\(Zheng et al\.,[2025a](https://arxiv.org/html/2607.03702#bib.bib42)\)\. 4\)Memory\-Augmented RL\-based Methods, including Mem0\+GRPO\(Chhikara et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib2)\), SimpleMem\+GRPO\(Liu et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib16)\), and SkillRL\(Xia et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib34)\)\. 5\)Reflect\-Retry RL\-based Methods, including MetaRL\(Jiang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib10)\)\. 6\)Classic Agentic Search Methods, including Search\-R1\(Jin et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib11)\), ZeroSearch\(Sun et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib28)\), and StepSearch\(Zheng et al\.,[2025b](https://arxiv.org/html/2607.03702#bib.bib43)\)\.

#### Training details\.

We use Qwen3\-4B as the base model and adopt GiGPO as the default optimization algorithm across all experiments\. For PivoARL training, we sampleN=3N=3episodes and set the group size to 8 for agent tasks and group size to 4 for search tasks\. All experiments are conducted on a cluster with 8 NVIDIA H100\-80GB GPUs\. Detailed training settings are provided in Appendix[C](https://arxiv.org/html/2607.03702#A3)\.

Table 1:Main results across four agent benchmarks with Qwen3\-4B\.Suc@1/2/3: success rate in 1, 2 or 3 rollouts \(%\);Avg\.denotes the mean of all success metrics across environments\. The best results and second best results are highlighted inredandblue, respectively\.Table 2:Pass@3 performance on search\-augmented QA tasks with Qwen3\-4B\. Models are trained on NQ and HotpotQA\.†\\daggerand⋆\\starindicate in\-domain and out\-of\-domain datasets, respectively\. The best results and second best results are highlighted inredandblue, respectively\.

### 4\.2Main Result

We evaluate all methods on four agent benchmarks and seven search\-based QA benchmarks\. Table[1](https://arxiv.org/html/2607.03702#S4.T1)summarizes the final performance on agent tasks, while Table[2](https://arxiv.org/html/2607.03702#S4.T2)reports comparisons with search\-related methods\. Figure[4](https://arxiv.org/html/2607.03702#S4.F4)shows the training curves across all four tasks\.

#### Comparison on Agent Tasks\.

Across all four agent environments, PivoARL consistently outperforms other RL training methods and Reflect\-Retry baselines on all Succ@2/3 metrics\. Overall, our method achieves an average 10\.5% improvement over the state\-of\-the\-art MetaRL\. In the Minesweeper environment, PivoARL surpasses GIGPO and MetaRL by 27\.7% and 16\.8%, respectively\. Moreover, benefiting from contrastive advantages at pivotal turns, PivoARL also obtains an average 6\.0% gain on most Succ@1 metrics, demonstrating its superior effectiveness\.

Comparison on Search Tasks\.Across seven single\-hop and multi\-hop retrieval tasks, PivoARL achieves the best Pass@3 performance on 6/7 benchmarks, and obtains the second\-best result on HotpotQA\. Compared with the GRPO baseline, PivoARL obtains a 22\.6% relative improvement\. Compared with classic agentic search methods, PivoARL improves performance by 20\.4% on average\. Compared with the reflect\-retry baseline MetaRL, PivoARL further achieves an 11\.6% relative improvement\. Detailed Pass@1 and Pass@2 results are provided in Appendix E\.

![Refer to caption](https://arxiv.org/html/2607.03702v1/x4.png)Figure 4:Training curves across four agent benchmarks on Qwen3\-4B\. The top, middle, and bottom rows show Success@1, Success@2, and Success@3, respectively, plotted against training steps\. We compare GiGPO \(gray, dotted\), MetaRL \(blue, dashed\), PivoARL \(blue, dashed\)\.
#### Comparison with Training Dynamics\.

We present the Pass@1/2/3 training curves of PivoARL, GiGPO, and MetaRL\. Overall, PivoARL shows more stable training advantages across different tasks\. On Minesweeper, PivoARL significantly accelerates learning convergence and consistently improves all three metrics\. On search tasks, PivoARL also brings moderate gains in Pass@1, with more pronounced improvements on Pass@2/3, indicating stronger advantages in multi\-attempt settings\.

### 4\.3Ablation studies

Table 3:Ablation Study of PivoARL component\.#### Ablation on Component\.

To analyze the contribution of each component in PivoARL, we conduct the following ablation experiments: a\)w/o Reflection: removing reflection experience injection; b\)w/o Pivotal: removing pivotal turn localization and instead performing a full retry from the initial state; and c\)w/o Isolation: removing the pivotal\-isolated credit assignment mechanism\. The results are shown in Table[3](https://arxiv.org/html/2607.03702#S4.T3)\. Without reflection experience, the model performs similarly to vanilla GiGPO, indicating that reflection injection is an important source of improved exploration efficiency\. Removing the pivotal mechanism reduces our method to a full\-retry paradigm similar to MetaRL\. In contrast, credit isolation plays a key role in pivotal retry by preventing gradient updates from mistakenly penalizing the correct prefix, bringing an average improvement of 10\.1% percentage points over the w/o Isolation, with a particularly large gain of 23\.5% percentage points on Minesweeper\.

#### Ablation on Credit Assignment\.

As shown in Figure[6](https://arxiv.org/html/2607.03702#S4.F6), PivoARL demonstrates superior exploration efficiency and prompt economy over baselines\. In terms of reflection capability \(Fig\.[6](https://arxiv.org/html/2607.03702#S4.F6)a\), PivoARL achieves a peak success rate of 83\.8%, outperforming MetaRL 58\.9% and the baseline\. When evaluating retry efficiency \(Fig\.[6](https://arxiv.org/html/2607.03702#S4.F6)b\), our method maintains a competitive average retry ratio of 0\.44, requiring significantly fewer redundant interactions than MetaRL \(constant 1\.0\)\. Notably, removing the CAPI mechanism \(w/o CAPI\) causes a noticeable drop in success rate and an increase in retry frequency, confirming CAPI’s necessity\. Furthermore, analysis of token overhead \(Fig\.[6](https://arxiv.org/html/2607.03702#S4.F6)c\) reveals that PivoARL consumes only 716 additional guidance tokens on average—comparable to the ablated variant and substantially fewer than the baseline\.

### 4\.4Analysis

#### Analysis of Preference Group Density\.

To understand why pivotal retry also improves Pass@1, we count the preference groups produced by different retry strategies\. A preference group refers to trajectories that share the same state but receive different rewards\. As shown in Figure[7\(a\)](https://arxiv.org/html/2607.03702#S5.F7.sf1), reflective retry consistently produces the highest number of preference groups throughout the training process, achieving an overall 48\.4% increase over MetaRL, mainly concentrated in the middle stages of trajectories\. This indicates that it provides denser optimization signals around critical decision regions, leading to stable Pass@1 improvement\. In contrast, random retry performs worse than full retry \(MetaRL\), suggesting that the gain comes from structured reflection\.

![Refer to caption](https://arxiv.org/html/2607.03702v1/x5.png)Figure 5:Comparison of interaction counts across training algorithms, with GRPO set to 100%\.
#### Analysis of Training Interaction Cost\.

PivoARL reuses the correct prefix through pivotal retry, avoiding redundant interactions caused by full retries\. To evaluate its interaction efficiency, we compare PivoARL with GRPO, GiGPO, and MetaRL, as shown in Figure[5](https://arxiv.org/html/2607.03702#S4.F5)\. We normalize the number of training interactions by setting GRPO to 100%\. Results show that PivoARL reduces the required interactions across all tasks, lowering interaction cost by about 29\.1% on average compared with GRPO and by about 13\.5% compared with MetaRL\. This demonstrates that PivoARL improves performance while offering higher training efficiency\.

![Refer to caption](https://arxiv.org/html/2607.03702v1/x6.png)Figure 6:Performance and efficiency comparison on MineSweeper\. We evaluate PivoARL, its ablated variant \(w/o Credit Assignment with Pivotal Isolation \(CAPI\)\), MetaRL, and the Baseline across three metrics: \(a\) Reflection Success Rate, \(b\) Average Retry Ratio, and \(c\) Prompt Length\.
#### Exhibits Stronger Test\-time Scaling\.

To evaluate whether PivoARL can better utilize additional test\-time attempts, we measure Pass@k by varying the number of rollouts during evaluation\. As shown in Figure[7\(b\)](https://arxiv.org/html/2607.03702#S5.F7.sf2), PivoARL consistently outperforms both standard RL and full\-retry reflection baselines across all values of k\. Averaged over Minesweeper and SearchQA, PivoARL improves from 47\.0% Pass@1 to 74\.7% Pass@8, while MetaRL improves from 41\.4% to 62\.4%\. These results indicate that PivoARL exhibits stronger test\-time scaling by learning to use additional attempts more effectively through reflective error localization\.

#### Scaling Performance with Episode Nums\.

We further study the training\-time scaling behavior of PivoARL by varying the number of training episodes \(N\)\. As shown in Figure[7\(c\)](https://arxiv.org/html/2607.03702#S5.F7.sf3), PivoARL generally achieves stronger Pass@k performance as \(N\) increases\. In particular, Pass@1 improves from 39\.8% at \(N=1\) to 53\.9% at \(N=5\), while the best overall result reaches 88\.4% under the largest episode budget\. These results suggest that increasing the number of training episodes provides richer retry and preference signals, enabling PivoARL to scale effectively during training\.

## 5Related Work

#### Agent Reinforcement Learning\.

RL plays a crucial role in helping LLM agents adapt to dynamic and open\-ended environments\. Recent work such as RAGEN\(Wang et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib31)\)integrates reasoning states with environment interactions into turn\-level responses and trains with trajectory\-level RL\. To address sparse rewards in multi\-turn interactions, IGPO\(Wang et al\.,[2026a](https://arxiv.org/html/2607.03702#bib.bib30)\)introduces information\-gain\-based intrinsic rewards derived from the model’s own confidence\. To better capture multi\-turn agent behaviors that trajectory\-level RL struggles with, Tree\-GRPO\(Ji et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib9)\)employs tree\-search\-based group relative policy optimization to enable learning from intermediate decisions\. In addition, GSPO\(Zheng et al\.,[2025a](https://arxiv.org/html/2607.03702#bib.bib42)\)CISPO\(MiniMax et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib19)\)and SAPO\(Gao et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib4)\)focus on the stability of policy optimization, mitigating high\-variance updates through different importance\-ratio clipping mechanisms\. However, these methods treat each interaction as an isolated event, making it difficult to leverage past trajectory experience for training\.

#### Self\-Feedback Evolving Agents\.

Recently, a growing number of methods have begun to emphasize the role of interaction experience in the continual evolution of agents\.*In\-context learning*approaches such as Reflect\-GRPO\(Bensal et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib1)\)and RLTF\(Song et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib27)\)treat natural language reflections as instructive experience\(Shinn et al\.,[2023](https://arxiv.org/html/2607.03702#bib.bib25); Yang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib36); Xiao et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib35)\)\. To internalize such short\-term experience into the model’s inherent capabilities, the*offline distillation*line of work—including ERL\(Shi et al\.,[2026a](https://arxiv.org/html/2607.03702#bib.bib23)\), R3L\(Shi et al\.,[2026b](https://arxiv.org/html/2607.03702#bib.bib24)\), and Critique\-GRPO\(Zhang et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib41)\)uses corrected trajectories or critiques to guide offline optimization\. Furthermore, self\-distillation, represented by SDAR\(Lu et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib17)\)and SDPO\(Hübotter et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib8)\), turns the model’s in\-context learning ability into a “self\-teacher,” internalizing feedback signals into parameters\. MetaRL\(Jiang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib10)\)and MR\-Search\(Xiao et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib35)\)view a trajectory as a continuous learning process across multiple retry episodes, enabling cross\-episode experience utilization\.

![Refer to caption](https://arxiv.org/html/2607.03702v1/x7.png)\(a\)Change of Preference groups
![Refer to caption](https://arxiv.org/html/2607.03702v1/x8.png)\(b\)Test\-time Scaling
![Refer to caption](https://arxiv.org/html/2607.03702v1/x9.png)\(c\)Train\-time Scaling

Figure 7:Analysis of the PivoARL mechanism\. We examine its impact through \(a\) dynamic preference\-group counts, \(b\) test\-time scalability, and \(c\) scaling with training episodes\.

## 6Conclusion

We proposed PivoARL, a self\-feedback reinforcement learning framework for exploiting failed trajectories in LLM agents\. Instead of restarting from scratch, PivoARL locates the pivotal erroneous turn and retries from that state, thereby reusing correct prefixes and reducing redundant interactions\. This design concentrates experience signals around critical error boundaries and prevents erroneous suffixes from receiving misleading credit\. Extensive experiments show that PivoARL consistently improves agent performance while reducing interaction cost\.

#### Limitation and Future Work\.

PivoARL depends on accurate pivotal\-turn localization, and incorrect reflection may reduce retry effectiveness\. It also introduces extra training cost compared with single\-rollout RL\. Future work may explore more reliable pivotal\-state verification, adaptive retry allocation, and extensions to more complex real\-world tool\-use agents\.

## References

- Bensal et al\. \(2025\)Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, et al\.Reflect, retry, reward: Self\-improving LLMs via reinforcement learning\.*arXiv preprint arXiv:2505\.24726*, 2025\.
- Chhikara et al\. \(2025\)Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav\.Mem0: Building production\-ready AI agents with scalable long\-term memory\.In*Proceedings of 28th European Conference on Artificial Intelligence \(EACL\)*, 2025\.
- Feng et al\. \(2025\)Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An\.Group\-in\-group policy optimization for LLM agent training\.In*Proceedings of the Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.
- Gao et al\. \(2025\)Chang Gao, Chujie Zheng, Xiong\-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin\.Soft adaptive policy optimization\.*arXiv preprint arXiv:2511\.20347*, 2025\.
- Guo et al\. \(2025\)Weiyang Guo, Jing Li, Wenya Wang, Yu Li, Daojing He, Jun Yu, and Min Zhang\.Mtsa: Multi\-turn safety alignment for llms through multi\-round red\-teaming\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 26424–26442, 2025\.
- Guo et al\. \(2026\)Weiyang Guo, Zesheng Shi, Liye Zhao, Jiayuan Ma, Zeen Zhu, Junxian He, Min Zhang, and Jing Li\.E3\-tir: Enhanced experience exploitation for tool\-integrated reasoning\.2026\.
- Ho et al\. \(2020\)Xanh Ho, Anh\-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa\.Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.In*Proceedings of the International Conference on Computational Linguistics \(COLING\)*, pp\. 6609–6625, 2020\.
- Hübotter et al\. \(2026\)Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause\.Reinforcement learning via self\-distillation\.*arXiv preprint arXiv:2601\.20802*, 2026\.
- Ji et al\. \(2025\)Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu\.Tree search for llm agent reinforcement learning\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2025\.
- Jiang et al\. \(2026\)Yulun Jiang, Liangze Jiang, Damien Teney, Michael Moor, and Maria Brbić\.Meta\-RL induces exploration in language agents\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2026\.
- Jin et al\. \(2025\)Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han\.Search\-r1: Training llms to reason and leverage search engines with reinforcement learning\.*arXiv preprint arXiv:2503\.09516*, 2025\.
- Joshi et al\. \(2017\)Mandar Joshi, Eunsol Choi, Daniel S\. Weld, and Luke Zettlemoyer\.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension\.In*Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2017\.
- Kwiatkowski et al\. \(2019\)Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N\. Toutanova, Llion Jones, Ming\-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov\.Natural questions: a benchmark for question answering research\.*Transactions of the Association for Computational Linguistics*, 2019\.
- Li et al\. \(2026\)Guocong Li, Jinjian Zhang, Ping Wang, Dongnan Liu, Tian Liang, Qiuyi Qi, Hao Huang, Siyan Guo, Mutian Bao, Wei Zhou, Linjian Mo, Hongxia Xu, and Jian Wu\.Mol: Adaptive mixture\-of\-length reasoning for efficient question answering with context\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=oWWAeLEdE3](https://openreview.net/forum?id=oWWAeLEdE3)\.
- Li et al\. \(2024\)Yinghao Li, Haorui Wang, and Chao Zhang\.Assessing logical puzzle solving in large language models: Insights from a minesweeper case study\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(ACL\)*, 2024\.
- Liu et al\. \(2026\)Jiaqi Liu, Yaofeng Su, Peng Xia, Siwei Han, Zeyu Zheng, Cihang Xie, Mingyu Ding, and Huaxiu Yao\.Simplemem: Efficient lifelong memory for LLM agents\.*arXiv preprint arXiv:2601\.02553*, 2026\.
- Lu et al\. \(2026\)Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi\-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen\.Self\-distilled agentic reinforcement learning\.*arXiv preprint arXiv:2605\.15155*, 2026\.
- Mallen et al\. \(2023\)Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi\.When not to trust language models: Investigating effectiveness of parametric and non\-parametric memories\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2023\.
- MiniMax et al\. \(2025\)MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, et al\.Minimax\-m1: Scaling test\-time compute efficiently with lightning attention\.*arXiv preprint arXiv:2506\.13585*, 2025\.
- Press et al\. \(2023\)Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A\. Smith, and Mike Lewis\.Measuring and narrowing the compositionality gap in language models\.In*Findings of the Association for Computational Linguistics \(EMNLP\)*, pp\. 5687–5711, 2023\.
- Racanière et al\. \(2017\)Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al\.Imagination\-augmented agents for deep reinforcement learning\.2017\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Shi et al\. \(2026a\)Taiwei Shi, Sihao Chen, Bowen Jiang, Linxin Song, Longqi Yang, and Jieyu Zhao\.Experiential reinforcement learning\.*arXiv preprint arXiv:2602\.13949*, 2026a\.
- Shi et al\. \(2026b\)Weijie Shi, Yanxi Chen, Zexi Li, Xuchen Pan, et al\.R3L: Reflect\-then\-retry reinforcement learning with language\-guided exploration, pivotal credit, and positive amplification\.*arXiv preprint arXiv:2601\.03715*, 2026b\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.In*Proceedings of the Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Shridhar et al\. \(2021\)Mohit Shridhar, Xingdi Yuan, Marc\-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht\.Alfworld: Aligning text and embodied environments for interactive learning\.2021\.
- Song et al\. \(2026\)Yuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, J\. Andrew Bagnell, Aarti Singh, and Andrea Zanette\.Expanding the capabilities of reinforcement learning via text feedback\.*arXiv preprint arXiv:2602\.02482*, 2026\.
- Sun et al\. \(2025\)Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou\.Zerosearch: Incentivize the search capability of llms without searching\.*arXiv preprint arXiv:2505\.04588*, 2025\.
- Trivedi et al\. \(2022\)Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal\.MuSiQue: Multihop questions via single\-hop question composition\.*Transactions of the Association for Computational Linguistics \(TACL\)*, 2022\.
- Wang et al\. \(2026a\)Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, et al\.Information gain\-based policy optimization: A simple and effective approach for multi\-turn search agents\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2026a\.
- Wang et al\. \(2025\)Rosie Wang et al\.RAGEN: Understanding self\-evolution in LLM agents via multi\-turn reinforcement learning\.*arXiv preprint arXiv:2504\.20073*, 2025\.
- Wang et al\. \(2026b\)Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei\-Fei, Lijuan Wang, Yejin Choi, and Manling Li\.Ragen\-2: Reasoning collapse in agentic rl\.*arXiv preprint arXiv:2604\.06268*, 2026b\.
- Wang et al\. \(2026c\)Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei\-Fei, Lijuan Wang, Yejin Choi, and Manling Li\.Ragen\-2: Reasoning collapse in agentic rl\.*arXiv preprint arXiv:2604\.06268*, 2026c\.
- Xia et al\. \(2026\)Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao\.Skillrl: Evolving agents via recursive skill\-augmented reinforcement learning\.*arXiv preprint arXiv:2602\.08234*, 2026\.
- Xiao et al\. \(2026\)Teng Xiao, Yige Yuan, Hamish Ivison, Huaisheng Zhu, et al\.Meta\-reinforcement learning with self\-reflection for agentic search\.*arXiv preprint arXiv:2603\.11327*, 2026\.
- Yang et al\. \(2026\)Lu Yang, Zelai Xu, Minyang Xie, Jiaxuan Gao, et al\.MAGE: Meta\-reinforcement learning for language agents toward strategic exploration and exploitation\.*arXiv preprint arXiv:2603\.03680*, 2026\.
- Yang et al\. \(2018\)Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W\. Cohen, Ruslan Salakhutdinov, and Christopher D\. Manning\.HotpotQA: A dataset for diverse, explainable multi\-hop question answering\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2018\.
- Yao et al\. \(2022\)Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan\.Webshop: Towards scalable real\-world web interaction with grounded language agents\.2022\.
- Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\.ReAct: Synergizing reasoning and acting in language models\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2023\.
- Zhang et al\. \(2026\)Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, et al\.The landscape of agentic reinforcement learning for llms: A survey\.*Trans\. Mach\. Learn\. Res\.*, 2026, 2026\.URL[https://openreview\.net/forum?id=RY19y2RI1O](https://openreview.net/forum?id=RY19y2RI1O)\.
- Zhang et al\. \(2025\)Xiaoying Zhang, Yipeng Zhang, Hao Sun, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng\.Critique\-GRPO: Advancing LLM reasoning with natural language and numerical feedback\.*arXiv preprint arXiv:2506\.03106*, 2025\.
- Zheng et al\. \(2025a\)Chujie Zheng, Shixuan Liu, Mingze Li, Xiong\-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin\.Group sequence policy optimization\.*arXiv preprint arXiv:2507\.18071*, 2025a\.
- Zheng et al\. \(2025b\)Xuhui Zheng, Kang An, Ziliang Wang, Yuhang Wang, and Yichao Wu\.Stepsearch: Igniting llms search ability via step\-wise proximal policy optimization\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2025b\.

## Appendix ATraining Algorithm Pipeline

We provide the complete training procedure of PivoARL in AlgorithmLABEL:alg:arl\_sfe2\. The algorithm summarizes the overall pipeline, including pivotal retry, pivotal\-aware credit assignment, experience distillation, and joint policy optimization\.

Algorithm 1Training Workflow of PivoARL1:Initial policy

πθ\\pi\_\{\\theta\}; task set

𝒬\\mathcal\{Q\}; maximum retry budget

NN
2:Initialize retry buffer

𝒟←∅\\mathcal\{D\}\\leftarrow\\varnothing
3:foreach training iteration

i=1,…,Ki=1,\\dots,Kdo

4:Phase 1: Self\-Feedback Pivotal Retry

5:foreach task

x∈𝒬x\\in\\mathcal\{Q\}do

6:Sample an initial rollout

τ\(0\)∼πθ\(⋅∣x\)\\tau^\{\(0\)\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)and evaluate

R​\(τ\(0\)\)R\(\\tau^\{\(0\)\}\)
7:Initialize trajectory chain

𝒯x←\{τ\(0\)\}\\mathcal\{T\}\_\{x\}\\leftarrow\\\{\\tau^\{\(0\)\}\\\}
8:for

n=0,…,N−2n=0,\\dots,N\-2do

9:if

τ\(n\)\\tau^\{\(n\)\}is successfulthen

10:break

11:endif

12:Generate structured reflection

Δreflect\(n\)=\(k∗\(n\),remark\(n\)\)∼πθ\(⋅∣x,c\(n\)\)\\Delta\_\{\\mathrm\{reflect\}\}^\{\(n\)\}=\\big\(k^\{\*\(n\)\},\\mathrm\{remark\}^\{\(n\)\}\\big\)\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,c^\{\(n\)\}\)
13:Retry from pivotal state

sk∗\(n\)\(n\)s\_\{k^\{\*\(n\)\}\}^\{\(n\)\}by reusing prefix

τ<k∗\(n\)\(n\)\\tau\_\{<k^\{\*\(n\)\}\}^\{\(n\)\}
14:Generate suffix conditioned on

remark\(n\)\\mathrm\{remark\}^\{\(n\)\}to obtain retry trajectory

τ\(n\+1\)\\tau^\{\(n\+1\)\}
15:Evaluate

R​\(τ\(n\+1\)\)R\(\\tau^\{\(n\+1\)\}\)
16:Append

τ\(n\+1\)\\tau^\{\(n\+1\)\}to

𝒯x\\mathcal\{T\}\_\{x\}
17:Store

\(τ\(n\),τ\(n\+1\),k∗\(n\),remark\(n\)\)\\big\(\\tau^\{\(n\)\},\\tau^\{\(n\+1\)\},k^\{\*\(n\)\},\\mathrm\{remark\}^\{\(n\)\}\\big\)in

𝒟\\mathcal\{D\}
18:endfor

19:endfor

20:Phase 2: Pivotal\-Aware Credit Assignment

21:foreach trajectory chain

𝒯x\\mathcal\{T\}\_\{x\}in

𝒟\\mathcal\{D\}do

22:Compute intra\-episode returns

gt\(n\)g\_\{t\}^\{\(n\)\}for each episode

τ\(n\)\\tau^\{\(n\)\}
23:Compute cross\-episode returns

Gt\(n\)G\_\{t\}^\{\(n\)\}backward over retry episodes

24:foreach failed episode

τ\(n\)\\tau^\{\(n\)\}with pivotal turn

k∗\(n\)k^\{\*\(n\)\}do

25:Assign retry credit to prefix steps

t<k∗\(n\)t<k^\{\*\(n\)\}
26:Isolate erroneous suffix steps

t≥k∗\(n\)t\\geq k^\{\*\(n\)\}from future retry credit

27:Compute reflection reward

rreflect\(n\)r\_\{\\mathrm\{reflect\}\}^\{\(n\)\}from the subsequent retry return

28:endfor

29:Estimate action advantages

A^t\(n\)\\hat\{A\}\_\{t\}^\{\(n\)\}and reflection advantages

A^reflect\(n\)\\hat\{A\}\_\{\\mathrm\{reflect\}\}^\{\(n\)\}
30:endfor

31:Phase 3: Joint Policy Optimization

32:Update

πθ\\pi\_\{\\theta\}by optimizing the joint self\-feedback objective:

ℒtotal=−𝔼​\[∑n,tlog⁡πθ​\(at\(n\)∣st\(n\)\)​A^t\(n\)\+∑nlog⁡πθ​\(Δreflect\(n\)∣x,c\(n\)\)​A^reflect\(n\)\]\.\\mathcal\{L\}\_\{\\mathrm\{total\}\}=\-\\mathbb\{E\}\\left\[\\sum\_\{n,t\}\\log\\pi\_\{\\theta\}\\big\(a\_\{t\}^\{\(n\)\}\\mid s\_\{t\}^\{\(n\)\}\\big\)\\hat\{A\}\_\{t\}^\{\(n\)\}\+\\sum\_\{n\}\\log\\pi\_\{\\theta\}\\big\(\\Delta\_\{\\mathrm\{reflect\}\}^\{\(n\)\}\\mid x,c^\{\(n\)\}\\big\)\\hat\{A\}\_\{\\mathrm\{reflect\}\}^\{\(n\)\}\\right\]\.
33:endfor

34:returnOptimized policy

πθ\\pi\_\{\\theta\}

## Appendix BBaseline and Environments

### B\.1Experiment Environment

#### Sokoban\.

A classic fully\-observable puzzle where an agent pushes boxes to target positions on a 2D grid\. Since there is no ”pull” operation, the agent must plan several moves ahead to avoid pushing boxes into corners where they become permanently stuck\. Our experiments use a6×66\\times 6board with 2 boxes\.

#### MineSweeper\.

A partially\-observable logic game where the goal is to reveal all safe cells without hitting hidden mines\. Based on numerical clues indicating adjacent mines, the agent must use logical deduction to navigate the board\. We utilize a6×66\\times 6grid containing 3 mines\.

#### Webshop\.

A text\-based environment simulating an e\-commerce interface\. Given a natural language instruction with specific product attributes, the agent must search, filter, and navigate product pages to purchase the item that best matches the criteria\. Success depends on how well the chosen product satisfies the original requirements\.

#### ALFWorld\.

A partially\-observable simulator for multi\-step household tasks\. Agents follow natural language instructions to navigate rooms and interact with objects \(e\.g\., cleaning or heating items\)\. We train on activities like ”Pick,” ”Look,” ”Clean,” and ”Heat,” then evaluate performance both in\-distribution and out\-of\-distribution using ”Cool” and ”Pick2” tasks\.

#### SearchQA\.

We evaluate our method on four open\-domain QA tasks in a search\-augmented setting\. We use Wikipedia as the retrieval corpus and E5 as the dense retriever\. For each query, the environment returns the top\-3 passages, each truncated to 1024 characters\. The agent can iteratively search, read evidence, and refine its answer, testing its multi\-turn evidence integration ability\.

### B\.2Introduction of Baseline

#### GiGPO\(Feng et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib3)\)\.

Group\-in\-Group Policy Optimization \(GiGPO\) is a critic\-free group\-based RL method designed for long\-horizon LLM agents\. It estimates advantages at both the trajectory level and the step level by grouping actions that originate from repeated anchor states, thereby providing more fine\-grained credit assignment under sparse or delayed rewards\.

#### GSPO\(Zheng et al\.,[2025a](https://arxiv.org/html/2607.03702#bib.bib42)\)\.

Group Sequence Policy Optimization \(GSPO\) improves the stability of RL training for LLMs by replacing token\-level importance ratios with sequence\-level likelihood ratios\. With sequence\-level clipping and optimization, GSPO reduces high\-variance updates and improves training efficiency, especially for large\-scale and MoE\-based models\.

Table 4:Hyperparameters and configurations for different environments\.This table summarizes the specific settings used for training and evaluation across different task domains\.
#### MetaRL\(Jiang et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib10)\)\.

Meta\-RL, instantiated as LaMer, trains language agents to explore and adapt across episodes\. It combines cross\-episode reinforcement learning with in\-context policy adaptation via reflection, enabling agents to learn from test\-time feedback without gradient updates and improving exploration in long\-horizon tasks\.

#### Mem0\+GRPO\(Chhikara et al\.,[2025](https://arxiv.org/html/2607.03702#bib.bib2)\)\.

Mem0 is a scalable long\-term memory framework for LLM agents that extracts, stores, and updates salient information from past interactions\. It maintains compact user\- and task\-specific memories through memory extraction, conflict , and relevance\-based retrieval, enabling agents to reuse prior experience without directly appending full histories to the context\. When paired with GRPO, Mem0 serves as the external memory module, while GRPO optimizes the policy using group\-based reinforcement learning over memory\-augmented trajectories\.

#### SimpleMem\+GRPO\(Liu et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib16)\)\.

SimpleMem is an efficient lifelong memory framework for LLM agents that compresses raw interaction histories into compact, structured memory units\. It combines semantic structured compression, recursive memory consolidation, and adaptive query\-aware retrieval to reduce redundant context while preserving task\-relevant information\. When paired with GRPO, SimpleMem serves as the memory module, while GRPO optimizes the policy using group\-based reinforcement learning over retrieved memory\-augmented trajectories\.

#### SkillRL\(Xia et al\.,[2026](https://arxiv.org/html/2607.03702#bib.bib34)\)\.

SkillRL is a recursive skill\-augmented reinforcement learning framework for LLM agents\. Instead of storing raw trajectories as memory, it distills past experiences into a hierarchical skill library, retrieves task\-relevant skills during interaction, and recursively evolves the skill bank together with the policy\.

Table 5:Notation used in this paper\.Summary of key variables and operations in PivoARL\. The*Definition*column indicates where each symbol first appears in the main text\.

## Appendix CDetails of Experiment Setup

### C\.1Training parameter config

This subsection reports the training hyperparameters and configuration settings used for the four agent environments: Sokoban, MineSweeper, WebShop, and ALFWorld\. To improve rollout efficiency in agentic loops, we use the non\-thinking mode during trajectory generation\. The experience guidanceremarkis attached to the original task prompt\. For a fair comparison, we keep most optimization\-related settings consistent across environments, while adapting environment\-specific limits such as maximum turns and prompt length according to task complexity\. In Table[4](https://arxiv.org/html/2607.03702#A2.T4), we list the detailed training configuration parameters used in each environment\.

### C\.2Symbol Notation in this paper

We include the main notation for agent\-environment interaction, self\-feedback pivotal retry, pivotal\-aware credit assignment, experience distillation, and information\-gain analysis\. The definitions Table[5](https://arxiv.org/html/2607.03702#A2.T5)are intended to provide a compact reference for the method formulation and experimental discussion\.

### C\.3Training Objective Details

We follow GiGPO to compute the action\-level advantage used in Eq\.[9](https://arxiv.org/html/2607.03702#S3.E9)\. Specifically, we combine an episode\-level relative advantage with a step\-level relative advantage\.

First, the episode\-level advantage is computed by normalizing the trajectory return within the retry group:

AE​\(τ\(n\)\)=R​\(τ\(n\)\)−mean​\(\{R​\(τ\(m\)\)\}m=0N−1\)Fnorm​\(\{R​\(τ\(m\)\)\}m=0N−1\),A^\{E\}\(\\tau^\{\(n\)\}\)=\\frac\{R\(\\tau^\{\(n\)\}\)\-\\mathrm\{mean\}\\left\(\\\{R\(\\tau^\{\(m\)\}\)\\\}\_\{m=0\}^\{N\-1\}\\right\)\}\{F\_\{\\mathrm\{norm\}\}\\left\(\\\{R\(\\tau^\{\(m\)\}\)\\\}\_\{m=0\}^\{N\-1\}\\right\)\},\(12\)whereR​\(τ\(n\)\)R\(\\tau^\{\(n\)\}\)denotes the total return of trajectoryτ\(n\)\\tau^\{\(n\)\}, andFnormF\_\{\\mathrm\{norm\}\}denotes the normalization factor, such as the standard deviation or a constant value\.

For step\-level credit assignment, we group actions generated from the same environment state:

𝒢S​\(s~\)=\{\(at\(n\),Gt\(n\)\)∣st\(n\)=s~\}\.\\mathcal\{G\}^\{S\}\(\\tilde\{s\}\)=\\left\\\{\\left\(a\_\{t\}^\{\(n\)\},G\_\{t\}^\{\(n\)\}\\right\)\\mid s\_\{t\}^\{\(n\)\}=\\tilde\{s\}\\right\\\}\.\(13\)For each actionat\(n\)a\_\{t\}^\{\(n\)\}in𝒢S​\(s~\)\\mathcal\{G\}^\{S\}\(\\tilde\{s\}\), the step\-level advantage is computed using the cross\-episode returnGt\(n\)G\_\{t\}^\{\(n\)\}:

AS​\(at\(n\)\)=Gt\(n\)−mean​\(\{Gt′\(m\)∣\(at′\(m\),Gt′\(m\)\)∈𝒢S​\(s~\)\}\)Fnorm​\(\{Gt′\(m\)∣\(at′\(m\),Gt′\(m\)\)∈𝒢S​\(s~\)\}\)\.A^\{S\}\(a\_\{t\}^\{\(n\)\}\)=\\frac\{G\_\{t\}^\{\(n\)\}\-\\mathrm\{mean\}\\left\(\\left\\\{G\_\{t^\{\\prime\}\}^\{\(m\)\}\\mid\\left\(a\_\{t^\{\\prime\}\}^\{\(m\)\},G\_\{t^\{\\prime\}\}^\{\(m\)\}\\right\)\\in\\mathcal\{G\}^\{S\}\(\\tilde\{s\}\)\\right\\\}\\right\)\}\{F\_\{\\mathrm\{norm\}\}\\left\(\\left\\\{G\_\{t^\{\\prime\}\}^\{\(m\)\}\\mid\\left\(a\_\{t^\{\\prime\}\}^\{\(m\)\},G\_\{t^\{\\prime\}\}^\{\(m\)\}\\right\)\\in\\mathcal\{G\}^\{S\}\(\\tilde\{s\}\)\\right\\\}\\right\)\}\.\(14\)
Finally, the action\-level advantage is obtained by combining the two terms:

At\(n\)=AE​\(τ\(n\)\)\+ω​AS​\(at\(n\)\),A\_\{t\}^\{\(n\)\}=A^\{E\}\(\\tau^\{\(n\)\}\)\+\\omega A^\{S\}\(a\_\{t\}^\{\(n\)\}\),\(15\)whereω\\omegacontrols the weight of the step\-level advantage\. This advantage is then used asAt\(n\)A\_\{t\}^\{\(n\)\}in the policy\-gradient objective\.

## Appendix DSupplementary Experiment

### D\.1Detailed Search Benchmark Result

We present a more detailed breakdown of the search benchmark results on Table[6](https://arxiv.org/html/2607.03702#A4.T6), including the pass@1, pass@2, and pass@3 metrics\.

Table 6:Main results across various search benchmarks with Qwen3\-4B\.Suc@1/2/3: success rate in 1, 2 or 3 rollouts \(%\)\. The best results and second best results are highlighted inredandblue, respectively\.

## Appendix ETheoretical Analysis

### E\.1Proof for Information Gain Analysis

Recall the per\-turn information gainIG​\(e,t\)=log⁡πθ​\(at∗∣st,e\)−log⁡πθ​\(at∗∣st\)\\mathrm\{IG\}\(e,t\)=\\log\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\},e\)\-\\log\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\}\)and the effective signal densityρ​\(e\)=1T​\|\{t:IG​\(e,t\)\>δ\}\|\\rho\(e\)=\\frac\{1\}\{T\}\|\\\{t:\\mathrm\{IG\}\(e,t\)\>\\delta\\\}\|\. We prove each case separately\.

#### Case \(i\): Global reflection \(MetaRL / LAMER\)\.

After a failed episodeτ\(n\)\\tau^\{\(n\)\}, the agent generates a holistic reflectionΔ\\Deltasummarizing the failure\. This sameΔ\\Deltais prepended to the context at*every*turnt∈\{0,…,T−1\}t\\in\\\{0,\\ldots,T\-1\\\}\.

SinceΔ\\Deltais generated from the entire failed trajectory rather than conditioned on any specific turn’s statests\_\{t\}, it provides no turn\-specific actionable information\. Formally,Δ\\Deltadoes not adapt to the local decision context at turntt, so the conditional distribution shift is uniformly small:

πθ​\(at∗∣st,Δ\)≈πθ​\(at∗∣st\)\+cT,∀t,\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\},\\Delta\)\\approx\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\}\)\+\\frac\{c\}\{T\},\\quad\\forall\\,t,\(16\)whereccis a constant reflecting the total information budget ofΔ\\Delta\. Taking logarithms and usinglog⁡\(1\+x\)≈x\\log\(1\+x\)\\approx xfor smallxx:

IG​\(Δ,t\)≈cT⋅πθ​\(at∗∣st\)=O​\(1/T\)∀t\.\\mathrm\{IG\}\(\\Delta,t\)\\approx\\frac\{c\}\{T\\cdot\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\}\)\}=O\(1/T\)\\quad\\forall\\,t\.\(17\)AsTTgrows,IG​\(Δ,t\)<δ\\mathrm\{IG\}\(\\Delta,t\)<\\deltafor alltt, henceρ​\(Δ\)→0\\rho\(\\Delta\)\\to 0\.

Cross\-trial noise accumulation\.Consider the accumulated context at retrynn:

ℋ\(n\)=\(τ\(0\),Δ\(0\),…,τ\(n−1\),Δ\(n−1\)\)\.\\mathcal\{H\}^\{\(n\)\}=\(\\tau^\{\(0\)\},\\Delta^\{\(0\)\},\\ldots,\\tau^\{\(n\-1\)\},\\Delta^\{\(n\-1\)\}\)\.\(18\)Each failed trajectoryτ\(m\)\\tau^\{\(m\)\}contains both correct steps \(t<k∗\(m\)t<k^\{\*\(m\)\}\) and erroneous steps \(t≥k∗\(m\)t\\geq k^\{\*\(m\)\}\), butΔ\(m\)\\Delta^\{\(m\)\}does not distinguish between them\. The noise tokens grow at least linearly:

\|noise tokens in​ℋ\(n\)\|≥∑m=0n−1\(T\(m\)−k∗\(m\)\),\|\\text\{noise tokens in \}\\mathcal\{H\}^\{\(n\)\}\|\\geq\\sum\_\{m=0\}^\{n\-1\}\(T^\{\(m\)\}\-k^\{\*\(m\)\}\),\(19\)while the reflection signal is bounded by\|Δ\(m\)\|\|\\Delta^\{\(m\)\}\|per retry\. The signal\-to\-noise ratio:

SNR​\(ℋ\(n\)\)≤n⋅\|Δ\|max∑m\(T\(m\)−k∗\(m\)\),\\mathrm\{SNR\}\(\\mathcal\{H\}^\{\(n\)\}\)\\leq\\frac\{n\\cdot\|\\Delta\|\_\{\\max\}\}\{\\sum\_\{m\}\(T^\{\(m\)\}\-k^\{\*\(m\)\}\)\},\(20\)decreases monotonically innn, further degrading the already diluted IG\.

#### Case \(ii\): Experience retrieval \(SkillsRL\)\.

At each turntt, the agent retrievese∗=Retrieve​\(ℳ,st\)e^\{\*\}=\\mathrm\{Retrieve\}\(\\mathcal\{M\},s\_\{t\}\)from a memory bankℳ=\{\(si,ai,ei\)\}i=1\|ℳ\|\\mathcal\{M\}=\\\{\(s\_\{i\},a\_\{i\},e\_\{i\}\)\\\}\_\{i=1\}^\{\|\\mathcal\{M\}\|\}via similaritysim​\(st,si\)\\mathrm\{sim\}\(s\_\{t\},s\_\{i\}\)\. For planning tasks with combinatorially large state spaces \(e\.g\., Sokoban with\|𝒮\|=Ω​\(\(100b\)⋅\(100−b\)\)\|\\mathcal\{S\}\|=\\Omega\(\\binom\{100\}\{b\}\\cdot\(100\-b\)\)\), the match probability vanishes:

P\(∃ei∈ℳ:sim\(st,si\)\>δ\)≤\|ℳ\|\|𝒮δ​\(st\)\|→0,P\(\\exists\\,e\_\{i\}\\in\\mathcal\{M\}:\\mathrm\{sim\}\(s\_\{t\},s\_\{i\}\)\>\\delta\)\\leq\\frac\{\|\\mathcal\{M\}\|\}\{\|\\mathcal\{S\}\_\{\\delta\}\(s\_\{t\}\)\|\}\\to 0,\(21\)where𝒮δ​\(st\)\\mathcal\{S\}\_\{\\delta\}\(s\_\{t\}\)is theδ\\delta\-neighborhood ofsts\_\{t\}\. When the retrieved experience does not match the current state:

sim​\(st,se∗\)≤δ⟹πθ​\(at∗∣st,e∗\)≈πθ​\(at∗∣st\),\\mathrm\{sim\}\(s\_\{t\},s\_\{e^\{\*\}\}\)\\leq\\delta\\implies\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\},e^\{\*\}\)\\approx\\pi\_\{\\theta\}\(a\_\{t\}^\{\*\}\\mid s\_\{t\}\),\(22\)and thereforeIG​\(e∗,t\)≈0\\mathrm\{IG\}\(e^\{\*\},t\)\\approx 0\. Since this holds for the vast majority of states encountered during rollout:

𝔼st∼πθ​\[IG​\(e∗,t\)\]≈0\.\\mathbb\{E\}\_\{s\_\{t\}\\sim\\pi\_\{\\theta\}\}\[\\mathrm\{IG\}\(e^\{\*\},t\)\]\\approx 0\.\(23\)This is the*template collapse*phenomenon\(ragen\_v2\): the retrieved experience becomes a fixed template the model learns to ignore, contributing no actionable information to the decision process\.

### E\.2Proof for Implicit Step\-level Preference Optimization

Pivotal retry can improve first\-attempt performance because it naturally induces a step\-level preference at the error boundary\. Given a failed trajectoryτ\(n\)\\tau^\{\(n\)\}, ARL\-SFE2identifies the pivotal turnk∗\(n\)k^\{\*\(n\)\}and retries from the same state\. For brevity, we omit the episode superscript and writek∗=k∗\(n\)k^\{\*\}=k^\{\*\(n\)\}\. Since the prefix beforek∗k^\{\*\}is reused, the failed action and the retry\-corrected action share the same statesk∗s\_\{k^\{\*\}\}:

a−=ak∗\(n\),a\+=ak∗\(n\+1\)\.a^\{\-\}=a\_\{k^\{\*\}\}^\{\(n\)\},\\qquad a^\{\+\}=a\_\{k^\{\*\}\}^\{\(n\+1\)\}\.If the retry succeeds while the original continuation fails, this induces a local preference

a\+≻a−atsk∗\.a^\{\+\}\\succ a^\{\-\}\\quad\\text\{at\}\\quad s\_\{k^\{\*\}\}\.
#### Proposition\.

Under a binary local outcome setting, the policy\-gradient update induced by pivotal retry has the same preference direction as step\-level DPO:

∇θJpivot\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{pivot\}\}=wpivot\[∇θlogπθ\(a\+∣sk∗,x\)\\displaystyle=w\_\{\\mathrm\{pivot\}\}\\Big\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\+\}\\mid s\_\{k^\{\*\}\},x\)−∇θlogπθ\(a−∣sk∗,x\)\]\.\\displaystyle\\quad\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\-\}\\mid s\_\{k^\{\*\}\},x\)\\Big\]\.wherewpivot\>0w\_\{\\mathrm\{pivot\}\}\>0is a scalar weight\.

#### Proof\.

Consider the local binary outcome at the pivotal state:

G\+=1,G−=0,G^\{\+\}=1,\\qquad G^\{\-\}=0,whereG\+G^\{\+\}andG−G^\{\-\}denote the returns of the corrected and failed continuations, respectively\. Let

p\+=πθ​\(a\+∣sk∗,x\),p−=πθ​\(a−∣sk∗,x\)\.p^\{\+\}=\\pi\_\{\\theta\}\(a^\{\+\}\\mid s\_\{k^\{\*\}\},x\),\\qquad p^\{\-\}=\\pi\_\{\\theta\}\(a^\{\-\}\\mid s\_\{k^\{\*\}\},x\)\.Using the local expected return as the baseline, we have

b=p\+​G\+\+p−​G−=p\+\.b=p^\{\+\}G^\{\+\}\+p^\{\-\}G^\{\-\}=p^\{\+\}\.Thus, the corresponding local advantages are

A\+=G\+−b=p−,A−=G−−b=−p\+\.A^\{\+\}=G^\{\+\}\-b=p^\{\-\},\\qquad A^\{\-\}=G^\{\-\}\-b=\-p^\{\+\}\.
The local policy\-gradient update aroundsk∗s\_\{k^\{\*\}\}can be written as

∇θJpivot\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{pivot\}\}=p\+​A\+​∇θlog⁡πθ​\(a\+∣sk∗,x\)\\displaystyle=p^\{\+\}A^\{\+\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\+\}\\mid s\_\{k^\{\*\}\},x\)\+p−​A−​∇θlog⁡πθ​\(a−∣sk∗,x\)\.\\displaystyle\\quad\+p^\{\-\}A^\{\-\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\-\}\\mid s\_\{k^\{\*\}\},x\)\.SubstitutingA\+=p−A^\{\+\}=p^\{\-\}andA−=−p\+A^\{\-\}=\-p^\{\+\}gives

∇θJpivot\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{pivot\}\}=p\+​p−​∇θlog⁡πθ​\(a\+∣sk∗,x\)\\displaystyle=p^\{\+\}p^\{\-\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\+\}\\mid s\_\{k^\{\*\}\},x\)−p−​p\+​∇θlog⁡πθ​\(a−∣sk∗,x\)\\displaystyle\\quad\-p^\{\-\}p^\{\+\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\-\}\\mid s\_\{k^\{\*\}\},x\)=p\+p−\[∇θlogπθ\(a\+∣sk∗,x\)\\displaystyle=p^\{\+\}p^\{\-\}\\Big\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\+\}\\mid s\_\{k^\{\*\}\},x\)−∇θlogπθ\(a−∣sk∗,x\)\]\.\\displaystyle\\quad\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\-\}\\mid s\_\{k^\{\*\}\},x\)\\Big\]\.Therefore,wpivot=p\+​p−\>0w\_\{\\mathrm\{pivot\}\}=p^\{\+\}p^\{\-\}\>0, and the proposition holds\.

This gradient has the same direction as the step\-level DPO gradient\. For the same preference pair\(a\+,a−\)\(a^\{\+\},a^\{\-\}\), step\-level DPO yields

∇θJDPO\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{DPO\}\}=wDPO\[∇θlogπθ\(a\+∣sk∗,x\)\\displaystyle=w\_\{\\mathrm\{DPO\}\}\\Big\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\+\}\\mid s\_\{k^\{\*\}\},x\)−∇θlogπθ\(a−∣sk∗,x\)\]\.\\displaystyle\\quad\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a^\{\-\}\\mid s\_\{k^\{\*\}\},x\)\\Big\]\.wherewDPO\>0w\_\{\\mathrm\{DPO\}\}\>0depends on the preference margin\. Thus, pivotal retry implicitly performs step\-level preference optimization without explicitly constructing DPO training data\.

Pivotal credit isolation prevents positive retry returns from leaking to the erroneous suffix containinga−a^\{\-\}\. Thus, PivoARL preserves a clean preference signal atk∗k^\{\*\}, encouraging the policy to choosea\+a^\{\+\}in the first attempt and improving Succ@1\.

## Appendix FPrompt Used in Experiment

This section presents the prompts used in our experiments\. For each environment, we provide a play prompt for agent\-environment interaction and a reflect prompt for pivotal error localization\.

### F\.1Prompt in Minesweeper Task

Figure[8](https://arxiv.org/html/2607.03702#A7.F8)shows the play prompt used for Minesweeper\. Figure[9](https://arxiv.org/html/2607.03702#A7.F9)shows the pivotal reflect prompt used for Minesweeper\.

### F\.2Prompt in ALFWorld Task

Figure[10](https://arxiv.org/html/2607.03702#A7.F10)shows the play prompt used for ALFWorld\. Figure[11](https://arxiv.org/html/2607.03702#A7.F11)shows the pivotal reflect prompt used for ALFWorld\.

### F\.3Prompt in SearchQA Task

Figure[12](https://arxiv.org/html/2607.03702#A7.F12)shows the play prompt used for SearchQA\. Figure[13](https://arxiv.org/html/2607.03702#A7.F13)shows the pivotal reflect prompt used for SearchQA\.

## Appendix GCase Study

This section provides qualitative examples of PivoARL across different environments\. Each case illustrates how the agent identifies the pivotal failure point and improves the subsequent retry\.

### G\.1Example in Minesweeper Environment

Figure[14](https://arxiv.org/html/2607.03702#A7.F14)shows a representative case study in the Minesweeper environment\.

### G\.2Example in ALFWorld Environment

Figure[15](https://arxiv.org/html/2607.03702#A7.F15)shows a representative case study in the ALFWorld environment\.

### G\.3Example in SearchQA Environment

Figure[16](https://arxiv.org/html/2607.03702#A7.F16)shows a representative case study in the SearchQA environment\.

Minesweeper Play PromptYou are an expert agent operating in the Minesweeper game\.
You will be given a two dimensional\{board\_size\}by\{board\_size\}board, with\{n\_mines\}hidden mines\.
The rows and columns are indexed from 1 to\{board\_size\}\.\# Cell States•Unopened cells \(?\): cells that are yet to be revealed and may contain a mine\.•Blank cells \(\.\): opened and non\-mine cells, and they have no neighboring mines\.•Numbered cells \(1\-8\): opened and non\-mine cells, and the number indicates how many mines are in the eight neighboring cells\.\# Your Goal•Clear the board by revealing all the cells that don’t contain mines\.•Use clues about the number of neighboring mines to reason\.\# Reveal Rules•Blank cell \(\.\): Triggers an auto\-cascade of contiguous blank cells\.•Numbered cell \(1–8\): Only that single cell is revealed\.•Mine \(\*\): The game ends immediately in a loss\.\# ObservationThe initial state of the game is:
\{init\_observation\}\{past\_trajectories\}\{current\_trajectory\}
Now it’s your turn to make a move\.Figure 8:The system prompt template for the Minesweeper agent\.Minesweeper Pivotal Reflection PromptNow reflect on the past experience:•First, analyze each turn’s decision and identify theroot causeof failure\.•Then determine theearliest action numberwhere the trajectory went wrong•\(the first shown action that should be changed\)\.•Finally, provide your reflection with the retry point\.Your response MUST end with:1\.A <turn\_idx\> tag containing the earliest turn number to retry from\.2\.A <remark\> tag containing your concise reflection and improved plan\.Format:
<turn\_idx\>integer</turn\_idx\>
<remark\>your concise reflection and improved plan </remark\>
Where:•The text inside <remark\> is a concise improved plan that accounts for the identified mistake\.•For example, if Action 3 is the first wrong action, output <turn\_idx\>3</turn\_idx\>\.•The integer inside <turn\_idx\> must be from 1 to\{max\_turn\}, matching the action numbers shown above\.Figure 9:The pivotal reflection prompt template for the Minesweeper agent\.ALFRED Play PromptYou are an expert agent operating in the ALFRED Embodied Environment\.
\{init\_observation\}\{past\_trajectories\_reflections\}\{trajectory\}\# Admissible Actions
Your admissible actions of the current situation are:
\[\{admissible\_actions\}\]\# Your Task
Now it’s your turn to take an action\.•If previous reflections are provided, use them as guidance but re\-evaluate the current state and justify your choice with your own reasoning\.•Do not copy a prior reflection or past action sequence verbatim; adapt principles to the current context\.•Your response should first provide step\-by\-step reasoning about the current situation\.•Once you have finished your reasoning, choose an admissible action for the current step and present it within<action\></action\>tags\.Format:
your step\-by\-step reasoning
<action\>admissible action</action\>Figure 10:The play prompt template for the ALFRED embodied agent\.ALFWorld Pivotal Reflection PromptNow reflect on the past experience:•First, analyze each turn’s decision and identify theroot causeof failure\.•Then determine theearliest action numberwhere the trajectory went wrong•\(the first shown action that should be changed\)\.•Finally, provide your reflection with the retry point\.Your response MUST end with:1\.A <turn\_idx\> tag containing the earliest turn number to retry from\.2\.A <remark\> tag containing your concise reflection and improved plan\.Format:
<turn\_idx\>integer</turn\_idx\>
<remark\>your concise reflection and improved plan </remark\>
Where:•The text inside <remark\> is a concise improved plan that accounts for the identified mistake\.•For example, if Action 3 is the first wrong action, output <turn\_idx\>3</turn\_idx\>\.•The integer inside <turn\_idx\> must be from 1 to\{max\_turn\}, matching the action numbers shown above\.Figure 11:The pivotal reflection prompt template for the ALFRED embodied agent\.SearchQA Play PromptYou are an expert agent tasked with answering the given question step\-by\-step\.
Your question:\{task\_description\}\# Your Task
Now it’s your turn to respond for the current step\.
You should first conduct reasoning process\. This process MUST be enclosed within<think\></think\>tags\.\# Action Space
After completing your reasoning, choose only one of the following actions\. Do not perform both:1\.If you lack some knowledge, call a search engine to get external information using the format:<search\>your query</search\>\.2\.If you have enough knowledge to answer confidently, provide your final answer within<answer\></answer\>tags, without detailed illustrations\.\# Reflection
On trial \#\{traj\_idx\}, the task was not solved well enough\. Your reflection is:
\{reflection\}Format:
<think\>your reasoning process</think\>
<search\>your query</search\>
or
<answer\>final answer</answer\>Figure 12:The play prompt template for the SearchQA agent\.SearchQA Pivotal Reflection PromptYou are an expert agent tasked with answering the given question step\-by\-step\.
Your question:\{task\_description\}\# Your Task
You will be given the history of a past attempt\.
Your job is to:1\.Reflect on the past search process and identify the earliest turn where a wrong or inefficient decision was made\.2\.Determine the earliest turn number whose action should be changed\.3\.Devise a concise improved plan starting from that turn\.\# Past Experience
\{current\_trajectory\}
The task is NOT successfully completed\.Now reflect on the past experience:•First, analyze the search process turn by turn and identify the root cause of failure\.•Then determine the earliest action number where the trajectory went wrong\.•The earliest action is the first shown search or answer step that should be changed\.•Focus on the earliest pivotal mistake, not a later downstream consequence\.•Finally, provide a concise improved plan for retrying from that turn\.Format:
<turn\_idx\>integer</turn\_idx\>
<remark\>
your concise reflection and improved plan
</remark\>Where:•The text inside<remark\>should summarize the corrected search strategy, missing checks, or improved answer criterion for the retry\.•The integer inside<turn\_idx\>must be from 1 to\{max\_turn\}\.•Use 1 if the very first shown step should be changed\.•For example, if Step 3 is the first wrong search or answer decision, output<turn\_idx\>3</turn\_idx\>\.Figure 13:The pivotal reflect prompt template for the SearchQA agent\.Case: Minesweeper \(Pivotal Retry\)Task:Reveal safe cells and avoid hidden mines \(6×\\times6 board, 3 mines\)\.Failed Agent Trajectory in First AttemptStep 1Obs:Row 1: 2 ? ? ? ? ?Row 2: ? ? ? ? ? ?Row 3: ? 1 ? ? ? ?Row 4: ? ? ? ? ? ?Row 5: ? ? ? ? ? ?Row 6: ? ? ? ? ? ?Think:Step\-by\-Step Reasoning: We are given a 6x6 Minesweeper board with 3 hidden mines\. The only known cell is at \(3,2\), which is a numbered cell with the value ’1’ …Action:\(3, 3\)Step 2Obs:Row 1: \. \. 1 ? 1 \.Row 2: \. \. 1 1 1 \.Row 3: 1 1 \. \. \. \.Row 4: ? 1 \. \. \. \.Row 5: ? 2 \. \. \. \.Row 6: ? 1 \. \. \. \.Think:Step\-by\-Step Reasoning: Let’s analyze the current state of the board after the first action\. We need to deduce where the remaining \*\*3 …Action:\(3, 2\)Step 3Obs:Row 1: \. \. 1 ? 1 \.Row 2: \. \. 1 1 1 \.Row 3: 1 1 \. \. \. \.Row 4: ? 1 \. \. \. \.Row 5: ? 2 \. \. \. \.Row 6: ? 1 \. \. \. \.Think:Step\-by\-Step Reasoning: Let’s analyze the current state of the board after the first action\. We need to deduce where the remaining \*\*3 …Action:\(4, 1\)FailedFirst attempt fails after an unsafe pivot decision branch\.Self\-Reflect on Failed TrajectoryReflectionReasoning:After Action 1 clicking \(3,3\), the board reveals substantial information\. Examining the neighbors of \(3,1\)=1: among its 8 surrounding cells, \(2,1\), \(2,2\), and \(3,2\) are all revealed …Retry Turn:<turn\_idx\>3</turn\_idx\>Experience:The mistake was revealing \(4, 1\) which turned out to be a mine\. The improved plan is to avoid guessing based on insufficient information and instead look for cells with unique numbers that can be deduced safely\.Success Agent Trajectory in Reflect RetryStep 3Obs:Row 1: \. \. 1 ? 1 \.Row 2: \. \. 1 1 1 \.Row 3: 1 1 \. \. \. \.Row 4: ? 1 \. \. \. \.Row 5: ? 2 \. \. \. \.Row 6: ? 1 \. \. \. \.Think:¡reflection¿The previous attempt revealed a mine at \(4, 1\), which was a mistake\. This highlights the need to be cautious and rely on logical deduction rather than random guessing\.Looking at the current board state, I notice that some cells have been revealed and are numbered\.Action:\(5, 1\)SuccessReflect retry succeeds from the same pivot observation state\.Figure 14:Case: Minesweeper \(Pivotal Retry\)Case: ALFWorld SprayBottle \(Pivotal Retry\)Task:put two spraybottle in cabinet\.Failed Agent Trajectory in First AttemptStep 1Obs:Room initialized; countertop 1 contains spraybottle 1 and spraybottle 2\.Think:To get two spray bottles, I need to check possible places where they could be located\. Let me start checking possible locations where they might be inside\.Action:go to countertop 1Step 2Obs:spraybottle 1 is available on countertop 1\.Think:To put two spraybottle in cabinet, I need to take two spraybottle\. On countertop 1, I notice spraybottle 1 and spraybottle 2 are available\.Action:take spraybottle 1 from countertop 1Step 3Obs:Agent holds spraybottle 1 and reaches cabinet area\.Think:I now hold spraybottle 1\. To put it in a cabinet, I can choose any of the available cabinets\.Action:go to cabinet 1Step 4Obs:You open the cabinet 1\. The cabinet 1 is open\. In it, you see nothingThink:I currently hold spraybottle 1\. The next step is to put it inside a cabinet\.Action:open cabinet 1Step 5Obs:Nothing happens\.Think:I currently am at cabinet 1\. I need to place two spraybottle in it\.Action:take spraybottle 1 from countertop 1Result: FailedThe first attempt fails after ineffective action at the pivot state\.Self\-Reflect on Failed TrajectoryReflectionReasoning:Looking at the history, I notice that the turn where the mistake was first made is Turn 5\. On that turn, I attempted to place spraybottle 1 into cabinet 1, but it didn’t go into the cabinet\. Additionally, I note that there are still items on countertops where items should not be placed\.Retry Turn:Step 4Experience:Use corrected placement action, then fetch and place the second spraybottle\.Success Agent Trajectory in Reflect RetryStep 4Obs:You open the cabinet 1\. The cabinet 1 is open\. In it, you see nothing\.Think:I will do that step by step\. Looking around, I notice that p̈ut spraybottle 1 in/on cabinet 1ïs available\.Action:put spraybottle 1 in/on cabinet 1Step 5Obs:You put the spraybottle 1 in/on the cabinet 1\.Think:I currently hold spraybottle 1\.Action:go to countertop 1Step 6Obs:Holding spraybottle 2\.Think:\- Current observations show I need to put two spraybottle in a cabinet\. So far, spraybottle 1 has been placed in cabinet 1\. To complete the task, there is another spraybottle to place in a cabinet\.Action:take spraybottle 2 from countertop 1Step 7Obs:You pick up the spraybottle 2 from the countertop 1\.Think:\- I currently hold spraybottle 2\. Now, to place it in a cabinet, I can do so to an available location\.Action:go to cabinet 1Step 8Obs:You arrive at loc 10\. The cabinet 1 is open\. In it, you see a spraybottle 1\.Think:\- I currently hold the spray bottle called spraybottle 2\.Action:put spraybottle 2 in/on cabinet 1Result: SuccessReflect retry succeeds from the same pivot observation state\.Figure 15:Case: ALFWorld Example \(Pivotal Retry\)Case: Complex Search QA \(Pivotal Retry\)Task:How tall is the ride that is nearly identical to the Goliath at Six Flags Over Texas?Failed Agent Trajectory in First AttemptStep 1Obs:Goliath at Six Flags Fiesta Texas \(formerly Batman: The Ride\) is an inverted roller coaster originally from Japan\. Meanwhile, Goliath at Six Flags Magic Mountain is described as a steel hypercoaster “nearly identical to Titan at Six Flags Over Texas,” though its height is not given in the returned snippet\.Think:The search results indicate that Goliath at Six Flags Magic Mountain is nearly identical to Titan at Over Texas, but the height of Goliath at Magic Mountain is not provided\. I need to look up the specific height of that ride\.Action:search: How tall is Goliath at Six Flags Magic Mountain?Step 2Obs:The result describes Goliath at Magic Mountain as a hypercoaster with a sub\-tropical Mayan theme\. It states “Goliath lacks the 540\-degree upward helix on Titan prior to the mid\-course brake run,” again confirming the nearly\-identical relationship to Titan\. However, the search snippet still does not enumerate a numeric height value\.Think:The information confirms Titan at Over Texas is Goliath at Magic Mountain’s twin\. The height of Goliath at Magic Mountain is not explicitly mentioned in the returned documents, so I will rely on my prior knowledge\.Action:answer: 175 feetResult: FailedThe agent conjectures a height without verification and answers incorrectly—the true target is Titan, not Goliath at Magic Mountain\.Self\-Reflect on Failed TrajectoryReflectionReasoning:The initial search \(Step 1\) revealed that Goliath at Magic Mountain is nearly identical to Titan at Over Texas\. The pivotal mistake occurred at Step 2, where the agent searched for the height of Goliath at Magic Mountain instead of directly searching for Titan\. The search for Goliath at Magic Mountain only returned a feature description without a height value, yet the agent still produced an unverified answer of “175 feet\.” The correct strategy is to search for Titan’s height, since Titan—not Goliath at Magic Mountain—is the ride that the question implicitly asks about\.Pivot Turn:Step 2Improved Plan:Retain the Step 1 observation \(the TItan–Goliath relationship\), then replace Step 2 with a direct search for “height of Titan at Six Flags Over Texas\.”Success Agent Trajectory in Reflect Retry \(Retry from Step 2\)Step 2Obs:Titan is a steel hyper coaster located at Six Flags Over Texas in Arlington\. As described in the search result, “It stands at 245 feet and contains a 255\-foot drop at 85 miles per hour\. It is the tallest, fastest, and longest coaster in Texas\.”\(corrected\)Think:The ride nearly identical to Goliath at Over Texas is Titan at Over Texas\. The search directly returns Titan’s height as 245 feet\.Action:search: height of Titan at Six Flags Over TexasStep 3Obs:The search result clearly states Titan stands at 245 feet\.Think:Titan, the ride nearly identical to Goliath at Over Texas, is confirmed to be 245 feet tall\. This directly answers the question\.Action:answer: 245 feetResult: SuccessRetrying from the pivot at Step 2 with the corrected search target yields the correct answer\.Figure 16:Case: SearchQA Example \(Pivotal Retry\)