RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

# Reverse-Turn Policy Optimization forStabilizing Agentic RL Training
Source: [https://arxiv.org/html/2608.18682](https://arxiv.org/html/2608.18682)
## RTPO: Reverse\-Turn Policy Optimization for Stabilizing Agentic RL Training

Yugu LiAffiliation:School of CSITAffiliation:Adelaide UniversityAffiliation:Adelaide, SA 5000, AustraliaEmail:[yugu\.li@adelaide\.edu\.au](mailto:)Jimmy CaoAffiliation:School of CSITAffiliation:Adelaide UniversityAffiliation:Adelaide, SA 5000, AustraliaEmail:[jimmy\.cao@adelaide\.edu\.au](mailto:)Jianglin QiaoAffiliation:ACFRAffiliation:The University of SydneyAffiliation:Camperdown, NSW 2050, AustraliaEmail:[jianglin\.qiao@sydney\.edu\.au](mailto:)Siyi HuAffiliation:School of EECMSAffiliation:Curtin UniversityAffiliation:Bentley, WA 6102, AustraliaEmail:[siyi\.hu@curtin\.edu\.au](mailto:)

###### Abstract

Training multi\-turn agentic workflows with reinforcement learning \(RL\) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single\-turn settings\. Yet multi\-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases\. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout–training context mismatch, weak turn\-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions\. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse\-turn formulation\. We propose Reverse\-Turn Policy Optimization \(RTPO\), which organizes multi\-turn rollouts as sparse reverse trees and performs turn\-level policy updates in temporal reverse order, aligning each decision with its downstream continuation\. RTPO enables causally consistent turn\-level credit assignment and on\-policy continuation to control asynchronous drift\. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn\-level formulation, reduces credit bias, and converges to recursive optimality\. Experiments on multi\-turn agentic RL benchmarks show that RTPO improves upon trajectory\- and turn\-level baselines by 21\.50% and 10\.76%, respectively, highlighting its potential to support more stable training for tool\-using agents\.

## 1Introduction

Reinforcement learning \(RL\) has become a central paradigm for post\-training large language models \(LLMs\), especially when supervision comes from final outcome rewards rather than dense token\-level labels\. Through outcome\-based optimization, RL encourages behaviors such as planning, self\-reflection, and verification, achieving strong results in single\-turn mathematical reasoning\([42](https://arxiv.org/html/2608.18682#bib.bib67);[59](https://arxiv.org/html/2608.18682#bib.bib3);[15](https://arxiv.org/html/2608.18682#bib.bib4);[27](https://arxiv.org/html/2608.18682#bib.bib39)\)and code generation\([25](https://arxiv.org/html/2608.18682#bib.bib41);[45](https://arxiv.org/html/2608.18682#bib.bib38);[49](https://arxiv.org/html/2608.18682#bib.bib2)\)\. Motivated by these advances, recent work extends RL to multi\-turn agentic workflows, especially Tool\-Integrated Reasoning \(TIR\)\([62](https://arxiv.org/html/2608.18682#bib.bib5);[14](https://arxiv.org/html/2608.18682#bib.bib6);[44](https://arxiv.org/html/2608.18682#bib.bib40);[9](https://arxiv.org/html/2608.18682#bib.bib13);[7](https://arxiv.org/html/2608.18682#bib.bib33);[30](https://arxiv.org/html/2608.18682#bib.bib34)\), where agents iteratively reason, call tools, receive feedback, and refine their behavior across turns\.

Despite this promise,RL training for multi\-turn workflows remains unstable: models that improve on short workflows often degrade as the number of turns increases\. This degradation is not merely due to longer sequences; it is amplified by temporal dependencies across turns, where each decision reshapes the context, environment state, and future decision distribution\. Existing methods partially mitigate this issue but provide limited analysis of its causes\. Trajectory\-level methods such as PPO\([41](https://arxiv.org/html/2608.18682#bib.bib7)\), GRPO\([42](https://arxiv.org/html/2608.18682#bib.bib67)\), and GSPO\([67](https://arxiv.org/html/2608.18682#bib.bib68)\)retain the flattened trajectory paradigm\. Turn\-decomposition methods such as SeeUPO\([20](https://arxiv.org/html/2608.18682#bib.bib66)\)improve update granularity but still condition on flattened histories\. Tree\-based methods such as TreeGRPO\([22](https://arxiv.org/html/2608.18682#bib.bib35)\)and ARPO\([13](https://arxiv.org/html/2608.18682#bib.bib36)\)use shared prefixes or branching rollouts, but their advantage estimates are not fully aligned with each turn’s causal contribution to downstream continuation\. Overall, the sources of instability remain underexplored at the training\-pipeline level, and existing methods \(seeAppendix[A](https://arxiv.org/html/2608.18682#A1)for details\) do not jointly address their shared origin\.

In this paper,we provide a theoretical analysis that identifies three coupled sources of instability: context mismatch between rollout and training, which breaks consistency between generated and optimized turn\-level contexts; weak turn\-level credit assignment, where sparse terminal rewards obscure the contribution of individual decisions; and asynchronous policy drift, where short and long trajectories are optimized under different versions of an evolving policy\. Although these issues arise from different components of the training pipeline, we show that they share a common structural origin\. We illustrate these sources inFigure[1](https://arxiv.org/html/2608.18682#S1.F1)and analyze them formally in Sec\.[2](https://arxiv.org/html/2608.18682#S2)\.

Core Challenge\.Motivated by our theoretical analysis, we ask: How can we improve multi\-turn agentic RL performance by stabilizing turn\-wise training with rollout–training consistency, turn\-level credit assignment, and controlled asynchronous policy drift?

To address this challenge,we propose Reverse\-Turn Policy Optimization \(RTPO\), a policy optimization framework for stabilizing multi\-turn agentic RL training, with theoretical details provided in Sec\.[3](https://arxiv.org/html/2608.18682#S3)\. The key idea is to formalize sampled interactions as sparse trees and optimize turn\-level policies in temporal reverse order, propagating continuation values from later turns to earlier ones through the reverse optimality guarantee\. By constructing sibling continuations for each turn, RTPO estimates turn\-level advantages under matched downstream conditions\. This removes context inconsistency induced by flattened trajectory optimization, reduces turn\-level credit bias through causal action alignment, and controls asynchronous policy drift through on\-policy continuation\.

Our contributions are fourfold: \(i\) We identify coupled sources of instability in multi\-turn agentic RL: context mismatch, weak turn\-level credit, and asynchronous policy drift\. \(ii\) We provide a theoretical analysis showing that these sources share a common structural origin in flattened trajectory optimization\. \(iii\) We propose RTPO, a reverse\-turn policy optimization framework with sparse reverse trees, turn\-level on\-policy updates, and theoretical guarantees on recursive optimality, context consistency, and reduced credit bias\. \(iv\) We validate RTPO on multi\-turn agentic RL benchmarks, where it outperforms strong baselines, including GRPO, TreeGRPO, ARPO, and SeeUPO, while further stabilizing the training pipeline\.

![Refer to caption](https://arxiv.org/html/2608.18682v1/figure1_jimmy.png)Figure 1:The identified training instability in multi\-turn agentic RL arises from: \(A\) rollout–training context mismatch, \(B\) trajectory\-only credit assignment, and \(C\) long\-horizon policy drift\.
## 2Theoretical Analysis: Training Instability

We argue that the instability of multi\-turn RL training stems fromrollout–training mismatch: rollouts are typically generated under truncated or summarized contexts, while training recomputes likelihood ratios under concatenated full\-history contexts\. This discrepancy biases policy optimization and worsens over long horizons\. Moreover, the flattened full\-history formulation provides only trajectory\-level credit, causing terminal rewards to obscure individual turns\. When trajectories are generated asynchronously, this mismatch further induces policy drift, as long trajectories generated under an older policy may be optimized after the policy has already been updated by shorter trajectories\.

Building on these observations and insights, we model a multi\-turn interaction as a hierarchical Markov decision process \(H\-MDP\)\([16](https://arxiv.org/html/2608.18682#bib.bib46)\)\. Given an initial promptqq, turnkkconsists of a model responselkl\_\{k\}and environment feedbackfkf\_\{k\}, producing the trajectory\(q,l0,f0,…,ln−1,fn−1\)\(q,l\_\{0\},f\_\{0\},\\ldots,l\_\{n\-1\},f\_\{n\-1\}\)\. The turn\-level state isSk=\(q,l0,f0,…,lk−1,fk−1\)S\_\{k\}=\(q,l\_\{0\},f\_\{0\},\\ldots,l\_\{k\-1\},f\_\{k\-1\}\), and the turn\-level action is the responselkl\_\{k\}\. Each response is generated autoregressively aslk=\(ak,1,…,ak,Tk\)l\_\{k\}=\(a\_\{k,1\},\\ldots,a\_\{k,T\_\{k\}\}\), where the token\-level state issk,t=\(Sk,ak,<t\)s\_\{k,t\}=\(S\_\{k\},a\_\{k,<t\}\)\. Existing PPO\- and GRPO\-style methods\([42](https://arxiv.org/html/2608.18682#bib.bib67);[67](https://arxiv.org/html/2608.18682#bib.bib68);[65](https://arxiv.org/html/2608.18682#bib.bib43);[64](https://arxiv.org/html/2608.18682#bib.bib45)\)typically flatten the full interaction into a single token sequence and optimize the resulting trajectory as follows:

Jflat​\(θ\)=𝔼⁡\[1G​∑i=1G1∑tmi,t​∑tmi,t​min⁡\(ρi,t​Ai,clip⁡\(ρi,t,1−ϵ,1\+ϵ\)​Ai\)\],J^\{\\mathrm\{flat\}\}\(\\theta\)=\\mathbb\{E\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\\sum\_\{t\}m\_\{i,t\}\}\\sum\_\{t\}m\_\{i,t\}\\min\\\!\\left\(\\rho\_\{i,t\}A\_\{i\},\\operatorname\{clip\}\(\\rho\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)A\_\{i\}\\right\)\\right\],\(1\)whereρi,t=πθ​\(ai,t∣xi,<t\)/πθold​\(ai,t∣xi,<t\)\\rho\_\{i,t\}=\\pi\_\{\\theta\}\(a\_\{i,t\}\\mid x\_\{i,<t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{i,t\}\\mid x\_\{i,<t\}\)is the token\-level importance\-sampling \(IS\) ratio andAiA\_\{i\}is a trajectory\-level advantage assigned uniformly to all unmasked tokens in trajectorygig\_\{i\}\(gig\_\{i\}belongs to a group ofGGtrajectories\)\. Full preliminaries are provided in Appendix[B](https://arxiv.org/html/2608.18682#A2)\.

### 2\.1Rollout–Training Mismatch

As illustrated inFigure[1](https://arxiv.org/html/2608.18682#S1.F1)\-A, in multi\-turn interactions, rollouts are often generated under a truncated or summarized contextϕ⁡\(x¯k\)\\phi\(\\bar\{x\}\_\{k\}\), while training recomputes token probabilities under the full flattened historyx¯k\\bar\{x\}\_\{k\}\. Thus, the IS ratio used in training differs from the true IS ratio induced by the rollout distribution:

ρk,tflat=πθ​\(ak,t∣x¯k\)πθold​\(ak,t∣ϕ⁡\(x¯k\)\)≠πθ​\(ak,t∣ϕ⁡\(x¯k\)\)πθold​\(ak,t∣ϕ⁡\(x¯k\)\)=ρk,ttrue\.\\rho\_\{k,t\}^\{\\mathrm\{flat\}\}=\\frac\{\\pi\_\{\\theta\}\(a\_\{k,t\}\\mid\\bar\{x\}\_\{k\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\}\\neq\\frac\{\\pi\_\{\\theta\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\}=\\rho\_\{k,t\}^\{\\mathrm\{true\}\}\.\(2\)Because the denominator inρk,tflat\\rho\_\{k,t\}^\{\\mathrm\{flat\}\}does not match the distribution that actually sampled the token, the resulting policy\-gradient estimate is biased\. This mismatch becomes more severe in later turns as the omitted history grows\. Moreover, whenϕ\\phiis non\-injective, distinct full\-history states can collapse into the same truncated observation, inducing state aliasing and restricting optimization to an observation\-induced policy class whose optimum may be strictly below the full\-history optimum\.

SeeAppendix[B\.1](https://arxiv.org/html/2608.18682#A2.SS1)for the full theoretical analysis of rollout–training mismatch\.

### 2\.2Trajectory\-Only Credit Assignment

Flattened training assigns a single trajectory\-level advantage to all turns, even though different turns may contribute unequally to the final outcome, as shown inFigure[1](https://arxiv.org/html/2608.18682#S1.F1)\-B\. For trajectorygig\_\{i\}, the population trajectory advantage can be decomposed at turnkkas

Ri−𝔼⁡\[R\]=Ri−Qπ​\(Si,k,li,k\)⏟downstream stochasticity\+Aπ​\(Si,k,li,k\)⏟true turn advantage\+Vπ​\(Si,k\)−μR⏟upstream state effect\.R\_\{i\}\-\\mathbb\{E\}\[R\]=\\underbrace\{R\_\{i\}\-Q^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)\}\_\{\\text\{downstream stochasticity\}\}\+\\underbrace\{A^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)\}\_\{\\text\{true turn advantage\}\}\+\\underbrace\{V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\}\_\{\\text\{upstream state effect\}\}\.\(3\)Here,RiR\_\{i\}denotes the final return of trajectorygig\_\{i\},Qπ\(Si,k,li,k\)=𝔼\[R∣Si,k,li,k\]Q^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)=\\mathbb\{E\}\[R\\mid S\_\{i,k\},l\_\{i,k\}\],Vπ​\(Si,k\)=𝔼⁡\[R∣Si,k\]V^\{\\pi\}\(S\_\{i,k\}\)=\\mathbb\{E\}\[R\\mid S\_\{i,k\}\], andμR=𝔼⁡\[R\]\\mu\_\{R\}=\\mathbb\{E\}\[R\]\. This decomposition shows that the true turn\-level advantage is only one component of the trajectory\-level signal\. When upstream state effects or downstream stochasticity dominate, the sign of the trajectory advantage may disagree with the true turn\-level advantage, leading to incorrect or even reversed policy updates\. Similarly, group\-relative baselines provide valid local comparisons only when trajectories share the same turn\-level state\. Under cross\-state grouping, the advantage estimator incurs additional biasBiascross=G−1G​\(Vπ​\(Si,k\)−μR\)\\mathrm\{Bias\}\_\{\\mathrm\{cross\}\}=\\frac\{G\-1\}\{G\}\\left\(V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\\right\), determined by the upstream trajectory without causal connection to the current action\.

SeeAppendix[B\.2](https://arxiv.org/html/2608.18682#A2.SS2)for the full theoretical analysis of trajectory\-only credit assignment issues\.

### 2\.3Long\-Horizon Policy Drift

In asynchronous training with parallel sampled trajectories \(seeFigure[1](https://arxiv.org/html/2608.18682#S1.F1)\-C\), shorter trajectories may complete and update the policy while longer trajectories are still being generated under an older policy\. When these longer trajectories are later optimized, they become off\-policy with respect to the current model\. The unbiased correction would require the full\-trajectory importance weightωi=∏t=1Tiρi,t\\omega\_\{i\}=\\prod\_\{t=1\}^\{T\_\{i\}\}\\rho\_\{i,t\}\. However, the PPO or GRPO method applies clipping independently at the token level, so the resulting correction differs from the true full\-trajectory IS ratio:

∏t=1Ticlip⁡\(ρi,t,1−ϵ,1\+ϵ\)≠∏t=1Tiρi,t=ωi\.\\prod\_\{t=1\}^\{T\_\{i\}\}\\operatorname\{clip\}\(\\rho\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)\\neq\\prod\_\{t=1\}^\{T\_\{i\}\}\\rho\_\{i,t\}=\\omega\_\{i\}\.\(4\)Therefore, token\-level clipping cannot faithfully correct long\-horizon policy drift\. The discrepancy compounds with trajectory length, while using the exact full\-trajectory ratio is impractical because its variance grows rapidly withTiT\_\{i\}\. This explains why standard PPO\- or GRPO\-style training methods can become unstable or even collapse in long\-horizon agentic RL\.

The full theoretical analysis of long\-horizon policy drift is provided inAppendix[B\.3](https://arxiv.org/html/2608.18682#A2.SS3)\.

## 3Method: Reverse\-Turn Policy Optimization \(RTPO\)

To address the training instability revealed by our theoretical analysis in Sec\.[2](https://arxiv.org/html/2608.18682#S2),we propose Reverse\-Turn Policy Optimization \(RTPO\) with theoretical guarantees for stabilizing agentic RL training, as shown inFigure[2](https://arxiv.org/html/2608.18682#S3.F2)\. RTPO first models multi\-turn interaction as a turn\-boundary MDP and defines an independent sub\-policy optimization objective for each turn \(Sec\.[3\.1](https://arxiv.org/html/2608.18682#S3.SS1)\)\. This formulation enables reverse\-order training to mitigate the mismatch between rollout and training contexts\. RTPO then develops sparse tree rollouts based on maximum\-value decomposition \(Sec\.[3\.2](https://arxiv.org/html/2608.18682#S3.SS2)\) to estimate true turn\-level advantages, enabling causally consistent turn\-level credit assignment for the case of cross\-trajectory comparison without state bias\. Finally, RTPO designs an on\-policy continuation mechanism \(Sec\.[3\.3](https://arxiv.org/html/2608.18682#S3.SS3)\) that eliminates the need for downstream IS\-ratio correction under PPO clipping, thereby addressing policy drift induced by asynchronous turns\.

![Refer to caption](https://arxiv.org/html/2608.18682v1/figure2_jimmy.png)Figure 2:Overview of RTPO in agentic RL training\. After rollout, RTPO performs reverse\-orderturn\-level policy optimizationfor each trajectory in the batch\. Starting from the final turnkk, the rollout \(old\) policyπold\\pi\_\{\\mathrm\{old\}\}generatesG−1G\{\-\}1sibling rollouts from the same turn boundary to estimate a group advantage and update the training \(turn\-level\) policy toπk\\pi\_\{k\}\. The procedure then proceeds backward through turnsk−1,k−2,…,0k\{\-\}1,k\{\-\}2,\\ldots,0, where each updated turn policy asπk−1,πk−2,…,π0\\pi\_\{k\-1\},\\pi\_\{k\-2\},\\ldots,\\pi\_\{0\}\. To generate state\-matched siblings forturn\-level credit assignment, we design a sparse tree structure that assigns group\-relative advantages to individual sibling rollouts at each turn and propagates optimization backward across the trajectory\. Finally, RTPO applieson\-policy continuationto coordinate asynchronous short\- and long\-trajectory updates fromπk\\pi\_\{k\}toπ0\\pi\_\{0\}, reducing policy drift\.### 3\.1Turn\-Level Policy Optimization

#### Turn\-boundary MDP for rollout\-training match\.

We model aKK\-turn agent episode as a turn\-boundary MDPℳ=⟨𝒳¯,𝒜H,PH,RH,γH⟩\\mathcal\{M\}=\\langle\\bar\{\\mathcal\{X\}\},\\mathcal\{A\}\_\{H\},P\_\{H\},R\_\{H\},\\gamma\_\{H\}\\rangle, where the augmented statex¯k=\(Sk,k\)\\bar\{x\}\_\{k\}=\(S\_\{k\},k\)encodes the interaction historySkS\_\{k\}and the turn indexkk\. At the turn\-kkboundary, the agent selects a macro\-actionuk≡lk∈𝒜H,k​\(Sk\)u\_\{k\}\\equiv l\_\{k\}\\in\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\), corresponding to the complete turn\-kkresponselk=\(ak,1,…,ak,Tk\)l\_\{k\}=\(a\_\{k,1\},\\ldots,a\_\{k,T\_\{k\}\}\)\. Executinguku\_\{k\}consumesτk=Tk\\tau\_\{k\}=T\_\{k\}token steps, after which the environment returns to the external tool feedbackfkf\_\{k\}, and the interaction history is updated asSk\+1=Sk∘\(lk,fk\)S\_\{k\+1\}=S\_\{k\}\\circ\(l\_\{k\},f\_\{k\}\)\. To define the conditioning context received by the model at turnkk, we letck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\), whereψ\\psican be the identity map, a truncation operator, or a summarization operator\. The turn\-level policy is factorized asπθ=\(πθ,0,…,πθ,K−1\)\\pi\_\{\\theta\}=\(\\pi\_\{\\theta,0\},\\ldots,\\pi\_\{\\theta,K\-1\}\), where each sub\-policy is autoregressive at the token level:πθ,k​\(uk\|ck\)=∏t=1Tkπθ​\(ak,t\|ck,ak,<t\)\\pi\_\{\\theta,k\}\(u\_\{k\}\|c\_\{k\}\)=\\prod\_\{t=1\}^\{T\_\{k\}\}\\pi\_\{\\theta\}\(a\_\{k,t\}\|c\_\{k\},a\_\{k,<t\}\)\.

This turn\-boundary MDP decomposition factorizes the episode policy intoKKturn\-level sub\-policies, with each sub\-policy mapping the conditioning contextck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\)to a complete responselkl\_\{k\}\. Here,ckc\_\{k\}is kept identical between rollout and training\. During rollout, RTPO records the exact contextckc\_\{k\}received by the model, including any truncation or summarization, together with the corresponding old\-policy log\-probabilities\. During training, the sameckc\_\{k\}is used as input, and the loss is computed only over the output tokens inlkl\_\{k\}\. Hence, the denominator of the IS ratio is evaluated under the same conditioning context as in rollout:

ρk,tRTPO=πθ​\(ak,t\|ck,ak,<t\)πθold​\(ak,t\|ck,ak,<t\)\\rho\_\{k,t\}^\{\\mathrm\{RTPO\}\}=\\frac\{\\pi\_\{\\theta\}\(a\_\{k,t\}\|c\_\{k\},a\_\{k,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\|c\_\{k\},a\_\{k,<t\}\)\}\(5\)

#### Per\-turn policy optimization under reverse\-order training\.

Training proceeds in reverse order through the turnsk=K−1,K−2,…,0k=K\{\-\}1,K\{\-\}2,\\ldots,0\. After turnkkis completed,πθ,k\\pi\_\{\\theta,k\}is frozen \(e\.g\., subsequent turns produce no gradients for turn\-kktokens\)\. This reverse ordering ensures that when turnkkis trained, the downstream policies in each turnπθ,k\+1:K−1\\pi\_\{\\theta,k\+1:K\-1\}have been optimized and fixed\.

During reverse\-order training for turn\-kk, the environment is forked from the trunk trajectory’s boundary stateSkS\_\{k\}to generateG−1G\{\-\}1sibling rollouts in the sparse tree\. Each sibling rolloutjjreceives a turn\-level advantageAj,kHA\_\{j,k\}^\{H\}\(the details of sibling rollout generation are provided in Sec\.[3\.2](https://arxiv.org/html/2608.18682#S3.SS2)\)\. The policy optimization objective for turnkkis then defined as:

Jk​\(θ\)=1G−1​∑j=1G−11Tj,k​∑t=1Tj,kmin⁡\(ρj,k,t​Aj,kH,clip⁡\(ρj,k,t,1−ϵ,1\+ϵ\)​Aj,kH\)J\_\{k\}\(\\theta\)=\\frac\{1\}\{G\{\-\}1\}\\sum\_\{j=1\}^\{G\-1\}\\frac\{1\}\{T\_\{j,k\}\}\\sum\_\{t=1\}^\{T\_\{j,k\}\}\\min\\\!\\Big\(\\rho\_\{j,k,t\}\\,A\_\{j,k\}^\{H\},\\;\\operatorname\{clip\}\\big\(\\rho\_\{j,k,t\},\\,1\{\-\}\\epsilon,\\,1\{\+\}\\epsilon\\big\)\\,A\_\{j,k\}^\{H\}\\Big\)\(6\)whereρj,k,t=πθ​\(aj,k,t\|ck,aj,k,<t\)/πθold​\(aj,k,t\|ck,aj,k,<t\)\\rho\_\{j,k,t\}=\\pi\_\{\\theta\}\(a\_\{j,k,t\}\|c\_\{k\},a\_\{j,k,<t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{j,k,t\}\|c\_\{k\},a\_\{j,k,<t\}\)is the token IS ratio andTj,kT\_\{j,k\}is the number of tokens generated by sibling rolloutjjat turnkk\. Only theseG−1G\{\-\}1sibling rollouts are used for gradient updates; the trunk trajectory is excluded\.

InTheorem 1, we show that RTPO has local and global convergence guarantees through reverse\-order turn\-level policy optimization\.

Theorem 1: Convergence to Recursive OptimalityUnder standard assumptions, finite state and macro\-action spaces, Robbins–Monro step sizes, sufficient per\-turn exploration, frozen downstream policies, and bounded rewards, reverse\-order turn\-level optimization satisfies:\(a\)Per\-turn convergence\.For each turnkk, fixing downstream policies reduces optimization to single\-step Q\-learning with a stationary continuation target; hence, per\-turn policyπk\\pi\_\{k\}converges under the stated assumptions\.\(b\)Recursive optimality\.Under reverse\-order training, applying the above argument recursively from turnK−1K\{\-\}1to turn00yields a sequence of per\-turn policies that is recursively optimal\.\(c\)Global optimality\.If the turn\-level macro\-action spaces are complete, i\.e\., the per\-turn policy class can represent any globally feasible trajectory\-level policy, then recursive optimality is equivalent to global optimality over the full trajectory\.

The proof of Theorem 1 is provided inAppendix[C\.2](https://arxiv.org/html/2608.18682#A3.SS2)\.

### 3\.2Turn\-Level Credit Assignment

#### Turn\-level advantage function\.

We construct sparse tree rollouts at turn\-level boundaries, elevating the advantage granularity from trajectory\-only reward to turn\-level credit while ensuring that all compared rollouts share the same state and are free from state bias\. In turnkk,G−1G\{\-\}1siblings independently generate turn\-kkresponses from the shared boundary stateSkS\_\{k\}and continue to the terminal\. The turn\-level advantage function for siblingjjis:Aj,kH=Q^j,k−V^kA^\{H\}\_\{j,k\}=\\hat\{Q\}\_\{j,k\}\-\\hat\{V\}\_\{k\}\.

Here, all siblings share the same stateSkS\_\{k\}, so advantage differences can only arise from two sources: different action choices at turnkkand independent downstream sampling noise\. The downstream noise has zero mean \(𝔼\[ξdown\|Sk,lj,k\]=0\\mathbb\{E\}\[\\xi\_\{\\mathrm\{down\}\}\|S\_\{k\},l\_\{j,k\}\]=0\) and introduces no systematic bias; the upstream state termξup=Vπ​\(Sk\)−V¯\\xi\_\{\\mathrm\{up\}\}=V^\{\\pi\}\(S\_\{k\}\)\-\\bar\{V\}from Eq\. \([3](https://arxiv.org/html/2608.18682#S2.E3)\) is exactly zero, because all siblings shareSkS\_\{k\}\. Furthermore,Aj,kHA^\{H\}\_\{j,k\}is assigned only to the output tokens of turnkk; the shared prefixSkS\_\{k\}serves as the prompt input but does not enter the loss, so each token appears exactly once in the training batch\.

#### Turn\-level value estimation\.

To identify turn\-level values, we separate the local reward at each turn from the downstream completion value over the full trajectory\. Based on the MAXQ principle\([12](https://arxiv.org/html/2608.18682#bib.bib44)\), we define the following action\-value decomposition for turnkk:Q~kπ​\(Sk,uk\)=rk\+γτk​Fkπ​\(Sk\+1\)\\tilde\{Q\}\_\{k\}^\{\\pi\}\(S\_\{k\},u\_\{k\}\)=r\_\{k\}\+\\gamma^\{\\tau\_\{k\}\}F\_\{k\}^\{\\pi\}\(S\_\{k\+1\}\)\.rkr\_\{k\}is the immediate reward at turnkk\(rk=0r\_\{k\}=0fork<K−1k<K\{\-\}1under sparse rewards\), andFkπ​\(Sk\+1\)F\_\{k\}^\{\\pi\}\(S\_\{k\+1\}\)is the downstream continuation value representing the expected cumulative return from turnk\+1k\{\+\}1onward under policyπ\\pi, with base caseFK−1π≡0F\_\{K\-1\}^\{\\pi\}\\equiv 0\. Then, the terminal rewardRjR\_\{j\}obtained by siblingjjrolling out the policy fromSkS\_\{k\}to the terminal is a single Monte Carlo sample of the above:

Q^j,k=Rj=rj,k\+γτk​F^j,k\\hat\{Q\}\_\{j,k\}=R\_\{j\}=r\_\{j,k\}\+\\gamma^\{\\tau\_\{k\}\}\\hat\{F\}\_\{j,k\}\(7\)whereF^j,k\\hat\{F\}\_\{j,k\}is a single sample ofFkπ​\(Sk\+1\)F\_\{k\}^\{\\pi\}\(S\_\{k\+1\}\)\.RjR\_\{j\}is an unbiased estimator ofQ~kπ\\tilde\{Q\}\_\{k\}^\{\\pi\}\. Note that the value estimation quality ofQ^j,k\\hat\{Q\}\_\{j,k\}depends onF^j,k\\hat\{F\}\_\{j,k\}, i\.e\., the quality of the sampled downstream continuation value\. If the downstream policy is not yet optimized, even a strong turn\-kkaction may still produceRj=0R\_\{j\}=0due to downstream errors, thereby contaminating the turn\-level advantage with downstream noise\. The case whereF^j,k\\hat\{F\}\_\{j,k\}is generated by an already optimized downstream policy is addressed in Sec\.[3\.3](https://arxiv.org/html/2608.18682#S3.SS3)\. After estimating the action values at each turn, we compute a state\-specific value baseline from the sibling rollouts\. Specifically, we defineV^k\\hat\{V\}\_\{k\}as the mean of the estimatedQQ\-values across theG−1G\{\-\}1siblings:V^k=1G−1​∑j=1G−1Rj\\hat\{V\}\_\{k\}=\\frac\{1\}\{G\{\-\}1\}\\sum\_\{j=1\}^\{G\-1\}R\_\{j\}\.

InTheorem 2, we show that RTPO obtains accurate turn\-level credit without bias from upstream and downstream effects\.

Theorem 2: Causally Consistent Turn\-Level Advantage EstimationConsider turnkkand suppose that theG−1G\{\-\}1sibling rollouts are forked from the same boundary stateSkS\_\{k\}, with each sibling rolloutj∈\{1,…,G−1\}j\\in\\\{1,\\ldots,G\{\-\}1\\\}independently sampling a turn\-kkmacro\-actionuj,k≡lj,ku\_\{j,k\}\\equiv l\_\{j,k\}\. LetAj,kHA^\{H\}\_\{j,k\}denote the turn\-level advantage assigned to sibling rolloutjj\. Then, under bounded rewards and independent sibling sampling, the turn\-level advantage estimator satisfies the following properties:\(a\)Local unbiasedness up to finite\-group bias\.Conditional onSkS\_\{k\}, the expectation ofAj,kHA^\{H\}\_\{j,k\}is proportional to the true turn\-level advantage, up to a finite\-group bias of orderO⁡\(1/G\)O\(1/G\)\. Because all comparisons are made from the same boundary state, the estimator removes the upstream state\-contamination term that appears in cross\-trajectory comparisons\.\(b\)Reduced value estimation error\.The mean squared error ofAj,kHA^\{H\}\_\{j,k\}is lower than that of trajectory advantage estimation whenever cross\-state value variance is non\-zero\. The improvement gap is governed by the variance of values across different boundary states, which can be large in long\-horizon, multi\-turn tasks\.\(c\)State\-matched causal actions\.Since all siblings share the same boundary stateSkS\_\{k\}, differences in their returns are causally attributable to the sampled turn\-kkmacro\-actionsuj,ku\_\{j,k\}, rather than to upstream trajectory differences\. The resulting advantage is assigned only to turn\-kkoutput tokens, excluding prefix tokens from the gradient\.

The proof of Theorem 2 is provided inAppendix[C\.3](https://arxiv.org/html/2608.18682#A3.SS3)\.

### 3\.3On\-Policy Continuation

#### On\-policy evolution\.

Asynchronous turn updates can induce policy drift that is not fully corrected by per\-token IS clipping\. In principle, one could correct this drift using the full trajectory\-level IS product, but its variance grows exponentially with the horizon length, making it unstable for long\-horizon multi\-turn training\. This drift directly affects the downstream continuation valueFkF\_\{k\}introduced in Sec\.[3\.2](https://arxiv.org/html/2608.18682#S3.SS2): under a stale or mismatched downstream policy,F^j,k\\hat\{F\}\_\{j,k\}can systematically deviate from the true continuation valueFkπθ\>kF\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}, while existing IS\-based corrections are insufficient to remove this deviation\. To avoid this issue, RTPO re\-generates sibling continuations on\-policy at each turn\. Letθ0\\theta\_\{0\}denote the parameters at the start of the policy evolution\. During reverse\-order training, the parameters are updated sequentially across turns\. By the time optimization reaches turnkk, the policies for downstream turnsK−1,…,k\+1K\{\-\}1,\\ldots,k\{\+\}1have already been updated; we denote the resulting current parameters byθ\>k\\theta\_\{\>k\}\. Thus, underπθ\>k\\pi\_\{\\theta\_\{\>k\}\}, the downstream turnsk\+1,…,K−1k\{\+\}1,\\ldots,K\{\-\}1use the optimized continuation policy, whereas turnkkand all upstream turns remain to be optimized\. Note that the trunk is a complete trajectory generated at the start of the policy evolution usingπθ0\\pi\_\{\\theta\_\{0\}\}\. It does not participate in gradient updates, nor does it enter the computation ofV^k\\hat\{V\}\_\{k\}\. Its sole role is to provide boundary statesSkS\_\{k\}and environment snapshotssnapk\\mathrm\{snap\}\_\{k\}as anchoring points for sibling forking\. The trunk’s policy nature affects which statesSkS\_\{k\}are visited during training \(state coverage\), but does not affect the correctness of advantage estimation, since all siblings contributing to the estimate are generated on\-policy\.

#### On\-policy sibling generation\.

At the start of turnkk, RTPO synchronizes the latest parametersθ\>k\\theta\_\{\>k\}to the inference engine, forks the environment fromsnapk\\mathrm\{snap\}\_\{k\}, and generatesG−1G\{\-\}1siblings usingπθ\>k\\pi\_\{\\theta\_\{\>k\}\}\. Each sibling rolloutjjgenerates a turn\-kkresponse, then continues withπθ\>k\\pi\_\{\\theta\_\{\>k\}\}through turnsk\+1k\{\+\}1toK−1K\{\-\}1, obtaining terminal rewardRjR\_\{j\}\. Since the sampling policy equals the current policy, the trajectory\-level IS weight is identically one:ωj=∏h=k\+1K−1∏t=1Tj,hπθ\>k​\(aj,h,t∣sj,h,t\)πθ\>k​\(aj,h,t∣sj,h,t\)≡1\.\\omega\_\{j\}=\\prod\_\{h=k\+1\}^\{K\-1\}\\prod\_\{t=1\}^\{T\_\{j,h\}\}\\frac\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid s\_\{j,h,t\}\)\}\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid s\_\{j,h,t\}\)\}\\equiv 1\.

Since turn\-level value estimation isQ^j,k=rj,k\+γτk​F^j,k\\hat\{Q\}\_\{j,k\}=r\_\{j,k\}\+\\gamma^\{\\tau\_\{k\}\}\\hat\{F\}\_\{j,k\}from Sec\.[3\.2](https://arxiv.org/html/2608.18682#S3.SS2), the sibling’s downstream continuation is performed under the current policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}, the sampleF^j,k\\hat\{F\}\_\{j,k\}is an unbiased draw fromFkπθ\>k​\(Sj,k\+1\)F\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{j,k\+1\}\)\. Therefore:

Q^j,k=rj,k\+γτk​F^j,kπθ\>k\\hat\{Q\}\_\{j,k\}=r\_\{j,k\}\+\\gamma^\{\\tau\_\{k\}\}\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(8\)is an unbiased Monte Carlo estimate ofQ~kπθ\>k​\(Sk,uj,k\)\\tilde\{Q\}\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{k\},u\_\{j,k\}\)\. The entire value estimate requires no IS correction, is unaffected by clip truncation, and is free of multiplicative variance explosion\.

InTheorem 3, we show that RTPO conducts on\-policy continuation to avoid policy drift and further reduces advantage\-estimation errors\.

Theorem 3: On\-Policy Continuation under Asynchronous TurnsIn the reverse\-order training procedure of RTPO, at the start of each asynchronous turnkk, sibling rolloutsj∈\{1,…,G−1\}j\\in\\\{1,\\ldots,G\{\-\}1\\\}are generated from the shared boundary stateSkS\_\{k\}using the current downstream policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}and are then continued to termination under the same policy\. Then the following two properties hold:\(a\)Drift\-free on\-policy continuation\.Each sibling’s terminal returnRjR\_\{j\}provides an unbiased Monte Carlo estimate of the turn\-level Q\-value under the current downstream policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}\. Because the sampling policy and the evaluation policy coincide throughout the sibling continuation, the trajectory\-level importance\-sampling weightωj\\omega\_\{j\}is identically one, and no trajectory\-level IS correction is required\.\(b\)Dynamic error reduction in advantage estimation\.As reverse\-order training improves the downstream policies, the continuation value estimatesF^j,kπθ\>k\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}become aligned with the current optimized downstream policy rather than a stale policy\. Under bounded binary or normalized rewards, when the induced success probabilities move away from the high\-uncertainty region around one\-half, the variance of the Monte Carlo Q\-value estimator decreases, thereby improving the signal\-to\-noise ratio of the resulting turn\-level advantage estimatesAj,kHA^\{H\}\_\{j,k\}\.

The proof of Theorem 3 is provided inAppendix[C\.4](https://arxiv.org/html/2608.18682#A3.SS4)\.

## 4Experimental Results and Analysis

Experimental Setting\.We use Qwen3\-8B\([58](https://arxiv.org/html/2608.18682#bib.bib8)\)as the backbone for the main experiments on RTPO and all baselines, enabling Qwen3’s thinking mode for all multi\-turn agentic RL rollouts\. We compare RTPO with trajectory\-level methods, including GRPO\([42](https://arxiv.org/html/2608.18682#bib.bib67)\), and turn/tree\-level credit methods, including ARPO\([13](https://arxiv.org/html/2608.18682#bib.bib36)\), TreeGRPO\([22](https://arxiv.org/html/2608.18682#bib.bib35)\), and SeeUPO\([20](https://arxiv.org/html/2608.18682#bib.bib66)\)\. We consider 12 experiments covering comprehensive mathematical and knowledge reasoning benchmarks, where agents perform multi\-turn RL training with external calculation and web\-search tools\. All methods are implemented in VeRL\([43](https://arxiv.org/html/2608.18682#bib.bib47)\), with vLLM\([24](https://arxiv.org/html/2608.18682#bib.bib48)\)for rollout generation and FSDP\([66](https://arxiv.org/html/2608.18682#bib.bib49)\)for distributed training\. We evaluate training performance and stability using task accuracy, tool\-call statistics, log\-probability comparisons, turn\-level credit effects, and the hit rates of on\-policy versus off\-policy outputs\. Detailed experimental setup, including base models, baselines, datasets, configurations, and evaluation metrics, is provided inAppendix[D](https://arxiv.org/html/2608.18682#A4)\. Implementation details, including pseudocode, source code, training details, and the Qwen3 chat template, are provided inAppendix[E](https://arxiv.org/html/2608.18682#A5)\.

### 4\.1Main Results

We compare RTPO against the trajectory\-level method GRPO and the turn\-level method SeeUPO, and additionally include the untuned vanilla model as a non\-RL reference\.Table[1](https://arxiv.org/html/2608.18682#S4.T1)reports both accuracy and the corresponding number of tool calls on each benchmark\. We evaluate overall performance on mathematical reasoning and knowledge\-intensive question answering across three difficulty tiers: easy \(GSM8K\), medium \(AMC23, MATH500\), and hard \(AIME24, AIME25, OE\-Math, HotpotQA, and 2Wiki\)\. Our RTPO achieves the best performance across all eight benchmarks, improving overall accuracy over vanilla by 66\.78%, outperforming GRPO \(\+21\.50%\) and SeeUPO \(\+10\.76%\)\. Further discussion and insights on tool calls are provided inAppendix[F\.1](https://arxiv.org/html/2608.18682#A6.SS1)\.

Table 1:Performance comparison across eight tool\-use agentic RL benchmarks\.Accdenotes Pass@1 for mathematical tasks and best\-span F1 for knowledge tasks, whileCallsdenotes the number of tool calls rounded to the nearest integer, such as Python and web\-search calls\.Overallsubscripts indicate relative changes in accuracy compared with the vanilla model, along with total tool calls \(M, K\)\.
### 4\.2Rollout–Training Consistency Analysis

To examine RTPO’s training\-time advantage, we measure at each step the geometric\-mean ratio between training\-stage and rollout\-stage tokenlogprobs\. This ratio captures rollout–training consistency: values near11indicate matched conditioning distributions, while deviations suggest that the full\-history training policy favors outputs different from those sampled during rollout\. The Kullback–Leibler \(KL\) divergence further quantifies the distributional gap, with smaller values indicating stronger rollout–training consistency\.Figure[3](https://arxiv.org/html/2608.18682#S4.F3)shows the log\-probability ratio and KL divergence during training\. On mathematical tasks, RTPO stays stable at1\.01\.0, while SeeUPO fluctuates around1\.01\.0and GRPO recovers from0\.890\.89to0\.970\.97after3030steps\. On knowledge tasks, RTPO again remains at1\.01\.0, whereas SeeUPO and GRPO recover only from around0\.800\.80to0\.890\.89and0\.830\.83after1414steps\. RTPO also achieves the lowest average KL divergence, indicating more consistent rollout\-training contexts than the baselines\.

![Refer to caption](https://arxiv.org/html/2608.18682v1/Figure/picture0.png)\(a\)Rollout\-train ratio \[M\]
![Refer to caption](https://arxiv.org/html/2608.18682v1/Figure/picture1.png)\(b\)Rollout\-train ratio \[K\]
![Refer to caption](https://arxiv.org/html/2608.18682v1/Figure/picture2.png)\(c\)KL divergence \(M\)
![Refer to caption](https://arxiv.org/html/2608.18682v1/Figure/picture3.png)\(d\)KL divergence \(K\)

Figure 3:Rollout–training consistency comparison using log\-probability ratios and KL divergence on mathematical \(M\) and knowledge \(K\) reasoning tasks\. Zoom in for better visualization\.A noteworthy observation is that, despite using different rollout and training contexts, baseline ratios still drift slowly toward11\. This resembles the bootstrapped alignment mechanisms in DAgger\([40](https://arxiv.org/html/2608.18682#bib.bib9)\)and SCoRe\([23](https://arxiv.org/html/2608.18682#bib.bib10)\)\. However, unlike RTPO’s structural consistency, this empirical alignment is incomplete, task\-dependent, noisy, and consumes additional optimization budget\. We provide further discussion and insights inAppendix[F\.2](https://arxiv.org/html/2608.18682#A6.SS2)\.

### 4\.3Effect of Turn\-Level Credit Assignment

To isolate the effect of RTPO’s turn\-level credit signal, we feed the full interaction history as the rollout input for all methods, controlling for the rollout–training context mismatch presented in Sec[4\.2](https://arxiv.org/html/2608.18682#S4.SS2)\. Under this control setting, performance differences mainly reflect the effect of credit assignment \(CA\) on advantage estimation\. Because full interaction histories are used, the accuracy scores in Table[2](https://arxiv.org/html/2608.18682#S4.T2)are substantially higher than those in Table[1](https://arxiv.org/html/2608.18682#S4.T1)\. RTPO\-CA achieves the highest overall average across the four mathematical reasoning benchmarks, as shown inTable[2](https://arxiv.org/html/2608.18682#S4.T2)\. It achieves the best results on AMC23 \(Pass@193\.3393\.33, Pass@4100\.0100\.0\), AIME25 \(Pass@170\.0070\.00, Pass@476\.6776\.67\), and MATH500 \(Pass@186\.6086\.60, Pass@489\.6089\.60\), while matching the best Pass@4 on AIME24 \(80\.0080\.00\)\. These results show that RTPO improves both solution coverage and the preference for correct answers, with the largest gain on the harder AIME25 benchmark\. This supports the value of turn\-level credit assignment: by forking sibling rollouts from the same boundary stateSkS\_\{k\}, RTPO forms a local baseline and attributes advantage directly to the current turn decision\.

Table 2:Mathematical reasoning performance under controlled turn\-level credit assignment \(CA\)\. Overall reports the average across benchmarks\. Bold indicates the best result\.
### 4\.4Policy Drift Correction

To isolate the effect of on\-policy continuation \(Sec[3\.3](https://arxiv.org/html/2608.18682#S3.SS3)\), we compare the default RTPO with an off\-policy variant that reuses downstream continuations generated byπθ0\\pi\_\{\\theta\_\{0\}\}during the initial rollout and corrects staleness using a clamped trajectory\-level IS weight; see Appendix[E\.2](https://arxiv.org/html/2608.18682#A5.SS2)for details\. We evaluate both variants on four knowledge\-reasoning deep\-search benchmarks \(GAIA, WebWalkerQA, HLE, and XBench\) using output\-hit accuracy\. As shown inTable[3](https://arxiv.org/html/2608.18682#S4.T3), default RTPO outperforms the off\-policy variant on GAIA \(\+5\.83%\+5\.83\\%, WebWalkerQA \(\+3\.50%\+3\.50\\%\), and XBench \(\+7\.00%\+7\.00\\%\)\. On HLE, the difference is negligible \(−0\.33%\-0\.33\\%\), indicating a near tie\. This pattern supportsTheorem 3\(b\): on\-policy continuation is most beneficial when downstream policies change substantially during reverse\-order training\. GAIA, WebWalkerQA, and XBench involve longer retrieval and interaction horizons, where stale\-rollout IS correction can introduce clamp\-truncation bias that on\-policy re\-sampling avoids\. In contrast, HLE is more closed\-ended and often requires shorter search horizons, leading to smaller gains; additional results inAppendix[F\.3](https://arxiv.org/html/2608.18682#A6.SS3)further support this interpretation\. In addition, we discuss the limitations, future work, and broader impacts of RTPO inAppendix[F\.4](https://arxiv.org/html/2608.18682#A6.SS4),[F\.5](https://arxiv.org/html/2608.18682#A6.SS5)\.

Table 3:Output\-hit comparison between on\- and off\-policy RTPO variants on knowledge tasks\. Values are rounded to the nearest integer\.Δ\\Deltadenotes the change from off\-policy to on\-policy hit rates\.

## 5Concluding Remarks

This work identifies rollout–training mismatch as a fundamental source of instability in multi\-turn agentic RL, particularly in tool\-augmented mathematical reasoning and deep\-search tasks\. We provide a theoretical analysis showing how existing training pipelines produce unstable optimization signals and propose RTPO as a principled framework to address this issue\. RTPO integrates rollout–training consistency, turn\-level credit assignment, and on\-policy continuation within a unified training pipeline\. Supported by theoretical guarantees and empirical results, RTPO improves multi\-turn optimization stability and provides a promising direction for training long\-horizon tool\-using agents\.

## References

- D\. Abel, N\. Umbanhowar, K\. Khetarpal, D\. Arumugam, D\. Precup, and M\. LittmanValue preserving state\-action abstractions\.InProceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics,Cited by:[§B\.1](https://arxiv.org/html/2608.18682#A2.SS1.SSS0.Px2.p1.1)\.
- Allenet al\.\(2021\)C\. Allen, N\. Parikh, O\. Gottesman, and G\. KonidarisLearning markov state abstractions for deep reinforcement learning\.Advances in Neural Information Processing Systems\.Cited by:[§B\.1](https://arxiv.org/html/2608.18682#A2.SS1.SSS0.Px2.p1.1),[§F\.2](https://arxiv.org/html/2608.18682#A6.SS2.p1.1)\.
- Bertsekas \(2025\)D\. P\. BertsekasNeuro\-dynamic programming\.InEncyclopedia of optimization,Cited by:[§C\.1](https://arxiv.org/html/2608.18682#A3.SS1.SSS0.Px3.p3.1)\.
- Browneet al\.\(2012\)C\. B\. Browne, E\. Powley, D\. Whitehouse, S\. M\. Lucas, P\. I\. Cowling, P\. Rohlfshagen, S\. Tavener, D\. Perez, S\. Samothrakis, and S\. ColtonA survey of monte carlo tree search methods\.IEEE Transactions on Computational Intelligence and AI in Games\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1)\.
- Caoet al\.\(2026a\)L\. Cao, H\. Ruan, Y\. Li, P\. Chao, W\. Ning, H\. Song, R\. Chen, and Y\. LiTreeAdv: tree\-structured advantage redistribution for group\-based rl\.arXiv preprint arXiv:2601\.03703\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1)\.
- Caoet al\.\(2026b\)R\. Cao, S\. Bai, F\. Yao, L\. Dong, J\. Xu, and L\. XiaoATPO: adaptive tree policy optimization for multi\-turn medical dialogue\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1)\.
- Changet al\.\(2026\)Q\. Chang, Z\. Zhang, P\. Hu, J\. Du, J\. Ma, Y\. Pan, J\. Zhang, Q\. Liu, and J\. GaoTHOR: tool\-integrated hierarchical optimization via RL for mathematical reasoning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Chenet al\.\(2025\)K\. Chen, Y\. Ren, Y\. Liu, X\. Hu, H\. Tian, T\. Xie, F\. Liu, H\. Zhang, H\. Liu, Y\. Gong,et al\.Xbench: tracking agents productivity scaling with profession\-aligned real\-world evaluations\.arXiv preprint arXiv:2506\.13651\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Chenet al\.\(2026\)Y\. Chen, G\. Dong, and Z\. DouToward effective tool\-integrated reasoning via self\-evolved preference learning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p2.1)\.
- Coulom \(2006\)R\. CoulomEfficient selectivity and backup operators in monte\-carlo tree search\.InProceedings of the 5th International Conference on Computers and Games,Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1),[§B\.1](https://arxiv.org/html/2608.18682#A2.SS1.SSS0.Px2.p1.1)\.
- Dietterich \(2000\)T\. G\. DietterichHierarchical reinforcement learning with the maxq value function decomposition\.Journal of artificial intelligence research\.Cited by:[§3\.2](https://arxiv.org/html/2608.18682#S3.SS2.SSS0.Px2.p1.1)\.
- Donget al\.\(2026\)G\. Dong, H\. Mao, K\. Ma, L\. Bao, Y\. Chen, Z\. Wang, Z\. Chen, J\. Du, H\. Wang, F\. Zhang, G\. Zhou, Y\. Zhu, J\. Wen, and Z\. DouAgentic reinforced policy optimization\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1),[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px2.p1.1),[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1),[§1](https://arxiv.org/html/2608.18682#S1.p2.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. NeubigPal: program\-aided language models\.InInternational conference on machine learning,Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Hauskrechtet al\.\(1998\)M\. Hauskrecht, N\. Meuleau, L\. P\. Kaelbling, T\. Dean, and C\. BoutilierHierarchical solution of markov decision processes using macro\-actions\.InProceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence,Cited by:[§2](https://arxiv.org/html/2608.18682#S2.p2.1)\.
- Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.arXiv preprint arXiv:2402\.14008\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.Thirty\-Fifth Annual Conference on Neural Information Processing Systems\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p2.1)\.
- Hoet al\.\(2020\)X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. AizawaConstructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Huet al\.\(2026\)T\. Hu, Q\. Fu, Y\. Chen, Z\. Liu, and B\. DingSeeUPO: sequence\-level agentic\-rl with convergence guarantees\.arXiv preprint arXiv:2602\.06554\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px1.p2.1),[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.18682#S1.p2.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Jaakkolaet al\.\(1994\)T\. Jaakkola, M\. I\. Jordan, and S\. P\. SinghOn the convergence of stochastic iterative dynamic programming algorithms\.Neural Computation\.Cited by:[§C\.1](https://arxiv.org/html/2608.18682#A3.SS1.SSS0.Px3.p3.1)\.
- Jiet al\.\(2026\)Y\. Ji, Z\. Ma, Y\. Wang, G\. Chen, X\. Chu, and L\. WuTree search for LLM agent reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1),[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.18682#S1.p2.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Kumaret al\.\(2025\)A\. Kumar, V\. Zhuang, R\. Agarwal, Y\. Su, J\. D\. Co\-Reyes, A\. Singh, K\. Baumli, S\. Iqbal, C\. Bishop, R\. Roelofs,et al\.Training language models to self\-correct via reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§F\.2](https://arxiv.org/html/2608.18682#A6.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.18682#S4.SS2.p2.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Leet al\.\(2022\)H\. Le, Y\. Wang, A\. D\. Gotmare, S\. Savarese, and S\. C\. H\. HoiCoderl: mastering code generation through pretrained models and deep reinforcement learning\.Advances in Neural Information Processing Systems\.Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Liet al\.\(2025\)K\. Li, Z\. Zhang, H\. Yin, L\. Zhang, L\. Ou, J\. Wu, W\. Yin, B\. Li, Z\. Tao, X\. Wang,et al\.Websailor: navigating super\-human reasoning for web agent\.arXiv preprint arXiv:2507\.02592\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Liet al\.\(2026\)Y\. Li, Z\. Cao, J\. Qiao, and S\. HuSSVPO: effective step\-level credit assignment for RL training of language models\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p2.1)\.
- Lindenbaueret al\.\(2025\)T\. Lindenbauer, I\. Slinko, L\. Felder, E\. Bogomolov, and Y\. ZharovThe complexity trap: simple observation masking is as efficient as llm summarization for agent context management\.arXiv preprint arXiv:2508\.21433\.Cited by:[§B\.1](https://arxiv.org/html/2608.18682#A2.SS1.p1.1)\.
- Liuet al\.\(2026\)M\. Liu, X\. Ma, Y\. Xie, Q\. Chen, X\. Liu, and Y\. MaROGA: scaling generalist agents for office productivity tasks via tool generation\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px4.p1.1)\.
- Luet al\.\(2025\)M\. Lu, W\. Sun, W\. Du, Z\. Ling, X\. Yao, K\. Liu, and J\. ChenScaling llm multi\-turn rl with end\-to\-end summarization\-based context management\.arXiv preprint arXiv:2510\.06727\.Cited by:[§F\.2](https://arxiv.org/html/2608.18682#A6.SS2.p1.1)\.
- Mathematical Association of America \(2023\)Mathematical Association of AmericaAmerican mathematics competitions \(AMC\)\.Mathematical Association of America\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p2.1)\.
- Mathematical Association of America \(2024\)Mathematical Association of AmericaAmerican invitational mathematics examination \(AIME\)\.Mathematical Association of America\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p2.1)\.
- Mathematical Association of America \(2025\)Mathematical Association of AmericaAmerican invitational mathematics examination \(AIME\)\.Mathematical Association of America\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p2.1)\.
- Mialonet al\.\(2023\)G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. ScialomGaia: a benchmark for general ai assistants\.arXiv preprint arXiv:2311\.12983\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Phanet al\.\(2025\)L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1)\.
- Robbins and Monro \(1951\)H\. Robbins and S\. MonroA stochastic approximation method\.The Annals of Mathematical Statistics\.Cited by:[§C\.2](https://arxiv.org/html/2608.18682#A3.SS2.SSS0.Px5.p7.1.1)\.
- Rosset al\.\(2011\)S\. Ross, G\. Gordon, and D\. BagnellA reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,Cited by:[§F\.2](https://arxiv.org/html/2608.18682#A6.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.18682#S4.SS2.p2.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.18682#S1.p1.1),[§1](https://arxiv.org/html/2608.18682#S1.p2.1),[§2](https://arxiv.org/html/2608.18682#S2.p2.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Shenget al\.\(2024\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px4.p1.1),[§E\.2](https://arxiv.org/html/2608.18682#A5.SS2.p1.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in Neural Information Processing Systems\.Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Shojaeeet al\.\(2023\)P\. Shojaee, A\. Jain, S\. Tipirneni, and C\. K\. ReddyExecution\-based code generation using deep reinforcement learning\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Silveret al\.\(2016\)D\. Silver, A\. Huang, C\. Maddison, A\. Guez, L\. Sifre, G\. Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot, S\. Dieleman, D\. Grewe, J\. Nham, N\. Kalchbrenner, I\. Sutskever, T\. Lillicrap, M\. Leach, K\. Kavukcuoglu, T\. Graepel, and D\. HassabisMastering the game of go with deep neural networks and tree search\.Nature\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1)\.
- Singhet al\.\(2000\)S\. Singh, T\. Jaakkola, M\. L\. Littman, and C\. SzepesváriConvergence results for single\-step on\-policyreinforcement\-learning algorithms\.Machine Learning\.Cited by:[§C\.1](https://arxiv.org/html/2608.18682#A3.SS1.SSS0.Px3.p3.1)\.
- Sunet al\.\(2025\)S\. Sun, H\. Song, Y\. Wang, R\. Ren, J\. Jiang, J\. Zhang, F\. Bai, J\. Deng, W\. X\. Zhao, Z\. Liu,et al\.Simpledeepsearcher: deep information seeking via web\-powered reasoning trajectory synthesis\.arXiv preprint arXiv:2505\.16834\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Teamet al\.\(2025\)K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao,et al\.Kimi k1\. 5: scaling reinforcement learning with llms\.arXiv preprint arXiv:2501\.12599\.Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Tianet al\.\(2026\)W\. Tian, S\. Zhang, K\. Zhang, X\. Chi, C\. Fan, J\. Lu, Y\. Luo, Q\. Zhou, Y\. Zhao, N\. Liu, S\. Lin, Z\. Qin, X\. Ju, S\. Zhang, and J\. TangSEEA\-r1: tree\-structured reinforcement fine\-tuning for self\-evolving embodied agents\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1)\.
- Tsitsiklis \(1994\)J\. N\. TsitsiklisAsynchronous stochastic approximation and q\-learning\.Machine learning\.Cited by:[§C\.1](https://arxiv.org/html/2608.18682#A3.SS1.SSS0.Px3.p3.1)\.
- Wanget al\.\(2025a\)X\. Wang, S\. Rosenberg, J\. Michelini, C\. Smith, H\. Tran, E\. Nyst, R\. Malhotra, X\. Zhou, V\. Chen, R\. Brennan,et al\.The openhands software agent sdk: a composable and extensible foundation for production agents\.arXiv preprint arXiv:2511\.03690\.Cited by:[§B\.1](https://arxiv.org/html/2608.18682#A2.SS1.p1.1)\.
- Wanget al\.\(2025b\)Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu,et al\.Ragen: understanding self\-evolution in llm agents via multi\-turn reinforcement learning\.arXiv preprint arXiv:2504\.20073\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px3.p1.1)\.
- Watkins and Dayan \(1992\)C\. J\. Watkins and P\. DayanQ\-learning\.Machine learning\.Cited by:[§C\.1](https://arxiv.org/html/2608.18682#A3.SS1.SSS0.Px3.p3.1)\.
- Wuet al\.\(2025a\)J\. Wu, W\. Yin, Y\. Jiang, Z\. Wang, Z\. Xi, R\. Fang, L\. Zhang, Y\. He, D\. Zhou, P\. Xie,et al\.Webwalker: benchmarking llms in web traversal\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Wuet al\.\(2025b\)X\. Wu, K\. Li, Y\. Zhao, L\. Zhang, L\. Ou, H\. Yin, Z\. Zhang, X\. Yu, D\. Zhang, Y\. Jiang,et al\.Resum: unlocking long\-horizon search intelligence via context summarization\.arXiv preprint arXiv:2509\.13313\.Cited by:[§B\.1](https://arxiv.org/html/2608.18682#A2.SS1.p1.1)\.
- Xueet al\.\(2026\)Z\. Xue, L\. Zheng, Q\. Liu, Y\. Li, X\. Zheng, Z\. MA, and B\. AnSimpleTIR: end\-to\-end reinforcement learning for multi\-turn tool\-integrated reasoning\.InFirst Workshop on Multi\-Turn Interactions in Large Language Models,Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin,et al\.Qwen2\. 5\-math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Yanget al\.\(2025b\)Z\. Yang, Z\. Guo, Y\. Huang, X\. Liang, Y\. Wang, and J\. TangTreerpo: tree relative policy optimization\.arXiv preprint arXiv:2506\.05183\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px3.p3.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18682#S1.p1.1)\.
- Yuet al\.\(2026a\)H\. Yu, T\. Chen, J\. Feng, J\. Chen, W\. Dai, Q\. Yu, Y\. Zhang, W\. Ma, J\. Liu, M\. Wang, and H\. ZhouMemAgent: reshaping long\-context LLM with multi\-conv RL\-based memory agent\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§F\.2](https://arxiv.org/html/2608.18682#A6.SS2.p1.1)\.
- Yuet al\.\(2026b\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, YuYue, W\. Dai, T\. Fan, G\. Liu, J\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, R\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.18682#S2.p2.1)\.
- Yueet al\.\(2025\)Y\. Yue, Y\. Yuan, Q\. Yu, X\. Zuo, R\. Zhu, W\. Xu, J\. Chen, C\. Wang, T\. Fan, Z\. Du,et al\.Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks\.arXiv preprint arXiv:2504\.05118\.Cited by:[§2](https://arxiv.org/html/2608.18682#S2.p2.1)\.
- Zhaoet al\.\(2023\)Y\. Zhao, A\. Gu, R\. Varma, L\. Luo, C\. Huang, M\. Xu, L\. Wright, H\. Shojanazeri, M\. Ott, S\. Shleifer,et al\.Pytorch fsdp: experiences on scaling fully sharded data parallel\.arXiv preprint arXiv:2304\.11277\.Cited by:[Appendix D](https://arxiv.org/html/2608.18682#A4.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.18682#S4.p1.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.18682#S1.p2.1),[§2](https://arxiv.org/html/2608.18682#S2.p2.1)\.
- Zhonget al\.\(2024\)Y\. Zhong, J\. G\. Kuba, X\. Feng, S\. Hu, J\. Ji, and Y\. YangHeterogeneous\-agent reinforcement learning\.Journal of Machine Learning Research\.Cited by:[Appendix A](https://arxiv.org/html/2608.18682#A1.SS0.SSS0.Px1.p2.1)\.

## Appendix ARelated Work

#### Policy optimization in agentic RL\.

Recent post\-training of large language models \(LLMs\) has shifted from supervised fine\-tuning toward reinforcement learning with verifiable rewards \(RLVR\)\. Among existing approaches, GRPO\[[42](https://arxiv.org/html/2608.18682#bib.bib67)\], built upon PPO, reduces variance through token\-level importance ratios and group\-relative advantage estimation, and has become a representative algorithm for agentic RL\. Its sequence\-level refinement, GSPO\[[67](https://arxiv.org/html/2608.18682#bib.bib68)\], further defines importance ratios and clipping operations at the sequence level, leading to more stable training dynamics on models such as Qwen3\. However, when these RL methods are directly transferred from single\-turn to multi\-turn agentic settings, a structural mismatch emerges between rollout and training\. Existing methods often treat the entire multi\-turn interaction as a single concatenated trajectory and distribute a single scalar reward uniformly across all tokens, thereby ignoring the actual contribution of each turn\. More importantly, the rollout may operate on truncated or summarized contexts, whereas the training recomputes importance ratios over the full interaction history\. This discrepancy induces a mismatch in conditioning distributions and weakens the assumptions under which PPO\-style optimization is expected to remain stable\. In long\-horizon multi\-turn scenarios, these issues can manifest as training divergence or eventual policy collapse\.

A noteworthy latest work is SeeUPO\[[20](https://arxiv.org/html/2608.18682#bib.bib66)\], which first models multi\-turn interaction as a sequentially executed multi\-agent bandit problem\. Under the heterogeneous\-agent RL framework\[[68](https://arxiv.org/html/2608.18682#bib.bib14)\], SeeUPO updates turn\-level virtual agents in reverse execution order\(T→T−1→⋯→1\)\(T\\to T\-1\\to\\cdots\\to 1\), thereby inheriting the monotonic\-improvement property and proving convergence to the globally optimal policy\. This provides strong motivation for our proposed RTPO design, particularly its reverse\-order turn updates\. However, each turn\-level agent in SeeUPO is still trained on the full chat history rather than on the turn\-level conditioning contextckc\_\{k\}actually observed by the model during rollout; therefore, the policy\-forward mismatch is not eliminated\. Moreover, within each turn, SeeUPO corrects downstream advantages using token\-level importance\-sampling reweighting\. As the reverse training recursion proceeds and the number of involved turns accumulates, the variance of the importance\-sampling ratio can grow multiplicatively\. Although clipping bounds this ratio from above, it also introduces systematic bias into the per\-turn advantage estimate\.

#### Tree\-based credit assignment\.

To mitigate the sparse\-credit problem caused by flattened trajectories in multi\-turn agentic RL training, a recent line of work reorganizes rollouts into tree structures with shared prefixes, thereby constructing finer\-grained credit signals\. ARPO\[[13](https://arxiv.org/html/2608.18682#bib.bib36)\]adaptively tree branches at high\-entropy nodes following tool calls and, through advantage attribution estimation, applies branch averaging to shared\-prefix tokens while estimating advantages independently for tokens on disjoint branches\. Afterward, TreeGRPO\[[22](https://arxiv.org/html/2608.18682#bib.bib35)\]abstracts each turn as a tree node and combines intra\-tree and inter\-tree group\-relative advantages, theoretically establishing gradient\-level equivalence with step\-level DPO\[[38](https://arxiv.org/html/2608.18682#bib.bib15)\]\. This tree\-based formulation is extended in SEEA\-R1\[[50](https://arxiv.org/html/2608.18682#bib.bib28)\], which integrates MCTS\[[11](https://arxiv.org/html/2608.18682#bib.bib26),[4](https://arxiv.org/html/2608.18682#bib.bib16),[46](https://arxiv.org/html/2608.18682#bib.bib27)\]into embodied\-agent settings and trains a multimodal generative reward model to densify sparse outcome rewards\. Tree\-structured credit assignment has also been explored in step/token levels\. TreeRPO\[[60](https://arxiv.org/html/2608.18682#bib.bib17)\]adopts anNN\-ary tree for mathematical reasoning and constructs step\-level rewards through bottom\-up Bellman expectations\. TreeAdv\[[5](https://arxiv.org/html/2608.18682#bib.bib18)\]employs entropy\-triggered branching and redistributes leaf advantages to tokens using inverse\-descendant\-count weighting\. Similarly, ATPO\[[6](https://arxiv.org/html/2608.18682#bib.bib20)\]operates under a hierarchical MDP and uses Bellman error andQQ\-value variance as uncertainty measures for adaptive expansion, while applying visit\-count\-based down\-weighting to suppress update imbalance caused by repeated nodes\.

Although these tree\-based rollouts improve the granularity of credit assignment in multi\-turn training, they do not fully eliminate the propagation of trajectory\-level credit mismatch to individual turns through the reconstructed tree\. For instance, in Tree\-GRPO and SEEA\-R1, advantages are still derived from leaf returns, typically in the formAi=Ri−mean⁡\(R\)std⁡\(R\)A\_\{i\}=\\frac\{R\_\{i\}\-\\mathrm\{mean\}\(R\)\}\{\\mathrm\{std\}\(R\)\}\. As a result, prefix nodes that appear on multiple paths inherit trajectory\-level signals from all descendant outcomes, causing the gradient direction at shared tokens to be perturbed by trajectory\-level scalars originating from different rollouts\. In TreeAdv, the1/\|S\|1/\|S\|normalization attenuates the signal more aggressively near the root, precisely suppressing early decisions where informative gradients are often most needed\. In addition, ARPO averages multi\-branch advantages on shared tokens, which can dilute the signal linearly with the number of branches, while ATPO relies on a learned critic and is therefore exposed to critic\-induced bias\. Therefore, under the rollout–training contextual mismatch settings, shared\-prefix tokens may still carry trajectory\-level credit signals with significant residual bias\.

#### Policy gradient correction\.

Another line of work studies stable training from the perspective of correcting \(off\-\)policy drift through interventions\. SimpleTIR\[[57](https://arxiv.org/html/2608.18682#bib.bib1)\], by decomposing the policy gradient on softmax logits, attributes gradient explosion in multi\-turn tool\-integrated reasoning \(TIR\) to the accumulation of low\-probability tokens under the distributional shift induced by tool feedback\. It proposes filtering out entire trajectories that contain*void turns*: turns produce neither a complete code block nor a final answer to block harmful gradients\. This method is empirically effective and plug\-and\-play; however, the void\-turn criterion is tightly coupled with the code\-execution setting of mathematical reasoning and is difficult to transfer to other scenarios, such as web search\. It also inevitably discards a non\-trivial fraction of training data\. RAGEN\[[53](https://arxiv.org/html/2608.18682#bib.bib19)\]discovers the inconsistency between rollout and training engines for mismatch correction through numerical mechanisms such as truncated importance sampling\. However, these approaches mainly address numerical discrepancies while overlooking the contextual mismatch between rollout conditioning and training recomputation\. In contrast, our proposed RTPO does not rely on trajectory dropping or post\-hoc numerical correction\. Instead, it restructures the training paradigm from a turn\-boundary MDP perspective, maintaining the on\-policy optimization throughout the entire training process\.

## Appendix BFull Theoretical Analysis: Training Instability

Here, we provide the full theoretical analysis of training instability in multi\-turn agentic RL:Single flattened\-trajectory policy optimization \(B\.1\):Standard rollouts are typically generated under truncated or summarized contexts for efficiency, whereas training re\-evaluates tokens under concatenated full\-history contexts without truncation\. This rollout–training context mismatch induces biased importance\-sampling \(IS\) ratios, thereby undermining stable policy optimization\.Trajectory\-only credit assignment \(B\.2\):In multi\-turn interactions, a given state may admit multiple valid actions, requiring accurate credit assignment for each action\. However, credit computed from a single trajectory\-level advantage can obscure the contribution of individual turns and therefore cannot provide a proper comparison among alternative actions\.Long\-horizon off\-policy drift \(B\.3\):In long\-horizon tasks, standard policy optimization with per\-token clipping is insufficient to correct policy drift under asynchronous training\. Since later states depend on earlier generated actions and tool feedback, once the policy drifts, the later\-turn states visited during rollout may no longer match those induced by the current policy\.

#### Preliminaries\.

A multi\-turn interaction with TIR is represented as a standard trajectory\(q,l0,f0,…,ln−1,fn−1\)\(q,l\_\{0\},f\_\{0\},\\ldots,l\_\{n\-1\},f\_\{n\-1\}\)spanningnnturns, whereqqis the initial prompt,lkl\_\{k\}is the response generated, andfkf\_\{k\}is the corresponding external tool feedback returned by the environment for each turnk∈\{0,…,n−1\}k\\in\\\{0,\\dots,n\-1\\\}\. We formulate this process as aHierarchical Markov Decision Process \(H\-MDP\), which captures turn\-level \(high\-level\) planning and token\-level \(low\-level\) execution in multi\-turn agentic RL\. At the turn level, for each turnkk, the agent stateSkS\_\{k\}represents all interaction history\(q,l0,f0,…,lk−1,fk−1\)\\left\(q,l\_\{0\},f\_\{0\},\\ldots,l\_\{k\-1\},f\_\{k\-1\}\\right\), available before the current turn, including previous responses, tool calls, and corresponding environment feedback\. The corresponding turn\-level action is the decision of what response to produce at the current turn, denoted bylk∈𝒜Hl\_\{k\}\\in\\mathcal\{A\}\_\{H\}\. After executing this action and receiving environmental feedbackfkf\_\{k\}, the turn\-level state evolves asSk\+1=Sk∘\(lk,fk\)S\_\{k\+1\}=S\_\{k\}\\circ\(l\_\{k\},f\_\{k\}\), where∘\\circdenotes concatenation\.

Each turn\-level action \(response\)lkl\_\{k\}is generated autoregressively at the token level aslk=\(ak,1,ak,2,…,ak,Tk\)l\_\{k\}=\(a\_\{k,1\},a\_\{k,2\},\\ldots,a\_\{k,T\_\{k\}\}\), whereak,t∈𝒜La\_\{k,t\}\\in\\mathcal\{A\}\_\{L\}denotes the token generated at steptt, andTkT\_\{k\}is the number of tokens in the response at turnkk\. The corresponding low\-level state issk,t=\(Sk,ak,1,…,ak,t−1\)∈𝒮Ls\_\{k,t\}=\(S\_\{k\},a\_\{k,1\},\\ldots,a\_\{k,t\-1\}\)\\in\\mathcal\{S\}\_\{L\}, which consists of the turn historySkS\_\{k\}together with the token prefix generated so far in the current turn\. Since the token only serves to generate the turn\-level action \(response\) and does not itself receive intermediate reward, we set the low\-level reward toRL=0R\_\{L\}=0and setγH=γL=1\\gamma\_\{H\}=\\gamma\_\{L\}=1as the discount factor\.

Our hierarchical formulation differs from existing policy optimization methods, which do not explicitly distinguish turn\-level planning from token\-level execution\. Instead, they flatten the entire multi\-turn interaction into a single token sequence, i\.e\., a single trajectory, and optimize it using clipped policy optimization methods in the PPO family\. Letq∼𝒟q\\sim\\mathcal\{D\}be an input prompt sampled from the task distribution, and let\{gi\}i=1G\\\{g\_\{i\}\\\}\_\{i=1\}^\{G\}be a group ofGGtrajectories of this form sampled from the old policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}conditioned onqq, whereθ\\thetadenotes the current policy parameters andθold\\theta\_\{\\mathrm\{old\}\}denotes the rollout policy parameters\. For trajectorygig\_\{i\}, we denote its turn\-kkstate and response bySi,kS\_\{i,k\}andli,kl\_\{i,k\}, respectively:

Jflat​\(θ\)=𝔼q∼𝒟\{gi\}i=1G∼πθold\(⋅∣q\)​\[1G​∑i=1G1∑t′mi,t′​∑tmi,t​Li,tCLIP​\(θ\)\]J^\{\\text\{flat\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}q\\sim\\mathcal\{D\}\\\\ \\\{g\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid q\)\\end\{subarray\}\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\\sum\_\{t^\{\\prime\}\}m\_\{i,t^\{\\prime\}\}\}\\sum\_\{t\}m\_\{i,t\}L\_\{i,t\}^\{\\text\{CLIP\}\}\(\\theta\)\\right\]\(9\)whereiiindexes the sampled trajectory,ttindexes flattened token positions, andLi,tCLIP​\(θ\)=min⁡\(ρi,t​\(θ\)​Ai,clip⁡\(ρi,t​\(θ\),1−ϵ,1\+ϵ\)​Ai\)L\_\{i,t\}^\{\\text\{CLIP\}\}\(\\theta\)=\\min\\\!\\left\(\\rho\_\{i,t\}\(\\theta\)A\_\{i\},\\,\\operatorname\{clip\}\\\!\\left\(\\rho\_\{i,t\}\(\\theta\),\\,1\-\\epsilon,\\,1\+\\epsilon\\right\)A\_\{i\}\\right\)is the standard clipped surrogate objective with clipping thresholdϵ\\epsilon\.ai,ta\_\{i,t\}is the token generated at flattened positionttin theii\-th sampled trajectory,mi,tm\_\{i,t\}is a binary mask indicating whether that token contributes to the policy gradient, andρi,t​\(θ\)=πθ​\(ai,t∣xi,<t\)πθold​\(ai,t∣xi,<t\)\\rho\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{i,t\}\\mid x\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{i,t\}\\mid x\_\{i,<t\}\)\}is the IS ratio\. The contextxi,<tx\_\{i,<t\}is the flattened interaction context preceding tokenai,ta\_\{i,t\}, including previously generated response tokens and inserted environment feedback\. The scalarAiA\_\{i\}denotes the trajectory\-level group\-relative advantage, which is uniformly assigned to all unmasked tokens in trajectorygig\_\{i\}\.

### B\.1Single flattened\-trajectory policy optimization: mismatch from rollout to training

In multi\-turn interactions, the accumulated history can become too long for the model to process in full\. For example, an agent may search for information, call a tool, revise its plan based on the returned result, and repeat this process over many turns\. By later turns, the prompt, previous responses, and tool feedback may already span tens of thousands of tokens\. In practice, rollouts therefore often rely on a truncated or summarized context rather than the complete interaction history\[[52](https://arxiv.org/html/2608.18682#bib.bib32),[29](https://arxiv.org/html/2608.18682#bib.bib31),[56](https://arxiv.org/html/2608.18682#bib.bib30)\]\. Therefore, existing methods typically reconstruct the whole interaction as a single concatenated sequence, or flattened trajectory, during training\. This flattened training formulation evaluates each generated token under the concatenated prefix, rather than under the original context that was actually used during rollout\. Consequently, tokens generated at turnkkare optimized under a conditioning context that can differ from the rollout context that produced them\. We refer to this discrepancy as a rollout\-to\-training mismatch induced by flattened\-trajectory policy optimization\. We next analyze how this mismatch distorts the likelihood\-ratio estimation underlying clipped policy optimization in the PPO/GRPO family\.

#### Policy optimization mismatch across turns\.

Under flat training, the optimization mismatch across turns arises because the same generated token is conditioned on different contexts during rollout and training\. We now formalize this mismatch and show that it induces a biased IS ratio\. Letx¯k\\bar\{x\}\_\{k\}denote the full interaction history before turnkk, and letϕ:𝒳¯→𝒵\\phi:\\bar\{\\mathcal\{X\}\}\\rightarrow\\mathcal\{Z\}be an observation map that truncates or summarizes history beyond the model’s effective context length\. During rollout, token\-level actions are sampled conditioned onϕ⁡\(x¯k\)\\phi\(\\bar\{x\}\_\{k\}\), so the true sampling distribution isπθold​\(ak,t∣ϕ⁡\(x¯k\)\)\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\. In clipped policy optimization of the PPO/GRPO family, the denominator of the IS ratio must match this rollout distribution\. However, under flat training, the same token is re\-evaluated under the concatenated full\-history context, yieldingπθold​\(ak,t∣x¯k\)\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\bar\{x\}\_\{k\}\)\. This leads to the mismatch

ρk,tflat=πθ​\(ak,t∣x¯k\)πθold​\(ak,t∣ϕ⁡\(x¯k\)\)≠πθ​\(ak,t∣ϕ⁡\(x¯k\)\)πθold​\(ak,t∣ϕ⁡\(x¯k\)\)=ρk,ttrue\.\\rho\_\{k,t\}^\{\\mathrm\{flat\}\}=\\frac\{\\pi\_\{\\theta\}\(a\_\{k,t\}\\mid\\bar\{x\}\_\{k\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\}\\neq\\frac\{\\pi\_\{\\theta\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\}=\\rho\_\{k,t\}^\{\\mathrm\{true\}\}\.\(10\)The denominator used in flat training,πθold​\(ak,t∣x¯k\)\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\bar\{x\}\_\{k\}\), does not match with the true rollout sampling probability,πθold​\(ak,t∣ϕ⁡\(x¯k\)\)\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid\\phi\(\\bar\{x\}\_\{k\}\)\)\. Consequently,ρk,tflat\\rho\_\{k,t\}^\{\\mathrm\{flat\}\}is a biased estimate of the correct IS ratio\. Since clipped policy\-gradient updates in the PPO/GRPO family depend on the IS ratio, the bias propagates into the gradient estimate and can distort the update direction, leading to instability in multi\-turn agentic RL training\. This mismatch becomes more severe for tokens generated in later turns, since the amount of history omitted during rollout typically grows withkk, while flat training continues to re\-evaluate these tokens under the concatenated training prefix\. In the worst case, the resulting discrepancy in token probability can become extremely large\.

#### State aliasing and projected suboptimality\.

Beyond gradient bias, truncated contexts also introducestate aliasing, where distinct full\-history states are mapped to the same truncated representation\. Whenϕ\\phiis non\-injective, different interaction histories may collapse into an identical observationzk=ϕ⁡\(x¯k\)z\_\{k\}=\\phi\(\\bar\{x\}\_\{k\}\)\[[11](https://arxiv.org/html/2608.18682#bib.bib26)\]\. The induced process overzkz\_\{k\}may not preserve the Markov property of the original full\-history process\[[2](https://arxiv.org/html/2608.18682#bib.bib11)\]\. As a result, any policy conditioned only onzkz\_\{k\}is confined to the observation\-induced policy classΠ​ϕ\\Pi\\phi\. Even if optimized exactly within this restricted class, such a policy can achieve only the projected optimumVΠϕ∗V^\{\*\}\_\{\\Pi\_\{\\phi\}\}\[[1](https://arxiv.org/html/2608.18682#bib.bib42)\], which can be strictly lower than the true optimumV∗V^\{\*\}\. Therefore, projected suboptimality can arise, withVΠϕ∗<V∗V^\{\*\}\_\{\\Pi\_\{\\phi\}\}<V^\{\*\}\. Under observation inconsistency, flattened\-trajectory training may be limited to the projected optimumVΠϕ∗V^\{\*\}\_\{\\Pi\_\{\\phi\}\}, which is strictly suboptimal relative to the full\-history optimumV∗V^\{\*\}\.

### B\.2Trajectory\-only credit assignment: mismatch across low\- and high\-quality turns

In multi\-turn interactions, trajectories generated from the same query can reach substantially different states by turnkk\. As a result, a trajectory\-level return no longer provides a reliable credit signal for evaluating actions taken at that turn\. Under flattened\-trajectory training, credit is assigned only at the trajectory level and then shared across tokens within a single trajectory, which mismatches the turn\-level structure of the decision process\. Here, we analyze how a trajectory advantage entangles the contribution of the current turn with both upstream state effect and downstream stochasticity\.

#### A trajectory advantage entangles turn\-level credit with context effects\.

A single trajectory advantage is typically defined asAi=Ri−R¯A\_\{i\}=R\_\{i\}\-\\bar\{R\}, whereRiR\_\{i\}denotes the final return of trajectorygig\_\{i\}andR¯\\bar\{R\}is the average return over the sampled group\. This trajectory advantage is then assigned to all turns in the trajectory, without identifying each turn’s individual contributions\. To expose the turn\-level credit hidden in this trajectory return, we analyze its population counterpart,Ri−𝔼⁡\[R\]R\_\{i\}\-\\mathbb\{E\}\[R\], by introducing the conditional expectations𝔼\[R∣Si,k,li,k\]\\mathbb\{E\}\[R\\mid S\_\{i,k\},l\_\{i,k\}\]and𝔼⁡\[R∣Si,k\]\\mathbb\{E\}\[R\\mid S\_\{i,k\}\]:

Ri−𝔼\[R\]=Ri−Qπ​\(Si,k,li,k\)⏟ξdown:downstream stochasticity\+Aπ​\(Si,k,li,k\)⏟true turn\-​k​advantage\+Vπ​\(Si,k\)−μR⏟ξup:upstream state effect,R\_\{i\}\-\\mathbb\{E\}\[R\]=\\underbrace\{R\_\{i\}\-Q^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)\}\_\{\\xi\_\{\\text\{down\}\}:\\;\\text\{downstream stochasticity\}\}\+\\underbrace\{A^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)\}\_\{\\text\{true turn\-\}k\\text\{ advantage\}\}\+\\underbrace\{V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\}\_\{\\xi\_\{\\text\{up\}\}:\\;\\text\{upstream state effect\}\},\(11\)whereQπ\(Si,k,li,k\)=𝔼\[R∣Si,k,li,k\]Q^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)=\\mathbb\{E\}\[R\\mid S\_\{i,k\},l\_\{i,k\}\],Vπ​\(Si,k\)=𝔼⁡\[R∣Si,k\]V^\{\\pi\}\(S\_\{i,k\}\)=\\mathbb\{E\}\[R\\mid S\_\{i,k\}\], andμR=𝔼⁡\[R\]\\mu\_\{R\}=\\mathbb\{E\}\[R\]\. The true turn\-kkadvantageAπ​\(Si,k,li,k\)A^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)is therefore only one component of the trajectory advantage\. The termξdown\\xi\_\{\\text\{down\}\}captures downstream stochasticity after turnkk, whileξup\\xi\_\{\\text\{up\}\}captures variation induced by the upstream state reached before turnkk\. Wheneverξdown\+ξup\\xi\_\{\\text\{down\}\}\+\\xi\_\{\\text\{up\}\}dominatesAπ​\(Si,k,li,k\)A^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)in magnitude, the sign of the trajectory advantage can disagree with that of the true turn advantage, i\.e\.,sign⁡\(Ai\)≠sign⁡\(Aπ​\(Si,k,li,k\)\)\\mathrm\{sign\}\(A\_\{i\}\)\\neq\\mathrm\{sign\}\(A^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)\), thereby reversing the gradient direction for turnkk\. Therefore, a trajectory advantage is not a valid turn\-level credit signal for turnkk, because it entangles the context effect of the current turn with both upstream state effects and downstream stochasticity\. When these two terms dominate, the resulting policy update can assign incorrect credit to the current turn and may even reverse the intended policy gradient direction\.

#### Cross\-trajectory baseline with state bias\.

Since group\-based policy optimization methods generate multiple trajectories from the same input query by sampling several rollouts from the current or old policy, they typically compute advantages of the formAi=Ri−bA\_\{i\}=R\_\{i\}\-b, whereRiR\_\{i\}is the return of trajectorygig\_\{i\}andb=1G​∑j=1GRjb=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}R\_\{j\}is the group baseline\. To determine whether the compared trajectories provide a valid turn\-level credit signal, we examine whether they share the same turn\-level stateSi,kS\_\{i,k\}\. If they do, the baseline compares alternative outcomes from the same context and is therefore matched\. Otherwise, the baseline mixes returns from different states and no longer reflects the local effect of the current turn, leading to mismatched credit assignment\.

Formally, viewingAiA\_\{i\}as an estimator of the true turn\-kkadvantageAπ​\(Si,k,li,k\)A^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)gives

𝔼\[Ai∣Si,k,li,k\]=G−1G\(Qπ\(Si,k,li,k\)−𝔼⁡\[Rj∣Si,k\]⏟depends on grouping\),j≠i\.\\mathbb\{E\}\[A\_\{i\}\\mid S\_\{i,k\},l\_\{i,k\}\]=\\frac\{G\-1\}\{G\}\\Big\(Q^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)\-\\underbrace\{\\mathbb\{E\}\[R\_\{j\}\\mid S\_\{i,k\}\]\}\_\{\\text\{depends on grouping\}\}\\Big\),\\qquad j\\neq i\.\(12\)
In the case of same\-state grouping, where all trajectories in the group share the same state at turnkk, we have𝔼⁡\[Rj∣Si,k\]=Vπ​\(Si,k\)\\mathbb\{E\}\[R\_\{j\}\\mid S\_\{i,k\}\]=V^\{\\pi\}\(S\_\{i,k\}\)\. In this case, the group baseline is anchored to the correct local decision context, and the resulting estimator differs from the true turn\-level advantage only by the multiplicative factor\(G−1\)/G\(G\-1\)/G\. However, under cross\-state grouping, trajectories within the same group may already reach different states by turnkk\. Then the baseline is no longer tied to the local stateSi,kS\_\{i,k\}, but is effectively centered around the global mean value as return, i\.e\.,𝔼⁡\[Rj\]=μR\\mathbb\{E\}\[R\_\{j\}\]=\\mu\_\{R\}\. This introduces the additional bias:

Biascross=G−1G​\(Vπ​\(Si,k\)−μR\)\.\\mathrm\{Bias\}\_\{\\text\{cross\}\}=\\frac\{G\-1\}\{G\}\\big\(V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\\big\)\.\(13\)
This bias is determined entirely by the upstream trajectory and has no causal connection to the action taken at turnkk\. In multi\-turn interactions, trajectories often diverge into semantically distinct environment states by turnkk\. For example, one trajectory may issue a search query while another is executing code\. As a result, the variance ofVπ​\(Si,k\)V^\{\\pi\}\(S\_\{i,k\}\)across states can be large, making the cross\-state bias comparable to, or even larger than, the true turn advantageAπ​\(Si,k,li,k\)A^\{\\pi\}\(S\_\{i,k\},l\_\{i,k\}\)\. Therefore, a cross\-trajectory baseline cannot guarantee a valid local comparison signal for turn\-level credit assignment\.

### B\.3Long\-horizon policy drift: PPO clipping mismatch asynchronous turns

In long\-horizon multi\-turn training, policy updates can become asynchronous across turns, inducing turn policy drift\. In practice, PPO\-style methods use token clipping to constrain the IS ratio and limit policy drift during updates\. However, this clipping mechanism is designed for near\-on\-policy updates with synchronized rollout data and does not explicitly account for turn\-wise discrepancies when different parts of a trajectory are generated or optimized under different policy versions\. Over long horizons, such mismatches can accumulate, shifting the update away from the near\-on\-policy learning and toward an off\-policy setting\.

This occurs because, in asynchronous multi\-turn training, policy updates may be performed before all trajectories have completed their rollouts\. For instance, shorter trajectories may complete first and immediately contribute to an update, while longer trajectories are still being generated under an older policy\. Specifically, short trajectories may finish first and update the policy fromθ0\\theta\_\{0\}toθ1\\theta\_\{1\}, while long trajectories are still being rolled out underπθ0\\pi\_\{\\theta\_\{0\}\}\. By the time these longer trajectories are used for training, the current policyπθ1\\pi\_\{\\theta\_\{1\}\}no longer matches the policy that generated them\.

In principle, this mismatch can be corrected by the full trajectory IS ratio:

ωi=∏t=1Tiρi,t=∏t=1Tiπθ1​\(ai,t∣xi,<t\)πθ0​\(ai,t∣xi,<t\),\\omega\_\{i\}=\\prod\_\{t=1\}^\{T\_\{i\}\}\\rho\_\{i,t\}=\\prod\_\{t=1\}^\{T\_\{i\}\}\\frac\{\\pi\_\{\\theta\_\{1\}\}\(a\_\{i,t\}\\mid x\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{0\}\}\(a\_\{i,t\}\\mid x\_\{i,<t\}\)\},\(14\)whereρi,t\\rho\_\{i,t\}is the token IS ratio,xi,<tx\_\{i,<t\}is the flattened multi\-turn interaction context preceding tokenai,ta\_\{i,t\}, andTiT\_\{i\}is the total number of tokens in the flattened trajectorygig\_\{i\}\. Weighting the loss byωi\\omega\_\{i\}would yield an unbiased correction to the policy objective\. However, clipped policy optimization in the PPO/GRPO family does not employ an IS ratio for the full trajectory\. Instead, it clips each token ratioρi,t\\rho\_\{i,t\}independently:

ℒclip=∑t=1Timin⁡\(ρi,t​Ai,clip⁡\(ρi,t,1−ϵ,1\+ϵ\)​Ai\)\.\\mathcal\{L\}\_\{\\text\{clip\}\}=\\sum\_\{t=1\}^\{T\_\{i\}\}\\min\\\!\\big\(\\rho\_\{i,t\}\\,A\_\{i\},\\;\\operatorname\{clip\}\(\\rho\_\{i,t\},\\,1\{\-\}\\epsilon,\\,1\{\+\}\\epsilon\)\\,A\_\{i\}\\big\)\.\(15\)
Therefore, if one composes the clipped token\-level ratios into a trajectory\-level correction, it generally differs from the true full\-trajectory IS ratio:

∏t=1Ticlip⁡\(ρi,t,1−ϵ,1\+ϵ\)≠∏t=1Tiρi,t=ωi\.\\prod\_\{t=1\}^\{T\_\{i\}\}\\operatorname\{clip\}\(\\rho\_\{i,t\},\\,1\{\-\}\\epsilon,\\,1\{\+\}\\epsilon\)\\neq\\prod\_\{t=1\}^\{T\_\{i\}\}\\rho\_\{i,t\}=\\omega\_\{i\}\.\(16\)
Because PPO clipping is nonlinear, the product of clipped token\-level ratios does not equal the true trajectory\-level importance weight\. In particular, each clipped ratio lies in\[1−ϵ,1\+ϵ\]\[1\{\-\}\\epsilon,\\,1\{\+\}\\epsilon\], so their product is restricted to\[\(1−ϵ\)T,\(1\+ϵ\)T\]\[\(1\{\-\}\\epsilon\)^\{T\},\\,\(1\{\+\}\\epsilon\)^\{T\}\], whereas the trueω\\omegacan in principle take any value in\(0,∞\)\(0,\\infty\)\. Wheneverω\\omegafalls outside this interval, per\-token clipping necessarily yields a biased trajectory correction\. Even whenω\\omegalies within the interval, the product of individually clipped ratios will generally differ fromω\\omegaas soon as anyρt\\rho\_\{t\}is clipped, since clipping and multiplication do not commute\. This discrepancy compounds with trajectory lengthTT: as more tokens are clipped, the gap between∏tclip⁡\(ρt\)\\prod\_\{t\}\\mathrm\{clip\}\(\\rho\_\{t\}\)andω\\omegacan grow progressively larger\. Removing clipping and using the exactω\\omegais not a practical solution, because its variance grows exponentially withTT, making gradient estimates increasingly uninformative over the long horizons typical of multi\-turn agentic RL\. Hence, token\-level clipping cannot faithfully reproduce the trajectory\-level correction required for long\-horizon policy drift\.

## Appendix CMethod: RTPO Theoretical Proofs

### C\.1Notation, Formal Problem Setup, and Technical Assumptions

#### Notation convention\.

We create a multi\-turn agentic episode, which consists ofKKturn\-level interactions, written as\(q,l0,f0,…,lK−1,fK−1\)\(q,l\_\{0\},f\_\{0\},\\ldots,l\_\{K\-1\},f\_\{K\-1\}\), whereqqis the initial prompt,lkl\_\{k\}is the complete response generated by the agent at turnkk, andfkf\_\{k\}is the external tool or environment feedback returned after executinglkl\_\{k\}\. The turn index is denoted byk∈\{0,…,K−1\}k\\in\\\{0,\\ldots,K\-1\\\}, while the token index within a response is denoted bytt\. When multiple rollouts are sampled, we useiiorjjto index the rollout or sibling trajectory\. Thus,Si,kS\_\{i,k\}denotes the turn\-kkstate in rolloutii, andai,k,ta\_\{i,k,t\}denotes thett\-th token generated at turnkkin rolloutii\. When no rollout index is needed, we can writeSkS\_\{k\},lkl\_\{k\}, andak,ta\_\{k,t\}for a generic trajectory\. We distinguish between a turn\-level macro\-action and its token\-level realization\. The macro\-action at turnkkis denoted byuk≡lku\_\{k\}\\equiv l\_\{k\}, wherelk=\(ak,1,…,ak,Tk\)l\_\{k\}=\(a\_\{k,1\},\\ldots,a\_\{k,T\_\{k\}\}\)is the complete response andTkT\_\{k\}is the number of generated tokens in that response\. The notationuku\_\{k\}is used in the MDP and value\-function definitions, whilelkl\_\{k\}emphasizes that the macro\-action is implemented as a language\-model response\. In addition, the reward in a task is assigned at the turn or trajectory level\. Tokens inside a response are treated as the low\-level realization of the turn\-level macro\-action and do not receive separate intermediate rewards\. Thus, the low\-level token reward is set to zero\. We writerkr\_\{k\}for the immediate turn\-level reward at turnkk\. In sparse\-reward tasks, we typically haverk=0r\_\{k\}=0fork<K−1k<K\-1, and the final task reward is observed only after the terminal turn\.

#### Turn\-boundary MDP for rollout\-training match\.

AKK\-turn agentic episode is modelled at turn boundaries asℳ=⟨𝒳¯,𝒜H,PH,RH,γH⟩\\mathcal\{M\}=\\langle\\bar\{\\mathcal\{X\}\},\\mathcal\{A\}\_\{H\},P\_\{H\},R\_\{H\},\\gamma\_\{H\}\\rangle, where𝒳¯\\bar\{\\mathcal\{X\}\}is the augmented turn\-boundary state space,𝒜H\\mathcal\{A\}\_\{H\}is the high\-level action space of complete responses,PHP\_\{H\}is the transition kernel induced by executing a complete response and receiving tool feedback,RHR\_\{H\}is the turn\-level reward function, andγH\\gamma\_\{H\}is the turn\-level discount factor\. The augmented state at turnkkisx¯k=\(Sk,k\)∈𝒳¯\\bar\{x\}\_\{k\}=\(S\_\{k\},k\)\\in\\bar\{\\mathcal\{X\}\}, whereSk=\(q,l0,f0,…,lk−1,fk−1\)S\_\{k\}=\(q,l\_\{0\},f\_\{0\},\\ldots,l\_\{k\-1\},f\_\{k\-1\}\)is the interaction history available before the current turn\. Adding the turn indexkkinx¯k\\bar\{x\}\_\{k\}can make the process Markov over a finite horizon, since the remaining number of turns can affect both the available decisions and the continuation value\. At turnkk, the agent selects a macro\-actionuk≡lk∈𝒜H,k​\(Sk\)u\_\{k\}\\equiv l\_\{k\}\\in\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\), where𝒜H,k​\(Sk\)\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\)denotes the set of feasible complete responses at stateSkS\_\{k\}and turnkk\. The macro\-action is realized autoregressively aslk=\(ak,1,…,ak,Tk\)l\_\{k\}=\(a\_\{k,1\},\\ldots,a\_\{k,T\_\{k\}\}\)and consumesτk=Tk\\tau\_\{k\}=T\_\{k\}token\-generation steps\. After executinguku\_\{k\}and receiving environment feedbackfkf\_\{k\}, the turn\-level history is updated by concatenation asSk\+1=Sk∘\(lk,fk\)S\_\{k\+1\}=S\_\{k\}\\circ\(l\_\{k\},f\_\{k\}\)\. Equivalently, sinceuk≡lku\_\{k\}\\equiv l\_\{k\}, one may writeSk\+1=Sk∘\(uk,fk\)S\_\{k\+1\}=S\_\{k\}\\circ\(u\_\{k\},f\_\{k\}\)\. In Sec[3](https://arxiv.org/html/2608.18682#S3), we adopt this MDP terminology for the turn\-boundary process\. Strictly speaking, because each macro\-action may span a variable number of token\-level stepsτk\\tau\_\{k\}, this process can alternatively be formulated as a finite\-horizon semi\-MDP\. This distinction does not affect our analysis, since policy optimization and credit assignment are defined exclusively over turn\-boundary states and macro\-actions\.

To maintain rollout–training consistency under conditional contexts, the policy need not condition on the full interaction historySkS\_\{k\}\. Instead, it may condition on a compressed or truncated context, provided that the same conditional context is used consistently during both rollout generation and policy optimization\. Instead, the actual conditioning context at turnkkisck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\), whereψ\\psimay be the identity map, a truncation operator, or a summarization operator\. This distinction is important in long\-context multi\-turn training: rollout may be performed under a truncated or summarized context, while training may otherwise recompute log\-probabilities under a different context\. RTPO avoids this mismatch by recording the exact contextckc\_\{k\}used during rollout and reusing the sameckc\_\{k\}during training\. The turn\-level policy factorizes asπθ=\(πθ,0,…,πθ,K−1\)\\pi\_\{\\theta\}=\(\\pi\_\{\\theta,0\},\\ldots,\\pi\_\{\\theta,K\-1\}\), where each sub\-policy maps the turn\-level contextckc\_\{k\}to a complete response\. Each sub\-policy is implemented autoregressively:

πθ,k​\(uk∣ck\)=∏t=1Tkπθ​\(ak,t∣ck,ak,<t\)\.\\pi\_\{\\theta,k\}\(u\_\{k\}\\mid c\_\{k\}\)=\\prod\_\{t=1\}^\{T\_\{k\}\}\\pi\_\{\\theta\}\(a\_\{k,t\}\\mid c\_\{k\},a\_\{k,<t\}\)\.\(17\)Here,ak,<t=\(ak,1,…,ak,t−1\)a\_\{k,<t\}=\(a\_\{k,1\},\\ldots,a\_\{k,t\-1\}\)is the token prefix generated within the current turn\. Because the sameckc\_\{k\}is used in rollout and training, the token\-level importance\-sampling \(IS\) ratio for turnkkis evaluated under matched conditioning contexts:

ρk,t=πθ​\(ak,t∣ck,ak,<t\)πθold​\(ak,t∣ck,ak,<t\)\.\\rho\_\{k,t\}=\\frac\{\\pi\_\{\\theta\}\(a\_\{k,t\}\\mid c\_\{k\},a\_\{k,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{k,t\}\\mid c\_\{k\},a\_\{k,<t\}\)\}\.\(18\)

#### Policy optimization with recursive optimality\.

A policy sequenceπrec=\(π0∗,…,πK−1∗\)\\pi^\{\\mathrm\{rec\}\}=\(\\pi\_\{0\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}\)is recursively optimal if, for every turnk∈\{0,…,K−1\}k\\in\\\{0,\\ldots,K\-1\\\}, the turn\-kksub\-policyπk∗\\pi\_\{k\}^\{\*\}maximizes the turn\-level augmented value given that all downstream sub\-policies have already been optimized and fixed\. Formally, for eachkk,πk∗∈arg⁡maxπk​Q~kπ∗​\(Sk,uk\)\\pi\_\{k\}^\{\*\}\\in\\arg\\max\_\{\\pi\_\{k\}\}\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\_\{k\}\), where the downstream policiesπk\+1∗,…,πK−1∗\\pi\_\{k\+1\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}are treated as fixed during the optimization ofπk∗\\pi\_\{k\}^\{\*\}\. This is a backward\-induction notion of optimality: the last turn is optimized first, then the preceding turn is optimized assuming the last\-turn policy is fixed, and so on until the first turn\.

Furthermore, we establish convergence to recursive optimality in a tabular finite\-horizon setting\. Assumptions below \(A1–A10\) are not intended to model the full neural implementation, but instead serve to isolate the theoretical effect of reverse\-order turn\-level policy optimization under standard stochastic approximation conditions\.

1. \(A1\)Finite turn\-boundary spaces\.The augmented state space𝒳¯\\bar\{\\mathcal\{X\}\}is finite, and for every turnkkand every reachable stateSkS\_\{k\}, the feasible high\-level action set𝒜H,k​\(Sk\)\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\)is finite\.
2. \(A2\)Proper finite\-horizon episodes\.The number of turnsKKis finite\. For every policy and every turnkk, the macro\-action duration satisfies𝔼πk​\[τk\]<∞\\mathbb\{E\}\_\{\\pi\_\{k\}\}\[\\tau\_\{k\}\]<\\infty\. Thus, each turn terminates almost surely in a finite expected token length\.
3. \(A3\)Discounting or finite\-horizon boundedness\.EitherγH∈\(0,1\)\\gamma\_\{H\}\\in\(0,1\), or the problem is finite\-horizon withK<∞K<\\inftyandγH∈\(0,1\]\\gamma\_\{H\}\\in\(0,1\]\. The latter case includes the undiscounted finite\-horizon settingγH=1\\gamma\_\{H\}=1\.
4. \(A4\)Tabular value representation\.The value estimateQk​\(S,u\)Q\_\{k\}\(S,u\)is stored separately for each turn\-state\-action tuple\(S,u,k\)\(S,u,k\)\. This assumption avoids approximation error and allows the proof to focus on the stochastic approximation dynamics induced by reverse\-order updates\.
5. \(A5\)Reverse\-order training with downstream freezing\.Training is proceeding in the orderk=K−1,K−2,…,0k=K\-1,K\-2,\\ldots,0\. During turnkk, the turn\-kkpolicy is updated while the downstream policiesπk\+1:K−1\\pi\_\{k\+1:K\-1\}are held fixed\. After turnkkis completed, the turn\-kkpolicy is also frozen before moving to turnk−1k\-1\.
6. \(A6\)GLIE exploration within each turn\.Within turnkk, the exploration schedule is greedy in the limit with infinite exploration \(GLIE\): every feasible turn\-level action is selected with strictly positive probability infinitely often, while the policy becomes greedy in the limit\. This ensures that all relevant action values at turnkkare sufficiently sampled before the policy is frozen\.
7. \(A7\)State coverage\.Every reachable boundary stateSkS\_\{k\}that can arise under the training process is visited infinitely often during turnkk\. This condition ensures that the tabular value estimate for each relevant state\-action pair receives infinitely many updates\.
8. \(A8\)Robbins–Monro step sizes\.For every tuple\(S,u,k\)\(S,u,k\), the learning rates satisfy∑n=1∞αn​\(S,u,k\)=∞\\sum\_\{n=1\}^\{\\infty\}\\alpha\_\{n\}\(S,u,k\)=\\inftyand∑n=1∞αn2​\(S,u,k\)<∞\\sum\_\{n=1\}^\{\\infty\}\\alpha\_\{n\}^\{2\}\(S,u,k\)<\\infty\. These are the standard stochastic approximation step\-size conditions\.
9. \(A9\)Bounded rewards\.The turn\-level rewards are uniformly bounded:\|rk\|≤rmax\|r\_\{k\}\|\\leq r\_\{\\max\}almost surely for allkk\.
10. \(A10\)Bounded iterates\.The tabular value iterates remain uniformly bounded:\|Qk​\(S,u\)\|≤Qmax\|Q\_\{k\}\(S,u\)\|\\leq Q\_\{\\max\}throughout learning\. This assumption is standard in stochastic approximation analyses and can be enforced by projection if necessary\.
11. \(A11\)Decision sufficiency of the conditioning context\.The observation mappingψ:Sk↦ck\\psi\\colon S\_\{k\}\\mapsto c\_\{k\}preserves all decision\-relevant information for following turns\. In other words, states that are indistinguishable underψ\\psishare the same optimal action\-value function, so the optimal policy at turnkkdepends onSkS\_\{k\}only throughckc\_\{k\}\. This implies thatQ~kπ∗\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}can be written as a function of\(ck,u\)\(c\_\{k\},u\)without loss, and the completeness conditionΠglobal⊆\{\(π0,…,πK−1\):πk\(⋅∣ck\)\}\\Pi\_\{\\mathrm\{global\}\}\\subseteq\\\{\(\\pi\_\{0\},\\ldots,\\pi\_\{K\-1\}\):\\pi\_\{k\}\(\\cdot\\mid c\_\{k\}\)\\\}in Theorem[1](https://arxiv.org/html/2608.18682#Thmtheorem1)\(c\) is automatically satisfied\.

Assumptions \(A1\)–\(A10\) are standard regularity conditions for the convergence of reinforcement learning algorithms; identical or closely analogous conditions appear in the foundational convergence proofs of Q\-learning\[[54](https://arxiv.org/html/2608.18682#bib.bib21),[51](https://arxiv.org/html/2608.18682#bib.bib22),[21](https://arxiv.org/html/2608.18682#bib.bib25)\], on\-policy GLIE control\[[47](https://arxiv.org/html/2608.18682#bib.bib23)\], and the systematic treatment in[3](https://arxiv.org/html/2608.18682#bib.bib24)\. They are not specific to RTPO but rather constitute the minimal set of conditions under which any stochastic\-approximation\-based value\-learning algorithm is known to converge\. These assumptions \(A1–A11\) are intentionally stronger than those required in the practical neural network implementation\. They are used to make the convergence argument mathematically clean in the tabular setting\. In the realistic RTPO implementation, the policy is represented by a shared neural language model rather than by independent tabular sub\-policies\. Therefore, “freezing” a turn\-level sub\-policy should be interpreted operationally: after a turn is completed, subsequent turns mask out the corresponding turn tokens from the loss, so that those turn\-level decisions no longer receive gradients\. The tabular analysis should thus be read as an idealized counterpart that clarifies the role of reverse\-turn optimization and downstream\-policy freezing, rather than as a claim of global convergence for arbitrary neural function approximation\. The extension to neural function approximation is discussed separately in Proposition[C\.2](https://arxiv.org/html/2608.18682#A3.SS2.SSS0.Px5)\. We do not rely on the universal approximation property alone to claim convergence of the neural algorithm; instead, the tabular result serves as a principled limiting case that motivates the reverse\-turn training design\.

#### Verification of decision sufficiency \(A11\) in RTPO\.

In the practical RTPO implementation,ψ\\psicorresponds to the chat\-template truncation operator that removes the model’s internal reasoning trace \(the content of<think\>blocks\) while preserving all externally observable elements: tool calls, tool results, and final answers\. Two statesSkS\_\{k\}andSk′S\_\{k\}^\{\\prime\}that differ only in their internal reasoning traces satisfyψ⁡\(Sk\)=ψ⁡\(Sk′\)\\psi\(S\_\{k\}\)=\\psi\(S\_\{k\}^\{\\prime\}\)\. Since the environment transition kernelPHP\_\{H\}depends exclusively on the executed tool calls and the returned feedback, not on the model’s internal reasoning, the next\-state distribution and thus the continuation valueFkπ∗​\(Sk\+1\)F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)are identical forSkS\_\{k\}andSk′S\_\{k\}^\{\\prime\}\. The turn\-level rewardrkr\_\{k\}likewise depends only on the externally observable action\. Therefore,Q~kπ∗​\(Sk,u\)=Q~kπ∗​\(Sk′,u\)\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\)=\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\}^\{\\prime\},u\)for alluu, and Assumption \(A11\) is satisfied\. Whenψ\\psiis the identity map \(full\-history conditioning\), \(A11\) holds trivially\.

#### Turn\-level value estimation\.

We formulate the interaction as a hierarchical MDP, which decomposes the value of each turn into the local effect of the current macro\-action and the downstream completion value, following a MAXQ\-style value decomposition\. The downstream continuation value after executing turnkkis defined as

Fkπ​\(Sk\+1\):=𝔼π​\[∑j=k\+1K−1γH∑m=k\+1j−1τm​rj\|Sk\+1\],FK−1π≡0\.F\_\{k\}^\{\\pi\}\(S\_\{k\+1\}\):=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{j=k\+1\}^\{K\-1\}\\gamma\_\{H\}^\{\\sum\_\{m=k\+1\}^\{j\-1\}\\tau\_\{m\}\}r\_\{j\}\\;\\middle\|\\;S\_\{k\+1\}\\right\],\\qquad F\_\{K\-1\}^\{\\pi\}\\equiv 0\.\(19\)The base caseFK−1π≡0F\_\{K\-1\}^\{\\pi\}\\equiv 0reflects that there are no downstream turns after the last turn\. The exponent∑m=k\+1j−1τm\\sum\_\{m=k\+1\}^\{j\-1\}\\tau\_\{m\}accounts for the number of token\-generation steps elapsed between the next stateSk\+1S\_\{k\+1\}and the future rewardrjr\_\{j\}\. WhenγH=1\\gamma\_\{H\}=1, this reduces to the undiscounted finite\-horizon setting used in many sparse\-reward agentic RL tasks\. The boundary\-level augmented action\-value function is:

Q~kπ\(Sk,uk\):=𝔼\[rk\+γHτkFkπ\(Sk\+1\)\|Sk,uk\]\.\\tilde\{Q\}\_\{k\}^\{\\pi\}\(S\_\{k\},u\_\{k\}\):=\\mathbb\{E\}\\\!\\left\[r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\\pi\}\(S\_\{k\+1\}\)\\;\\middle\|\\;S\_\{k\},u\_\{k\}\\right\]\.\(20\)This value measures the expected return obtained by choosing the complete turn\-kkresponseuku\_\{k\}at stateSkS\_\{k\}, followed by downstream policy execution fromSk\+1S\_\{k\+1\}onward\. In sparse\-reward settings, the immediate termrkr\_\{k\}is often zero for nonterminal turns, so the quality of a turn\-level action is primarily reflected through the downstream continuation value\. The corresponding optimal value satisfiesQ~k∗\(Sk,uk\)=𝔼\[rk\+γHτkFk∗\(Sk\+1\)∣Sk,uk\]\\tilde\{Q\}\_\{k\}^\{\*\}\(S\_\{k\},u\_\{k\}\)=\\mathbb\{E\}\[r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\*\}\(S\_\{k\+1\}\)\\mid S\_\{k\},u\_\{k\}\], whereFk∗F\_\{k\}^\{\*\}is obtained by the usual backward Bellman optimality recursion over turn boundaries\.

### C\.2Proof of Theorem 1: Convergence to Recursive Optimality

We formalize Theorem 1 from Sec[3\.1](https://arxiv.org/html/2608.18682#S3.SS1)before presenting the proof\.

Reverse\-Order Training to Recursive Optimality###### Theorem 1\(Convergence to Recursive Optimality\)\. Under Assumptions \(A1\)–\(A10\), including finite state and macro\-action spaces, Robbins–Monro step sizes, sufficient exploration at each turn, exact freezing of optimized downstream policies, and bounded rewards, the reverse\-order backward induction of turn\-level policy optimization satisfies the following properties:\(a\)Per\-turn convergence under fixed downstream policies\.For each turnkk, once the downstream policiesπk\+1:K−1∗\\pi\_\{k\+1:K\-1\}^\{\*\}are fixed, the downstream continuation value becomes a stationary function:Fkπ∗\(Sk\+1\)=𝔼π∗k\+1:K−1\[∑j=k\+1K−1γH∑m=k\+1j−1τmrj\|Sk\+1\]\.F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)=\\mathbb\{E\}\_\{\\pi^\{\*\}\_\{k\+1:K\-1\}\}\\\!\\left\[\\sum\_\{j=k\+1\}^\{K\-1\}\\gamma\_\{H\}^\{\\sum\_\{m=k\+1\}^\{j\-1\}\\tau\_\{m\}\}r\_\{j\}\\;\\middle\|\\;S\_\{k\+1\}\\right\]\.\(21\)Therefore, the turn\-kktargetYk:=rk\+γHτk​Fkπ∗​\(Sk\+1\)Y\_\{k\}:=r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)\(22\)is stationary conditional on\(Sk,uk\)\(S\_\{k\},u\_\{k\}\), with conditional mean𝔼\[Yk∣Sk,uk\]=Q~kπ∗\(Sk,uk\)\.\\mathbb\{E\}\[Y\_\{k\}\\mid S\_\{k\},u\_\{k\}\]=\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\_\{k\}\)\.\(23\)The corresponding tabular updateQk\(n\+1\)​\(S,u\)←\(1−αn\)​Qk\(n\)​\(S,u\)\+αn​YkQ\_\{k\}^\{\(n\+1\)\}\(S,u\)\\leftarrow\(1\-\\alpha\_\{n\}\)Q\_\{k\}^\{\(n\)\}\(S,u\)\+\\alpha\_\{n\}Y\_\{k\}\(24\)is a single\-step stochastic approximation problem and converges toQ~kπ∗\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}under the stated assumptions\. Hence, the learned per\-turn policy satisfiesπ^k→a\.s\.πk∗\\hat\{\\pi\}\_\{k\}\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\pi\_\{k\}^\{\*\}\.\(b\)Recursive optimality under reverse\-order training\.Applying the per\-turn convergence argument in reverse order,k=K−1,K−2,…,0,k=K\{\-\}1,K\{\-\}2,\\ldots,0,\(25\)yields a sequence of optimized and frozen downstream policies\. At each turnkk, the per\-turn optimizer converges toπk∗​\(Sk\)∈arg⁡maxuk∈𝒜H,k​\(Sk\)​Q~kπ∗​\(Sk,uk\),\\pi\_\{k\}^\{\*\}\(S\_\{k\}\)\\in\\arg\\max\_\{u\_\{k\}\\in\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\)\}\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\_\{k\}\),\(26\)given the fixed downstream policiesπk\+1∗,…,πK−1∗\\pi\_\{k\+1\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}\. The resulting policy sequenceπrec=\(π0∗,…,πK−1∗\)\\pi^\{\\mathrm\{rec\}\}=\(\\pi\_\{0\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}\)\(27\)is recursively optimal\.\(c\)Global optimality under context\-sufficient macro\-action completeness\.If the turn\-level macro\-action spaces are complete with respect to the conditioning contextsck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\), i\.e\., every globally feasible trajectory\-level policy can be represented by a sequence of turn\-level policies acting on these contexts,Πglobal⊆\{\(π0,…,πK−1\):πk\(⋅∣ck\),uk∈𝒜H,k\(Sk\)\},\\Pi\_\{\\mathrm\{global\}\}\\subseteq\\left\\\{\(\\pi\_\{0\},\\ldots,\\pi\_\{K\-1\}\):\\pi\_\{k\}\(\\cdot\\mid c\_\{k\}\),\\;u\_\{k\}\\in\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\)\\right\\\},\(28\)Then, recursive optimality is equivalent to global optimality over the full trajectory:Vπrec​\(x¯0\)=V∗​\(x¯0\)\.V^\{\\pi^\{\\mathrm\{rec\}\}\}\(\\bar\{x\}\_\{0\}\)=V^\{\*\}\(\\bar\{x\}\_\{0\}\)\.\(29\)

###### Proof\.

We prove parts \(a\)–\(c\) of Theorem 1 as follows\.

#### Proof of Theorem 1\(a\): per\-turn convergence under fixed downstream policies\.

Fix a turnkkand suppose the downstream sub\-policiesπk\+1∗,…,πK−1∗\\pi\_\{k\+1\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}are already optimized and frozen\. By the definition in Theorem[1](https://arxiv.org/html/2608.18682#Thmtheorem1), the downstream continuation value is

Fkπ∗\(Sk\+1\)=𝔼π∗k\+1:K−1\[∑j=k\+1K−1γH∑m=k\+1j−1τmrj\|Sk\+1\]\.F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)=\\mathbb\{E\}\_\{\\pi^\{\*\}\_\{k\+1:K\-1\}\}\\\!\\left\[\\sum\_\{j=k\+1\}^\{K\-1\}\\gamma\_\{H\}^\{\\sum\_\{m=k\+1\}^\{j\-1\}\\tau\_\{m\}\}r\_\{j\}\\;\\middle\|\\;S\_\{k\+1\}\\right\]\.\(30\)Because the downstream policiesπk\+1:K−1∗\\pi\_\{k\+1:K\-1\}^\{\*\}are frozen,Fkπ∗​\(Sk\+1\)F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)is a fixed function ofSk\+1S\_\{k\+1\}throughout turnkk\. Define the augmented one\-step target

Yk=rk\+γHτk​Fkπ∗​\(Sk\+1\)\.Y\_\{k\}=r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)\.\(31\)Conditional on\(Sk,uk\)\(S\_\{k\},u\_\{k\}\), the distribution ofrkr\_\{k\},τk\\tau\_\{k\}, andSk\+1S\_\{k\+1\}is induced by the time\-invariant turn\-boundary transition kernelPHP\_\{H\}\. SinceFkπ∗F\_\{k\}^\{\\pi^\{\*\}\}is fixed throughout turnkk, the conditional distribution ofYkY\_\{k\}given\(Sk,uk\)\(S\_\{k\},u\_\{k\}\)is stationary\.

The conditional mean of the target is exactly the augmented turn\-level action value:

𝔼\[Yk∣Sk,uk\]=𝔼\[rk\+γHτkFkπ∗\(Sk\+1\)\|Sk,uk\]=Q~kπ∗\(Sk,uk\)\.\\mathbb\{E\}\[Y\_\{k\}\\mid S\_\{k\},u\_\{k\}\]=\\mathbb\{E\}\\\!\\left\[r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)\\;\\middle\|\\;S\_\{k\},u\_\{k\}\\right\]=\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\_\{k\}\)\.\(32\)The target also has a bounded second moment\. Since rewards are bounded,\|rj\|≤rmax\|r\_\{j\}\|\\leq r\_\{\\max\}almost surely\. Hence, for any policyπ\\pi,

\|Fkπ​\(Sk\+1\)\|\\displaystyle\\bigl\|F\_\{k\}^\{\\pi\}\(S\_\{k\+1\}\)\\bigr\|=\|𝔼π​\[∑j=k\+1K−1γH∑m=k\+1j−1τm​rj\|Sk\+1\]\|\\displaystyle=\\left\|\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{j=k\+1\}^\{K\-1\}\\gamma\_\{H\}^\{\\sum\_\{m=k\+1\}^\{j\-1\}\\tau\_\{m\}\}r\_\{j\}\\;\\middle\|\\;S\_\{k\+1\}\\right\]\\right\|≤𝔼π​\[∑j=k\+1K−1γH∑m=k\+1j−1τm​\|rj\|\|Sk\+1\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{j=k\+1\}^\{K\-1\}\\gamma\_\{H\}^\{\\sum\_\{m=k\+1\}^\{j\-1\}\\tau\_\{m\}\}\|r\_\{j\}\|\\;\\middle\|\\;S\_\{k\+1\}\\right\]≤\(K−k−1\)​rmax≤K​rmax,\\displaystyle\\leq\(K\-k\-1\)r\_\{\\max\}\\leq Kr\_\{\\max\},\(33\)where we usedγH∈\(0,1\]\\gamma\_\{H\}\\in\(0,1\]and the finite horizonKK\. Therefore,

\|Yk\|≤\|rk​\|\+γHτk\|​Fkπ∗​\(Sk\+1\)\|≤\(K\+1\)​rmax,\|Y\_\{k\}\|\\leq\|r\_\{k\}\|\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}\|F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)\|\\leq\(K\+1\)r\_\{\\max\},\(34\)soYkY\_\{k\}has uniformly bounded conditional second moment\.

During turnkk, the tabular update is

Qk\(n\+1\)​\(S,u\)←\(1−αn\)​Qk\(n\)​\(S,u\)\+αn​Yk\.Q\_\{k\}^\{\(n\+1\)\}\(S,u\)\\leftarrow\(1\-\\alpha\_\{n\}\)Q\_\{k\}^\{\(n\)\}\(S,u\)\+\\alpha\_\{n\}Y\_\{k\}\.\(35\)For each fixed\(S,u,k\)\(S,u,k\), this is a Robbins–Monro stochastic approximation to the stationary conditional mean𝔼\[Yk∣S,u\]=Q~kπ∗\(S,u\)\\mathbb\{E\}\[Y\_\{k\}\\mid S,u\]=\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S,u\)\. The finite state and macro\-action spaces imply that there are finitely many entries to estimate\. Sufficient exploration ensures that every relevant\(S,u\)\(S,u\)pair is visited infinitely often during turnkk\. The step sizes satisfy

∑n=1∞αn​\(S,u,k\)=∞,∑n=1∞αn2​\(S,u,k\)<∞\.\\sum\_\{n=1\}^\{\\infty\}\\alpha\_\{n\}\(S,u,k\)=\\infty,\\qquad\\sum\_\{n=1\}^\{\\infty\}\\alpha\_\{n\}^\{2\}\(S,u,k\)<\\infty\.\(36\)Together with bounded targets and bounded iterates, standard stochastic approximation gives

Qk\(n\)​\(S,u\)→a\.s\.Q~kπ∗​\(S,u\)for every reachable​\(S,u\)\.Q\_\{k\}^\{\(n\)\}\(S,u\)\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S,u\)\\qquad\\text\{for every reachable \}\(S,u\)\.\(37\)As exploration vanishes, the induced greedy policy converges to an optimal greedy selector:

π^k​\(S\)∈arg⁡maxu​Qk\(n\)​\(S,u\)⟹π^k​\(S\)→a\.s\.πk∗​\(S\)∈arg⁡maxu​Q~kπ∗​\(S,u\)\.\\hat\{\\pi\}\_\{k\}\(S\)\\in\\arg\\max\_\{u\}Q\_\{k\}^\{\(n\)\}\(S,u\)\\quad\\Longrightarrow\\quad\\hat\{\\pi\}\_\{k\}\(S\)\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\pi\_\{k\}^\{\*\}\(S\)\\in\\arg\\max\_\{u\}\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S,u\)\.\(38\)This completes the proof of Theorem 1\(a\)\.

#### Proof of Theorem 1\(b\): reverse\-order recursion gives recursive optimality\.

We now apply Theorem 1\(a\) in reverse order here\. At the final turnk=K−1k=K\-1, there are no downstream turns, so

FK−1π​\(SK\)≡0\.F\_\{K\-1\}^\{\\pi\}\(S\_\{K\}\)\\equiv 0\.\(39\)The augmented target reduces to

YK−1=rK−1\.Y\_\{K\-1\}=r\_\{K\-1\}\.\(40\)By claim Theorem 1\(a\), the final\-turn policy converges:

π^K−1→a\.s\.πK−1∗\.\\hat\{\\pi\}\_\{K\-1\}\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\pi\_\{K\-1\}^\{\*\}\.\(41\)
Now assume that for somek∈\{0,…,K−2\}k\\in\\\{0,\\ldots,K\-2\\\}, the downstream policiesπ^k\+1,…,π^K−1\\hat\{\\pi\}\_\{k\+1\},\\ldots,\\hat\{\\pi\}\_\{K\-1\}have already converged almost surely toπk\+1∗,…,πK−1∗\\pi\_\{k\+1\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}and have been frozen before turnkkbegins\. Then the continuation value

Fkπ∗\(Sk\+1\)=𝔼π∗k\+1:K−1\[∑j=k\+1K−1γH∑m=k\+1j−1τmrj\|Sk\+1\]F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)=\\mathbb\{E\}\_\{\\pi^\{\*\}\_\{k\+1:K\-1\}\}\\\!\\left\[\\sum\_\{j=k\+1\}^\{K\-1\}\\gamma\_\{H\}^\{\\sum\_\{m=k\+1\}^\{j\-1\}\\tau\_\{m\}\}r\_\{j\}\\;\\middle\|\\;S\_\{k\+1\}\\right\]\(42\)is fixed throughout turnkk\. Therefore, claim \(a\) applies to turnkk, giving

π^k→a\.s\.πk∗,πk∗​\(Sk\)∈arg⁡maxuk​Q~kπ∗​\(Sk,uk\)\.\\hat\{\\pi\}\_\{k\}\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\pi\_\{k\}^\{\*\},\\qquad\\pi\_\{k\}^\{\*\}\(S\_\{k\}\)\\in\\arg\\max\_\{u\_\{k\}\}\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\_\{k\}\)\.\(43\)By backward induction fromK−1K\{\-\}1to00, every turn\-level policy converges to its optimal selector given the optimized and frozen downstream policies\. Therefore, the resulting sequence

πrec=\(π0∗,…,πK−1∗\)\\pi^\{\\mathrm\{rec\}\}=\(\\pi\_\{0\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}\)\(44\)is recursively optimal\.

Till here, we have completed the proof of Theorem 1\(b\)\.

#### Proof of Theorem 1\(c\): context\-sufficient macro\-action completeness implies global optimality\.

Assume the turn\-level macro\-action spaces are complete with respect to the conditioning contextsck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\), meaning that every globally feasible trajectory\-level policy can be represented as a sequence of turn\-level sub\-policies\(π0,…,πK−1\)\(\\pi\_\{0\},\\ldots,\\pi\_\{K\-1\}\), with eachπk\\pi\_\{k\}acting onckc\_\{k\}and selecting a feasible macro\-actionuk∈𝒜H,k​\(Sk\)u\_\{k\}\\in\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\)\.

We prove by backward induction that

Vkπrec​\(Sk\)=Vk∗​\(Sk\)for every reachable​Sk\.V\_\{k\}^\{\\pi^\{\\mathrm\{rec\}\}\}\(S\_\{k\}\)=V\_\{k\}^\{\*\}\(S\_\{k\}\)\\qquad\\text\{for every reachable \}S\_\{k\}\.\(45\)At the last turn,FK−1π≡0F\_\{K\-1\}^\{\\pi\}\\equiv 0, so

Q~K−1π\(SK−1,u\)=𝔼\[rK−1∣SK−1,u\]\.\\tilde\{Q\}\_\{K\-1\}^\{\\pi\}\(S\_\{K\-1\},u\)=\\mathbb\{E\}\[r\_\{K\-1\}\\mid S\_\{K\-1\},u\]\.\(46\)SinceπK−1∗\\pi\_\{K\-1\}^\{\*\}maximizes this quantity,

VK−1πrec\(SK−1\)=maxu∈𝒜H,K−1​\(SK−1\)𝔼\[rK−1∣SK−1,u\]=VK−1∗\(SK−1\)\.V\_\{K\-1\}^\{\\pi^\{\\mathrm\{rec\}\}\}\(S\_\{K\-1\}\)=\\max\_\{u\\in\\mathcal\{A\}\_\{H,K\-1\}\(S\_\{K\-1\}\)\}\\mathbb\{E\}\[r\_\{K\-1\}\\mid S\_\{K\-1\},u\]=V\_\{K\-1\}^\{\*\}\(S\_\{K\-1\}\)\.\(47\)
Assume now thatVk\+1πrec​\(Sk\+1\)=Vk\+1∗​\(Sk\+1\)V\_\{k\+1\}^\{\\pi^\{\\mathrm\{rec\}\}\}\(S\_\{k\+1\}\)=V\_\{k\+1\}^\{\*\}\(S\_\{k\+1\}\)for every reachableSk\+1S\_\{k\+1\}\. Then

Fkπrec​\(Sk\+1\)=Fk∗​\(Sk\+1\)\.F\_\{k\}^\{\\pi^\{\\mathrm\{rec\}\}\}\(S\_\{k\+1\}\)=F\_\{k\}^\{\*\}\(S\_\{k\+1\}\)\.\(48\)By Assumption \(A11\), the optimal action\-valueQ~kπ∗​\(Sk,u\)\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\)depends onSkS\_\{k\}only throughck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\)\. Therefore, the greedy policyπk∗∈arg⁡maxu​Q~kπ∗​\(Sk,u\)\\pi\_\{k\}^\{\*\}\\in\\arg\\max\_\{u\}\\tilde\{Q\}\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\},u\)is well\-defined as a function ofckc\_\{k\}alone, and we writeπk∗​\(ck\)\\pi\_\{k\}^\{\*\}\(c\_\{k\}\)without ambiguity\. Applying this to the inductive step:

Vkπrec​\(Sk\)\\displaystyle V\_\{k\}^\{\\pi^\{\\mathrm\{rec\}\}\}\(S\_\{k\}\)=Q~kπrec​\(Sk,πk∗​\(ck\)\)\\displaystyle=\\tilde\{Q\}\_\{k\}^\{\\pi^\{\\mathrm\{rec\}\}\}\\bigl\(S\_\{k\},\\pi\_\{k\}^\{\*\}\(c\_\{k\}\)\\bigr\)=𝔼\[rk\+γHτkFkπrec\(Sk\+1\)\|Sk,πk∗\(Sk\)\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\\pi^\{\\mathrm\{rec\}\}\}\(S\_\{k\+1\}\)\\;\\middle\|\\;S\_\{k\},\\pi\_\{k\}^\{\*\}\(S\_\{k\}\)\\right\]=𝔼\[rk\+γHτkFk∗\(Sk\+1\)\|Sk,πk∗\(Sk\)\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\*\}\(S\_\{k\+1\}\)\\;\\middle\|\\;S\_\{k\},\\pi\_\{k\}^\{\*\}\(S\_\{k\}\)\\right\]=maxu∈𝒜H,k​\(Sk\)𝔼\[rk\+γHτkFk∗\(Sk\+1\)\|Sk,u\]\\displaystyle=\\max\_\{u\\in\\mathcal\{A\}\_\{H,k\}\(S\_\{k\}\)\}\\mathbb\{E\}\\\!\\left\[r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\*\}\(S\_\{k\+1\}\)\\;\\middle\|\\;S\_\{k\},u\\right\]=Vk∗​\(Sk\)\.\\displaystyle=V\_\{k\}^\{\*\}\(S\_\{k\}\)\.\(49\)ThusVkπrec​\(Sk\)=Vk∗​\(Sk\)V\_\{k\}^\{\\pi^\{\\mathrm\{rec\}\}\}\(S\_\{k\}\)=V\_\{k\}^\{\*\}\(S\_\{k\}\)for allkkby induction\. In particular,

Vπrec​\(x¯0\)=V∗​\(x¯0\)\.V^\{\\pi^\{\\mathrm\{rec\}\}\}\(\\bar\{x\}\_\{0\}\)=V^\{\*\}\(\\bar\{x\}\_\{0\}\)\.\(50\)This proves Theorem 1\(c\), and completes the proof of Theorem 1\. ∎

#### Interpretation\.

As the key technical point of this proof, turn\-kktargetYk=rk\+γHτk​Fkπ∗​\(Sk\+1\)Y\_\{k\}=r\_\{k\}\+\\gamma\_\{H\}^\{\\tau\_\{k\}\}F\_\{k\}^\{\\pi^\{\*\}\}\(S\_\{k\+1\}\)is stationary only, when the downstream policiesπk\+1∗,…,πK−1∗\\pi\_\{k\+1\}^\{\*\},\\ldots,\\pi\_\{K\-1\}^\{\*\}are already optimized and frozen\. Reverse\-order training provides exactly this condition\. Without reverse\-order freezing,FkπF\_\{k\}^\{\\pi\}would change during turnkk, the targetYkY\_\{k\}would no longer be stationary, and the single\-turn stochastic approximation argument would not apply\.

#### Proposition of Theorem 1: Asymptotic recursive optimality under neural function approximation\.

The convergence guarantee in Theorem[1](https://arxiv.org/html/2608.18682#Thmtheorem1)relies on Assumption \(A4\) from Appendix[C\.1](https://arxiv.org/html/2608.18682#A3.SS1), which postulates a tabular representation ofQkQ\_\{k\}\. In the practical implementation of RTPO,QkQ\_\{k\}is realized implicitly by a neural sub\-policyπθk\\pi\_\{\\theta\_\{k\}\}operating on the conditioning contextck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\)\. We now state the corresponding asymptotic guarantee under the neural function approximation; the proof follows the standard reduction from tabular Q\-learning to projected Q\-learning under a function classℱθ\\mathcal\{F\}\_\{\\theta\}\. This statement is formalized as aProposition, which appears below as a neural\-function\-approximation extension of Theorem[1](https://arxiv.org/html/2608.18682#Thmtheorem1)\.

Proposition of Theorem 1: Asymptotic recursive optimality under neural function approximation\.

Suppose the assumptions of Theorem[1](https://arxiv.org/html/2608.18682#Thmtheorem1)hold, with current assumption \(A4\) from Appendix[C\.1](https://arxiv.org/html/2608.18682#A3.SS1)replaced by new assumption \(A4’\) below:

1. \(A4’\)Universal approximation\.For every fixedFkπ∗F\_\{k\}^\{\\pi^\{\*\}\}, there existsθk∗∈Θ\\theta\_\{k\}^\{\*\}\\in\\Thetasuch that the induced greedy policyπθk∗\\pi\_\{\\theta\_\{k\}^\{\*\}\}realizesπk∗\\pi\_\{k\}^\{\*\}on every reachableSkS\_\{k\}in the support of the visitation distribution\.

Then, reverse\-order training of RTPO produces a policy sequence\(π^0,…,π^K−1\)\(\\hat\{\\pi\}\_\{0\},\\ldots,\\hat\{\\pi\}\_\{K\-1\}\)that asymptotically realizes the recursively optimal policy\. That is, for everykkand every reachableSkS\_\{k\},

limn→∞πθk\(n\)\(⋅∣ck\)=πk∗\(⋅∣ck\)a\.s\.\\lim\_\{n\\to\\infty\}\\pi\_\{\\theta\_\{k\}^\{\(n\)\}\}\(\\cdot\\mid c\_\{k\}\)=\\pi\_\{k\}^\{\*\}\(\\cdot\\mid c\_\{k\}\)\\qquad\\text\{a\.s\.\}
###### Proof: Proposition of Theorem 1\.

Step 1\. Reduction to projected Q\-learning: Replace the tabular update with a parametric update on the function class\{πθk:θk∈Θ\}\\\{\\pi\_\{\\theta\_\{k\}\}:\\theta\_\{k\}\\in\\Theta\\\},

θk\(n\+1\)←θk\(n\)−αn​∇θk𝔼\(S,u\)∼μn​\[\(Qθk\(n\)​\(S,u\)−r~k\)2\],\\theta\_\{k\}^\{\(n\+1\)\}\\leftarrow\\theta\_\{k\}^\{\(n\)\}\-\\alpha\_\{n\}\\,\\nabla\_\{\\theta\_\{k\}\}\\mathbb\{E\}\_\{\(S,u\)\\sim\\mu\_\{n\}\}\\\!\\left\[\\big\(Q\_\{\\theta\_\{k\}^\{\(n\)\}\}\(S,u\)\-\\widetilde\{r\}\_\{k\}\\big\)^\{2\}\\right\],\(51\)whereμn\\mu\_\{n\}is the visitation distribution induced by the current greedy\-with\-exploration policy andQθkQ\_\{\\theta\_\{k\}\}is the implicit Q\-function realized byπθk\\pi\_\{\\theta\_\{k\}\}\. The fixed point of this update is the projection ofQ~k∗\\widetilde\{Q\}\_\{k\}^\{\*\}onto the function class,ΠΘ​Q~k∗\\Pi\_\{\\Theta\}\\widetilde\{Q\}\_\{k\}^\{\*\}, in the weightedL2​\(μ\)L^\{2\}\(\\mu\)sense\.

Step 2: Asymptotic realizability: By assumption \(A4’\), there existsθk∗∈Θ\\theta\_\{k\}^\{\*\}\\in\\Thetasuch that the induced greedy policy realizesπk∗\\pi\_\{k\}^\{\*\}on every reachableSkS\_\{k\}in the support ofμ\\mu\. This impliesΠΘ​Q~k∗=Q~k∗\\Pi\_\{\\Theta\}\\widetilde\{Q\}\_\{k\}^\{\*\}\\;=\\;\\widetilde\{Q\}\_\{k\}^\{\*\}on the support ofμ\\mu, so the projection error vanishes for the quantities relevant to the induced greedy policy\. Combined with standard asymptotic results for stochastic gradient descent on smooth nonconvex objectives under Robbins–Monro step sizes\[[39](https://arxiv.org/html/2608.18682#bib.bib29)\],Qθk\(n\)​\(S,u\)→Q~k∗​\(S,u\)Q\_\{\\theta\_\{k\}^\{\(n\)\}\}\(S,u\)\\to\\widetilde\{Q\}\_\{k\}^\{\*\}\(S,u\)a\.s\. for every\(S,u\)\(S,u\)in the support ofμ\\mu\.

Step 3\. Backward induction in the function\-class regime: Replace each step in the proof of Theorem[1](https://arxiv.org/html/2608.18682#Thmtheorem1)by Steps 1–2 above\. The reverse induction structure is unchanged: the determinism ofFkπ∗F\_\{k\}^\{\\pi^\{\*\}\}under frozen downstream sub\-policies is purely a property of the SMDP and the freezing schedule \(A5\), independent of whetherQkQ\_\{k\}is tabular or parametric\. Hence, the backward induction goes through and yieldsπθk\(n\)→πk∗\\pi\_\{\\theta\_\{k\}^\{\(n\)\}\}\\to\\pi\_\{k\}^\{\*\}a\.s\. on the support ofμ\\mufor everykk\. ∎

Remark on the assumption gap between \(A4\) and \(A4’\)\.Assumption \(A4’\) is strictly weaker than what would be required for finite\-sample convergence rates: it asks only that the function class be*rich enough*to contain the optimal sub\-policy in the asymptotic limit, not that the gradient dynamics realize this optimum at any finite iteration\. For the modern Transformer architectures used in our experiments with Qwen3 models, the universal approximation property is widely accepted as a working assumption, and our experimental results in Sec[4](https://arxiv.org/html/2608.18682#S4)provide empirical evidence that the asymptotic guarantee transfers to practice\.

### C\.3Proof of Theorem 2: Causally Consistent Turn\-Level Advantage Estimation

We formalize Theorem 2 from Sec[3\.2](https://arxiv.org/html/2608.18682#S3.SS2)before presenting the proof\. We compare the state\-matched sibling advantage estimator used by RTPO with the trajectory\-level advantage estimator used in flat\-trajectory training\. The key distinction is whether the baseline is computed from rollouts sharing the same boundary stateSkS\_\{k\}, or from complete trajectories that may have reached different boundary states by turnkk\.

Turn\-Level Credit Assignment without State Bias###### Theorem 2\(Causally Consistent Turn\-Level Advantage Estimation\)\. Consider turnkkand suppose thatG−1G\{\-\}1sibling rollouts are forked from the same boundary stateSkS\_\{k\}\. Each sibling rolloutj∈\{1,…,G−1\}j\\in\\\{1,\\ldots,G\{\-\}1\\\}independently samples a turn\-kkmacro\-actionuj,k≡lj,ku\_\{j,k\}\\equiv l\_\{j,k\}and then continues to the terminal\. LetQ^j,k=Rj,V^k=1G−1​∑j=1G−1Q^j,k,Aj,kH=Q^j,k−V^k\\hat\{Q\}\_\{j,k\}=R\_\{j\},\\qquad\\hat\{V\}\_\{k\}=\\frac\{1\}\{G\-1\}\\sum\_\{j=1\}^\{G\-1\}\\hat\{Q\}\_\{j,k\},\\qquad A^\{H\}\_\{j,k\}=\\hat\{Q\}\_\{j,k\}\-\\hat\{V\}\_\{k\}\(52\)denote the Monte Carlo turn\-level action\-value estimate, the state\-matched sibling baseline, and the resulting turn\-level advantage for sibling rolloutjj, respectively\. Let the corresponding flat trajectory\-level estimator beR¯=1G​∑i=1GRi,Aitraj=Ri−R¯,\\bar\{R\}=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}R\_\{i\},\\qquad A\_\{i\}^\{\\mathrm\{traj\}\}=R\_\{i\}\-\\bar\{R\},\(53\)wheregig\_\{i\}denotes theii\-th complete trajectory andSi,kS\_\{i,k\}is its boundary state at turnkk\. DefineQπ\(Sk,uj,k\)=𝔼\[R∣Sk,uj,k\],Vπ\(Sk\)=𝔼u∼π\(⋅∣ck\)\[Qπ\(Sk,u\)\],Q^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)=\\mathbb\{E\}\[R\\mid S\_\{k\},u\_\{j,k\}\],\\qquad V^\{\\pi\}\(S\_\{k\}\)=\\mathbb\{E\}\_\{u\\sim\\pi\(\\cdot\\mid c\_\{k\}\)\}\[Q^\{\\pi\}\(S\_\{k\},u\)\],\(54\)andAπ​\(Sk,uj,k\)=Qπ​\(Sk,uj,k\)−Vπ​\(Sk\),μR=𝔼⁡\[R\]\.A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)=Q^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\-V^\{\\pi\}\(S\_\{k\}\),\\qquad\\mu\_\{R\}=\\mathbb\{E\}\[R\]\.\(55\)Then, under bounded rewards and independent sibling sampling, the RTPO sibling advantage estimator satisfies the following properties:\(a\)Local unbiasedness up to finite\-group bias\.Conditional on the shared boundary stateSkS\_\{k\}and the sampled macro\-actionuj,ku\_\{j,k\}, the state\-matched sibling estimator satisfies:𝔼\[Aj,kH∣Sk,uj,k\]=G−2G−1Aπ\(Sk,uj,k\)\.\\mathbb\{E\}\\\!\\left\[A^\{H\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\\right\]=\\frac\{G\-2\}\{G\-1\}A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\.\(56\)Thus,Aj,kHA^\{H\}\_\{j,k\}estimates the true turn\-level advantage up to a finite\-group bias of orderO⁡\(1/G\)O\(1/G\)and contains no upstream state\-contamination term\. In contrast, the trajectory\-level estimator satisfies:𝔼\[Aitraj∣Si,k,ui,k\]=G−1GAπ\(Si,k,ui,k\)\+G−1G\(Vπ\(Si,k\)−μR\),\\mathbb\{E\}\\\!\\left\[A\_\{i\}^\{\\mathrm\{traj\}\}\\mid S\_\{i,k\},u\_\{i,k\}\\right\]=\\frac\{G\-1\}\{G\}A^\{\\pi\}\(S\_\{i,k\},u\_\{i,k\}\)\+\\frac\{G\-1\}\{G\}\\bigl\(V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\\bigr\),\(57\)where the second term is the upstream state\-contamination term\.\(b\)Reduced value estimation error\.LetσV2:=VarSk⁡\[Vπ​\(Sk\)\]\\sigma\_\{V\}^\{2\}:=\\operatorname\{Var\}\_\{S\_\{k\}\}\\\!\\left\[V^\{\\pi\}\(S\_\{k\}\)\\right\]\(58\)denote the variance of boundary\-state values\. Compared with the trajectory\-level estimator, the sibling estimator removes the cross\-state value componentVπ​\(Si,k\)−μRV^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\. Therefore, under a matched downstream\-noise comparison, the trajectory estimator contains an additional MSE contribution proportional toσV2\\sigma\_\{V\}^\{2\}, while the sibling estimator does not\. In particular, whenσV2\>0\\sigma\_\{V\}^\{2\}\>0, the state\-matched estimator removes a non\-zero source of value\-estimation error induced by cross\-state baselines\.\(c\)State\-matched causal actions\.All sibling rollouts share the same boundary stateSkS\_\{k\}\. Therefore, differences in their returns are attributable to the sampled turn\-kkmacro\-actionsuj,ku\_\{j,k\}and independent downstream sampling noise, rather than to upstream trajectory differences\. The resulting advantageAj,kHA^\{H\}\_\{j,k\}is assigned only to the output tokens of turnkk,lj,k=\(aj,k,1,…,aj,k,Tj,k\),l\_\{j,k\}=\(a\_\{j,k,1\},\\ldots,a\_\{j,k,T\_\{j,k\}\}\),\(59\)while the prefix tokens contained in the shared contextck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\)are excluded from the gradient\.

###### Proof\.

We prove parts \(a\)–\(c\) of Theorem 2 as follows\.

#### Proof of Theorem 2\(a\): local unbiasedness up to finite\-group bias\.

Fix a turnkkand a shared boundary stateSkS\_\{k\}\. RTPO forksG−1G\{\-\}1sibling rollouts from the same environment snapshotsnapk\\mathrm\{snap\}\_\{k\}\. Conditional onSkS\_\{k\}, each sibling rollout independently samples its turn\-kkmacro\-action and downstream continuation\. Thus, for any two distinct sibling rolloutsj≠j′j\\neq j^\{\\prime\}, the returnRj′R\_\{j^\{\\prime\}\}of siblingj′j^\{\\prime\}is conditionally independent of the actionuj,ku\_\{j,k\}sampled by siblingjj:

Rj′⟂⟂uj,k\|Sk\.R\_\{j^\{\\prime\}\}\\perp\\\!\\\!\\\!\\perp u\_\{j,k\}\\mid S\_\{k\}\.\(60\)Consequently,

𝔼\[Rj′∣Sk,uj,k\]=𝔼\[Rj′∣Sk\]=Vπ\(Sk\)\.\\mathbb\{E\}\[R\_\{j^\{\\prime\}\}\\mid S\_\{k\},u\_\{j,k\}\]=\\mathbb\{E\}\[R\_\{j^\{\\prime\}\}\\mid S\_\{k\}\]=V^\{\\pi\}\(S\_\{k\}\)\.\(61\)Meanwhile, for the own return of siblingjj,

𝔼\[Rj∣Sk,uj,k\]=Qπ\(Sk,uj,k\)\.\\mathbb\{E\}\[R\_\{j\}\\mid S\_\{k\},u\_\{j,k\}\]=Q^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\.\(62\)
By definition,

Aj,kH\\displaystyle A^\{H\}\_\{j,k\}=Rj−1G−1​∑r=1G−1Rr\\displaystyle=R\_\{j\}\-\\frac\{1\}\{G\-1\}\\sum\_\{r=1\}^\{G\-1\}R\_\{r\}=G−2G−1​Rj−1G−1​∑r≠jRr\.\\displaystyle=\\frac\{G\-2\}\{G\-1\}R\_\{j\}\-\\frac\{1\}\{G\-1\}\\sum\_\{r\\neq j\}R\_\{r\}\.\(63\)Taking conditional expectation given\(Sk,uj,k\)\(S\_\{k\},u\_\{j,k\}\)and using Eq\. \([61](https://arxiv.org/html/2608.18682#A3.E61)\) and Eq\. \([62](https://arxiv.org/html/2608.18682#A3.E62)\),

𝔼\[Aj,kH∣Sk,uj,k\]\\displaystyle\\mathbb\{E\}\[A^\{H\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\]=G−2G−1​Qπ​\(Sk,uj,k\)−G−2G−1​Vπ​\(Sk\)\\displaystyle=\\frac\{G\-2\}\{G\-1\}Q^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\-\\frac\{G\-2\}\{G\-1\}V^\{\\pi\}\(S\_\{k\}\)=G−2G−1​\(Qπ​\(Sk,uj,k\)−Vπ​\(Sk\)\)\\displaystyle=\\frac\{G\-2\}\{G\-1\}\\bigl\(Q^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\-V^\{\\pi\}\(S\_\{k\}\)\\bigr\)=G−2G−1​Aπ​\(Sk,uj,k\)\.\\displaystyle=\\frac\{G\-2\}\{G\-1\}A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\.\(64\)Therefore, the sibling estimator is locally unbiased up to the finite\-group multiplicative factor\(G−2\)/\(G−1\)\(G\-2\)/\(G\-1\)\. Equivalently, the conditional bias relative to the true turn\-level advantage is

𝔼\[Aj,kH∣Sk,uj,k\]−Aπ\(Sk,uj,k\)=−1G−1Aπ\(Sk,uj,k\),\\mathbb\{E\}\[A^\{H\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\]\-A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)=\-\\frac\{1\}\{G\-1\}A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\),\(65\)which isO⁡\(1/G\)O\(1/G\)and contains no term depending onVπ​\(Sk\)−μRV^\{\\pi\}\(S\_\{k\}\)\-\\mu\_\{R\}\.

We now contrast this with the trajectory\-level estimator\. In flat\-trajectory training, theGGtrajectories are independently sampled from the same prompt but may reach different boundary states by turnkk\. The trajectory\-level advantage is

Aitraj\\displaystyle A\_\{i\}^\{\\mathrm\{traj\}\}=Ri−1G​∑r=1GRr\\displaystyle=R\_\{i\}\-\\frac\{1\}\{G\}\\sum\_\{r=1\}^\{G\}R\_\{r\}=G−1G​Ri−1G​∑r≠iRr\.\\displaystyle=\\frac\{G\-1\}\{G\}R\_\{i\}\-\\frac\{1\}\{G\}\\sum\_\{r\\neq i\}R\_\{r\}\.\(66\)The own return satisfies

𝔼\[Ri∣Si,k,ui,k\]=Qπ\(Si,k,ui,k\)\.\\mathbb\{E\}\[R\_\{i\}\\mid S\_\{i,k\},u\_\{i,k\}\]=Q^\{\\pi\}\(S\_\{i,k\},u\_\{i,k\}\)\.\(67\)Forr≠ir\\neq i, trajectoryrris generated by an independent upstream rollout and reaches its own boundary stateSr,kS\_\{r,k\}\. Thus,

𝔼\[Rr∣Si,k,ui,k\]=𝔼Sr,k\[Vπ\(Sr,k\)\]=μR\.\\mathbb\{E\}\[R\_\{r\}\\mid S\_\{i,k\},u\_\{i,k\}\]=\\mathbb\{E\}\_\{S\_\{r,k\}\}\\\!\\left\[V^\{\\pi\}\(S\_\{r,k\}\)\\right\]=\\mu\_\{R\}\.\(68\)Substituting Eq\. \([67](https://arxiv.org/html/2608.18682#A3.E67)\) and Eq\. \([68](https://arxiv.org/html/2608.18682#A3.E68)\) into Eq\. \([66](https://arxiv.org/html/2608.18682#A3.E66)\) gives

𝔼\[Aitraj∣Si,k,ui,k\]\\displaystyle\\mathbb\{E\}\[A\_\{i\}^\{\\mathrm\{traj\}\}\\mid S\_\{i,k\},u\_\{i,k\}\]=G−1G​\(Qπ​\(Si,k,ui,k\)−μR\)\\displaystyle=\\frac\{G\-1\}\{G\}\\bigl\(Q^\{\\pi\}\(S\_\{i,k\},u\_\{i,k\}\)\-\\mu\_\{R\}\\bigr\)=G−1G​\(Qπ​\(Si,k,ui,k\)−Vπ​\(Si,k\)\)\+G−1G​\(Vπ​\(Si,k\)−μR\)\\displaystyle=\\frac\{G\-1\}\{G\}\\bigl\(Q^\{\\pi\}\(S\_\{i,k\},u\_\{i,k\}\)\-V^\{\\pi\}\(S\_\{i,k\}\)\\bigr\)\+\\frac\{G\-1\}\{G\}\\bigl\(V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\\bigr\)=G−1G​Aπ​\(Si,k,ui,k\)\+G−1G​\(Vπ​\(Si,k\)−μR\)\.\\displaystyle=\\frac\{G\-1\}\{G\}A^\{\\pi\}\(S\_\{i,k\},u\_\{i,k\}\)\+\\frac\{G\-1\}\{G\}\\bigl\(V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\\bigr\)\.\(69\)The second term in Eq\. \([69](https://arxiv.org/html/2608.18682#A3.E69)\) depends only on the upstream boundary stateSi,kS\_\{i,k\}and the global mean returnμR\\mu\_\{R\}\. It has no causal dependence on the current turn actionui,ku\_\{i,k\}and is precisely the cross\-state contamination term\. Therefore, the sibling estimator removes the upstream state\-contamination term present in the trajectory\-level estimator\. This proves part \(a\) of Theorem 2\.

#### Proof of Theorem[2](https://arxiv.org/html/2608.18682#Thmtheorem2)\(b\): reduced value estimation error\.

We compare the error of each estimator against its corresponding true turn\-level advantage\. For the sibling estimator, the target isAπ​\(Sk,uj,k\)A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\); for the trajectory estimator, the target isAπ​\(Si,k,ui,k\)A^\{\\pi\}\(S\_\{i,k\},u\_\{i,k\}\)\. The trajectory estimator contains the additional cross\-state term

Ci,k:=G−1G​\(Vπ​\(Si,k\)−μR\),C\_\{i,k\}:=\\frac\{G\-1\}\{G\}\\bigl\(V^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}\\bigr\),\(70\)whereas the sibling estimator does not contain such a term\. Marginalizing over the boundary\-state distribution,

𝔼⁡\[Ci,k\]=0,Var⁡\(Ci,k\)=\(G−1G\)2​VarSk​\[Vπ​\(Sk\)\]=\(G−1G\)2​σV2\.\\mathbb\{E\}\[C\_\{i,k\}\]=0,\\qquad\\operatorname\{Var\}\(C\_\{i,k\}\)=\\left\(\\frac\{G\-1\}\{G\}\\right\)^\{2\}\\operatorname\{Var\}\_\{S\_\{k\}\}\\\!\\left\[V^\{\\pi\}\(S\_\{k\}\)\\right\]=\\left\(\\frac\{G\-1\}\{G\}\\right\)^\{2\}\\sigma\_\{V\}^\{2\}\.\(71\)Thus, cross\-trajectory baselines introduce an additional MSE component proportional toσV2\\sigma\_\{V\}^\{2\}, while state\-matched sibling baselines remove it\.

In addition to the squared\-bias contribution above, the trajectory\-level baselineR¯=1G​∑r≠iRr\\bar\{R\}=\\frac\{1\}\{G\}\\sum\_\{r\\neq i\}R\_\{r\}introduces further variance from cross\-state effects\. Conditional on\(Si,k,ui,k\)\(S\_\{i,k\},u\_\{i,k\}\), eachRrR\_\{r\}\(r≠ir\\neq i\) is independent of the conditioning, so its conditional variance equals its unconditional variance:

Var⁡\(Rr∣Si,k,ui,k\)=Var⁡\(Rr\)=𝔼Sr,k​\[Var⁡\(Rr∣Sr,k\)\]⏟σ¯down2\+σV2,\\mathrm\{Var\}\(R\_\{r\}\\mid S\_\{i,k\},u\_\{i,k\}\)=\\mathrm\{Var\}\(R\_\{r\}\)=\\underbrace\{\\mathbb\{E\}\_\{S\_\{r,k\}\}\\\!\\left\[\\mathrm\{Var\}\(R\_\{r\}\\mid S\_\{r,k\}\)\\right\]\}\_\{\\displaystyle\\bar\{\\sigma\}\_\{\\mathrm\{down\}\}^\{2\}\}\+\\;\\sigma\_\{V\}^\{2\},\(72\)where theσV2\\sigma\_\{V\}^\{2\}term arises becauseSr,kS\_\{r,k\}varies across trajectories\. This contributesG−1G2​σV2\\frac\{G\-1\}\{G^\{2\}\}\\sigma\_\{V\}^\{2\}to the conditional variance ofAitrajA\_\{i\}^\{\\mathrm\{traj\}\}\. By contrast, in the sibling estimator, all returns share the same boundary stateSkS\_\{k\}, soVar⁡\(Rr∣Sk\)\\mathrm\{Var\}\(R\_\{r\}\\mid S\_\{k\}\)is a purely within\-state quantity and contains noσV2\\sigma\_\{V\}^\{2\}component\.

More explicitly, the mean squared error can be decomposed as

MSE⁡\(A^\)=Var⁡\(A^\)\+Bias⁡\(A^\)2\.\\operatorname\{MSE\}\(\\hat\{A\}\)=\\operatorname\{Var\}\(\\hat\{A\}\)\+\\operatorname\{Bias\}\(\\hat\{A\}\)^\{2\}\.\(73\)For the trajectory estimator, the bias contains the cross\-state termCi,kC\_\{i,k\}from Eq\. \([70](https://arxiv.org/html/2608.18682#A3.E70)\)\. For the sibling estimator, the only systematic bias from part \(a\) is the finite\-group term

−1G−1​Aπ​\(Sk,uj,k\),\-\\frac\{1\}\{G\-1\}A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\),\(74\)which isO⁡\(1/G\)O\(1/G\)\. Therefore, under a matched downstream\-noise comparison, the totalσV2\\sigma\_\{V\}^\{2\}coefficient in the trajectory estimator’s marginal MSE combines the squared\-bias contribution \(Eq\. \([71](https://arxiv.org/html/2608.18682#A3.E71)\)\) with the baseline\-variance contribution \(Eq\. \([72](https://arxiv.org/html/2608.18682#A3.E72)\)\):

\(G−1G\)2⏟Bias2\+G−1G2⏟Var=\(G−1\)2\+\(G−1\)G2=G−1G\.\\underbrace\{\\left\(\\frac\{G\-1\}\{G\}\\right\)^\{\\\!2\}\}\_\{\\mathrm\{Bias\}^\{2\}\}\+\\;\\underbrace\{\\frac\{G\-1\}\{G^\{2\}\}\}\_\{\\mathrm\{Var\}\}\\;=\\;\\frac\{\(G\-1\)^\{2\}\+\(G\-1\)\}\{G^\{2\}\}\\;=\\;\\frac\{G\-1\}\{G\}\.\(75\)The sibling estimator incurs noσV2\\sigma\_\{V\}^\{2\}contribution from either source\. The leading MSE difference is thus:

MSE⁡\(Aitraj\)−MSE⁡\(Aj,kH\)=G−1G​σV2\+Δdown−O⁡\(1/G2\),\\operatorname\{MSE\}\(A\_\{i\}^\{\\mathrm\{traj\}\}\)\-\\operatorname\{MSE\}\(A^\{H\}\_\{j,k\}\)=\\frac\{G\-1\}\{G\}\\,\\sigma\_\{V\}^\{2\}\+\\Delta\_\{\\mathrm\{down\}\}\-O\(1/G^\{2\}\),\(76\)whereΔdown≥0\\Delta\_\{\\mathrm\{down\}\}\\geq 0collects differences in downstream Monte Carlo noise between the two estimators under matched conditions\. Hence, wheneverσV2\>0\\sigma\_\{V\}^\{2\}\>0, the state\-matched sibling estimator has strictly lower MSE\. This proves part \(b\) of Theorem 2\.

#### Proof of Theorem[2](https://arxiv.org/html/2608.18682#Thmtheorem2)\(c\): state\-matched causal actions\.

All sibling rollouts are forked from the same boundary stateSkS\_\{k\}\. Therefore, conditional onSkS\_\{k\}, sibling rolloutjjand sibling rolloutj′j^\{\\prime\}differ only in their sampled turn\-kkmacro\-actions and their independent downstream randomness:

\(uj,k,ξj\)versus\(uj′,k,ξj′\),\(u\_\{j,k\},\\xi\_\{j\}\)\\quad\\text\{versus\}\\quad\(u\_\{j^\{\\prime\},k\},\\xi\_\{j^\{\\prime\}\}\),\(77\)whereξj\\xi\_\{j\}denotes the downstream randomness of siblingjj\. Since the shared prefix stateSkS\_\{k\}is identical across siblings, any systematic difference in conditional expected return is attributable to the sampled turn\-kkmacro\-action:

𝔼\[Rj−Rj′∣Sk,uj,k,uj′,k\]=Qπ\(Sk,uj,k\)−Qπ\(Sk,uj′,k\)\.\\mathbb\{E\}\[R\_\{j\}\-R\_\{j^\{\\prime\}\}\\mid S\_\{k\},u\_\{j,k\},u\_\{j^\{\\prime\},k\}\]=Q^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\-Q^\{\\pi\}\(S\_\{k\},u\_\{j^\{\\prime\},k\}\)\.\(78\)The downstream randomness contributes to Monte Carlo noise but does not create an upstream state bias, because the upstream state is shared and fixed\.

Finally, RTPO assigns the resulting advantage only to the output tokens of turnkk\. For sibling rolloutjj, these tokens are

lj,k=\(aj,k,1,…,aj,k,Tj,k\)\.l\_\{j,k\}=\(a\_\{j,k,1\},\\ldots,a\_\{j,k,T\_\{j,k\}\}\)\.\(79\)The shared prefixck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\)is used only as the conditioning input and is excluded from the gradient\. Equivalently, the gradient support of the turn\-kkobjective satisfies

supp\(∇θJk\)⊆\{\(j,k,t\):j∈\{1,…,G−1\},t∈\{1,…,Tj,k\}\}\.\\operatorname\{supp\}\(\\nabla\_\{\\theta\}J\_\{k\}\)\\subseteq\\\{\(j,k,t\):j\\in\\\{1,\\ldots,G\-1\\\},\\;t\\in\\\{1,\\ldots,T\_\{j,k\}\\\}\\\}\.\(80\)Thus, prefix tokens receive zero gradient, and the credit signal is assigned only to the sampled turn\-kkoutput action\. This proves part \(c\) of Theorem 2, and completes the proof of Theorem[2](https://arxiv.org/html/2608.18682#Thmtheorem2)\. ∎

#### Interpretation\.

The proof shows that the key difference between RTPO and flat\-trajectory training is the baseline state\. The sibling estimator compares alternative turn\-kkactions from the same boundary stateSkS\_\{k\}, so the baseline estimates the local valueVπ​\(Sk\)V^\{\\pi\}\(S\_\{k\}\)\. In contrast, a flat\-trajectory baseline compares returns from trajectories that may have reached different boundary states by turnkk, so it is centered around the global meanμR\\mu\_\{R\}\. The resulting termVπ​\(Si,k\)−μRV^\{\\pi\}\(S\_\{i,k\}\)\-\\mu\_\{R\}is an upstream state\-contamination term: it can dominate the true turn\-level advantage even though it is unrelated to the current turn action\. RTPO removes this term by constructing state\-matched sibling rollouts and assigning the resulting advantage only to the output tokens of the current turn\.

### C\.4Proof of Theorem 3: On\-Policy Continuation under Asynchronous Turns

We formalize Theorem 3 from Sec[3\.3](https://arxiv.org/html/2608.18682#S3.SS3)before presenting the proof\. The purpose is to show that RTPO avoids the off\-policy drift induced by stale downstream rollouts\. In asynchronous multi\-turn training, a trajectory generated under an old policy may later be evaluated under an updated policy, which would require a long\-horizon trajectory\-level importance\-sampling correction\. RTPO avoids this issue by regenerating sibling continuations on\-policy at each reverse\-order turn\.

#### Setup\.

Letθ0\\theta\_\{0\}denote the rollout parameters at the beginning of a training iteration, and letθ\>k\\theta\_\{\>k\}denote the current parameters available at the start of turnkk, after downstream turnsk\+1,…,K−1k\+1,\\ldots,K\-1have already been optimized\. A stale\-rollout alternative would retain sibling continuations generated underπθ0\\pi\_\{\\theta\_\{0\}\}and correct them using the trajectory\-level importance\-sampling weight:

ωjold=∏h=k\+1K−1∏t=1Tj,hπθ\>k​\(aj,h,t∣cj,h,aj,h,<t\)πθ0​\(aj,h,t∣cj,h,aj,h,<t\)\.\\omega\_\{j\}^\{\\mathrm\{old\}\}=\\prod\_\{h=k\+1\}^\{K\-1\}\\prod\_\{t=1\}^\{T\_\{j,h\}\}\\frac\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}\{\\pi\_\{\\theta\_\{0\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}\.\(81\)In contrast, RTPO synchronizes the latest parametersθ\>k\\theta\_\{\>k\}to the inference engine at the start of turnkk, forksG−1G\{\-\}1sibling rollouts from the shared boundary stateSkS\_\{k\}, and samples each sibling rolloutj∈\{1,…,G−1\}j\\in\\\{1,\\ldots,G\{\-\}1\\\}usingπθ\>k\\pi\_\{\\theta\_\{\>k\}\}from turnkkuntil termination\. We writeuj,k≡lj,ku\_\{j,k\}\\equiv l\_\{j,k\}for the sampled turn\-kkmacro\-action of siblingjj, andRjR\_\{j\}for its terminal return\. The Monte Carlo turn\-level Q\-value estimate is

Q^j,k=Rj=rj,k\+γHτj,k​F^j,kπθ\>k,\\hat\{Q\}\_\{j,k\}=R\_\{j\}=r\_\{j,k\}\+\\gamma\_\{H\}^\{\\tau\_\{j,k\}\}\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\},\(82\)whereF^j,kπθ\>k\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}is the sampled downstream continuation value generated under the same current policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}\. The corresponding sibling advantage is

Aj,kH=Q^j,k−V^k,V^k=1G−1​∑r=1G−1Q^r,k\.A^\{H\}\_\{j,k\}=\\hat\{Q\}\_\{j,k\}\-\\hat\{V\}\_\{k\},\\qquad\\hat\{V\}\_\{k\}=\\frac\{1\}\{G\-1\}\\sum\_\{r=1\}^\{G\-1\}\\hat\{Q\}\_\{r,k\}\.\(83\)
On\-Policy Continuation without Drift###### Theorem 3\(On\-Policy Continuation under Asynchronous Turns\)\. Consider turnkkin the reverse\-order training procedure of RTPO\. At the start of turnkk, RTPO synchronizes the current downstream policy parametersθ\>k\\theta\_\{\>k\}to the inference engine, forksG−1G\{\-\}1sibling rollouts from the shared boundary stateSkS\_\{k\}, and continues each sibling rolloutjjto termination under the same policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}\. Then the following two properties hold:\(a\)Drift\-free on\-policy continuation\.Each sibling’s terminal returnRjR\_\{j\}is an unbiased Monte Carlo estimate of the current\-policy turn\-level Q\-value:𝔼\[Q^j,k∣Sk,uj,k\]=Q~kπθ\>k\(Sk,uj,k\)\.\\mathbb\{E\}\\\!\\left\[\\hat\{Q\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\\right\]=\\tilde\{Q\}\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{k\},u\_\{j,k\}\)\.\(84\)Moreover, because the sampling policy and the evaluation policy coincide throughout the sibling continuation, the trajectory\-level importance\-sampling weight satisfiesωj=∏h=k\+1K−1∏t=1Tj,hπθ\>k​\(aj,h,t∣cj,h,aj,h,<t\)πθ\>k​\(aj,h,t∣cj,h,aj,h,<t\)≡1\.\\omega\_\{j\}=\\prod\_\{h=k\+1\}^\{K\-1\}\\prod\_\{t=1\}^\{T\_\{j,h\}\}\\frac\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}\\equiv 1\.\(85\)\(b\)Dynamic error reduction in advantage estimation\.Under bounded binary or normalized rewards, the conditional variance of the Monte Carlo Q\-value estimator is controlled by the current\-policy success probability\. In the binary case, ifpj,k:=Q~kπθ\>k​\(Sk,uj,k\)∈\[0,1\],p\_\{j,k\}:=\\tilde\{Q\}\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{k\},u\_\{j,k\}\)\\in\[0,1\],\(86\)thenVar⁡\(Q^j,k∣Sk,uj,k\)=pj,k​\(1−pj,k\)\.\\operatorname\{Var\}\(\\hat\{Q\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\)=p\_\{j,k\}\(1\-p\_\{j,k\}\)\.\(87\)Therefore, when reverse\-order training improves downstream policies so thatpj,kp\_\{j,k\}moves away from the high\-uncertainty region around1/21/2, the Q\-value estimation variance decreases\. SinceAj,kHA^\{H\}\_\{j,k\}is computed from the sibling Q\-value estimates, this improves the signal\-to\-noise ratio of the resulting turn\-level advantage estimator\.

###### Proof\.

We prove parts \(a\) and \(b\) of Theorem 3 as follows\.

#### Proof of Theorem[3](https://arxiv.org/html/2608.18682#Thmtheorem3)\(a\): drift\-free on\-policy continuation\.

Fix a turnkkand a sibling rolloutjj\. At the start of turnkk, RTPO synchronizes the current downstream policy parametersθ\>k\\theta\_\{\>k\}to the inference engine\. The sibling rollout is forked from the shared boundary stateSkS\_\{k\}, samples the turn\-kkmacro\-actionuj,k≡lj,ku\_\{j,k\}\\equiv l\_\{j,k\}, and then continues through turnsk\+1,…,K−1k\+1,\\ldots,K\-1under the same policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}\.

By the definition of the turn\-level augmented action value,

Q~kπθ\>k\(Sk,uj,k\)=𝔼\[rj,k\+γHτj,kFkπθ\>k\(Sj,k\+1\)\|Sk,uj,k\],\\tilde\{Q\}\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{k\},u\_\{j,k\}\)=\\mathbb\{E\}\\\!\\left\[r\_\{j,k\}\+\\gamma\_\{H\}^\{\\tau\_\{j,k\}\}F\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{j,k\+1\}\)\\;\\middle\|\\;S\_\{k\},u\_\{j,k\}\\right\],\(88\)whereSj,k\+1S\_\{j,k\+1\}is the next boundary state reached by siblingjjafter executinguj,ku\_\{j,k\}\. Since the sampled downstream continuationF^j,kπθ\>k\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}is generated by rolling out the same current policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}fromSj,k\+1S\_\{j,k\+1\}to termination, it is an unbiased Monte Carlo draw from the downstream continuation value:

𝔼⁡\[F^j,kπθ\>k∣Sj,k\+1\]=Fkπθ\>k​\(Sj,k\+1\)\.\\mathbb\{E\}\\\!\\left\[\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\\mid S\_\{j,k\+1\}\\right\]=F\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{j,k\+1\}\)\.\(89\)Substituting Eq\. \([89](https://arxiv.org/html/2608.18682#A3.E89)\) into Eq\. \([82](https://arxiv.org/html/2608.18682#A3.E82)\) gives

𝔼\[Q^j,k∣Sk,uj,k\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\hat\{Q\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\\right\]=𝔼\[rj,k\+γHτj,kF^j,kπθ\>k\|Sk,uj,k\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[r\_\{j,k\}\+\\gamma\_\{H\}^\{\\tau\_\{j,k\}\}\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\\;\\middle\|\\;S\_\{k\},u\_\{j,k\}\\right\]=𝔼\[rj,k\+γHτj,kFkπθ\>k\(Sj,k\+1\)\|Sk,uj,k\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[r\_\{j,k\}\+\\gamma\_\{H\}^\{\\tau\_\{j,k\}\}F\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{j,k\+1\}\)\\;\\middle\|\\;S\_\{k\},u\_\{j,k\}\\right\]=Q~kπθ\>k​\(Sk,uj,k\)\.\\displaystyle=\\tilde\{Q\}\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{k\},u\_\{j,k\}\)\.\(90\)Therefore, each sibling terminal return provides an unbiased Monte Carlo estimate of the turn\-level Q\-value under the current downstream policy\.

Next, because the sibling is both sampled and evaluated under the same policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}, the full trajectory\-level importance\-sampling weight is

ωj=∏h=k\+1K−1∏t=1Tj,hπθ\>k​\(aj,h,t∣cj,h,aj,h,<t\)πθ\>k​\(aj,h,t∣cj,h,aj,h,<t\)=1\.\\omega\_\{j\}=\\prod\_\{h=k\+1\}^\{K\-1\}\\prod\_\{t=1\}^\{T\_\{j,h\}\}\\frac\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}=1\.\(91\)Thus, no trajectory\-level IS correction is required\. This proves part \(a\) of Theorem 3\.

#### Proof of Theorem[3](https://arxiv.org/html/2608.18682#Thmtheorem3)\(b\): dynamic error reduction in advantage estimation\.

We first consider the binary\-reward case, whereRj∈\{0,1\}R\_\{j\}\\in\\\{0,1\\\}\. Conditional on\(Sk,uj,k\)\(S\_\{k\},u\_\{j,k\}\), define

pj,k=ℙπθ\>k​\(Rj=1∣Sk,uj,k\)=Q~kπθ\>k​\(Sk,uj,k\)\.p\_\{j,k\}=\\mathbb\{P\}\_\{\\pi\_\{\\theta\_\{\>k\}\}\}\(R\_\{j\}=1\\mid S\_\{k\},u\_\{j,k\}\)=\\tilde\{Q\}\_\{k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\(S\_\{k\},u\_\{j,k\}\)\.\(92\)ThenQ^j,k=Rj\\hat\{Q\}\_\{j,k\}=R\_\{j\}is a Bernoulli random variable with meanpj,kp\_\{j,k\}, and therefore

Var⁡\(Q^j,k∣Sk,uj,k\)=pj,k​\(1−pj,k\)\.\\operatorname\{Var\}\(\\hat\{Q\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\)=p\_\{j,k\}\(1\-p\_\{j,k\}\)\.\(93\)The functionf⁡\(p\)=p⁡\(1−p\)f\(p\)=p\(1\-p\)is maximized atp=1/2p=1/2and decreases asppmoves toward either00or11\. Equivalently,

p⁡\(1−p\)=14−\(p−12\)2\.p\(1\-p\)=\\frac\{1\}\{4\}\-\\left\(p\-\\frac\{1\}\{2\}\\right\)^\{2\}\.\(94\)Therefore, if reverse\-order training improves the downstream continuation policy so that the induced success probability moves away from the high\-uncertainty region around1/21/2, then the conditional variance of the Q\-value estimator decreases\. Formally,

\|pj,knew−12\|\>\|pj,kold−12\|⟹pj,knew​\(1−pj,knew\)<pj,kold​\(1−pj,kold\)\.\\left\|p\_\{j,k\}^\{\\mathrm\{new\}\}\-\\frac\{1\}\{2\}\\right\|\>\\left\|p\_\{j,k\}^\{\\mathrm\{old\}\}\-\\frac\{1\}\{2\}\\right\|\\quad\\Longrightarrow\\quad p\_\{j,k\}^\{\\mathrm\{new\}\}\(1\-p\_\{j,k\}^\{\\mathrm\{new\}\}\)<p\_\{j,k\}^\{\\mathrm\{old\}\}\(1\-p\_\{j,k\}^\{\\mathrm\{old\}\}\)\.\(95\)
For normalized rewardsRj∈\[0,1\]R\_\{j\}\\in\[0,1\], the same statement holds as a bounded\-variance control rather than an exact Bernoulli identity\. In particular, by the Bhatia–Davis bound for random variables supported on\[0,1\]\[0,1\],

Var\(Rj∣Sk,uj,k\)≤𝔼\[Rj∣Sk,uj,k\]\(1−𝔼\[Rj∣Sk,uj,k\]\)\.\\operatorname\{Var\}\(R\_\{j\}\\mid S\_\{k\},u\_\{j,k\}\)\\leq\\mathbb\{E\}\[R\_\{j\}\\mid S\_\{k\},u\_\{j,k\}\]\\bigl\(1\-\\mathbb\{E\}\[R\_\{j\}\\mid S\_\{k\},u\_\{j,k\}\]\\bigr\)\.\(96\)Thus, moving the conditional mean away from the high\-uncertainty middle region also reduces the worst\-case variance bound for normalized returns\.

Finally, RTPO computes the turn\-level advantage by subtracting the sibling baseline:

Aj,kH=Q^j,k−V^k,V^k=1G−1​∑r=1G−1Q^r,k\.A^\{H\}\_\{j,k\}=\\hat\{Q\}\_\{j,k\}\-\\hat\{V\}\_\{k\},\\qquad\\hat\{V\}\_\{k\}=\\frac\{1\}\{G\-1\}\\sum\_\{r=1\}^\{G\-1\}\\hat\{Q\}\_\{r,k\}\.\(97\)SinceAj,kHA^\{H\}\_\{j,k\}is a centered function of the sibling Q\-value estimates, reducing the Monte Carlo noise inQ^j,k\\hat\{Q\}\_\{j,k\}reduces the noise entering the advantage estimator\. Consequently, as reverse\-order training improves downstream policies and the on\-policy continuation values become more confident, the signal\-to\-noise ratio ofAj,kHA^\{H\}\_\{j,k\}improves\. This proves part \(b\) of Theorem 3, and completes the proof of Theorem[3](https://arxiv.org/html/2608.18682#Thmtheorem3)\. ∎

#### Comparison with stale trajectory\-level IS correction\.

The drift\-free property above should be contrasted with a stale\-rollout alternative that reuses continuations generated underπθ0\\pi\_\{\\theta\_\{0\}\}and then applies the trajectory\-level IS weightωjold\\omega\_\{j\}^\{\\mathrm\{old\}\}in Eq\. \([81](https://arxiv.org/html/2608.18682#A3.E81)\)\. Let

Nj=∑h=k\+1K−1Tj,hN\_\{j\}=\\sum\_\{h=k\+1\}^\{K\-1\}T\_\{j,h\}\(98\)denote the number of tokens in the continuation from turnkkto termination\. If the per\-token divergence between the current policy and the stale rollout policy is nonzero along this continuation, the variance of the full product IS weight can grow multiplicatively withNjN\_\{j\}\. To see this, suppose that for each token position\(h,t\)\(h,t\), the conditional second moment of the token ratio

ρj,h,told=πθ\>k​\(aj,h,t∣cj,h,aj,h,<t\)πθ0​\(aj,h,t∣cj,h,aj,h,<t\)\\rho\_\{j,h,t\}^\{\\mathrm\{old\}\}=\\frac\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}\{\\pi\_\{\\theta\_\{0\}\}\(a\_\{j,h,t\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\)\}\(99\)satisfies

𝔼πθ0\[\(ρj,h,told\)2∣cj,h,aj,h,<t\]≥1\+δfor someδ\>0\.\\mathbb\{E\}\_\{\\pi\_\{\\theta\_\{0\}\}\}\\\!\\left\[\\left\(\\rho\_\{j,h,t\}^\{\\mathrm\{old\}\}\\right\)^\{2\}\\mid c\_\{j,h\},a\_\{j,h,<t\}\\right\]\\geq 1\+\\delta\\qquad\\text\{for some \}\\delta\>0\.\(100\)Then the second moment of the product weight scales as

𝔼πθ0​\[\(ωjold\)2\]=𝔼πθ0​\[∏h=k\+1K−1∏t=1Tj,h\(ρj,h,told\)2\]≳\(1\+δ\)Nj,\\mathbb\{E\}\_\{\\pi\_\{\\theta\_\{0\}\}\}\\\!\\left\[\\left\(\\omega\_\{j\}^\{\\mathrm\{old\}\}\\right\)^\{2\}\\right\]=\\mathbb\{E\}\_\{\\pi\_\{\\theta\_\{0\}\}\}\\\!\\left\[\\prod\_\{h=k\+1\}^\{K\-1\}\\prod\_\{t=1\}^\{T\_\{j,h\}\}\\left\(\\rho\_\{j,h,t\}^\{\\mathrm\{old\}\}\\right\)^\{2\}\\right\]\\gtrsim\(1\+\\delta\)^\{N\_\{j\}\},\(101\)up to the usual conditioning on autoregressive histories\. Since𝔼πθ0​\[ωjold\]=1\\mathbb\{E\}\_\{\\pi\_\{\\theta\_\{0\}\}\}\[\\omega\_\{j\}^\{\\mathrm\{old\}\}\]=1, this implies

Varπθ0⁡\(ωjold\)=𝔼πθ0​\[\(ωjold\)2\]−1≳\(1\+δ\)Nj−1\.\\operatorname\{Var\}\_\{\\pi\_\{\\theta\_\{0\}\}\}\(\\omega\_\{j\}^\{\\mathrm\{old\}\}\)=\\mathbb\{E\}\_\{\\pi\_\{\\theta\_\{0\}\}\}\\\!\\left\[\\left\(\\omega\_\{j\}^\{\\mathrm\{old\}\}\\right\)^\{2\}\\right\]\-1\\gtrsim\(1\+\\delta\)^\{N\_\{j\}\}\-1\.\(102\)This illustrates the long\-horizon instability of trajectory\-level IS correction: even a small nonzero per\-token policy mismatch can compound into a high\-variance product over many tokens\. Per\-token PPO clipping does not remove this mismatch at the trajectory level, because clipping and multiplication do not commute:

∏h,tclip⁡\(ρj,h,told,1−ϵ,1\+ϵ\)≠clip⁡\(∏h,tρj,h,told,1−ϵ,1\+ϵ\)\.\\prod\_\{h,t\}\\operatorname\{clip\}\\\!\\left\(\\rho\_\{j,h,t\}^\{\\mathrm\{old\}\},1\-\\epsilon,1\+\\epsilon\\right\)\\neq\\operatorname\{clip\}\\\!\\left\(\\prod\_\{h,t\}\\rho\_\{j,h,t\}^\{\\mathrm\{old\}\},1\-\\epsilon,1\+\\epsilon\\right\)\.\(103\)Therefore, stale\-rollout correction remains fundamentally different from RTPO’s on\-policy sibling continuation, where the corresponding weight isωj≡1\\omega\_\{j\}\\equiv 1\.

#### Interpretation\.

Theorem 3 shows why RTPO regenerates sibling continuations on\-policy instead of reusing stale downstream rollouts\. In asynchronous multi\-turn training, stale continuations estimate values under outdated downstream policies and would require a long\-horizon trajectory\-level IS correction\. RTPO avoids this by synchronizingθ\>k\\theta\_\{\>k\}before sibling generation and rolling out each sibling to termination under the same current policy\. As a result, the full trajectory IS weight is one, the Q\-value estimate targets the current downstream policy, and the resulting turn\-level advantage avoids policy\-drift contamination\.

## Appendix DExperimental Setup

#### Base Models\.

We use Qwen3 models\[[58](https://arxiv.org/html/2608.18682#bib.bib8)\]as the backbone for RTPO and all baselines\. These models support context lengths of up to 32,768 tokens, which is sufficient to accommodate the multi\-turn interaction histories required by long\-horizon agentic RL tasks\. Since Qwen3 natively supports a thinking inference mode, we enable this mode consistently across all multi\-turn agentic RL experiments to ensure that the model generates complete tool\-integrated reasoning trajectories during rollout\. We use Qwen3\-8B as the main backbone because search\-based agentic tasks require strong base\-model capabilities for multi\-turn evidence acquisition, information integration, and tool\-augmented reasoning\. This choice is supported by our main results in Sec\.[4\.1](https://arxiv.org/html/2608.18682#S4.SS1), the rollout–training consistency analysis in Sec\.[4\.2](https://arxiv.org/html/2608.18682#S4.SS2), and the policy\-drift analysis in Sec\.[4\.4](https://arxiv.org/html/2608.18682#S4.SS4)\. To isolate the effect of turn\-level credit assignment in Sec\.[4\.3](https://arxiv.org/html/2608.18682#S4.SS3), we use the smaller Qwen3\-4B model for mathematical reasoning tasks\. This is because credit assignment in these tasks depends more directly on the model’s intrinsic reasoning ability than on external knowledge retrieval, making the gains from turn\-level credit assignment more discernible\.

#### Baselines\.

We compare RTPO with two categories of state\-of\-the\-art multi\-turn agentic RL methods\. The first category consists of trajectory\-only policy optimization methods, including GRPO\[[42](https://arxiv.org/html/2608.18682#bib.bib67)\]and\. GRPO uses token\-level importance ratios with group\-relative advantage estimation\. The second category includes turn\-level and tree\-based policy optimization methods, including ARPO\[[13](https://arxiv.org/html/2608.18682#bib.bib36)\], Tree\-GRPO\[[22](https://arxiv.org/html/2608.18682#bib.bib35)\], and SeeUPO\[[20](https://arxiv.org/html/2608.18682#bib.bib66)\]\. ARPO performs entropy\-driven adaptive branching at uncertain tool\-call nodes and estimates advantages separately for shared\-prefix and branch\-specific tokens\. Tree\-GRPO represents multi\-turn agent interaction as a tree and constructs group\-relative advantages at both intra\-tree and inter\-tree levels by sharing prefixes\. SeeUPO treats each turn as an independent agent and performs sequential per\-turn updates under a heterogeneous multi\-agent learning, with a theoretical guarantee of monotonic improvement\.

#### Datasets\.

To ensure a fair comparison across methods in multi\-turn agentic scenarios, we consider two representative tool\-use tasks: mathematical reasoning and knowledge reasoning from search\. Both require agents to interact with external tools, perform multi\-turn reasoning, and adapt their actions based on intermediate feedback\.

For the mathematical reasoning task, we use the MATH dataset\[[18](https://arxiv.org/html/2608.18682#bib.bib51)\]for training, which covers challenging multi\-step reasoning problems spanning algebra, geometry, number theory, probability, and other topics\. The task requires agents to decompose complex problems, invoke external Python computation tools when necessary, and integrate intermediate results into the final answer, where successful solutions often depend on iterative calculation, verification, and correction\. At the evaluation stage, we test generalization at two difficulty levels: \(1\) standard mathematical reasoning benchmarks, including GSM8K\[[10](https://arxiv.org/html/2608.18682#bib.bib50)\]and MATH\-500\[[28](https://arxiv.org/html/2608.18682#bib.bib56)\]; and \(2\) competition\-level benchmarks, including AMC’23\[[33](https://arxiv.org/html/2608.18682#bib.bib54)\], AIME’24\[[34](https://arxiv.org/html/2608.18682#bib.bib52)\], AIME’25\[[35](https://arxiv.org/html/2608.18682#bib.bib53)\], and OE\-Math\[[17](https://arxiv.org/html/2608.18682#bib.bib55)\], which feature problems that demand extended chains of tool\-augmented reasoning and advanced problem\-solving capabilities\. Since no training data are available for the competition\-level benchmarks, all evaluations are conducted in a zero\-shot setting\.

For the knowledge reasoning task on web search, we adopt the hard\-search training set constructed by ARPO\[[13](https://arxiv.org/html/2608.18682#bib.bib36)\], consisting of 1,000 high\-difficulty search samples drawn from two open\-source deep\-search data sources: SimpleDeepSearcher\[[48](https://arxiv.org/html/2608.18682#bib.bib57)\]and WebSailor\[[26](https://arxiv.org/html/2608.18682#bib.bib58)\]\. These samples require extensive web retrieval, multi\-source evidence integration, long\-context reasoning, and frequent tool calls, providing a rigorous testbed for evaluating the model stability and sample efficiency of multi\-turn agentic RL\. On this task, we evaluate knowledge\-intensive multi\-hop question answering on HotpotQA\[[61](https://arxiv.org/html/2608.18682#bib.bib64)\]and 2WikiMultiHopQA\[[19](https://arxiv.org/html/2608.18682#bib.bib63)\], following the ARPO evaluation protocol\[[13](https://arxiv.org/html/2608.18682#bib.bib36)\]with LLM\-as\-Judge scoring based on Qwen2\.5\-72B\-Instruct and report F1 scores\. In addition, to examine the effect of on\-policy continuation, we compare default RTPO with its off\-policy variant, and conduct additional evaluation on four challenging deep\-search benchmarks: GAIA\[[36](https://arxiv.org/html/2608.18682#bib.bib62)\], which evaluates general AI\-assistant capabilities across three difficulty levels \(Lv\.1–Lv\.3\); WebWalkerQA\[[55](https://arxiv.org/html/2608.18682#bib.bib61)\], which focuses on interactive web navigation and multi\-hop question answering; Humanity’s Last Exam\[[37](https://arxiv.org/html/2608.18682#bib.bib60)\], which covers expert\-level problems in sciences, engineering, and humanities; and xBench\[[8](https://arxiv.org/html/2608.18682#bib.bib59)\], which evaluates cross\-lingual deep\-search capability\. Since the training data are drawn exclusively from the hard\-search source, all four benchmarks serve as held\-out evaluation sets\. We report Pass@1 with sampling temperature set to0\.60\.6and top\-ppset to0\.950\.95\.

#### Configurations\.

All methods are implemented on top of the VeRL framework\[[43](https://arxiv.org/html/2608.18682#bib.bib47)\], with vLLM\[[24](https://arxiv.org/html/2608.18682#bib.bib48)\]for rollout generation and FSDP\[[66](https://arxiv.org/html/2608.18682#bib.bib49)\]for post\-training\. All experiments are conducted on8×8\\timesNVIDIA A100 GPUs\. Unless otherwise specified, the training batch size is6464and the maximum single\-turn response length is40964096tokens\. The maximum interaction horizon is set according to task type:K=3K=3turns for mathematical reasoning andK=6K=6turns for web search\. The learning rate is fixed at1×10−61\\times 10^\{\-6\}with the AdamW\[[31](https://arxiv.org/html/2608.18682#bib.bib65)\]optimizer and a weight decay of0\.010\.01\. For branching\-based baselines \(ARPO and TreeGRPO\), we follow the hyperparameters recommended in their original papers, including an entropy threshold of0\.40\.4and an initial sampling size of88\. Full hyperparameter configurations are provided in Appendix[E\.3](https://arxiv.org/html/2608.18682#A5.SS3)\.

#### Evaluation Metrics\.

We evaluate RTPO and all baselines along multiple dimensions\. For overall performance, we use Pass@1 accuracy as the primary indicator of final policy quality after optimization \(Sec[4\.1](https://arxiv.org/html/2608.18682#S4.SS1)\)\. We also measure the average number of tool calls to assess behavioral differences across methods under a fixed compute budget\. For training stability, rollout–training consistency is evaluated using the log\-probability ratio and KL divergence, as described in Sec\.[4\.2](https://arxiv.org/html/2608.18682#S4.SS2)\. We further conduct controlled experiments to examine whether RTPO’s turn\-level credit assignment \(Sec[4\.3](https://arxiv.org/html/2608.18682#S4.SS3)\) and on\-policy continuation \(Sec[4\.4](https://arxiv.org/html/2608.18682#S4.SS4)\) are consistent with the theoretical analysis\. Each experiment is repeated three times to mitigate randomness, and we report the average performance\.

## Appendix EImplementation Details

### E\.1Pseudocode

The pseudocode for our proposed RTPO is shown in Algorithm[1](https://arxiv.org/html/2608.18682#alg1)\.

Algorithm 1Reverse\-Turn Policy Optimization \(RTPO\)1:Query dataset

𝒟\\mathcal\{D\}; policy

πθ\\pi\_\{\\theta\}; inference engine

ℰ\\mathcal\{E\}; sibling count

GG; PPO clip

ϵ\\epsilon; max turns

KK; observation map

ψ\\psi; PPO epochs

EE\.

2:Updated parameters

θ\\theta\.

3:Phase 1: Trunk Rollout

4:For each query

q∈𝒟q\\\!\\in\\\!\\mathcal\{D\}, generate a complete trajectory

\(S0,u0,f0,…,uK−1,fK−1\)\(S\_\{0\},u\_\{0\},f\_\{0\},\\ldots,u\_\{K\-1\},f\_\{K\-1\}\)under

πθ\\pi\_\{\\theta\}by interacting with the environment\. Record boundary states and environment snapshots

\{Sk,snapk\}k=0K−1\\\{S\_\{k\},\\mathrm\{snap\}\_\{k\}\\\}\_\{k=0\}^\{K\-1\}, and set conditioning contexts

ck=ψ⁡\(Sk\)c\_\{k\}=\\psi\(S\_\{k\}\)\. The trunk provides anchors only and receives no gradient\.

5:Phase 2: Reverse\-Order Training

6:for

k=K−1,K−2,…,0k=K\{\-\}1,\\;K\{\-\}2,\\;\\ldots,\\;0do

7:Synchronize\.Push current parametersθ\\theta\(denotedθ\>k\\theta\_\{\>k\}, reflecting downstream turns already optimized\) toℰ\\mathcal\{E\}; freezeθold←θ\>k\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\theta\_\{\>k\}as the IS denominator\.

8:Sibling generation \(on\-policy\)\.For eachq∈𝒟q\\\!\\in\\\!\\mathcal\{D\}and each siblingj=1,…,G−1j=1,\\ldots,G\{\-\}1: restore environment fromsnapk\\mathrm\{snap\}\_\{k\}; sample turn\-kkresponseuj,k∼πθ\>k\(⋅∣ck\)u\_\{j,k\}\\sim\\pi\_\{\\theta\_\{\>k\}\}\(\\cdot\\mid c\_\{k\}\); continue underπθ\>k\\pi\_\{\\theta\_\{\>k\}\}through turnsk\+1,…,K−1k\{\+\}1,\\ldots,K\{\-\}1to terminal; receive rewardRj∈\{0,1\}R\_\{j\}\\in\\\{0,1\\\}\. Record turn\-kktokens\{aj,k,t\}t=1Tj,k\\\{a\_\{j,k,t\}\\\}\_\{t=1\}^\{T\_\{j,k\}\}and their log\-probs underθold\\theta\_\{\\mathrm\{old\}\}\.

9:Turn\-level advantage\.For eachq∈𝒟q\\\!\\in\\\!\\mathcal\{D\}, compute the sibling baseline and advantage:V^k=1G−1​∑j=1G−1Rj,Aj,kH=Rj−V^k\.\\hat\{V\}\_\{k\}=\\frac\{1\}\{G\{\-\}1\}\\sum\_\{j=1\}^\{G\-1\}R\_\{j\},\\qquad A\_\{j,k\}^\{H\}=R\_\{j\}\-\\hat\{V\}\_\{k\}\.All siblings shareSkS\_\{k\}, soAj,kHA\_\{j,k\}^\{H\}is free of upstream state contamination \(Theorem[2](https://arxiv.org/html/2608.18682#Thmtheorem2)\)\.

10:PPO update \(turn\-kktokens only\)\.

11:forepoch

=1,…,E=1,\\ldots,Edo

12:foreach mini\-batch from sibling turn\-

kktokensdo

13:Compute IS ratios

ρj,k,t=πθ​\(aj,k,t∣ck,aj,k,<t\)/πθold​\(aj,k,t∣ck,aj,k,<t\)\\rho\_\{j,k,t\}=\\pi\_\{\\theta\}\(a\_\{j,k,t\}\\mid c\_\{k\},a\_\{j,k,<t\}\)\\,/\\,\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{j,k,t\}\\mid c\_\{k\},a\_\{j,k,<t\}\)\.

14:Update

θ\\thetavia the clipped objective:

Jk​\(θ\)=1G−1​∑j=1G−11Tj,k​∑t=1Tj,kmin⁡\(ρj,k,t​Aj,kH,clip⁡\(ρj,k,t,1−ϵ,1\+ϵ\)​Aj,kH\)\.J\_\{k\}\(\\theta\)=\\frac\{1\}\{G\{\-\}1\}\\sum\_\{j=1\}^\{G\-1\}\\frac\{1\}\{T\_\{j,k\}\}\\sum\_\{t=1\}^\{T\_\{j,k\}\}\\min\\\!\\Big\(\\rho\_\{j,k,t\}\\,A\_\{j,k\}^\{H\},\\;\\mathrm\{clip\}\\bigl\(\\rho\_\{j,k,t\},\\,1\{\-\}\\epsilon,\\,1\{\+\}\\epsilon\\bigr\)\\,A\_\{j,k\}^\{H\}\\Big\)\.
15:endfor

16:endfor

17:

⊳\\trianglerightTurnkkcomplete;πθ,k\\pi\_\{\\theta,k\}frozen; next turn inherits updatedθ\\theta\.

18:endfor

### E\.2Source Code

We provide the source code of RTPO in theSupplementary Material\. The repository includes complete installation instructions and versioned dependency requirements\. Our implementation is built uponVeRL\[[43](https://arxiv.org/html/2608.18682#bib.bib47)\], whose core additions are: \(1\) summarizing the output of each round, so that the next round can rollout without having access to the complete history, \(2\) on\-policy tree rollouts, and \(3\) computation of per\-turn advantage and reverse updates\. We also provide out\-of\-the\-box training and evaluation scripts to reproduce our main results on mathematical and knowledge reasoning tasks\.

### E\.3Training Details

We provide the detailed hyperparameters for all experiments in Table[4](https://arxiv.org/html/2608.18682#A5.T4)\. Unless otherwise noted, the maximum prompt/response lengths are kept the same across all experiments\. For computational fairness, the global sampling budget is set to80​k80\\text\{k\}rollouts for mathematical reasoning \(5​k5\\text\{k\}prompts×\\times1616rollouts per prompt\) and16​k16\\text\{k\}for knowledge reasoning \(1​k1\\text\{k\}prompts×\\times1616rollouts per prompt\)\. For the Qwen3 series, we use a sampling temperature of0\.90\.9, the AdamW optimizer with a learning rate of1×10−61\\times 10^\{\-6\}, weight decay0\.010\.01, and an initial KL coefficient of0\.020\.02\. The log\-probability clipping range is set to\[−10,10\]\[\-10,10\]\. The training batch size and the rollout batch size are both set to6464\. The maximum prompt length is81928192tokens and the maximum single\-turn response length is40964096tokens for both tasks\. The maximum interaction horizon is set toK=3K=3turns for mathematical reasoning andK=6K=6turns for web search\. All methods are allocated a uniform budget of1616rollouts per prompt \(see Table[4](https://arxiv.org/html/2608.18682#A5.T4)\)\.

Table 4:Hyperparameter settings for all experiments\.For GRPO, we sample1616independent chains per prompt\. For SeeUPO, we generate1616independent chains with sequential turn\-level updates in reverse execution order\. For ARPO, we set the initial sampling size toN=8N=8and the global rollout budget toM=16M=16, with entropy weightβ=0\.2\\beta=0\.2, base probabilityα=0\.5\\alpha=0\.5, and branching thresholdτ=0\.5\\tau=0\.5; rollouts that do not trigger entropy\-driven branching are supplemented with independent chains until the budget of1616is reached\. For TreeGRPO, we set the number of initial trees toM=4M=4, the number of expansion nodes per iteration toN=3N=3, and the number of expansion iterations toL=1L=1, yieldingM×\(L×N\+1\)=16M\\times\(L\\times N\+1\)=16rollouts per prompt via random node expansion\. For RTPO, we first generate multiple trunk trajectories per prompt to establish turn\-boundary states:44trunks for mathematical reasoning \(K=3K\{=\}3\) and22trunks for web search \(K=6K\{=\}6\), since in practice most trunks do not reach the maximum interaction horizon and terminate early with fewer tool calls\. At each realized turn boundarySkS\_\{k\},G−1=2G\{\-\}1=2sibling rollouts are forked and continued to termination\.

To ensure computational fairness with baselines, RTPO enforces the same global sampling budget as all other methods \(80​k80\\text\{k\}for mathematical reasoning,16​k16\\text\{k\}for knowledge reasoning\), so that the total number of generated rollouts across all prompts does not exceed that of any baseline\. Per\-prompt rollout counts may vary depending on early termination, but the global budget constraint guarantees that RTPO consumes no more rollout compute than the1616\-chain baselines in aggregate\. All methods use a discount factor ofγ=1\\gamma=1\.

All experiments are conducted on a single node equipped with8×8\\timesNVIDIA A100 80GB GPUs,256256CPU cores, and256256GB of system memory, running Ubuntu 22\.04 LTS\. The math task requires approximately 15 hours of training, whereas the knowledge task requires approximately 5 hours\.

### E\.4Qwen3 Chat Template

![Refer to caption](https://arxiv.org/html/2608.18682v1/chat_template.png)Figure 4:Examples of Qwen3 chat template\.At each interaction turn, Qwen3 receives the current context and generates three components: \(A\) an internal reasoning trace enclosed by<think\>\.\.\.</think\>, \(B\) textual commentary after</think\>, and \(C\) a tool\-call instruction, e\.g\.,<tool\_call\>\.\.\.</tool\_call\>\. The tool call is executed by the external environment, which returns the result as an observation\. Before constructing the next\-turn input, the chat\-template filter removes the<think\>\.\.\.</think\>segment, while retaining the textual commentary, the tool\-call instruction, and the appended tool\-execution result\. Thus, the input to the second turn consists of: \(1\) the original user request, \(2\) the previous textual commentary, \(3\) the previous tool call, and \(4\) the tool result\. This filtered context enables the model to condition on the action history and environment feedback without carrying redundant internal reasoning traces, keeping the context length manageable while preserving decision\-relevant information\. The process is illustrated in Figure[4](https://arxiv.org/html/2608.18682#A5.F4)\.

## Appendix FAdditional Results and Insights

### F\.1Discussion of Main Results

For the main experiments, we use Qwen3\-8B as the base model and evaluate on two categories of multi\-turn tool\-use tasks: mathematical reasoning and web search\. Qwen3 models automatically produce chain\-of\-thought content wrapped in<think\>\.\.\.</think\>tags during inference\. In our setup, each turn’s input is constructed using the standard Qwen3 chat template, which post\-processes prior assistant content by stripping out the<think\>\.\.\.</think\>segments so that subsequent turns observe only the final answer content following</think\>\. This design keeps the per\-turn input context concise and aligns naturally with the turn\-boundary MDP formulation underlying RTPO: each turn\-level policy is conditioned on the visible contextckc\_\{k\}rather than on the full raw interaction history\. In addition, we report the complete accuracy results with mean and standard deviation, extending Table[1](https://arxiv.org/html/2608.18682#S4.T1)to Table[5](https://arxiv.org/html/2608.18682#A6.T5)\.

Table 5:Accuracy comparison across eight benchmarks\. Values are reported as mean±stdover three evaluation runs\.Accdenotes Pass@1 \(%\) for mathematical tasks and best\-span F1 \(%\) for knowledge reasoning tasks\. Standard deviations are reported in percentage points\.#### Performance and tool\-use patterns on mathematical reasoning tasks\.

On mathematical tasks, tool\-call frequency exhibits a trend that runs almost opposite to accuracy: Vanilla makes 17 Python calls on AIME24 yet solves only one problem, whereas RTPO solves ten with just four calls\. This pattern reflects an intrinsic property of mathematical reasoning: the knowledge and derivations required to solve a math problem are primarily internalized in the model’s parameters, while the external Python tool serves as an auxiliary aid for verification and numerical computation rather than as a source of new information\. Vanilla’s frequent tool use, therefore, largely reflects an inefficient strategy of*repeated verification used to mask reasoning uncertainty*\. After RL training, the model gains stronger control over its own reasoning process, and tool calls collapse from redundant repeated verification into precise invocations at critical computation steps; consequently, the number of calls decreases while accuracy increases\. RTPO’s advantage becomes particularly pronounced on the hardest math benchmarks: it reaches 33\.33% Pass@1 on AIME24, surpassing SeeUPO \(20\.00%\) and GRPO \(10\.00%\) by 13\.3 and 23\.3 absolute points, respectively, and maintains a clear lead on AIME25 and OE\-Math\. RTPO simultaneously achieves the fewest tool calls and the highest accuracy on these hard problems, indicating that it learns the most efficient tool\-use strategy\. We further note that the gap between methods on mathematical tasks remains relatively small overall, since math problems typically require only a few interaction turns, and the context discrepancy between rollout and training stays at a manageable scale under short horizons; the robustness of standard training dynamics alone is sufficient for baselines to reach near\-optimal performance in this regime\. We provide an example to support our discussion, as shown in Figure[5](https://arxiv.org/html/2608.18682#A6.F5)\.

![Refer to caption](https://arxiv.org/html/2608.18682v1/mathtaskcase.drawio.png)Figure 5:An example of different policy performance on a mathematical\-reasoning task through Python tools\.
#### Performance and tool\-use patterns on knowledge reasoning tasks\.

On knowledge\-intensive question answering, RTPO reaches 64\.89 F1 on HotpotQA, exceeding SeeUPO \(54\.27\) and GRPO \(53\.39\) by 10\.6 and 11\.5 points, respectively, and maintains a clear lead on 2Wiki\. In contrast to mathematical tasks, the tool\-use pattern on knowledge reasoning exhibits a positive correlation between call count and accuracy: RTPO issues 613 and 867 search calls on HotpotQA and 2Wiki, far exceeding the roughly 200 calls observed for the other methods\. This contrast admits a natural explanation grounded in task structure: the factual information required to answer such questions does not reside in the model’s parametric knowledge and must be acquired through external retrieval, while multi\-hop questions further demand cross\-turn integration of multiple pieces of evidence\. The lower retrieval counts of GRPO and SeeUPO indicate that they fail to learn to issue sustained follow\-up queries and to expand retrieval across multiple turns\. The underlying cause is that such hard tasks require longer interaction horizons and more frequent reasoning revision, so the rollout–training context mismatch is amplified as turns accumulate, and baseline methods must spend a larger share of their optimization budget compensating for this drift\. RTPO eliminates this source of bias structurally, allowing the entire optimization budget to act directly on the task objective, which yields more stable improvements on long\-horizon hard tasks\. We provide an example to support our discussion, as shown in Figure[6](https://arxiv.org/html/2608.18682#A6.F6)\.

![Refer to caption](https://arxiv.org/html/2608.18682v1/searchtaskcase_cropped.png)Figure 6:An example of different policy performance on a knowledge\-reasoning task through web search tools\.Taking both task categories together, RTPO learns to perform mathematical reasoning with*fewer calls but more reliable internal reasoning*, while performing knowledge reasoning with*denser calls and more thorough external information integration*\. This bidirectional adaptation of tool\-use behavior indicates that the advantage of RTPO does not stem from simply encouraging or suppressing tool calls, but rather from its turn\-level optimization objective, which is able to learn a tool\-use strategy matched to the underlying nature of each task\.

### F\.2Insights from Rollout–Training Consistency Analysis

Follow\-up discussion from Sec[4\.2](https://arxiv.org/html/2608.18682#S4.SS2), a noteworthy observation is that, despite using different conditioning contexts at the rollout and training stages, the baselines’ ratios still exhibit a slow drift toward11\. We find that this phenomenon is closely related to the mechanism revealed by DAgger\[[40](https://arxiv.org/html/2608.18682#bib.bib9)\]: when the training data is continually drawn from the distribution induced by the policy itself, the model gradually adapts to the distribution it actually operates on\. In our setting, trajectories generated by summary\-rollouts are reinforced under full\-history\-training, and parameter sharing causes the behaviors under the two conditionings to converge indirectly; the next rollout therefore falls closer to the region considered reasonable under the training context\. SCoRe\[[23](https://arxiv.org/html/2608.18682#bib.bib10)\]observes the same bootstrapped alignment process in multi\-turn online RL\. In addition, some works point out that the summary can serve as a learnable sufficient statistic: if the summary preserves the decision\-relevant information, the optimal policies under the two contexts can converge in an information\-theoretic sense\[[2](https://arxiv.org/html/2608.18682#bib.bib11)\]; recent work on multi\-turn RL further shows that the summary context can evolve into a learnable compact decision state under end\-to\-end optimization\[[32](https://arxiv.org/html/2608.18682#bib.bib12),[63](https://arxiv.org/html/2608.18682#bib.bib37)\]\.

However, this empirical alignment differs from the structural consistency of RTPO in three fundamental ways:

\(1\) Alignment is incomplete and strongly depends on task complexity\. GRPO recovers only to0\.970\.97on math after3030steps and only to0\.830\.83on search after1414steps; the longer the horizon and the faster the context accumulates, the harder the alignment becomes, which is reflected in the knowledge reasoning task in Table[1](https://arxiv.org/html/2608.18682#S4.T1)\.

\(2\)The alignment process is accompanied by oscillation\. SeeUPO produces a1\.021\.02spike at step99on math and exhibits sustained small fluctuations, because the bootstrapped loop is driven by the advantage signal, and the variance of advantage estimates propagates directly into step\-to\-step jitter of the ratio\. RTPO’s ratio is structurally guaranteed and is only affected by differences between the training and inference engines\.

\(3\) Empirical alignment consumes additional optimization budget\. Bootstrapped alignment essentially allocates part of the policy’s capacity to an implicit objective—pulling the behavior under the summary toward the optimal behavior under the full history\. RTPO removes this hidden cost, so that the optimization budget can act directly on the task objective\. This is consistent with the lower search\-call counts of baselines on long\-horizon knowledge reasoning observed in Table[1](https://arxiv.org/html/2608.18682#S4.T1): part of their training dynamics is diverted topatching the mismatch\.

Overall, the rollout–training ratio reveals not that baselines necessarily fail, but that baselines must rely on training dynamics to compensate for a gap that RTPO does not have by construction, and this compensatory mechanism becomes substantially less effective on long\-horizon tasks\.

### F\.3Additional Findings for Hard\-Search Scenarios

Table 6:Rechecked output\-hit comparison between standard RTPO \(on\-policy\) and its off\-policy variant\. Values are rounded to the nearest integer for better readability\. Output\-hit marks a sample correct if the output contains the gold answer or an alias\.To quantify the independent contribution of on\-policy continuation \(Sec\.[3\.3](https://arxiv.org/html/2608.18682#S3.SS3)\), we use Qwen3\-8B as the base model and compare standard RTPO with its off\-policy variant\. Both share the same reverse\-order turn\-level training and state\-matched sibling structure; the only difference lies in how sibling downstream continuations are obtained\. RTPO \(on\-policy\) synchronizes the latest parametersθ\>k\\theta\_\{\>k\}to the inference engine at the start of turnkkand regenerates sibling continuations underπθ\>k\\pi\_\{\\theta\_\{\>k\}\}until termination, so the trajectory\-level IS weight is identically one, and the Q\-value estimate is

Q^j,kon=rj,k\+γτj,k​F^j,kπθ\>k\.\\hat\{Q\}^\{\\text\{on\}\}\_\{j,k\}=r\_\{j,k\}\+\\gamma^\{\\tau\_\{j,k\}\}\\,\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\.\(104\)RTPO with off\-policy variant instead reuses the downstream continuations already generated underπθ0\\pi\_\{\\theta\_\{0\}\}during the initial rollout stage and corrects for the staleness via a clamped trajectory\-level importance weight:

Q^j,koff=rj,k\+γτj,k​ω¯j,k​F^j,kπθ0,ω¯j,k=clamp⁡\(∏h=k\+1K−1∏t=1Tj,hπθ\>k​\(aj,h,t∣sj,h,t\)πθ0​\(aj,h,t∣sj,h,t\),ρmin,ρmax\)\.\\hat\{Q\}^\{\\text\{off\}\}\_\{j,k\}=r\_\{j,k\}\+\\gamma^\{\\tau\_\{j,k\}\}\\,\\bar\{\\omega\}\_\{j,k\}\\,\\hat\{F\}\_\{j,k\}^\{\\pi\_\{\\theta\_\{0\}\}\},\\qquad\\bar\{\\omega\}\_\{j,k\}=\\mathrm\{clamp\}\\\!\\Bigg\(\\prod\_\{h=k\+1\}^\{K\-1\}\\prod\_\{t=1\}^\{T\_\{j,h\}\}\\frac\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid s\_\{j,h,t\}\)\}\{\\pi\_\{\\theta\_\{0\}\}\(a\_\{j,h,t\}\\mid s\_\{j,h,t\}\)\},\\;\\rho\_\{\\min\},\\;\\rho\_\{\\max\}\\Bigg\)\.\(105\)We compare the on\-policy RTPO and its off\-policy variant on four knowledge\-reasoning deep\-search benchmarks, as shown in Table[6](https://arxiv.org/html/2608.18682#A6.T6)\. We use output\-hit accuracy, which marks a sample correct if the full output contains the gold answer or an alias\. The benchmarks include GAIA with three difficulty levels, WebWalkerQA, XBench, and HLE with eight subject subsets\.

### F\.4Limitations and Future Work

Although RTPO provides stronger theoretical guarantees and empirical performance than existing flat\-trajectory methods, its algorithmic design has several limitations that motivate future work\.

#### Dependence on trunk quality\.

RTPO uses the boundary statesSkS\_\{k\}of a trunk trajectory as the forking points for sibling branches\. As a result, the training signal at each reverse phase is anchored to the state sequence visited by the trunk\. If the trunk makes a poor decision at an early turn, such ask=0k=0ork=1k=1, later boundary states may lie in low\-value regions where even strong turn\-level actions fail to obtain positive terminal rewards\. In this case, the turn\-level advantagesAj,kHA\_\{j,k\}^\{H\}may degenerate into near\-zero signals, making the corresponding gradient update ineffective\. In contrast, beam\-style search methods, such as beam search or best\-of\-NN, maintain multiple candidate prefixes and can discard low\-quality paths earlier\. RTPO currently relies on a single trunk anchor and does not explicitly incorporate trunk\-level diversity or post\-hoc trunk selection\. A natural extension is*multi\-trunk sampling*or*retroactive trunk selection*, where multiple complete trunks are generated during rollout and a high\-reward trunk is selected as the anchor to improve boundary\-state coverage\. This would increase rollout cost, but would not change the reverse\-order training formulation\.

#### Overhead of reverse multi\-turn training\.

RTPO decomposes aKK\-turn episode intoKKsequential turn\-level optimization phases\. Each phase requires sibling generation, environment restoration, on\-policy continuation to termination, and a PPO\-style update\. Compared with flat\-trajectory methods, which perform a single optimization pass over the full trajectory, RTPO incurs additional cost that scales with the number of turnsKKand the number of sibling rollouts\. Moreover, each reverse phase requires synchronizing the latest policy parameters to the inference engine before generating on\-policy continuations, introducing additional inference\-training latency\. In our VeRL\-based implementation, this overhead is partially mitigated by batched parallel sibling generation and asynchronous engine scheduling, but it cannot be fully removed\. Designing more efficient sibling generation and update schedules is therefore an important direction for future work\.

#### Under\-utilization of training tokens\.

RTPO assigns the turn\-level advantage in phasekkonly to the sibling’s turn\-kkoutput tokens\. Prefix tokens inckc\_\{k\}and downstream continuation tokens from turnsk\+1,…,K−1k\{\+\}1,\\ldots,K\{\-\}1receive no gradient\. In addition, the trunk trajectory is used only as an anchor and does not directly contribute to gradient updates\. Thus, although sibling continuations are necessary for estimating terminal returns, their downstream tokens are discarded during policy optimization\. For example, in aK=5K=5episode with an average of 200 tokens per turn, the sibling continuation at phasek=2k=2may generate around 600 downstream tokens, while only the 200 turn\-kktokens are used for the PPO update\. This reduced token utilization is the cost of causal turn\-level credit assignment: by withholding gradients from non\-turn\-kktokens, RTPO avoids assigning credit to actions that are not causally responsible for the turn\-kkcomparison, as stated inTheorem 2\(c\)\. Future work may explore auxiliary objectives, such as language\-modeling losses or self\-play rewards on downstream tokens, to improve token efficiency while preserving the causal consistency of the turn\-level advantage\.

### F\.5Broader Impacts

The potential positive impact of RTPO is that more stable agentic RL training can reduce failed tool\-use trajectories, improve sample efficiency, and support more reliable deployment of LLM agents in research, education, software engineering, and decision\-support settings\. RTPO may also make multi\-turn RL training easier to analyze by separating turn\-level decisions from full\-trajectory outcomes\. By enabling turn\-level monitoring, RTPO can further improve our understanding of how agentic workflows learn to plan, search, and use tools over multiple turns\. At the same time, stronger turn\-refined agentic workflows may increase the capability of LLM agents to act autonomously across long\-horizon tasks\. If deployed without appropriate safeguards, such systems could produce incorrect outputs with high confidence, misuse external tools, or amplify harmful automation\. Therefore, practical deployment should include safety constraints, tool\-use monitoring, privacy\-preserving data handling, and human oversight, especially in high\-stakes domains\.

## Appendix GSupplementary Theoretical Clarifications, Implementation Details, and Extended Experiments

### G\.1Stateful Tool\-Agent Evaluation onτ3\\tau^\{3\}\-Airline

#### Experimental setting\.

We evaluate Qwen3\-1\.7B on a fixed set of 30 training tasks and the 20 held\-out test tasks provided by theτ3\\tau^\{3\}\-Airline environment\. The agent must query and modify an airline database through multi\-turn tool interactions, and earlier actions change the state observed in later turns\.

Table 7:Online training success rate onτ3\\tau^\{3\}\-Airline\.RTPO obtains the highest online success rate from Step 20 onward and peaks at 56\.25% at Step 40\. The reduction to 40\.63% at Step 50 indicates that continued optimization can produce late\-stage degradation when the training set is small and the reward is sparse\.

Table 8:Held\-out evaluation on the 20τ3\\tau^\{3\}\-Airline test tasks\. Pass4follows the definition used in the main paper\.RTPO achieves the highest Pass@1 and Pass@4 and the highest normal environment\-termination rate\. Relative to Tree\-GRPO, it reduces the generation\-truncation rate from 47\.50% to 42\.50% and the average response length from 2,538\.2 to 1,688\.8 tokens\. Relative to REFUEL, RTPO improves Pass@1 from 11\.25% to 17\.50%, improves Pass@4 from 20\.00% to 30\.00%, and reduces average response length by approximately 35\.6%\.

Table 9:Tool\-execution quality on the held\-outτ3\\tau^\{3\}\-Airline tasks\.RTPO attempts fewer tool calls than GRPO but completes more calls without execution errors\. Tool\-error calls decrease from 50 to 9, and the execution success rate increases from 57\.63% to 89\.16%\. RTPO also requires approximately 9,650\.5 response tokens per successful trajectory, which is 38\.2% lower than Tree\-GRPO and 56\.0% lower than GRPO\.

### G\.2Sensitivity to the Sibling Group Size

The group\-size hyperparameter isGG; at each boundary, RTPO forksG−1G\-1sibling continuations from the same turn\-boundary stateSkS\_\{k\}\. The turn\-level estimator is

A^j,kH=Rj−1G−1​∑r=1G−1Rr\.\\widehat\{A\}^\{H\}\_\{j,k\}=R\_\{j\}\-\\frac\{1\}\{G\-1\}\\sum\_\{r=1\}^\{G\-1\}R\_\{r\}\.\(106\)Theorem[2](https://arxiv.org/html/2608.18682#Thmtheorem2)\(a\) gives

𝔼\[A^j,kH∣Sk,uj,k\]=G−2G−1Aπ\(Sk,uj,k\)\.\\mathbb\{E\}\\\!\\left\[\\widehat\{A\}^\{H\}\_\{j,k\}\\mid S\_\{k\},u\_\{j,k\}\\right\]=\\frac\{G\-2\}\{G\-1\}A^\{\\pi\}\(S\_\{k\},u\_\{j,k\}\)\.\(107\)Thus, the finite\-group bias is of orderO⁡\(1/G\)O\(1/G\), and the multiplicative coefficient approaches one asGGincreases\. Averaging more sibling returns also reduces sampling noise in the Monte Carlo baseline\. Because all continuations begin from the same boundary state, this comparison does not reintroduce upstream\-state contamination\.

Table 10:Effect of sibling group size on signal density and held\-out performance inτ3\\tau^\{3\}\-Airline\.IncreasingGGfrom 3 to 4 reduces the zero\-advantage rate from 89\.61% to 82\.66%, improves Pass@1 from 17\.50% to 20\.00%, and reduces inference tokens per successful trajectory from 9,650\.5 to 7,477\.9\. The larger group therefore provides denser relative learning signals, but requires more offline exploration\.

Table 11:Performance–cost trade\-off for sibling group size\.Moving fromG=3G=3toG=4G=4increases GPU\-hours by approximately 30\.6% and training\-generation tokens by approximately 152\.8%, while improving Pass@1 by 2\.5 percentage points and leaving Pass@4 unchanged\. The resulting policy uses approximately 36\.2% fewer tool calls and 11\.4% fewer response tokens at inference time\. Consequently,G=3G=3provides the stronger default cost–performance trade\-off, whereasG=4G=4is useful when Pass@1 and concise inference behavior are prioritized\.

ForG=2G=2, the coefficient in Eq\. \([107](https://arxiv.org/html/2608.18682#A7.E107)\) is zero, so the conditional expectation of the relative advantage degenerates to zero\. Therefore,G=3G=3, corresponding to two sibling continuations, is the minimum viable configuration that preserves an informative relative signal\.

### G\.3Trunk Quality and Failure Dynamics

Table 12:Aggregate trunk outcomes in theτ3\\tau^\{3\}\-Airline training run\.Only 6\.88% of trunks fail within the first turn, indicating that catastrophic early failure is not the dominant failure mode\. Most errors occur in the middle or later stages\. Moreover, an observed early failure under finite sampling is not equivalent to a strict dead state, because finite continuations cannot establish that every possible future policy is unable to recover\.

Table 13:Evolution of failure modes during RTPO training\.The fraction of trajectories without a terminal environment signal decreases from 37\.50% at Step 10 to 0% at Step 50, while the average failed\-trajectory length decreases from 13\.30 to 2\.84 turns\. RTPO therefore substantially reduces trajectories that stall for a long time or fail to complete the interaction\. Later failures increasingly take the form of fast but incorrect termination rather than persistent generation until truncation\. The drop in success after Step 40 may reflect over\-optimization on a limited task set under sparse binary rewards\.

The implementation does not use heuristic trunk filtering\. All trunks are sampled on\-policy from the current model\. Filtering low\-quality trunks could reduce sibling\-sampling cost, but would alter the actually visited state distribution and concentrate optimization on manually selected states\. Instead, RTPO retains all trunks and uses same\-state sibling returns to determine whether a boundary supplies an informative relative signal\.

### G\.4Selective\-Gradient Optimization and Training Efficiency

#### Why the loss is restricted to current\-turn tokens\.

RTPO applies the policy loss only to the output tokens of the current\-turn siblings\. Prefix tokens and the trunk trajectory define the state and conditioning context that were actually reached, while the downstream continuation supplies the return used to evaluate the current action\. Assigning the same turn\-level advantage to prefix or downstream\-continuation tokens would reintroduce the trajectory\-level credit contamination analyzed in Theorem[2](https://arxiv.org/html/2608.18682#Thmtheorem2)\. Although only one turn receives gradients at a particular reverse stage, every turn is optimized when it becomes the current turn during the complete reverse sweep\. The selective loss is therefore intended to isolate causal credit rather than to discard particular turns from training\.

Table 14:Training\-generation cost and deployment\-time response efficiency onτ3\\tau^\{3\}\-Airline\.RTPO generates approximately 8\.3% more training tokens than Tree\-GRPO and 40\.0% more than GRPO, while attaining the highest Pass@1\. Relative to Tree\-GRPO, it reduces response tokens per successful trajectory by 38\.2%; relative to GRPO, the reduction is approximately 56\.0%\. The additional offline continuation sampling therefore does not translate into more verbose deployment\-time inference\.

#### Wall\-clock and GPU\-hour overhead\.

Under the same Qwen3\-1\.7B model, four A100 GPUs, and 160 training trunks, the measured cost is:

Table 15:End\-to\-end training cost under the same hardware\.RTPO increases wall\-clock time and GPU\-hours by approximately 41\.3%, and training\-generation tokens by approximately 40\.0%\. The additional cost comes primarily from sibling\-continuation generation, turn\-by\-turn reverse updates, and synchronization of the latest policy between reverse stages\. RTPO thus trades higher training\-time computation for state\-matched credit assignment and current\-policy downstream continuations\.

#### Compute\-matched comparison\.

The following comparison uses approximately the same total GPU\-hour budget\. It is distinct from the fully trained RTPO result at 21\.48 GPU\-hours\.

Table 16:GRPO and compute\-matched RTPO under approximately equal training cost\.With only approximately 2% more GPU\-hours, RTPO matches GRPO on Pass@1, improves Pass@4 from 25\.00% to 35\.00%, and increases environment completion from 20\.00% to 30\.00%\. It also reduces truncation, total tool calls, and average response length while improving tool\-execution success\.

#### Matched maximum inference budget\.

All methods below use the same limits of 20 turns, 2,048 tokens per turn, and 8,192 response tokens per trajectory\.

Table 17:Performance under the same maximum inference budget\.Under the same maximum inference budget, RTPO attains the highest Pass@1, Pass@4, and number of successful trajectories per 1,000 response tokens, while producing the shortest average responses\. Its gains therefore do not come from allowing longer test\-time trajectories\.

### G\.5Tool\-Use Behavior on GSM8K

The total number of tool calls should be interpreted relative to dataset size: GSM8K contains 1,319 distinct test problems, while AIME24 and AIME25 each contain 30\. Across all three benchmarks, RTPO improves accuracy while reducing the total number of calls relative to the vanilla model\.

Table 18:Accuracy and total tool calls across mathematical benchmarks\.We further audit the complete trajectories for the first 100 distinct GSM8K test problems generated by the RTPO Qwen3\-4B checkpoint\.

Table 19:Audit of Python use in 100 distinct GSM8K trajectories\.There are no repeated questions and no trajectory invokes Python more than once\. All 36 calls contain valid syntax and execute successfully\. In 34 of the 36 tool\-using trajectories, the model has already derived the correct numerical result before invoking Python\. The calls are one\-shot arithmetic checks rather than multi\-step tool search, repeated code, or duplicate execution\. For example, on GSM8K QID 55, the model first derives30−2=2830\-2=28and28/2=1428/2=14, then invokes Python once to verify the result before returning14\\boxed\{14\}\. Thus, the GSM8K call total reflects many distinct problems receiving a single low\-cost numerical verification, rather than repeated tool use within a small set of trajectories\.

### G\.6On\-Policy Continuation and Synchronization Frequency

As characterized by Theorem[3](https://arxiv.org/html/2608.18682#Thmtheorem3), standard RTPO synchronizes the rollout model after every reverse stage\. When turnkkis optimized, its downstream continuation is therefore generated by the current policyπθ\>k\\pi\_\{\\theta\_\{\>k\}\}, after turnsk\+1,…,K−1k\+1,\\ldots,K\-1have been updated\. The resulting estimator is

Q^j,kon=rj,k\+γτj,k​F^j,kπθ\>k,ωj,k≡1\.\\widehat\{Q\}^\{\\mathrm\{on\}\}\_\{j,k\}=r\_\{j,k\}\+\\gamma^\{\\tau\_\{j,k\}\}\\widehat\{F\}^\{\\pi\_\{\\theta\_\{\>k\}\}\}\_\{j,k\},\\qquad\\omega\_\{j,k\}\\equiv 1\.\(108\)This construction assigns the turn\-kkaction a return under the current downstream policy and avoids a product of trajectory\-level importance ratios\.

At the opposite synchronization endpoint, the off\-policy variant reuses downstream continuations generated by the initial rollout policyπθ0\\pi\_\{\\theta\_\{0\}\}and applies a clamped trajectory\-level importance weight:

Q^j,koff\\displaystyle\\widehat\{Q\}^\{\\mathrm\{off\}\}\_\{j,k\}=rj,k\+γτj,k​ω¯j,k​F^j,kπθ0,\\displaystyle=r\_\{j,k\}\+\\gamma^\{\\tau\_\{j,k\}\}\\overline\{\\omega\}\_\{j,k\}\\widehat\{F\}^\{\\pi\_\{\\theta\_\{0\}\}\}\_\{j,k\},\(109\)ω¯j,k\\displaystyle\\overline\{\\omega\}\_\{j,k\}=clamp⁡\(∏h=k\+1K−1∏t=1Tj,hπθ\>k​\(aj,h,t∣sj,h,t\)πθ0​\(aj,h,t∣sj,h,t\),ρmin,ρmax\)\.\\displaystyle=\\operatorname\{clamp\}\\\!\\left\(\\prod\_\{h=k\+1\}^\{K\-1\}\\prod\_\{t=1\}^\{T\_\{j,h\}\}\\frac\{\\pi\_\{\\theta\_\{\>k\}\}\(a\_\{j,h,t\}\\mid s\_\{j,h,t\}\)\}\{\\pi\_\{\\theta\_\{0\}\}\(a\_\{j,h,t\}\\mid s\_\{j,h,t\}\)\},\\rho\_\{\\min\},\\rho\_\{\\max\}\\right\)\.\(110\)
Table 20:Output\-hit accuracy of off\-policy and per\-stage on\-policy RTPO with Qwen3\-8B\.Per\-stage on\-policy continuation yields consistent gains on the long\-horizon deep\-search tasks, with the largest improvement on GAIA Level 3\. The two variants are broadly comparable on HLE, which is dominated by more static and shorter retrieval\. These results localize the cost of stale continuations to the long\-horizon settings targeted by RTPO\. Fixed\-interval synchronization and policy\-KL\-based adaptive synchronization lie between the fully on\-policy and fully off\-policy endpoints: they can reduce synchronization cost, but no longer strictly satisfyω¯j,k≡1\\overline\{\\omega\}\_\{j,k\}\\equiv 1\.

### G\.7Budget Consumption Across Training

The initial policy produces longer trajectories and more tool calls, so early training stages are more expensive per trunk\. The training schedule does not, however, reserve a fixed number of tool calls for each stage\. Every stage continues under the predefined trunk\-sampling and update schedule, and the actual per\-trunk cost decreases as the policy becomes more efficient\.

Table 21:Evolution of success, tool use, and generation cost during RTPO training\.From Step 10 to Step 50, tool calls decrease from 7\.34 to 0\.94 per trunk, generated tokens decrease from 8,013\.5 to 1,669\.3 per trunk, and tool\-execution success increases from 74\.04% to 93\.33%\. Tool use therefore becomes less frequent and more reliable as training progresses\.

### G\.8Further Distinctions from Related Methods

#### SeeUPO\.

SeeUPO and RTPO both use reverse\-order sequential updates at the procedural level, but they differ in motivation, theoretical object, and algorithmic mechanism\. SeeUPO abstracts multi\-turn interaction as sequentially executed multi\-agent bandits and uses backward induction to establish monotonic improvement and global convergence for critic\-free backbone algorithms\. RTPO instead begins from three structural inconsistencies in a flattened multi\-turn training pipeline: Rollout–Training Mismatch, Trajectory\-Only Credit Assignment, and Long\-Horizon Policy Drift\. Its reverse order is one component of a turn\-boundary formulation that is combined with state\-matched sibling comparison and on\-policy continuation\. The contribution is therefore not the isolated use of reverse order, but the unified diagnosis, formalization, algorithm, and guarantees for the three coupled failure mechanisms\.

#### ArCHer\.

ArCHer addresses delayed reward in long\-horizon multi\-turn interaction through a hierarchical actor–critic design\. Its high\-level component learns turn\-level values with off\-policy value\-based RL, and its low\-level component uses the critic to train the token\-level policy within each turn\. RTPO does not learn an explicit critic; it constructs a turn\-level Monte Carlo advantage from sibling returns sampled from the same boundary state\. ArCHer also does not directly target the rollout–training conditioning mismatch or the asynchronous downstream\-continuation drift analyzed by RTPO\.

#### REFUEL\.

REFUEL and RTPO both avoid an independent critic, but they address different mismatches\. REFUEL uses covariate shift to describe the difference between training histories generated by a reference policy and deployment histories generated by the current learner\. It iteratively collects self\-generated data and reformulates multi\-turn optimization as relative\-future regression tasks\. RTPO’s Rollout–Training Mismatch occurs within the same sampled batch, when rollout and likelihood recomputation condition on different contexts for the same tokens\. Thus, REFUEL addresses policy\-induced covariate shift across data\-collection stages, whereas RTPO addresses context inconsistency between rollout and training\-time recomputation\. The matchedτ3\\tau^\{3\}\-Airline results in Table[8](https://arxiv.org/html/2608.18682#A7.T8)additionally show higher final success, more reliable termination, and shorter responses for RTPO\.

#### R3and SRL\.

R3mitigates sparse\-reward exploration by moving the curriculum starting point backward along an expert reasoning trajectory; its reverse mechanism is a demonstration\-based reverse curriculum\. SRL also relies on expert trajectories and derives step\-wise supervision from similarity between model and expert actions\. RTPO remains outcome\-supervised and on\-policy, estimates turn\-level advantages from sibling continuations at the same boundary state, and jointly addresses conditioning\-context mismatch, upstream\-state contamination, and asynchronous continuation drift\.

### G\.9Additional Limitations

The supplementary results expose several limitations\. First, the strict convergence guarantee in Theorem[1](https://arxiv.org/html/2608.18682#Thmtheorem1)belongs to the turn\-level tabular formulation with exactly fixed downstream policies\. A shared neural policy only approximates this recursive structure and does not inherit a global convergence guarantee for non\-convex optimization\.

Second, RTPO is more expensive to train than GRPO\. WithG=3G=3, wall\-clock time and GPU\-hours increase by approximately 41\.3%\. Increasing toG=4G=4improves Pass@1 and produces more concise inference, but raises training\-generation tokens by approximately 152\.8%, revealing a substantial performance–cost trade\-off\.

Third, even withG=4G=4, the zero\-advantage rate in the sparse binary\-reward Airline environment remains above 80%\. Denser rewards, adaptive sibling sampling, or prioritized selection of turn boundaries may improve the density and efficiency of the learning signal\.

Fourth, the online Airline success rate decreases from 56\.25% at Step 40 to 40\.63% at Step 50, indicating possible late\-stage over\-optimization on a limited set of sparse\-reward training tasks\.

Finally, the original experiments report averages over three runs and include Qwen3\-4B and Qwen3\-8B, withK=3K=3turns for mathematical reasoning andK=6K=6turns for web\-search tasks\. Broader per\-seed stability curves, additional model scales and turn lengths, and intermediate synchronization schemes based on a fixed interval or policy KL remain useful directions for future work\.