Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs

# Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs
Source: [https://arxiv.org/html/2609.19636](https://arxiv.org/html/2609.19636)
###### Abstract

Reinforcement learning now trains language\-model agents that act over dozens of steps in live environments\. The gains are large, and they are read as better decision\-making\. An agent in a closed loop writes its own inputs\. Each observation follows from its own earlier actions, so the states it meets late in an episode are partly of its own making\. An SFT checkpoint and an RL checkpoint are then scored from different states, even on identical tasks\. Endpoint success mixes two changes: where the agent arrives, and what it does once it is there\. Restricting the comparison to states both policies reach does not separate them\. That restriction selects on an outcome, and in our data it flips the sign of the effect\. We introduce checkpoint handoff, an evaluation protocol that clones a state one released checkpoint reached and hands it to another, with no retraining\. Crossing a reacher role and a solver role over SFT and RL splits an endpoint gain intoReachandSolve\.Reachis how often a policy arrives at a state the environment confirms is a fixed number of actions from success\.Solveis how often it finishes from an identical cloned state\. Across two benchmarks and two independently released pipelines, the reacher by solver interaction is positive in all five conditions\. An RL history is worth more to an RL solver than the same history is to an SFT solver\. On ALFWorld, RL improves both terms, and the SFT solver never succeeds where the RL solver fails\. IndependentReachandSolvegaps predict the aggregate interaction\. Handoff asks only that one checkpoint’s history can be replayed under another, so long\-horizon evaluation can report arrival and completion beside endpoint success\.

## 1Introduction

Language\-model agents have rapidly become a central application of foundation models\. They search, call tools, and interact with environments over multiple turns\([Xie et al\., 2024](https://arxiv.org/html/2609.19636#bib.bib5);[Yao et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib7);[Shridhar et al\., 2021](https://arxiv.org/html/2609.19636#bib.bib8)\)\. Supervised fine\-tuning \(SFT\) develops these abilities through demonstrations\. AgentTuning mixes agent interaction trajectories with general instruction data\([Zeng et al\., 2024](https://arxiv.org/html/2609.19636#bib.bib1)\), while Agent\-FLAN redesigns the training corpus to separate format following from agent reasoning\([Chen et al\., 2024](https://arxiv.org/html/2609.19636#bib.bib2)\)\. Agentic reinforcement learning \(RL\) instead improves a policy through interaction and reward, with recent work studying trajectory\-level optimization and stability \(RAGEN\), transition\-level credit assignment \(Agent Lightning\), progressively longer interaction horizons \(AgentGym\-RL\), and reusable skill libraries \(SkillRL\)\([Wang et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib9);[Luo et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib10);[Xi et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib11);[Xia et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib4)\)\. Yet these advances leave unclear whether RL’s gains over SFT come from reaching more useful states, acting better from a given state, or both\. This distinction matters for understanding what RL learns and deciding which part of training or evaluation to improve\.

The difficulty arises because an agent’s later inputs depend on its earlier actions\. In a closed loop, actions change the environment and hence subsequent observations\([Sutton and Barto, 2018](https://arxiv.org/html/2609.19636#bib.bib28)\)\. SFT and RL checkpoints can thus receive the same task while generating different histories and being scored from different states\. In Figure[1](https://arxiv.org/html/2609.19636#S1.F1), one checkpoint reaches the wrong cabinet with raw beef, while the other has already cooked and placed it\. We call this policy dependence of evaluation inputs the*endogenous state problem*\. Endpoint comparisons change both the states reached and the actions taken from them, leaving the source of a gain unresolved\.

![Refer to caption](https://arxiv.org/html/2609.19636v1/fig_endogenous_state.png)Figure 1:The endogenous state problem\.Two checkpoints receive one task and act\. Their early actions leave them in different states, and each is then scored from the state it produced\.Several evaluation families look inside an endpoint\. Progress evaluators score subgoal completion, process evaluators assess intermediate decisions and procedural compliance, and proxy\-state methods verify the environment state left by a tool call\([Ma et al\., 2024](https://arxiv.org/html/2609.19636#bib.bib12);[Gritta et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib13);[Chuang et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib16)\)\. Exploration analyses measure entry into useful regions, while failure diagnosis separates errors made before and after a capability becomes exercisable\([Ye et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib14);[Ji et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib17);[Shao et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib19)\)\. Causal decompositions further attribute outcomes to actions, transitions, or perception\([Triantafyllou et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib25);[Wang et al\., 2026a](https://arxiv.org/html/2609.19636#bib.bib26)\)\. These methods provide useful descriptions of a checkpoint’s realized rollout, but they do not reveal how another checkpoint would continue from the same state and history\. Filtering to reached states also requires care because arrival depends on both the policy and task difficulty\. Comparing subsequent scores after this policy\-dependent selection can mix solver ability with the difficulty of the surviving tasks, introducing selection bias\([Hernán et al\., 2004](https://arxiv.org/html/2609.19636#bib.bib31)\)\. Restricting further to tasks both policies reach discards cases in which only one reaches the subgoal and changes the target from all tasks to a selected subset\. Even within a shared task, reaching the same subgoal need not yield identical histories or budgets\. Attributing the overall gain therefore requires comparing solvers from identical states while retaining non\-arrivals in the population\-level comparison\.

To address this attribution gap, we propose*checkpoint handoff*, which intervenes on who continues from a state\. Because early actions determine where an agent arrives and later actions determine whether it finishes, we assign separate*reacher*and*solver*roles\. The reacher produces the state distribution, and the solver acts from a state held fixed\. We cross these roles over\{SFT,RL\}\\\{\\mathrm\{SFT\},\\mathrm\{RL\}\\\}without retraining\. At an environment\-verified frontier, we clone the environment, history, and remaining budget and let both solvers continue from identical inputs\.Reachmeasures arrival frequency, whileSolvemeasures completion conditional on arrival\. Non\-arrivals remain failures in the full\-population endpoint comparison\. This intervention separates policy\-induced state visitation from state\-conditional value\([Kakade and Langford, 2002](https://arxiv.org/html/2609.19636#bib.bib20)\)on released checkpoints\. Profiles measured on disjoint data then test whether the two components predict how the solver gain from SFT to RL depends on the reacher\.

We study SFT/RL checkpoint pairs from two independent training pipelines on TravelPlanner and ALFWorld\. We organize the comparison around three questions\.RQ1asks whether the gain from switching the solver from SFT to RL depends on whether the reacher is SFT or RL\.RQ2asks whether RL’s advantage over SFT comes from reaching the solvable frontier more often, solving better from identical states, or both\.RQ3asks whether independently measured SFT/RL differences inReachandSolvepredict the dependence of the solver gain on the reacher identified in RQ1\.

This paper makes three contributions\.

- •We characterize the endogenous state problem of closed\-loop agents by decomposing endpoint success into state visitation and state\-conditional value, and show on released checkpoints that neither endpoint comparison nor filtering to reached states can attribute an RL gain\.
- •We propose checkpoint handoff, which separatesReachfromSolveon released checkpoints without retraining, using an environment\-verified frontier, cloned\-state continuation, and prediction from independentReachandSolvegaps\.
- •Experiments on two benchmarks and two training pipelines show a positive reacher by solver interaction in every condition, and exact replay on ALFWorld locates the RL gain in both components\.

## 2The Endogenous State Problem

### 2\.1Setting and Estimand

Let a taskTTbe drawn from a probability distributionp𝒯p\_\{\\mathcal\{T\}\}\. At steptt, letxtx\_\{t\}be the environment configuration,oto\_\{t\}the observation,ata\_\{t\}the action,ht=\(o0,a0,…,at−1,ot\)h\_\{t\}=\(o\_\{0\},a\_\{0\},\\ldots,a\_\{t\-1\},o\_\{t\}\)the visible history, andbtb\_\{t\}the remaining action budget\. We call

Σt=\(T,xt,ht,bt\)∈𝒮\\Sigma\_\{t\}=\(T,x\_\{t\},h\_\{t\},b\_\{t\}\)\\in\\mathcal\{S\}\(1\)the evaluation state: the input from which the rest of the episode is scored\. The checkpoint sees onlyhth\_\{t\}; a replay\-capable evaluator also retainsxtx\_\{t\}\. The episode ends at a stepttermt\_\{\\mathrm\{term\}\}by submission, termination, or budget exhaustion, and a task verifiergT:𝒮→\{0,1\}g\_\{T\}:\\mathcal\{S\}\\to\\\{0,1\\\}scores the endpointY=gT​\(Σterm\)Y=g\_\{T\}\(\\Sigma\_\{\\mathrm\{term\}\}\), whereΣterm=Σtterm\\Sigma\_\{\\mathrm\{term\}\}=\\Sigma\_\{t\_\{\\mathrm\{term\}\}\}\. Let𝒞=\{SFT,RL\}\\mathcal\{C\}=\\\{\\mathrm\{SFT\},\\mathrm\{RL\}\\\}index released checkpoints andπc​\(a∣h\)\\pi\_\{c\}\(a\\mid h\),c∈𝒞c\\in\\mathcal\{C\}, their induced policies on any visible historyhh\. Decoding is fixed, so we intervene onccand read the consequences offπc\\pi\_\{c\}\. For a fixed stateσ∈𝒮\\sigma\\in\\mathcal\{S\}and a random continuation seed, letY⁡\(R,σ\)∈\{0,1\}Y\(R;\\sigma\)\\in\\\{0,1\\\}be the potential endpoint when checkpointR∈𝒞R\\in\\mathcal\{C\}continues fromσ\\sigmato termination, in the sense of potential outcomes\([Rubin, 1974](https://arxiv.org/html/2609.19636#bib.bib29)\); its meanVR​\(σ\)=Pr⁡\(Y⁡\(R,σ\)=1\)V\_\{R\}\(\\sigma\)=\\Pr\(Y\(R;\\sigma\)=1\)is the continuation value, a property of the state and the solver alone\. For a fixed handoff distribution, solver performance is its mean continuation value\.

We define a handoff rule as a prespecified map from the reacher’s trajectory to a cutτ\\tau, selected independently of solver assignment and continuation outcomes\. RunningW∈𝒞W\\in\\mathcal\{C\}alone up toτ\\tauinduces a distributiondWτd\_\{W\}^\{\\tau\}overΣτ\\Sigma\_\{\\tau\}, and we let a second checkpointRRcontinue\. The reacherWWsets the measure, the solverRRsets the integrand,

Jτ​\(W,R\)=𝔼Σ∼dWτ​\[VR​\(Σ\)\]\.J\_\{\\tau\}\(W,R\)=\\mathbb\{E\}\_\{\\Sigma\\sim d\_\{W\}^\{\\tau\}\}\\\!\\left\[V\_\{R\}\(\\Sigma\)\\right\]\.\(2\)Assignments withW=RW=Rprovide same\-checkpoint controls; those withW≠RW\\neq Rswitch checkpoints at handoff\.

#### The attribution question\.

The diagonal controls run one checkpoint in both roles and reportJτ​\(c,c\)J\_\{\\tau\}\(c,c\)\. WritingJJforJτJ\_\{\\tau\}, the RL gain then splits into a reach term and a solve term along either of two paths, by the same telescoping that underlies the performance difference lemma\([Kakade and Langford, 2002](https://arxiv.org/html/2609.19636#bib.bib20)\):

J⁡\(RL,RL\)−J⁡\(SFT,SFT\)=\\displaystyle J\(\\mathrm\{RL\},\\mathrm\{RL\}\)\-J\(\\mathrm\{SFT\},\\mathrm\{SFT\}\)=\{\}J⁡\(RL,RL\)−J⁡\(SFT,RL\)⏟reach, under the RL solver\+J⁡\(SFT,RL\)−J⁡\(SFT,SFT\)⏟solve, on SFT states\\displaystyle\\underbrace\{J\(\\mathrm\{RL\},\\mathrm\{RL\}\)\-J\(\\mathrm\{SFT\},\\mathrm\{RL\}\)\}\_\{\\text\{reach, under the RL solver\}\}\+\\underbrace\{J\(\\mathrm\{SFT\},\\mathrm\{RL\}\)\-J\(\\mathrm\{SFT\},\\mathrm\{SFT\}\)\}\_\{\\text\{solve, on SFT states\}\}=\\displaystyle=\{\}J⁡\(RL,SFT\)−J⁡\(SFT,SFT\)⏟reach, under the SFT solver\+J⁡\(RL,RL\)−J⁡\(RL,SFT\)⏟solve, on RL states\.\\displaystyle\\underbrace\{J\(\\mathrm\{RL\},\\mathrm\{SFT\}\)\-J\(\\mathrm\{SFT\},\\mathrm\{SFT\}\)\}\_\{\\text\{reach, under the SFT solver\}\}\+\\underbrace\{J\(\\mathrm\{RL\},\\mathrm\{RL\}\)\-J\(\\mathrm\{RL\},\\mathrm\{SFT\}\)\}\_\{\\text\{solve, on RL states\}\}\.\(3\)Every right\-hand term is a cross assignment in which one checkpoint continues from states produced by the other\. Endpoint evaluation observes onlyJτ​\(c,c\)J\_\{\\tau\}\(c,c\), so it leavesdWτd\_\{W\}^\{\\tau\}andVRV\_\{R\}confounded; the two paths can therefore disagree\. The three research questions follow directly: RQ1 varies the reacher while measuring the solver gain, RQ2 attributes RL’s advantage to reach and solve, and RQ3 tests the interaction prediction below\.

### 2\.2Endpoint Comparisons Change Two Factors at Once

The same\-checkpoint contrast on the left of Equation[3](https://arxiv.org/html/2609.19636#S2.E3)changes the measure and integrand together, so it cannot attribute the difference to either\. Evaluating all four assignments instead yields the risk\-difference interaction of a two\-by\-two factorial design\([VanderWeele, 2015](https://arxiv.org/html/2609.19636#bib.bib30)\),

Iτ=\\displaystyle I\_\{\\tau\}=\{\}\[Jτ​\(RL,RL\)−Jτ​\(RL,SFT\)\]\\displaystyle\\big\[J\_\{\\tau\}\(\\mathrm\{RL\},\\mathrm\{RL\}\)\-J\_\{\\tau\}\(\\mathrm\{RL\},\\mathrm\{SFT\}\)\\big\]−\[Jτ​\(SFT,RL\)−Jτ​\(SFT,SFT\)\]\.\\displaystyle\-\\big\[J\_\{\\tau\}\(\\mathrm\{SFT\},\\mathrm\{RL\}\)\-J\_\{\\tau\}\(\\mathrm\{SFT\},\\mathrm\{SFT\}\)\\big\]\.\(4\)IτI\_\{\\tau\}is the difference between the two solve terms of Equation[3](https://arxiv.org/html/2609.19636#S2.E3), and equally between the two reach terms\. A positiveIτI\_\{\\tau\}means that the advantage of the RL solver is larger on the states produced by the RL reacher: the two roles are coupled, and the two paths disagree\. It does not say how much of the gain sits in each\.

### 2\.3Arrival Diverges Long Before Success Does

Figure 2:Arrival atℱ2\\mathcal\{F\}\_\{2\}by reacher action\.ALFWorld unseen split,120120paired trajectories per checkpoint\.Figure[2](https://arxiv.org/html/2609.19636#S2.F2)shows that the two checkpoints already diverge by the third reacher action; by the end of the budget, RL reaches the solvable frontier about six times as often as SFT\. The curves record environment membership inℱ2\\mathcal\{F\}\_\{2\}, the set of states exactly two valid actions from success with enough budget to complete them \(the generalℱD\\mathcal\{F\}\_\{D\}is defined in Section[3\.1](https://arxiv.org/html/2609.19636#S3.SS1)\)\. This verified frontier gives the common boundary for solver comparison in Section[3](https://arxiv.org/html/2609.19636#S3)\. Endpoint success still combines arrival and completion, whereas reached\-only conditioning changes the target population; Section[2\.4](https://arxiv.org/html/2609.19636#S2.SS4)and Appendix[C](https://arxiv.org/html/2609.19636#A3)quantify the resulting selection and budget effects\.

### 2\.4Keeping Only Reached States Changes the Estimand

Figure 3:IτI\_\{\\tau\}under two estimands\.Same ALFWorld unseen\-split episodes, scored two ways;95%95\\%paired bootstrap\.Figure[3](https://arxiv.org/html/2609.19636#S2.F3)evaluates the same ALFWorld episodes under two different estimands\. The full\-population estimand keeps every initial episode\. An episode whose reacher never arrives has no continuation state and therefore counts as a failure\. This quantity combines how often a reacher arrives with what a solver does after arrival\. The reached\-only estimand conditions on the reacher having arrived and averages only over the states that this reacher actually produced\. It therefore measures conditional completion, not the overall endpoint gain, and it does not compare a common set of states across the two reachers\.

The two estimands answer different questions\. A reacher that arrives mainly on easy tasks can have a different conditional solver gain from one that also reaches harder tasks: the full\-population estimand retains arrival frequency, whereas reached\-only normalizes it away\. In our data the full\-population interaction is positive while the reached\-only estimate has the opposite direction and a wide interval; Appendix[C](https://arxiv.org/html/2609.19636#A3)quantifies the support loss behind that interval\.

Conditioning on arrival targets conditional completion, but using it to explain the population\-level RL gain induces selection because arrival depends on the policy and task difficulty\([Hernán et al\., 2004](https://arxiv.org/html/2609.19636#bib.bib31)\)\. Restricting further to episodes that both policies reach changes the target again\. The protocol therefore retains non\-arrivals for the population endpoint and uses cloned states for the conditional solver effect; the retention diagnostic appears in Appendix[C](https://arxiv.org/html/2609.19636#A3)\.

## 3Checkpoint Handoff

![Refer to caption](https://arxiv.org/html/2609.19636v1/overview.png)Figure 4:Checkpoint handoff and interaction prediction\.\(A\)A benchmark\-specific rule selects a handoff from each reacher’s trajectory\. Both solvers continue with matched inputs, yielding four endpoint success rates and their interaction\.\(B\)On ALFWorld, target arrival gaps and independentSolvegaps predict the interaction for comparison with the observed value \(Equation[9](https://arxiv.org/html/2609.19636#S3.E9)\)\.Checkpoint handoff crosses released reachers and solvers without retraining \(Equation[2](https://arxiv.org/html/2609.19636#S2.E2)\)\. Solvers receive no label identifying the reacher\. Each solver continues live from the reconstructed prefix, as in branching model\-switching evaluation\([Gonuguntla, 2026](https://arxiv.org/html/2609.19636#bib.bib27)\)\. Figure[4](https://arxiv.org/html/2609.19636#S3.F4)summarizes the design and the ALFWorld prediction test; Appendix[A](https://arxiv.org/html/2609.19636#A1), Table[5](https://arxiv.org/html/2609.19636#A1.T5)maps the controls to the quantities they identify\.

### 3\.1Handoff at Equal Remaining Distance

For replayable environments such as ALFWorld, we align handoffs by remaining distance: after the same number of actions, one reacher may be one action from success and another ten\. Distance to success is used as a goal\-conditioned value\([Kaelbling, 1993](https://arxiv.org/html/2609.19636#bib.bib37)\), a proximity measure in embodied navigation evaluation\([Anderson et al\., 2018](https://arxiv.org/html/2609.19636#bib.bib38)\), and a reward\-shaping potential\([Ng et al\., 1999](https://arxiv.org/html/2609.19636#bib.bib33)\)\. We use it to index states, not to reward them\. For any valid action sequence𝐚\\mathbf\{a\}from stateσ\\sigma, letΦT​\(σ,𝐚\)\\Phi\_\{T\}\(\\sigma,\\mathbf\{a\}\)be the state obtained by exact replay and\|𝐚\|\|\\mathbf\{a\}\|its length\. The remaining distance, computed by replay in the environment, is

ℓT​\(σ\)=min𝐚​valid from​σ⁡\{\|𝐚\|:gT​\(ΦT​\(σ,𝐚\)\)=1\},\\ell\_\{T\}\(\\sigma\)=\\min\_\{\\mathbf\{a\}\\ \\text\{valid from\}\\ \\sigma\}\\\!\\left\\\{\|\\mathbf\{a\}\|:\\ g\_\{T\}\\\!\\left\(\\Phi\_\{T\}\(\\sigma,\\mathbf\{a\}\)\\right\)=1\\right\\\},\(5\)withℓT​\(σ\)=∞\\ell\_\{T\}\(\\sigma\)=\\inftyif no successful sequence exists\. ForD≥1D\\geq 1the solvable frontier is

ℱD\(T\)=\{σ∈𝒮:ℓT\(σ\)=D,b\(σ\)≥D\},\\mathcal\{F\}\_\{D\}\(T\)=\\left\\\{\\sigma\\in\\mathcal\{S\}:\\ \\ell\_\{T\}\(\\sigma\)=D,\\ b\(\\sigma\)\\geq D\\right\\\},\(6\)whereb⁡\(σ\)b\(\\sigma\)is the budget component ofσ\\sigma\. Membership depends only on the environment, never on which checkpoint produced the state: every retained state is exactlyDDvalid actions from success and has budget to get there\. With early budgetBearlyB\_\{\\mathrm\{early\}\}, the handoff is the first hitting timeτD=inf\{t≤Bearly:Σt∈ℱD​\(T\)\}\\tau\_\{D\}=\\inf\\\{t\\leq B\_\{\\mathrm\{early\}\}:\\Sigma\_\{t\}\\in\\mathcal\{F\}\_\{D\}\(T\)\\\}, and an empty set yields the non\-arrival⊥\\botwithVR​\(⊥\)=0V\_\{R\}\(\\bot\)=0\.τD\\tau\_\{D\}depends on the reacher alone, as Section[2\.1](https://arxiv.org/html/2609.19636#S2.SS1)requires; we writeJD=JτDJ\_\{D\}=J\_\{\\tau\_\{D\}\}andID=IτDI\_\{D\}=I\_\{\\tau\_\{D\}\}\. We useD=Bcont=2D=B\_\{\\mathrm\{cont\}\}=2, the shortest horizon that lets a solver act, observe feedback, and act again\. Enumerating one\- and two\-action continuations certifies this distance against the environment’s success flag \(Appendix[A](https://arxiv.org/html/2609.19636#A1)\)\. Every retained state requires two actions to succeed, and both solvers receive that same continuation budget\.

### 3\.2ReachandSolvefrom Cloned States

LetZW=𝟏\{ΣτD≠⊥\}Z\_\{W\}=\\mathbf\{1\}\\\{\\Sigma\_\{\\tau\_\{D\}\}\\neq\\bot\\\}indicate that reacherWWarrived\.Reachis the arrival rateAW=Pr⁡\(ZW=1\)A\_\{W\}=\\Pr\(Z\_\{W\}=1\), the share of episodes in whichWWreaches the frontier at all\. Non\-arrivals are neither discarded nor imputed\. Conditional on arrival, and defined whenAW\>0A\_\{W\}\>0, letqWDq\_\{W\}^\{D\}be the natural frontier\-state distribution, namelydWτDd\_\{W\}^\{\\tau\_\{D\}\}conditioned onΣτD≠⊥\\Sigma\_\{\\tau\_\{D\}\}\\neq\\bot\. We restore each reached state twice and assign the SFT and RL solvers under identical environment, history, admissible actions, budget, and seed schedule\. We pair seeds because seed choice alone moves reported agent performance by a wide margin\([Henderson et al\., 2018](https://arxiv.org/html/2609.19636#bib.bib34)\)\.Solveis the conversion rateCW,RD=𝔼Σ∼qWD​\[VR​\(Σ\)\]C\_\{W,R\}^\{D\}=\\mathbb\{E\}\_\{\\Sigma\\sim q\_\{W\}^\{D\}\}\[V\_\{R\}\(\\Sigma\)\], the share of those arrivals that solverRRturns into success\. WithVR​\(⊥\)=0V\_\{R\}\(\\bot\)=0the endpoint is the product of the two rates,

JD​\(W,R\)=AW​CW,RD,J\_\{D\}\(W,R\)=A\_\{W\}\\,C\_\{W,R\}^\{D\},\(7\)so the reacher moves the first factor and the solver the second\. Matching the environment state and visible history fixes task progress and available information; switching only the continuation checkpoint identifies the same\-stateSolvegapδ⁡\(σ\)=VRL​\(σ\)−VSFT​\(σ\)\\delta\(\\sigma\)=V\_\{\\mathrm\{RL\}\}\(\\sigma\)\-V\_\{\\mathrm\{SFT\}\}\(\\sigma\), and

ID=ARL​𝔼Σ∼qRLD​\[δ⁡\(Σ\)\]−ASFT​𝔼Σ∼qSFTD​\[δ⁡\(Σ\)\]\.I\_\{D\}=A\_\{\\mathrm\{RL\}\}\\,\\mathbb\{E\}\_\{\\Sigma\\sim q\_\{\\mathrm\{RL\}\}^\{D\}\}\[\\delta\(\\Sigma\)\]\-A\_\{\\mathrm\{SFT\}\}\\,\\mathbb\{E\}\_\{\\Sigma\\sim q\_\{\\mathrm\{SFT\}\}^\{D\}\}\[\\delta\(\\Sigma\)\]\.\(8\)The interaction is the difference between two products of an arrival rate and aSolvegap\. It is positive when the RL solver’s advantage is worth more on states the RL reacher produced\. For a prespecified set of statesμD\\mu\_\{D\}, chosen independently of solver assignment and outcomes, we also reportΔsolve​\(μD\)=𝔼Σ∼μD​\[δ⁡\(Σ\)\]\\Delta\_\{\\mathrm\{solve\}\}\(\\mu\_\{D\}\)=\\mathbb\{E\}\_\{\\Sigma\\sim\\mu\_\{D\}\}\[\\delta\(\\Sigma\)\]\. We reportδ\\deltaseparately on SFT\-produced and RL\-produced states, so aSolvegain cannot be an artefact of RL states being easier\.

### 3\.3Predicting Interaction from Reach and Solve Gaps

Equation[7](https://arxiv.org/html/2609.19636#S3.E7)is an identity on the same natural states\. A stronger test forecasts the interaction from components measured elsewhere, before the target continuations are run\. Letk∈𝒦k\\in\\mathcal\{K\}index task strata with prevalencepkp\_\{k\}\. Natural rollouts on the target split give the arrival rateAW​\(k\)=Pr⁡\(ZW=1∣k\)A\_\{W\}\(k\)=\\Pr\(Z\_\{W\}=1\\mid k\)\. An independently selected set ofℱD\\mathcal\{F\}\_\{D\}states gives each solver’sSolverateCRext​\(k\)C\_\{R\}^\{\\mathrm\{ext\}\}\(k\)on states the target checkpoints never visited; its selection and scoring never touch target continuation outcomes\. Under this component model, the interaction is the prevalence\-weighted product of the two gaps,

I^Dcomp=∑kpk​\[ARL​\(k\)−ASFT​\(k\)\]​\[CRLext​\(k\)−CSFText​\(k\)\]\.\\widehat\{I\}\_\{D\}^\{\\mathrm\{comp\}\}=\\sum\_\{k\}p\_\{k\}\\,\\big\[A\_\{\\mathrm\{RL\}\}\(k\)\-A\_\{\\mathrm\{SFT\}\}\(k\)\\big\]\\big\[C\_\{\\mathrm\{RL\}\}^\{\\mathrm\{ext\}\}\(k\)\-C\_\{\\mathrm\{SFT\}\}^\{\\mathrm\{ext\}\}\(k\)\\big\]\.\(9\)The residualRcomp=ID−I^DcompR\_\{\\mathrm\{comp\}\}=I\_\{D\}\-\\widehat\{I\}\_\{D\}^\{\\mathrm\{comp\}\}is a forecast error: nothing in Equation[9](https://arxiv.org/html/2609.19636#S3.E9)is estimated from the observedIDI\_\{D\}\. Its transport condition, in the sense of[Pearl and Bareinboim \(2011\)](https://arxiv.org/html/2609.19636#bib.bib32), is that withinkkthe external rate approximates continuation success on naturally reached target states regardless of their reacher\. The per\-type residuals quantify how well this condition holds across the target task distribution\.

## 4Experiments

### 4\.1Setup

TravelPlanner\([Xie et al\., 2024](https://arxiv.org/html/2609.19636#bib.bib5)\)tests RQ1 across model scales using recorded\-history handoffs\. ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2609.19636#bib.bib8)\)supports verified\-frontier handoffs for RQ2 and RQ3\. Agent\-STAR\([Wu et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib3)\)and SkillRL\([Xia et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib4)\)are independent Qwen\-family training pipelines\. Decoding settings were identical within each SFT/RL pair and fixed before any handoff ran \(Appendix[A](https://arxiv.org/html/2609.19636#A1)\)\.

#### TravelPlanner and Agent\-STAR\.

We cross released 1\.5B, 3B, and 7B Agent\-STAR SFT/RL checkpoints on the same180180TravelPlanner validation tasks; their immutable revisions are listed in Appendix[A](https://arxiv.org/html/2609.19636#A1), Table[4](https://arxiv.org/html/2609.19636#A1.T4)\. TravelPlanner’s two\-stage setting separates information collection from plan generation over a static database\([Xie et al\., 2024](https://arxiv.org/html/2609.19636#bib.bib5)\)\. Agent\-STAR records tool interactions in a message history and terminates with a submitted plan\([Wu et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib3)\)\. We therefore use the recorded trajectory as the handoff input: task prompt, model messages, tool calls, and tool responses\. Each reacher runs to termination under the native6060\-decision limit; we cut immediately before its final model call\. This retains the reacher’s accumulated information and intermediate reasoning while leaving plan completion to the solver\. The cut depends only on the recorded reacher run\. We reset the task, replay the prefix, and verify identical reconstructed histories\. Both solvers receive the same record and at mostB=4B=4model decisions, allowing further tool use before submission under a common budget\. A fixed Formatter converts the result for the official evaluator\. Crossing reachers and solvers gives four assignments, each evaluated on the same 180 tasks, or 720 rows per scale \(Appendix[A](https://arxiv.org/html/2609.19636#A1)\)\.

#### ALFWorld and SkillRL\.

ALFWorld’s TextWorld engine represents household states and action effects symbolically\([Shridhar et al\., 2021](https://arxiv.org/html/2609.19636#bib.bib8)\)\. We use its admissible actions and success flag to verify remaining distance by enumeration\. The unseen split places tasks in rooms held out from training, and the seen split reuses training rooms\. We run released SkillRL 7B SFT/RL checkpoints on3030prespecified unseen\-split tasks, five from each official task type, with four paired seeds\.Reachcomes from natural rollouts: a reacher acts for at most4848steps, and after each action the environment is asked whether the current state lies onℱ2\\mathcal\{F\}\_\{2\}\(Section[3\.1](https://arxiv.org/html/2609.19636#S3.SS1)\)\.Solvecomes from replay\. We reset the task under the same seed and replay the recorded reacher actions to restore the arrival state\. Observation, admissible actions, and rendered prompt must match the recording\. Each solver then takes two actions from that clone, in a separate replay of the same state\. This yields240240natural trajectories,115115verified frontier states, and480480endpoint rows, scored by the environment’s nativewonflag\. A separately selected seen\-split sample of the same design supplies the externalSolveprofile \(Appendix[B](https://arxiv.org/html/2609.19636#A2)\) and a replication\. All replay, solvability, pairing, prompt, and completeness checks pass \(Appendix[A](https://arxiv.org/html/2609.19636#A1)\)\.

#### Inference\.

Arrival and endpoint rates are empirical means over prespecified tasks and paired seeds, with zero endpoint success for every non\-arrival\. All uncertainty uses100,000100\{,\}000task\-level paired bootstrap resamples\([Efron and Tibshirani, 1994](https://arxiv.org/html/2609.19636#bib.bib36)\)that keep seeds, the four checkpoint assignments, and task\-type counts together\. Tasks are the unit that varies, and few\-run evaluations call for interval estimates rather than point scores\([Agarwal et al\., 2021](https://arxiv.org/html/2609.19636#bib.bib35)\)\. For the component\-based prediction, each draw independently samples tasks within each task type from the target and external samples, retains all paired seeds for every sampled task, and recomputes the observed interaction, predicted interaction, and residual from that draw\. On ALFWorld, same\-state contrasts identify the solver effect because the environment decidesτD\\tau\_\{D\}before any solver runs, both solvers start from exact clones, and only the solver changes\.

Figure 5:Endpoint success for the four reacher by solver assignments\.Non\-parallel lines are the interaction\.Table 1:Endpoint success for the four reacher \(WW\) by solver \(RR\) assignments\.Subscripts on theRR=RL columns give the solver effect under that reacher, and the last column is their difference,IτI\_\{\\tau\}of Equation[4](https://arxiv.org/html/2609.19636#S2.E4)\. Brackets are95%95\\%paired\-bootstrap intervals\.Endpoint success \(%\)SettingnnWW=SFTWW=SFTWW=RLWW=RLInteractionRR=SFTRR=RLRR=SFTRR=RL\(pp\)*TravelPlanner / Agent\-STAR*1\.5B1806\.114\.4\+8\.37\.833\.9\+26\.1\+\+17\.8 \[9\.4, 26\.1\]3B18012\.224\.4\+12\.222\.248\.9\+26\.7\+\+14\.4 \[6\.1, 22\.8\]7B18020\.029\.4\+9\.438\.958\.9\+20\.0\+\+10\.6 \[0\.6, 20\.6\]*ALFWorld / SkillRL*Seen split1205\.09\.2\+4\.243\.365\.0\+21\.7\+\+17\.5 \[6\.7, 27\.5\]Unseen split1205\.812\.5\+6\.748\.379\.2\+30\.8\+\+24\.2 \[13\.3, 35\.0\]

### 4\.2RQ1: The Solver Gain from SFT to RL Depends on the Reacher

#### The Solver Effect Is Two to Five Times Larger on RL\-Produced States

Table[1](https://arxiv.org/html/2609.19636#S4.T1)compares the SFT and RL solvers separately under each reacher\. In Figure[5](https://arxiv.org/html/2609.19636#S4.F5)the vertical gap between the two lines is the solver effect, so a gap that changes across reachers indicates an interaction\. The interaction is positive in all five conditions, and every lower bound in Table[1](https://arxiv.org/html/2609.19636#S4.T1)lies above zero\. The size of the dependence is the informative part\. On TravelPlanner, switching to the RL solver gains between two and three times as much on RL\-produced histories as on SFT\-produced ones, at every scale\. On ALFWorld the ratio rises to about five\. The SFT reacher rarely produces a state that any solver can finish within budget: no SFT\-reacher entry exceeds12\.5%12\.5\\%, whichever solver continues\. The late checkpoint is worth more on states the early checkpoint prepared well, and the dependence is strongest where the two reachers produce the most different states\.

#### With Scale, the Gain Moves From the Solver to the Reacher

At 1\.5B the SFT reacher with the RL solver beats the RL reacher with the SFT solver,14\.4%14\.4\\%against7\.8%7\.8\\%; an RL history is worth little to an SFT solver\. At 7B the order flips,29\.4%29\.4\\%against38\.9%38\.9\\%, so the RL history alone now carries most of the gain\. The interaction shrinks from 1\.5B to 7B, but the three intervals overlap and the 7B lower bound sits close to zero\. We read the decline as an observation, not a trend\. The intervals support a positive interaction at every scale and under two independently trained pipelines\.

### 4\.3RQ2a: RL Reaches the Frontier More Often than SFT

#### RL Reaches an Equally Solvable State Six Times as Often

On ALFWorld each entry of Table[1](https://arxiv.org/html/2609.19636#S4.T1)equals the arrival rate in Table[2](https://arxiv.org/html/2609.19636#S4.T2)\(a\) times the completion rate in Table[2](https://arxiv.org/html/2609.19636#S4.T2)\(b\), to the reported precision\.

Figure 6:Reach,Solve, and the endpoint interaction\.Filled markers are the unseen split, open the seen split\.Equation[2](https://arxiv.org/html/2609.19636#S2.E2)is an identity here, and the question is how much of the gain sits in each term\. On the unseen split the SFT reacher reachesℱ2\\mathcal\{F\}\_\{2\}in13\.3%13\.3\\%of trajectories and the RL reacher in82\.5%82\.5\\%\. Every frontier state is exactly two actions from success, so the contrast counts arrivals at a common boundary\. Mean actions to arrival or stopping,44\.044\.0against17\.817\.8, separate two regimes: the SFT reacher usually runs to the4848\-action limit without arriving, the RL reacher usually arrives within the first third of the budget\. The gap replicates on the seen split,15\.8%15\.8\\%against81\.7%81\.7\\%, andReachis the component on which the two splits agree most closely \(Figure[6](https://arxiv.org/html/2609.19636#S4.F6)\)\. Appendix[C](https://arxiv.org/html/2609.19636#A3)reports the split replication, budget contrast \(Figure[10](https://arxiv.org/html/2609.19636#A3.F10)and Table[7](https://arxiv.org/html/2609.19636#A3.T7)\), and non\-arrival audit: RL with eight actions already out\-reaches SFT with forty\-eight, and almost every SFT non\-arrival is a loop that repeats actions without changing the state\. More budget does not close the deficit\.

Table 2:ReachandSolveon ALFWorld\.Each row is a reacher\. \(a\) Arrival atℱ2\\mathcal\{F\}\_\{2\}over120120trajectories, with mean actions to arrival or stopping\. \(b\) Completion by each solver from identical replayed states that the reacher produced\. Brackets are task\-stratified paired\-bootstrap95%95\\%intervals; a dash marks a stratum too small for one\.

### 4\.4RQ2b: RL Outperforms SFT from Identical States

Table 3:Solvesplit into choosing a bridge action and finishing from it\.Shares of reached states with95%95\\%bootstrap intervals; the last two columns count states completed by only one solver \(n=115n=115on the unseen split,117117on the seen split\)\.#### The RL Solver Is Near Ceiling Whoever Produced the State

From two exact clones of each of the115115reached unseen\-split states, the SFT solver completes56\.5%56\.5\\%and the RL solver95\.7%95\.7\\%\(Table[2](https://arxiv.org/html/2609.19636#S4.T2)b\)\. The clones share task, history, admissible actions, two\-action budget, and seed schedule, so the gap is a solver effect, and its lower bound lies well above zero on both splits\. The origin of the state matters only to the SFT solver: it completes43\.8%43\.8\\%of the states its own reacher produced and58\.6%58\.6\\%of the RL\-produced ones, while the RL solver completes93\.8%93\.8\\%and96\.0%96\.0\\%\. The same\-state effect is larger on SFT\-produced states, the opposite of what an easier\-RL\-states account predicts\. Of the115115states,4545are won by the RL solver only and none by the SFT solver only \(Table[3](https://arxiv.org/html/2609.19636#S4.T3)\)\. The same table splitsSolveinto choosing a bridge action and finishing from it\. SFT loses at both stages, and the RL advantage at the second action is the less stable across splits\. Appendix[C](https://arxiv.org/html/2609.19636#A3)gives the corresponding action\-stage breakdown\. Appendix[D](https://arxiv.org/html/2609.19636#A4)\(Table[8](https://arxiv.org/html/2609.19636#A4.T8)\) walks through one state: holding the pencil, the SFT solver spends its first action looking around while the RL solver walks to a shelf and places it\.

### 4\.5RQ3: Predicting Reacher–Solver Interaction from Reach and Solve Gaps

Figure 7:Predicted against observed interaction, per task type\.The dashed line isy=xy=x; the black diamond is the unweighted mean across the six task types\.#### Independent Reach and Solve Gaps Predict the Aggregate Interaction

We test whether the interaction can be predicted from its two component gaps\. For each task type, we multiply the RL–SFT difference in frontier arrival by the RL–SFT difference in continuation success, then average across the six balanced types\. Arrival rates come from the target split, while continuation success is measured on an independent seen\-split sample with no state or continuation outcome shared with the115115replayed target states\. The targetvalid\_unseenstates are reserved for evaluating the observed interaction; the independentvalid\_seensample supplies the externalSolveprofile, making this an out\-of\-sample transport test across splits conditional on task type\. Equation[9](https://arxiv.org/html/2609.19636#S3.E9)predicts an interaction of20\.720\.7percentage points, compared with an observed value of24\.224\.2points \(Figure[7](https://arxiv.org/html/2609.19636#S4.F7), Appendix[B](https://arxiv.org/html/2609.19636#A2)\)\. The black\-diamond overall point is the unweighted mean of the six task\-type predictions and observations,\(20\.7,24\.2\)\(20\.7,24\.2\)\. The prediction recovers the positive interaction and its aggregate scale; the residual is3\.53\.5points \(95% CI\[−7\.2,14\.2\]\[\-7\.2,14\.2\]\), quantifying the remaining difference between prediction and observation\. Per\-type residuals map how continuation success transfers across state distributions\. These results show that Reach and Solve gaps capture the aggregate reacher–solver interaction, while task type leaves measurable state\-level heterogeneity\. Appendix[B](https://arxiv.org/html/2609.19636#A2)gives the per\-type comparison, including Figures[8](https://arxiv.org/html/2609.19636#A2.F8)and[9](https://arxiv.org/html/2609.19636#A2.F9), along with their source rates in Table[6](https://arxiv.org/html/2609.19636#A2.T6)\.

## 5Related Work

#### Training and interpreting agentic RL\.

Multi\-turn RL for agents optimizes whole trajectories: RAGEN treats the rollout as the unit of optimization and studies when it collapses, Agent Lightning decouples agent execution from training and assigns credit at the transition level, and AgentGym\-RL lengthens the interaction horizon during training\([Wang et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib9);[Luo et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib10);[Xi et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib11)\)\. SkillRL grows a recursive skill library, and Agent\-STAR distills a recipe for long\-horizon tool use\([Xia et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib4);[Wu et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib3)\)\. Diagnostics of these runs localize reasoning collapse and token\-level action bottlenecks\([Wang et al\., 2026b](https://arxiv.org/html/2609.19636#bib.bib15);[He et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib21)\)\. Comparisons of SFT and RL report that RL generalizes where SFT memorizes, that RL fine\-tuning restores out\-of\-distribution ability lost in SFT, and that SFT generalization depends on data and model capability\([Chu et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib22);[Jin et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib24);[Ren et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib23)\); causal decompositions attribute multi\-agent outcomes to counterfactual actions and transitions, or vision\-language errors to perception against reasoning\([Triantafyllou et al\., 2025](https://arxiv.org/html/2609.19636#bib.bib25);[Wang et al\., 2026a](https://arxiv.org/html/2609.19636#bib.bib26)\)\. All of these score each checkpoint on the states it reached for itself, so early states and the checkpoint acting on them later are never separated\. Checkpoint handoff makes that separation on released checkpoints, realizing the classical distinction between policy\-induced state visitation and state\-conditional value\([Kakade and Langford, 2002](https://arxiv.org/html/2609.19636#bib.bib20)\)as an intervention; the claim is behavioral, not mechanistic\.

#### Agent evaluation beyond endpoint success\.

AgentBoard scores progress toward subgoals rather than the endpoint alone, process evaluation grades each step of an agentic trajectory, and proxy\-state rewards verify the environment state a tool call leaves behind\([Ma et al\., 2024](https://arxiv.org/html/2609.19636#bib.bib12);[Gritta et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib13);[Chuang et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib16)\)\. Exploration analyses measure how often a policy enters useful regions, guide it there with action\-level supervision, or assign credit across several timescales of environment feedback\([Ye et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib14);[Ji et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib17);[Huo et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib18)\)\. Failure diagnosis separates errors made before a capability becomes exercisable from errors made once it is\([Shao et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib19)\)\. In the terms of Section[2](https://arxiv.org/html/2609.19636#S2), these methods either restrict to states each policy selected for itself \(Section[2\.4](https://arxiv.org/html/2609.19636#S2.SS4)\) or change the state distribution and the continuation value together \(Section[2\.2](https://arxiv.org/html/2609.19636#S2.SS2)\)\. Concurrent work on model switching shows that replaying one model’s logged prefix and pasting in another model’s recorded outputs scores a different world from a live rollout\([Gonuguntla, 2026](https://arxiv.org/html/2609.19636#bib.bib27)\)\. Checkpoint handoff runs the solver live from an exact clone, hands off at an environment\-verified frontier rather than an arbitrary step, and keeps non\-arrivals in the estimand\.

## 6Conclusion

A closed\-loop agent shapes the states it is later scored from, so endpoint success conflates where an agent arrives with what it does once there, and keeping only the reached states changes the question rather than answering it\. Checkpoint handoff separates the two by moving naturally reached, environment\-verified states between released checkpoints\. Across two benchmarks and two training pipelines, the reacher by solver interaction is positive in every condition\. On ALFWorld, exact replay shows that RL reaches the solvable frontier far more often and completes far more often from identical states, and independent Reach and Solve gaps predict the aggregate interaction\. Handoff needs only the ability to give one checkpoint’s history to another, so long\-horizon evaluation can report arrival and completion alongside endpoint success\.

## AI Use Statement

This study uses AI tools to write parts of the code, refine the idea, and adjust the LaTeX formatting\.

## Reproducibility Statement

Appendix[A](https://arxiv.org/html/2609.19636#A1)reports the released checkpoints and revisions, task samples, generation settings, estimators, and acceptance checks\. Appendix[B](https://arxiv.org/html/2609.19636#A2)documents the independent Solve\-profile sample and its transport assumption, and Appendix[C](https://arxiv.org/html/2609.19636#A3)reports the split replication and two post\-hoc diagnostics\. The anonymous supplementary archive contains the prespecified configurations, continuation rows, raw Formatter replies, official per\-example rewards, task\-level effects, and terminal summaries behind every reported number\.

## References

- Agarwalet al\.\(2021\)R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. Courville, and M\. G\. BellemareDeep reinforcement learning at the edge of the statistical precipice\.InAdvances in Neural Information Processing Systems,Cited by:[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.SSS0.Px3.p1.1)\.
- Andersonet al\.\(2018\)P\. Anderson, A\. Chang, D\. S\. Chaplot, A\. Dosovitskiy, S\. Gupta, V\. Koltun, J\. Kosecka, J\. Malik, R\. Mottaghi, M\. Savva, and A\. R\. ZamirOn evaluation of embodied navigation agents\.arXiv preprint arXiv:1807\.06757\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1807.06757),[Link](https://arxiv.org/abs/1807.06757)Cited by:[§3\.1](https://arxiv.org/html/2609.19636#S3.SS1.p1.1)\.
- Chenet al\.\(2024\)Z\. Chen, K\. Liu, Q\. Wang, W\. Zhang, J\. Liu, D\. Lin, K\. Chen, and F\. ZhaoAgent\-FLAN: designing data and methods of effective agent tuning for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 9354–9366\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.557)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1)\.
- Chuet al\.\(2025\)T\. Chu, Y\. Zhai, J\. Yang, S\. Tong, S\. Xie, D\. Schuurmans, Q\. V\. Le, S\. Levine, and Y\. MaSFT memorizes, RL generalizes: a comparative study of foundation model post\-training\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 10818–10838\.External Links:[Link](https://proceedings.mlr.press/v267/chu25c.html)Cited by:[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Chuanget al\.\(2026\)Y\. Chuang, C\. Kulkarni, A\. M\. Chiu, A\. Thangali, Z\. Pan, S\. Shekhar, Y\. Ge, Y\. Li, U\. Kona, L\. Pang, and P\. MehrotraToward scalable verifiable reward: proxy state\-based evaluation for multi\-turn tool\-calling LLM agents\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics: Industry Track,pp\. 1251–1264\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-industry.87)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Efron and Tibshirani \(1994\)B\. Efron and R\. J\. TibshiraniAn introduction to the bootstrap\.Chapman & Hall/CRC\.Cited by:[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.SSS0.Px3.p1.1)\.
- Gonuguntla \(2026\)A\. GonuguntlaThe replay gap: static evaluation of model switching in LLM agents scores the wrong world\.InEfficient Reasoning Workshop at COLM,External Links:[Link](https://openreview.net/forum?id=8gqqiNrzyA)Cited by:[§3](https://arxiv.org/html/2609.19636#S3.p1.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Grittaet al\.\(2026\)M\. Gritta, D\. Paul, X\. Li, L\. Shang, J\. Wang, and G\. LampourasProcess evaluation for agentic systems\.InFindings of the Association for Computational Linguistics: EACL 2026,pp\. 2678–2692\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.140)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Heet al\.\(2026\)L\. He, J\. Zhu, Y\. Zhou, Z\. Gu, J\. Liu, W\. Huang, H\. P\. Zou, D\. Wipf, P\. S\. Yu, and Q\. WuResolving action bottleneck: agentic reinforcement learning informed by token\-level energy\.arXiv preprint arXiv:2605\.14558\.Cited by:[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Hendersonet al\.\(2018\)P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. MegerDeep reinforcement learning that matters\.InAAAI Conference on Artificial Intelligence,Cited by:[§3\.2](https://arxiv.org/html/2609.19636#S3.SS2.p1.1)\.
- Hernánet al\.\(2004\)M\. A\. Hernán, S\. Hernández\-Díaz, and J\. M\. RobinsA structural approach to selection bias\.Epidemiology15\(5\),pp\. 615–625\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§2\.4](https://arxiv.org/html/2609.19636#S2.SS4.p3.1)\.
- Holtzmanet al\.\(2020\)A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. ChoiThe curious case of neural text degeneration\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.19636#A1.SS0.SSS0.Px2.p1.1)\.
- Huoet al\.\(2026\)Y\. Huo, S\. Xing, C\. Wang, P\. Feng, Q\. He, Y\. Ding, A\. Ma, Y\. Gao, T\. Liu, T\. Xiao, and J\. ZhuLearning from environmental feedback: credit assignment across multiple timescales for agentic reinforcement learning\.arXiv preprint arXiv:2608\.08255\.Cited by:[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Jiet al\.\(2026\)Y\. Ji, Z\. Wang, Y\. Wang, S\. Yang, Z\. Ma, G\. Chen, Z\. Sun, L\. Wu, and X\. ChuLearning agentic policy from action guidance\.arXiv preprint arXiv:2605\.12004\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Jinet al\.\(2025\)H\. Jin, S\. Luan, T\. Ni, S\. Lyu, G\. Rabusseau, R\. Rabbany, D\. Precup, and M\. HamdaqaRL fine\-tuning heals OOD forgetting in SFT\.arXiv preprint arXiv:2509\.12235\.Note:Presented at the NeurIPS 2025 Workshop on Foundations of Reasoning in Language ModelsCited by:[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Kaelbling \(1993\)L\. P\. KaelblingLearning to achieve goals\.InInternational Joint Conference on Artificial Intelligence,pp\. 1094–1099\.Cited by:[§3\.1](https://arxiv.org/html/2609.19636#S3.SS1.p1.1)\.
- Kakade and Langford \(2002\)S\. Kakade and J\. LangfordApproximately optimal approximate reinforcement learning\.InProceedings of the 19th International Conference on Machine Learning,pp\. 267–274\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.19636#S2.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Luoet al\.\(2025\)X\. Luo, Y\. Zhang, Z\. He, Z\. Wang, S\. Zhao, D\. Li, L\. K\. Qiu, and Y\. YangAgent lightning: train any AI agents with reinforcement learning\.arXiv preprint arXiv:2508\.03680\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Maet al\.\(2024\)C\. Ma, J\. Zhang, Z\. Zhu, C\. Yang, Y\. Yang, Y\. Jin, Z\. Lan, L\. Kong, and J\. HeAgentBoard: an analytical evaluation board of multi\-turn LLM agents\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 74325–74362\.External Links:[Document](https://dx.doi.org/10.52202/079017-2365)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Nget al\.\(1999\)A\. Y\. Ng, D\. Harada, and S\. RussellPolicy invariance under reward transformations: theory and application to reward shaping\.InInternational Conference on Machine Learning,pp\. 278–287\.Cited by:[§3\.1](https://arxiv.org/html/2609.19636#S3.SS1.p1.1)\.
- Pearl and Bareinboim \(2011\)J\. Pearl and E\. BareinboimTransportability of causal and statistical relations: a formal approach\.InProceedings of the 25th AAAI Conference on Artificial Intelligence,pp\. 247–254\.Cited by:[§3\.3](https://arxiv.org/html/2609.19636#S3.SS3.p1.2)\.
- Renet al\.\(2026\)Q\. Ren, P\. Wang, R\. Cai, S\. Shao, D\. Guo, Y\. Xie, Y\. Li, Q\. Zhang, X\. Hu, J\. Shao, and D\. LiuRethinking generalization in reasoning SFT: a conditional analysis on optimization, data, and model capability\.arXiv preprint arXiv:2604\.06628\.Note:Presented at the FoGen Workshop at ICML 2026 \(non\-archival\)Cited by:[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Rubin \(1974\)D\. B\. RubinEstimating causal effects of treatments in randomized and nonrandomized studies\.Journal of Educational Psychology66\(5\),pp\. 688–701\.Cited by:[§2\.1](https://arxiv.org/html/2609.19636#S2.SS1.p1.2)\.
- Shaoet al\.\(2026\)W\. Shao, C\. Fang, Z\. Tan, Z\. Liang, S\. Rafatirad, A\. Sasan, and H\. HomayounBeyond end\-to\-end success: diagnosing failures in long\-horizon security LLM agents\.arXiv preprint arXiv:2608\.20563\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Shridharet al\.\(2021\)M\. Shridhar, X\. Yuan, M\. Cote, Y\. Bisk, A\. Trischler, and M\. HausknechtALFWorld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.p1.1)\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.2 edition,MIT Press\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p2.1)\.
- Triantafyllouet al\.\(2025\)S\. Triantafyllou, A\. Sukovic, Y\. Zolfimoselo, and G\. RadanovicCounterfactual effect decomposition in multi\-agent sequential decision making\.InProceedings of the 42nd International Conference on Machine Learning,pp\. 60072–60098\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- VanderWeele \(2015\)T\. J\. VanderWeeleExplanation in causal inference: methods for mediation and interaction\.Oxford University Press\.Cited by:[§2\.2](https://arxiv.org/html/2609.19636#S2.SS2.p1.1)\.
- Wanget al\.\(2026a\)H\. Wang, Q\. Xu, C\. Wang, T\. Xue, C\. Peng, W\. Chen, and F\. LinBad seeing or bad thinking? rewarding perception for multimodal reasoning\.InProceedings of the 43rd International Conference on Machine Learning,Note:To appearExternal Links:[Link](https://icml.cc/virtual/2026/poster/62726)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026b\)Z\. Wang, C\. Gui, X\. Jin, Q\. Wang, L\. Liu, K\. Wang, S\. Chen, L\. Li, Z\. Yang, P\. Zhang, Y\. Lu, J\. Wu, L\. Fei\-Fei, L\. Wang, Y\. Choi, and M\. LiRAGEN\-2: reasoning collapse in agentic RL\.InProceedings of the 43rd International Conference on Machine Learning,Note:Oral presentation; to appearExternal Links:[Link](https://ragen-ai.github.io/)Cited by:[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu, E\. Gottlieb, Y\. Lu, K\. Cho, J\. Wu, L\. Fei\-Fei, L\. Wang, Y\. Choi, and M\. LiRAGEN: understanding self\-evolution in LLM agents via multi\-turn reinforcement learning\.arXiv preprint arXiv:2504\.20073\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.20073),[Link](https://arxiv.org/abs/2504.20073)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2026\)X\. Wu, Q\. Sun, R\. Zhang, C\. Song, J\. Wu, Y\. Qi, and H\. ChengDemystifying reinforcement learning for long\-horizon tool\-using agents: a comprehensive recipe\.InProceedings of the 2026 Conference on Empirical Methods in Natural Language Processing,Note:To appearExternal Links:[Link](https://github.com/WxxShirley/Agent-STAR)Cited by:[Appendix A](https://arxiv.org/html/2609.19636#A1.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.p1.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Xiet al\.\(2026\)Z\. Xi, J\. Huang, C\. Liao, B\. Huang, J\. Liu, H\. Guo, Y\. Yang, R\. Zheng, J\. Ye, J\. Zhang, W\. Chen, W\. He, Y\. Ding, G\. Li, Z\. Chen, Z\. Du, X\. Yao, Y\. Xu, J\. Chen, T\. Gui, Z\. Wu, Q\. Zhang, X\. Huang, and Y\. JiangAgentGym\-RL: an open\-source framework to train LLM agents for long\-horizon decision making via multi\-turn RL\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZgCCDwcGwn)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Xiaet al\.\(2026\)P\. Xia, J\. Chen, H\. Wang, J\. Liu, K\. Zeng, Y\. Wang, S\. Han, Y\. Zhou, X\. Zhao, H\. Chen, Z\. Zheng, C\. Xie, and H\. YaoSkillRL: evolving agents via recursive skill\-augmented reinforcement learning\.InICLR 2026 Workshop on Memory for LLM\-Based Agentic Systems,Note:Oral presentationExternal Links:[Link](https://openreview.net/forum?id=FYc2IygegR)Cited by:[Appendix A](https://arxiv.org/html/2609.19636#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.19636#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.p1.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px1.p1.1)\.
- Xieet al\.\(2024\)J\. Xie, K\. Zhang, J\. Chen, T\. Zhu, R\. Lou, Y\. Tian, Y\. Xiao, and Y\. SuTravelPlanner: a benchmark for real\-world planning with language agents\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 54590–54613\.External Links:[Link](https://proceedings.mlr.press/v235/xie24j.html)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.19636#S4.SS1.p1.1)\.
- Yaoet al\.\(2025\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1)\.
- Yeet al\.\(2026\)Z\. Ye, W\. Shi, Y\. Liu, Y\. Wang, Z\. Cai, Y\. Shi, Q\. Gu, X\. Cai, and F\. FengLook before you leap: autonomous exploration for LLM agents\.arXiv preprint arXiv:2605\.16143\.Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p3.1),[§5](https://arxiv.org/html/2609.19636#S5.SS0.SSS0.Px2.p1.1)\.
- Zenget al\.\(2024\)A\. Zeng, M\. Liu, R\. Lu, B\. Wang, X\. Liu, Y\. Dong, and J\. TangAgentTuning: enabling generalized agent abilities for LLMs\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 3053–3077\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.181)Cited by:[§1](https://arxiv.org/html/2609.19636#S1.p1.1)\.

## Appendix AReproducibility Details

#### Released checkpoints\.

We evaluate existing checkpoints without retraining them\. Table[4](https://arxiv.org/html/2609.19636#A1.T4)lists the immutable revisions used throughout the paper\. Agent\-STAR\([Wu et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib3)\)provides three SFT/RL pairs for TravelPlanner, and SkillRL\([Xia et al\., 2026](https://arxiv.org/html/2609.19636#bib.bib4)\)provides one SFT/RL pair for ALFWorld\. The two releases constitute independent training pipelines, but both use architectures from the broad Qwen family; the experiment therefore does not establish transfer across unrelated base\-model architectures\.

Table 4:Released checkpoints and fixed revisions\.
#### TravelPlanner protocol\.

For each scale, Val180 contains 60 easy, 60 medium, and 60 hard tasks\. The primary analysis crosses two reachers and two solvers at budgetB=4B=4, yielding 720 rows per scale\. Inference uses Python 3\.10\.20, PyTorch 2\.6\.0 with CUDA 11\.8, vLLM 0\.8\.1 with CUDA 11\.8, FP16, and a 32K\-token context\. We use nucleus sampling\([Holtzman et al\., 2020](https://arxiv.org/html/2609.19636#bib.bib6)\)with temperature 0\.6, top\-pp0\.95, and seed 2027\. Confidence intervals use 100,000 task\-level paired bootstrap resamples\. The archived Formatter alias isdeepseek\-chat; its raw replies are retained\. All requests completed, and recomputing the official Agent\-STAR evaluator produced zero per\-example reward differences\. The three formal terminal summaries pass their protocol acceptance checks\.

#### TravelPlanner handoff construction\.

Each reacher first runs its released checkpoint under the native 60\-decision and 60\-tool limits\. The handoff is the point immediately before that trajectory’s final model call, rather than a common elapsed step\. The rule is a function of the recorded reacher trajectory alone, so it satisfies Section[2\.1](https://arxiv.org/html/2609.19636#S2.SS1), but its states are late rather than early\. We reset the task and replay every preceding assistant action and tool response, then require the reconstructed message list to equal the recorded list\. For each history, the SFT and RL solvers receive that entire list and the recorded final\-turn seed; no message is summarized, reordered, or semantically repaired\. The recorded first solver decision is theB=1B=1snapshot\. If it is a valid tool call, the evaluator executes it, appends its native tool response, and permits at most three further decisions, using consecutive paired seeds, to formB=4B=4\. Answers, invalid actions, and context limits are absorbing\. ThusB=4B=4counts four post\-handoff model decisions, not the four reacher–solver assignments\. The Formatter is applied only after the solver terminates\. At every scale, theB=4B=4audit contains 720 unique rows, 180 per assignment, with equal paired input hashes, no Formatter request failures, and no official\-score recomputation differences\.

#### Formatter measurement audit\.

The fixed\-handoff Val180 run initially contained two unique empty Formatter envelopes, corresponding to four endpoint rows\. The separately registered repair protocol retried each input exactly once with the same model, prompt, JSON contract, and disabled\-thinking setting, using no new trajectories and changing no checkpoint assignment; it writes to an isolated artifact\. Both requests returned valid envelopes\. The repaired directory contains 292 valid unique Formatter samples, 720 formatted rows, and 720 officially scored rows; the evaluator recomputation reported zero mismatches\. On the 716 unaffected rows, all identity fields, formatted\-plan hashes, official metrics, andrsumr\_\{\\mathrm\{sum\}\}are identical to the source artifact\. Eight diagnosticreward\_detailsmessages differ only in the order of an unordered list, which is recorded separately and does not change any score\. The fixed\-handoff interaction remains7\.227\.22points with 95% CI\[1\.67,12\.78\]\[1\.67,12\.78\], so the positive interaction is not explained by the two transient measurement failures\.

#### ALFWorld protocol\.

The unseen\-split sample \(valid\_unseenin the ALFWorld release; the seen\-split sample isvalid\_seen\) contains 30 tasks stratified across the six ALFWorld task types\. Each released checkpoint receives 48 early environment steps\. Natural arrival is measured at the environment\-verified distance\-two frontierℱ2\\mathcal\{F\}\_\{2\}\. Every arriving state is replayed, verified, cloned, and continued by both solvers with two remaining environment steps and four paired seed slots\. Inference uses vLLM 0\.8\.1, BF16, a 4,096\-token prompt limit, a 512\-token response limit\. We use temperature\-scaled multinomial sampling with temperature 0\.4 and top\-pp1 \(no probability truncation\), with selection seed 2029\. The outcome is the environment’s nativewonflag\. The main protocol produces 115 verified frontier states and passes all replay, pairing, and completion checks\.

#### ALFWorld solvability check and clone audit\.

For a candidate action prefix, the check first rejects terminal states, then replays every currently admissible action\. It rejects the candidate if any first action wins; otherwise it replays every admissible second action from each nonterminal successor and accepts only if at least one two\-action path wins\. This exhaustive local search includes alternative shortest paths and states from which some actions are irreversible\. Across the 115 accepted handoffs, it tested 3,227 first actions and 89,511 second actions, and never failed\.

The environment is restored by deterministic reset with the same task seed and exact replay of the recorded reacher actions; we do not serialize a Python environment object\. After reset and every reacher action, equality is checked on a JSON\-safe record containing the observation, ordered admissible\-action list, score, terminal flag, andwonflag\. At handoff, the visible history, rendered prompt, prompt\-token count, and token\-ID digest must also agree\. The two solvers are run in separate replays from this operational clone\. All 230 arrival\-solver replays passed, all 115 solver pairs had equal inputs and seed schedules, and no prompt exceeded 4,096 tokens\. Table[5](https://arxiv.org/html/2609.19636#A1.T5)maps each common attribution concern to the protocol control that defines the reported quantity\.

Table 5:Design questions and the protocol control that answers each\.

## Appendix BIndependent Solve Profile Sample

The externalSolveprofile is selected before inspecting continuation outcomes on the targetvalid\_unseensample\. It comes from a separatevalid\_seenexperiment containing five tasks per task type\. Four paired seeds yield 120 distance\-two states per solver\. We combine these six type\-specific conversion rates with target\-split type\-specific arrival rates\. This estimator requires transportability of continuation success within task type from the independent sample to the target split\. It deliberately does not assume that task type is sufficient for every aspect of state difficulty\.

The resulting interaction prediction is 20\.7 percentage points, compared with an observed interaction of 24\.2 points\. The residual is 3\.5 points with 95% CI\[−7\.2,14\.2\]\[\-7\.2,14\.2\], quantifying the remaining difference between prediction and observation and the uncertainty from the finite task samples\. Figure[8](https://arxiv.org/html/2609.19636#A2.F8)plots both solver levels by task type rather than their difference:*pick & place*gains only 5 points because the SFT solver already completes 95\.2% of those states, whereas*clean then place*gains 100 points from a floor of zero\. Figure[9](https://arxiv.org/html/2609.19636#A2.F9)shows the per\-type residuals behind the 3\.5\-point aggregate: type\-level errors are larger and have both signs\. Table[6](https://arxiv.org/html/2609.19636#A2.T6)lists the same residuals with the arrival and conversion rates they are computed from\. Type\-level residuals run from\+25\.0\+25\.0on look at object in light to−12\.0\-12\.0on heat then place and cancel in the unweighted mean\. Comparing the external conversion gap with the same\-stateSolvegap on the target’s own replayed states \(Figure[8](https://arxiv.org/html/2609.19636#A2.F8)\) locates the error\. For look at object in light the external profile gives a2525\-point gap where the target states show7070, and the prediction falls2525points short\. For pick and place it gives1515where the target shows4\.84\.8, and it overshoots by10\.210\.2\. Five of the six residuals share the sign of this difference; heat then place is the exception, with gaps of2020and21\.721\.7that agree\. The error sits in the transport of the within\-typeSolveprofile, which is the assumption the external test was designed to expose\.

Table 6:Per\-task\-type interaction prediction \(Equation[9](https://arxiv.org/html/2609.19636#S3.E9)\)\.Arrival is from the targetvalid\_unseensplit and theSolveprofile from the independent sample\. The overall row is the unweighted mean over types, the stratified bootstrap unit; dashes mark cells it leaves undefined\.Figure 8:Same\-stateSolveby task type onvalid\_unseen\.Rows are sorted by the RL\-minus\-SFT gap\. The source data report no per\-type intervals\.Figure 9:Per\-task\-type interaction\-prediction residual, observed minus predicted\.Positive means the external profile underpredicts\. The source data report no per\-type intervals\.
## Appendix CRobustness Checks

#### Seen\-split replication\.

On a separately selected, type\-balanced 30\-taskvalid\_seensample, RL improves frontier arrival by 65\.8 points with 95% CI\[54\.2,77\.5\]\[54\.2,77\.5\], improves same\-state completion by 26\.5 points with CI\[11\.2,41\.3\]\[11\.2,41\.3\], and yields a positive endpoint interaction of 17\.5 points with CI\[6\.7,27\.5\]\[6\.7,27\.5\]\. All three effects replicate on a different split with positive lower confidence bounds\.

#### Solveby solver action\.

Every frontier state admits bridge actions, first actions from which one further action wins, soSolvefactors into choosing a bridge action and finishing from it \(Table[3](https://arxiv.org/html/2609.19636#S4.T3)\)\. Onvalid\_unseenthe SFT solver chooses a bridge action on72\.2%72\.2\\%of states and finishes from one on78\.3%78\.3\\%of those; the RL solver does so on97\.4%97\.4\\%and98\.2%98\.2\\%\. The two stages contribute comparably to the SFT deficit,25\.225\.2and19\.919\.9points, and their products reproduce the completion rates of56\.5%56\.5\\%and95\.7%95\.7\\%\. Of the115115states,4545are won by the RL solver only and none by the SFT solver only\. Onvalid\_seenthe RL solver alone wins4141states and the SFT solver alone wins1010\. The SFT solver behaves almost identically on the two splits,72\.6%72\.6\\%against72\.2%72\.2\\%at the bridge stage and49\.6%49\.6\\%against56\.5%56\.5\\%overall\. The change is on the RL side: its finishing rate given a bridge action falls from98\.2%98\.2\\%to82\.4%82\.4\\%, while its bridge rate falls only from97\.4%97\.4\\%to92\.3%92\.3\\%\. The replication preserves the direction and the two\-stage structure of theSolveeffect; the RL advantage at the second action is the less stable of the two\.

#### Early\-action budget diagnostic\.

We reuse archivedvalid\_seentrajectories at common early\-action budgets 8, 16, and 48\. Table[7](https://arxiv.org/html/2609.19636#A3.T7)lists the arrival levels and their contrasts\. This post\-hoc diagnostic shows a positive RL arrival advantage at eight steps that remains large as both checkpoints receive more steps\. In the direct cross\-budget comparison, RL at eight actions exceeds SFT at 48 actions by 31\.7 points with CI\[19\.2,44\.2\]\[19\.2,44\.2\]\. Budget data exist forvalid\_seenonly; no acceptedvalid\_unseenbudget artefact exists, so these intervals are not attached to thevalid\_unseencurve of Figure[2](https://arxiv.org/html/2609.19636#S2.F2)\. The claim is bounded by the protocol: cumulative arrival within 48 reacher actions under frozen policies, with no extrapolation to additional sampling, search, or replanning\.

Figure 10:Frontier arrival against reacher action budget onvalid\_seen\.Arrival at three budgets on the separately selectedvalid\_seensample, with task\-stratified paired\-bootstrap95%95\\%intervals\. Thevalid\_unseencurve is Figure[2](https://arxiv.org/html/2609.19636#S2.F2)\.Table 7:Frontier arrival by reacher budget onvalid\_seen\.The bolded row contrasts RL at 8 actions with SFT at 48\. Brackets are task\-stratified paired\-bootstrap 95% intervals from100,000100\{,\}000resamples\.
#### Non\-arrival failure modes\.

Everyvalid\_unseentrajectory that never reachesℱ2\\mathcal\{F\}\_\{2\}\(104104SFT,2121RL, all running the full4848actions\) is assigned one of seven mutually exclusive labels by an automatic audit of its action sequence\. For SFT,102102trajectories are loops that repeat actions without changing the state and22miss a subgoal; for RL,1818are loops,22wander, and11never picks up the target object\. No trajectory shows format collapse\. At the step level, SFT repeats its immediately preceding action on66\.7%66\.7\\%of steps \(RL43\.4%43\.4\\%\), revisits an earlier state on86\.3%86\.3\\%\(75\.2%75\.2\\%\), and leaves the state unchanged on76\.2%76\.2\\%\(47\.2%47\.2\\%\); the median number of distinct actions per trajectory is44for SFT and88for RL\. Trajectory\-level bootstrap intervals from100,000100\{,\}000resamples put the RL\-minus\-SFT difference in loop share at−12\.4\-12\.4points with CI\[−28\.6,1\.0\]\[\-28\.6,1\.0\]\. The audit is descriptive and makes no mechanism claim\.

#### Non\-arrival retention diagnostic\.

We compare two estimands on the accepted 480\-rowvalid\_unseenmatrix\. The endpoint estimand retains non\-arrivals as zero outcomes\. The survivor\-only estimand computes the paired RL\-minus\-SFT solver effect separately on the states reached by each reacher, then differences those two effects\. Both use 100,000 task\-stratified paired\-bootstrap draws with all four seeds and checkpoint assignments kept together\. Fifty\-five draws have no SFT arrival and are excluded under the fixed empty\-denominator rule, a rate of0\.055%0\.055\\%against the1%1\\%stopping threshold\.

The endpoint interaction is24\.224\.2points with 95% CI\[13\.3,35\.0\]\[13\.3,35\.0\]\. The survivor\-only estimate is−12\.6\-12\.6points with CI\[−45\.8,22\.7\]\[\-45\.8,22\.7\], and its paired difference from the endpoint interaction is−36\.8\-36\.8points with CI\[−68\.6,−7\.3\]\[\-68\.6,\-7\.3\]\. SFT reachesℱ2\\mathcal\{F\}\_\{2\}in only 16 task and seed trajectories from six tasks and has no arrivals in two of the six task types; RL reaches it in 99 trajectories from 28 tasks\. This support loss explains the wide survivor\-only interval\. The diagnostic was specified after its point estimates were observed, so it establishes neither a confirmatory effect nor a negative conditional interaction\. It shows that deleting non\-arrivals does not preserve the endpoint estimand in this setting\.

## Appendix DA Handoff State in Detail

Table[8](https://arxiv.org/html/2609.19636#A4.T8)shows one of the4545valid\_unseenstates that the RL solver completes and the SFT solver does not, selected by a prespecified rule \(SFT\-produced state, simplest task type, SFT first action not a bridge action\)\. The task is “put a pencil in shelf”\. The SFT reacher produced the state by taking the pencil from the desk; the cloned observation reads “You pick up the pencil 1 from the desk 1”, and the solvability check lists six bridge actions, walking to any of shelves 1 to 6, each followed by the single winning action of moving the pencil there\. Both solvers receive the identical clone; the two handoff prompts hash to the same value\. The SFT solver’s reasoning names the right plan, “head straight to the nearest shelf”, and then spends its first action onlook; its second action walks to a shelf, one action too late\. The RL solver walks to a shelf and places the pencil\.

Table 8:Two solvers on one cloned state\.Bridge actions in blue\. The SFT solver’s first\-step reasoning is quoted from its archived reply\.