Li$^{2}$SAO：通过组相对行动者-评论家自举提升轻量级单次轨迹采样异步强化学习优化

# Li2SAO: Lifting Light Single-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
Source: [https://arxiv.org/html/2610.40726](https://arxiv.org/html/2610.40726)
Yanli Wang⋄\\diamondYang Liu⋄\\diamondJunhao ShiXueqin LiuJinguo SunAffiliation:Muye Li, Shaoqiang Zheng†\\dagger, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng CaoAffiliation:JD\.Com

###### Abstract

Asynchronous reinforcement learning \(RL\) is becoming a mainstream paradigm for post\-training large language models \(LLMs\), as it improves training efficiency by updating the model as rollouts arrive\. Recent work argues that group\-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long\-tailed distribution, and proposes Single\-rollout Asynchronous Optimization \(SAO\) to improve stability and off\-policy behavior, especially on agentic tasks\. However, SAO relies on a closed\-source value model obtained from scaling value pretraining\. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single\-turn reasoning and multi\-turn agentic tasks\. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks\. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps\. To address this issue, we proposeLi2SAO, whichLiftsLightSingle\-rolloutAsynchronousOptimization with group\-relative actor\-critic bootstrapping: while the critic is cold, a group\-relative advantage trains the actor and a group\-centered return trains the critic, after which training switches to single\-rollout GAE\. Experiments show that Li2SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling\. Code is available at[https://github\.com/jd\-cc\-coder/Li2SAO](https://github.com/jd-cc-coder/Li2SAO)\.

## 1Introduction

Reinforcement learning \(RL\) has become a central approach for eliciting long chain of thought reasoning and multi\-turn agentic behavior in large language models \(LLMs\), including tool use and environment interaction\([Ouyang et al\., 2022](https://arxiv.org/html/2610.40726#bib.bib4);[Shao et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib16);[DeepSeek\-AI, 2025](https://arxiv.org/html/2610.40726#bib.bib17)\)\. Most existing RL systems couple generation and training by completing a full batch of trajectories under a fixed policy before each update, so variable lengths on long reasoning and tool use tasks force fast rollouts to wait for stragglers\([Yao et al\., 2023](https://arxiv.org/html/2610.40726#bib.bib5);[Hu et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib6);[Sheng et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib7)\)\. Asynchronous RL removes this coupling by running rollout generation and training concurrently, allowing trajectories to enter training as they become available without draining the rollout workers, which substantially improves training efficiency\([Noukhovitch et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib8);[Sheng et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib7);[Zhong et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib9);[Wu et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib10)\)\. A fully asynchronous extension is the*in\-flight*weight\-update scheme, in which rollout workers receive updated parameters and continue unfinished trajectories without discarding generated tokens\([Fu et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib11);[Piché et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib12)\)\.

However, asynchronous RL introduces two challenges\. First, under the*in\-flight*weight\-update scheme, a trajectory can span multiple policy versions, compounding policy lag with within trajectory distribution shift and destabilizing optimization\([Huang et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib14)\)\. Second, the widely used Group Relative Policy Optimization \(GRPO\) is structurally mismatched to asynchronous training\([Shao et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib16);[DeepSeek\-AI, 2025](https://arxiv.org/html/2610.40726#bib.bib17)\)\. GRPO must wait for all responses to the same prompt before computing group relative advantages, which recreates a synchronization barrier and increases the staleness of early responses\. Its sequence level advantage also provides limited temporal credit assignment for long agentic trajectories with multiple decisions and environment interactions\.

To address these challenges,[Hou et al\. \(2026\)](https://arxiv.org/html/2610.40726#bib.bib1)proposes Single\-rollout Asynchronous Optimization \(SAO\), which replaces group wise sampling with one rollout per prompt and consumes each trajectory as soon as it arrives\. SAO stabilizes off policy updates with Direct Double Sided Importance Sampling \(DIS\) and supports agentic credit assignment with Skip Observation token level GAE\. However, controlling the high variance of single rollout optimization requires a well calibrated value model, and SAO relies on scaled value pretraining whose model and construction pipeline are not publicly available\. This raises a natural question:

*How can we recover the benefits of SAO without a pretrained value model?*

To answer this question, we first reproduce a lightweight form of SAO \(LiSAO\) without value pretraining\. Our implementation is built on the open sourceslimeRL framework111[https://github\.com/THUDM/slime](https://github.com/THUDM/slime)and we implement the*in\-flight*weight\-update asynchronous training for both single\-turn reasoning and multi\-turn sandbox\-based agentic tasks\. We find that GRPO and LiSAO exhibit complementary strengths, with GRPO performing better on single\-turn reasoning and LiSAO on multi\-turn agentic tool use\. More importantly, LiSAO consistently lags behind GRPO during early optimization in both settings\. Our analysis attributes this early gap to critic calibration\. A critic initialized from the policy checkpoint yields inaccurate temporal differences and noisy GAE estimates until it tracks the return, and this recovery is slower on tool use trajectories\. Once the critic tracks the return, single\-rollout GAE assigns credit along long tool interactions, while single\-turn reasoning gains little beyond a sequence\-level baseline\.

These findings motivateLi2SAO, whichLiftsLightSingle\-rolloutAsynchronousOptimization through group relative actor\-critic bootstrapping before transitioning to single rollout optimization\. During critic cold start, Li2SAO samples a small response group per prompt, trains the actor with normalized group relative advantages, and trains the critic on group centered residual returns\. Once the critic is deemed sufficiently calibrated, Li2SAO switches once to single rollout GAE\. Our policy gradient analysis explains how group comparisons align early updates with success probability and how a calibrated critic lets single\-rollout GAE recover that direction\. Experiments on reasoning and agentic tasks show that Li2SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling\. Our contributions are summarized as follows:

- •Lightweight Reproduction\.We implement in\-flight asynchronous RL onslimeand reproduce LiSAO without the inaccessible scaled value pretraining stage for single\-turn reasoning and multi\-turn agentic tasks\.
- •Cold\-critic Diagnosis\.We conduct a controlled comparison of GRPO and LiSAO across both task families\. Our study reveals their distinct strengths, identifies the cold critic as the source of the early LiSAO gap, and provides insights into group based and value based asynchronous optimization\.
- •Group\-relative Lifting\.We propose Li2SAO, which uses group\-relative actor\-critic bootstrapping and then switches once to single rollout GAE\. Our theoretical analysis and empirical evidence showing that this design improves early optimization while retaining the later benefits of temporal credit assignment\.

## 2Preliminaries

In reinforcement learning for language models, letq∼𝒟q\\sim\\mathcal\{D\}denote a prompt and letπθ\\pi\_\{\\theta\}denote the policy\. A trajectoryτ\\tauis generated byπθ\\pi\_\{\\theta\}interacting with an environment\. For a single\-turn task,τ\\taucontains only one model response\. For a multi\-turn agentic task, model\-generated actions are interleaved with observation sequences returned by the environment\. We serialize all action and observation tokens afterqqasz1:Tz\_\{1:T\}and define the statest=\(q,z<t\)s\_\{t\}=\(q,z\_\{<t\}\)\. We setmt=1m\_\{t\}=1whenztz\_\{t\}is generated by the model andmt=0m\_\{t\}=0when it belongs to an environment observation\. At a model\-generated position, we writeat=zta\_\{t\}=z\_\{t\}, soata\_\{t\}is the next model\-generated token sampled fromπθ\(⋅∣st\)\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\. Policy loss, value loss, and advantage estimation are restricted to positions wheremt=1m\_\{t\}=1\. The environment assigns a trajectory rewardR⁡\(τ\)R\(\\tau\)at the end of the interaction\. We writertr\_\{t\}for the reward assigned to model\-generated tokenata\_\{t\}\. In this work,rt=0r\_\{t\}=0except at the final model\-generated position, where it equalsR⁡\(τ\)R\(\\tau\)\.

### 2\.1In\-Flight Asynchronous RL

In synchronous RL, rollout workers generate a complete batch under a frozen policy, and the trainer starts an update only after the batch is complete\([Sheng et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib7)\)\. Asynchronous RL instead runs generation and training concurrently\([Noukhovitch et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib8)\)\. Completed trajectories enter a buffer immediately, and the trainer repeatedly consumes available trajectories while rollout workers continue generation\. If the trainer holds policy versionvlv\_\{l\}, a trajectory produced by versionvs<vlv\_\{s\}<v\_\{l\}has*policy lag*vl−vsv\_\{l\}\-v\_\{s\}\. Large lag makes the data increasingly off\-policy and can destabilize optimization\([Zheng et al\., 2025b](https://arxiv.org/html/2610.40726#bib.bib13);[Huang et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib14)\)\.

In this paper, we study the*in\-flight*weight\-update asynchronous RL setting implemented onslime\. During a weight update, an unfinished rollout is paused without discarding its generated prefix and, after synchronization, resumes from that prefix under the latest policy\. Consequently, a model\-generated tokenata\_\{t\}may be sampled from a rollout policyπr,vt\\pi\_\{\\mathrm\{r\},v\_\{t\}\}, where the versionvtv\_\{t\}can change within the same trajectory\. The RL system we study storeslog⁡πr,vt​\(at∣st\)\\log\\pi\_\{\\mathrm\{r\},v\_\{t\}\}\(a\_\{t\}\\mid s\_\{t\}\)in the rollout engine during generation\. Thus, for the current policyπθ\\pi\_\{\\theta\}, the token\-level importance ratio can be computed asρt​\(θ\)=πθ​\(at∣st\)πr,vt​\(at∣st\)\\rho\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\mathrm\{r\},v\_\{t\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}\. Given an advantage estimateA^t\\hat\{A\}\_\{t\}, the clipped policy objective is

𝒥⁡\(θ\)=𝔼q∼𝒟,τ∼\(πr,ℰ\)​\[1∑t=1Tmt​∑t=1Tmt​ℓt​\(θ\)\],\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\_\{q\\sim\\mathcal\{D\},\\,\\tau\\sim\(\\pi\_\{\\mathrm\{r\}\},\\mathcal\{E\}\)\}\\left\[\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}m\_\{t\}\}\\sum\_\{t=1\}^\{T\}m\_\{t\}\\ell\_\{t\}\(\\theta\)\\right\],\(1\)whereℰ\\mathcal\{E\}is the environment,πr\\pi\_\{\\mathrm\{r\}\}denotes the \(possibly version\-changing\) rollout policy, andℓt​\(θ\)=min⁡\(ρt​\(θ\)​A^t,clip⁡\(ρt​\(θ\),1−ϵℓ,1\+ϵh\)​A^t\)\\ell\_\{t\}\(\\theta\)=\\min\\bigl\(\\rho\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\},\\allowbreak\\operatorname\{clip\}\(\\rho\_\{t\}\(\\theta\),1\-\\epsilon\_\{\\ell\},1\+\\epsilon\_\{h\}\)\\hat\{A\}\_\{t\}\\bigr\)\. After an update, leftover trajectories may remain in the rollout buffer and become increasingly off\-policy\. In this paper, we cap this staleness with a maximum policy lagkk: samples whose lag exceeds this threshold are discarded\. Different RL algorithms mainly differ in how they estimateA^t\\hat\{A\}\_\{t\}and how they handle extreme importance ratios\.

### 2\.2Group\-wise Sampling and GRPO

GRPO avoids learning a value function by constructing a sampled baseline from the rewards of multiple trajectories for each prompt\([Shao et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib16)\)\. Given a promptqqand a group ofnntrajectories\{τi\}i=1n\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{n\}with rewards\{Ri\}i=1n\\\{R\_\{i\}\\\}\_\{i=1\}^\{n\}, the group\-relative advantage for trajectoryiiis

A^iGRPO=Ri−1n​∑j=1nRj1n−1​∑j=1n\(Rj−1n​∑k=1nRk\)2\+ε\.\\hat\{A\}\_\{i\}^\{\\mathrm\{GRPO\}\}=\\frac\{R\_\{i\}\-\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}R\_\{j\}\}\{\\sqrt\{\\frac\{1\}\{n\-1\}\\sum\_\{j=1\}^\{n\}\\bigl\(R\_\{j\}\-\\frac\{1\}\{n\}\\sum\_\{k=1\}^\{n\}R\_\{k\}\\bigr\)^\{2\}\}\+\\varepsilon\}\.\(2\)This scalar is assigned to every model\-generated token in trajectoryii\. Equal group rewards yield zero advantage and no policy gradient\.

GRPO offers a strong critic\-free estimator when multiple comparable samples can be collected efficiently\. However, computing group statistics requires waiting for the slowest trajectory, creating a prompt\-level synchronization barrier that increases policy lag\. In multi\-turn trajectories, the sequence\-level advantage in Eq\.\([2](https://arxiv.org/html/2610.40726#S2.E2)\) also assigns the same advantage to every reasoning step and tool call\.

### 2\.3Single\-rollout Optimization and SAO

Single rollout optimization setsn=1n=1and removes the need to assemble prompt level groups\. Since the group\-relative advantage degenerates, it learns a criticVϕ​\(st\)V\_\{\\phi\}\(s\_\{t\}\)that predicts the expected return from statests\_\{t\}\. PPO commonly uses GAE to construct token\-level advantages\([Schulman et al\., 2016](https://arxiv.org/html/2610.40726#bib.bib20);[Schulman et al\., 2017](https://arxiv.org/html/2610.40726#bib.bib19)\)\. For adjacent model\-generated positionsttandt\+1t\+1, the temporal\-difference \(TD\) residual and GAE recursion are

δt=rt\+γ​Vϕ​\(st\+1\)−Vϕ​\(st\),\\displaystyle\\delta\_\{t\}=r\_\{t\}\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\-V\_\{\\phi\}\(s\_\{t\}\),\(3\)A^tGAE=δt\+γ​λ​A^t\+1GAE,\\displaystyle\\hat\{A\}\_\{t\}^\{\\mathrm\{GAE\}\}=\\delta\_\{t\}\+\\gamma\\lambda\\hat\{A\}\_\{t\+1\}^\{\\mathrm\{GAE\}\},\(4\)whereγ\\gammais the discount factor andλ\\lambdacontrols the bias and variance tradeoff\. The corresponding return target isG^t=A^tGAE\+Vϕ​\(st\)\\hat\{G\}\_\{t\}=\\hat\{A\}\_\{t\}^\{\\mathrm\{GAE\}\}\+V\_\{\\phi\}\(s\_\{t\}\), which the critic fits by minimizing

ℒV​\(ϕ\)=𝔼t​\[mt​\(Vϕ​\(st\)−G^t\)2\]\.\\mathcal\{L\}\_\{V\}\(\\phi\)=\\mathbb\{E\}\_\{t\}\\left\[m\_\{t\}\\left\(V\_\{\\phi\}\(s\_\{t\}\)\-\\hat\{G\}\_\{t\}\\right\)^\{2\}\\right\]\.\(5\)Unlike the trajectory\-level GRPO advantage in Eq\.\([2](https://arxiv.org/html/2610.40726#S2.E2)\), GAE produces token\-level advantages that can vary across model\-generated positions within a trajectory\. Its quality, however, directly depends on the accuracy ofVϕV\_\{\\phi\}\.

Agentic trajectories contain observation tokens that are produced by the environmentℰ\\mathcal\{E\}rather than sampled fromπr\\pi\_\{\\mathrm\{r\}\}\. SAO\([Hou et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib1)\)handles them with skip observation GAE\. Letu1<⋯<uMu\_\{1\}<\\cdots<u\_\{M\}be the positions for whichmt=1m\_\{t\}=1, whereM=∑t=1TmtM=\\sum\_\{t=1\}^\{T\}m\_\{t\}\. The skip observation GAE recursion is evaluated only on this subsequence as

δuj=ruj\+γ​Vϕ​\(suj\+1\)−Vϕ​\(suj\),\\displaystyle\\delta\_\{u\_\{j\}\}=r\_\{u\_\{j\}\}\+\\gamma V\_\{\\phi\}\(s\_\{u\_\{j\+1\}\}\)\-V\_\{\\phi\}\(s\_\{u\_\{j\}\}\),\(6\)A^uj=δuj\+γ​λ​A^uj\+1,\\displaystyle\\hat\{A\}\_\{u\_\{j\}\}=\\delta\_\{u\_\{j\}\}\+\\gamma\\lambda\\hat\{A\}\_\{u\_\{j\+1\}\},\(7\)forj<Mj<M\. At the terminal model\-generated token, zero bootstrapping givesδuM=ruM−Vϕ​\(suM\)\\delta\_\{u\_\{M\}\}=r\_\{u\_\{M\}\}\-V\_\{\\phi\}\(s\_\{u\_\{M\}\}\)andA^uM=δuM\\hat\{A\}\_\{u\_\{M\}\}=\\delta\_\{u\_\{M\}\}\. Observation positions receive no policy or value loss\. This links consecutive model\-generated tokens, including tokens separated by an observation sequence, while avoiding value transitions between externally generated tokens\.

Figure 1:GRPO vs\. LiSAOon single\-turn mathematical reasoning \(left\) and ReTool agentic tool use \(right\)\. Each half shows Pass@1 across training steps and Gap \(pp\) across Pass@kk\. Gap \(pp\) is GRPO minus LiSAO, and a positive value means GRPO is ahead\. Bold curves average AIME 2024, 2025, and 2026\. Thin curves are the individual years\.The complete per\-year pass@kkresults are in Figure[6](https://arxiv.org/html/2610.40726#A4.F6)of Appendix[E](https://arxiv.org/html/2610.40726#A5)\.SAO further applies direct double\-sided importance sampling \(DIS\), which formsρt​\(θ\)\\rho\_\{t\}\(\\theta\)from the stored rollout probabilities and masks tokens whose ratios fall outside a prescribed interval:

ℓtDIS​\(θ\)=f⁡\(ρt​\(θ\),ϵℓ,ϵh\)​A^t,\\displaystyle\\ell\_\{t\}^\{\\mathrm\{DIS\}\}\(\\theta\)=f\\bigl\(\\rho\_\{t\}\(\\theta\);\\,\\epsilon\_\{\\ell\},\\epsilon\_\{h\}\\bigr\)\\hat\{A\}\_\{t\},\(8\)f⁡\(x,ϵℓ,ϵh\)=\{x,1−ϵℓ<x<1\+ϵh,0,otherwise\.\\displaystyle f\(x;\\epsilon\_\{\\ell\},\\epsilon\_\{h\}\)=\\begin\{cases\}x,&1\-\\epsilon\_\{\\ell\}<x<1\+\\epsilon\_\{h\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(9\)
Unlike the PPO clip in Eq\.\([1](https://arxiv.org/html/2610.40726#S2.E1)\), out\-of\-range tokens receive a zero multiplier and do not enter the policy gradient\. However, SAO\([Hou et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib1)\)identifies cold\-start value estimation as a major bottleneck and finds scaled value pretraining essential for robustly initializingVϕV\_\{\\phi\}during early\-stage optimization\. Without that initialization, the critic begins uncalibrated and the resulting GAE errors become the central challenge in this work\.

## 3LiSAO without Value Pretraining

In this section, we empirically test whether SAO remains effective without scaled value pretraining\. We compare GRPO and LiSAO under the same in\-flight asynchronous pipeline onslime\. GRPO estimates a sequence\-level advantage from multiple rollouts of the same prompt\([Shao et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib16)\)\. LiSAO is our lightweight reproduction of SAO\([Hou et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib1)\)and uses the same single\-rollout update, DIS, and skip\-observation GAE\. Its critic is a separate network initialized from the policy checkpoint, with attention frozen and only the MLP blocks and a scalar value head trained\. We train Qwen3\-4B\([Yang et al\., 2025a](https://arxiv.org/html/2610.40726#bib.bib26)\)for 500 steps on DAPO\-Math\-17K\([Yu et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib24)\)with a binary outcome reward and a 32K context window, and disable thinking throughout\. The comparison covers single\-turn mathematical reasoning, which starts from the base checkpoint and generates a text solution, and multi\-turn agentic tool use under ReTool\([Feng et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib23)\), which starts from a tool\-use supervised checkpoint and interleaves reasoning with a Python interpreter\. We evaluate on AIME 2024, AIME 2025, and AIME 2026\([Mathematical Association of America, 2026](https://arxiv.org/html/2610.40726#bib.bib28)\), drawing 16 samples per problem at temperature1\.01\.0and reporting the unbiased pass@kkestimator of[Chen et al\. \(2021\)](https://arxiv.org/html/2610.40726#bib.bib27)\. Experimental details are in Appendix[C](https://arxiv.org/html/2610.40726#A3), and the choices that distinguish this reproduction from the original SAO pipeline are in Appendix[D](https://arxiv.org/html/2610.40726#A4)\.

### 3\.1Phenomenon: A Shared Early Gap Precedes Complementary Strengths

#### GRPO leads on single\-turn mathematical reasoning\.

The left half of Figure[1](https://arxiv.org/html/2610.40726#S2.F1)follows this comparison across training steps and acrosskk\. GRPO leads in pass@1 for most of training, and that lead then closes\. Averaged over AIME 2024, 2025, and 2026, the margin is 9\.9% at step 100 and 11\.1% at step 200, and it falls to 1\.7% at step 500\. At that checkpoint LiSAO is ahead by 2\.5% on AIME 2024\. Drawing more samples narrows the same lead\. Averaged over the three years and all five checkpoints, the margin drops from 6\.7% at pass@1 to 1\.5% at pass@16\. The step 100 checkpoint on AIME 2024 is the exception, where the margin widens askkgrows, as shown in Figure[6](https://arxiv.org/html/2610.40726#A4.F6)\.

#### LiSAO reaches a higher ceiling on multi\-turn agentic tool use\.

The right half of Figure[1](https://arxiv.org/html/2610.40726#S2.F1)gives the ReTool results in the same two panels\. GRPO leads in pass@1 early, and LiSAO then moves ahead\. Averaged over AIME 2024, 2025, and 2026, LiSAO trails by 6\.0% at step 100 and leads by 1\.4% at step 500\. At that checkpoint it is ahead on all three years\. This final lead widens as more samples are drawn\. At step 500, averaged over the three years, the margin rises from 1\.4% at pass@1 to 2\.2% at pass@16\. More details are in Figure[6](https://arxiv.org/html/2610.40726#A4.F6)\.

#### A shared early\-training deficit\.

Despite their complementary late\-stage strengths, LiSAO substantially trails GRPO over the early steps in both task families, as shown in panels 2 and 4 of Figure[1](https://arxiv.org/html/2610.40726#S2.F1)\. Figure[2](https://arxiv.org/html/2610.40726#S3.F2)shows that this deficit is already present on the training batches: LiSAO’s binary accuracy remains below GRPO through roughly step 200 in both single\-turn math and ReTool\. On single\-turn math, the two methods start with similar training accuracy, but LiSAO begins to fall behind after roughly 50 steps and the gap subsequently widens\. On ReTool, by contrast, LiSAO starts with a substantial deficit during the first 100 steps, after which the gap gradually narrows\. Thus, although GRPO holds an early advantage in both settings, that advantage emerges at different stages of training\. As the next subsection shows, this delayed takeoff stems from the cold critic: single\-rollout GAE becomes competitive only after the critic learns useful value estimates\.

### 3\.2Mechanism: Critic Dynamics Explain the Early Gap and GAE’s Informative Turn

Figure 2:Training accuracyof GRPO and LiSAO\. Curves are a two\-sided EMA \(span 10\) of per\-step 0/1 accuracy\. The inset shows the first 200 steps\. LiSAO starts at step 10, after critic warmup\.#### A cold critic makes early GAE noisy\.

Without value pretraining,VϕV\_\{\\phi\}is initialized from the policy checkpoint before it predicts returns\. Explained variance,1−Vart⁡\(G^t−Vϕ​\(st\)\)/Vart⁡\(G^t\)1\-\\operatorname\{Var\}\_\{t\}\(\\hat\{G\}\_\{t\}\-V\_\{\\phi\}\(s\_\{t\}\)\)/\\operatorname\{Var\}\_\{t\}\(\\hat\{G\}\_\{t\}\), is then negative, so the critic is worse than a predictor of the mean ofG^t\\hat\{G\}\_\{t\}\. Figure[3](https://arxiv.org/html/2610.40726#S3.F3)shows the fit over the first 200 steps\. At step 10, when actor updates begin, explained variance is still about−3\.0\-3\.0on reasoning and about−5\.0\-5\.0on ReTool, with value loss about0\.90\.9and1\.51\.5\. On reasoning, explained variance reaches about0\.00\.0by step 30, and value loss falls below0\.40\.4by step 20\. On ReTool, explained variance is still about−1\.0\-1\.0at step 20 and crosses0\.00\.0around step 36\. Intermediate rewards are zero, so the residual in Eq\.\([6](https://arxiv.org/html/2610.40726#S2.E6)\) reduces toγ​Vϕ​\(suj\+1\)−Vϕ​\(suj\)\\gamma V\_\{\\phi\}\(s\_\{u\_\{j\+1\}\}\)\-V\_\{\\phi\}\(s\_\{u\_\{j\}\}\), and Eq\.\([7](https://arxiv.org/html/2610.40726#S2.E7)\) accumulates that error into the advantage\. Early actor updates therefore follow critic noise, and this noise lasts longer on ReTool, where the training accuracy deficit appears immediately\.

#### Return tracking separates the two tasks\.

After this recovery, value loss keeps falling on both tasks, and explained variance stays positive\. On reasoning, one terminal reward leaves little structure among tokens inside a completed solution, and a group of solutions already supplies a sequence\-level baseline\. Training accuracy falls behind GRPO after about step 50, when explained variance is already positive\. On ReTool, the terminal reward folds an early plan, a tool call, and a later mistake into one scalar\. Skip\-observation GAE separates those turns onceVϕV\_\{\\phi\}tracks the return\. Explained variance there rises from about0\.10\.1at step 50 to about0\.40\.4at step 200\. The accuracy gap narrows over the same window, and LiSAO later moves ahead\. These dynamics motivate training the actor with a group\-relative advantage and the critic with a group\-centered return untilVϕV\_\{\\phi\}is ready, then switching to single\-rollout GAE\.

Figure 3:Critic Dynamicsof LiSAO over the first 200 steps\. Insets magnify steps 5 to 50 and steps 50 to 200 to show the cold\-start recovery and the later positive fit\. The critic is updated from step 0, including 10 warmup steps before actor updates\.

## 4Li2SAO: Lifting LiSAO via Group\- Relative Actor\-Critic Bootstrapping

### 4\.1Bootstrapping Actor and Critic

Li2SAO addresses the cold critic without a separate value pretraining stage\. It begins withnb\>1n\_\{b\}\>1responses for each prompt and jointly bootstraps the actor and critic from the resulting group\. Letggdenote a prompt group and letRiR\_\{i\}be the raw outcome reward of responseii\. We define the group meanμg=1nb​∑j∈gRj\\mu\_\{g\}=\\frac\{1\}\{n\_\{b\}\}\\sum\_\{j\\in g\}R\_\{j\}and standard deviationσg=1nb−1​∑j∈g\(Rj−μg\)2\\sigma\_\{g\}=\\sqrt\{\\frac\{1\}\{n\_\{b\}\-1\}\\sum\_\{j\\in g\}\(R\_\{j\}\-\\mu\_\{g\}\)^\{2\}\}\.

#### Group relative actor update\.

During this bootstrap phase, the actor usesA^i,tactor=Ri−μgσg\+ε\\hat\{A\}\_\{i,t\}^\{\\mathrm\{actor\}\}=\\frac\{R\_\{i\}\-\\mu\_\{g\}\}\{\\sigma\_\{g\}\+\\varepsilon\}at every model generated token in responseii\. This is the GRPO advantage in Eq\.\([2](https://arxiv.org/html/2610.40726#S2.E2)\), and these advantages enter the clipped policy objective in Eq\.\([1](https://arxiv.org/html/2610.40726#S2.E1)\)\. The importance\-ratio correction for in\-flight samples otherwise follows LiSAO in Eq\.\([8](https://arxiv.org/html/2610.40726#S2.E8)\) and Eq\.\([9](https://arxiv.org/html/2610.40726#S2.E9)\)\.

#### Group residual critic target\.

In vanilla LiSAO, Eq\.\([5](https://arxiv.org/html/2610.40726#S2.E5)\) fitsVϕ​\(si,t\)V\_\{\\phi\}\(s\_\{i,t\}\)to the absolute returnG^i,t\\hat\{G\}\_\{i,t\}built from the outcome rewardRiR\_\{i\}on the final token\. Inside a prompt group, the outcome in that return splits asRi=μg\+\(Ri−μg\)R\_\{i\}=\\mu\_\{g\}\+\(R\_\{i\}\-\\mu\_\{g\}\), whereμg\\mu\_\{g\}is identical for every response and reflects only how easy the current prompt is\. Predicting this shared mean from the prompt lowers Eq\.\([5](https://arxiv.org/html/2610.40726#S2.E5)\) even when a successful response and an unsuccessful response receive the same value\. Thus, Li2SAO setsG~i,t=G^i,t−μg\\widetilde\{G\}\_\{i,t\}=\\hat\{G\}\_\{i,t\}\-\\mu\_\{g\}and drops groups with identical rewards, whereRi−μg=0R\_\{i\}\-\\mu\_\{g\}=0, so the squared loss supervises the critic by which response in the group succeeds\.

Figure 4:Overview of Li2SAO\.GRPO waits for a complete prompt group and assigns one group\-relative advantage to every action token\. LiSAO consumes each rollout on arrival, but a cold critic makes early GAE noisy\. Li2SAO bootstraps the actor with group\-relative advantages and the critic with centered returns, divergence masking, and terminal ranking, then switches to single\-rollout GAE when residual explained variance and ranking accuracy exceed the thresholds\.
#### Divergence masked residual loss\.

Responses with different outcomes can share a common prefix\. Li2SAO does not trainVϕV\_\{\\phi\}on states before those responses diverge, because a residual target there depends on future tokens that the state has not generated\. For responseii, letd⁡\(i,j\)d\(i,j\)be the first response position at which it differs from responsejj, and setdi=maxj∈g:Rj≠Rid\(i,j\)d\_\{i\}=\\max\_\{j\\in g:R\_\{j\}\\neq R\_\{i\}\}d\(i,j\)\. Li2SAO sets the critic maskci,t=1c\_\{i,t\}=1on model generated tokens at and afterdid\_\{i\}\. When the local training partition contains no response withRj≠RiR\_\{j\}\\neq R\_\{i\}, Li2SAO instead keeps only the last action token\. With this mask, Li2SAO freezesVi,told=Vϕold​\(si,t\)V^\{\\mathrm\{old\}\}\_\{i,t\}=V\_\{\\phi\_\{\\mathrm\{old\}\}\}\(s\_\{i,t\}\)once before the critic passes on a batch\. WritingVi,t=Vϕ​\(si,t\)V\_\{i,t\}=V\_\{\\phi\}\(s\_\{i,t\}\)andϵv=0\.2\\epsilon\_\{v\}=0\.2, it minimizes

V¯i,t=Vi,told\+clip⁡\(Vi,t−Vi,told,−ϵv,ϵv\),ℒres​\(ϕ\)=∑g,i,tci,t​max⁡\{\(Vi,t−G~i,t\)2,\(V¯i,t−G~i,t\)2\}∑g,i,tci,t\.\\begin\{gathered\}\\bar\{V\}\_\{i,t\}=V^\{\\mathrm\{old\}\}\_\{i,t\}\+\\operatorname\{clip\}\(V\_\{i,t\}\-V^\{\\mathrm\{old\}\}\_\{i,t\},\-\\epsilon\_\{v\},\\epsilon\_\{v\}\),\\\\ \\mathcal\{L\}\_\{\\mathrm\{res\}\}\(\\phi\)=\\frac\{\\sum\_\{g,i,t\}c\_\{i,t\}\\max\\left\\\{\\begin\{gathered\}\(V\_\{i,t\}\-\\widetilde\{G\}\_\{i,t\}\)^\{2\},\\\\ \(\\bar\{V\}\_\{i,t\}\-\\widetilde\{G\}\_\{i,t\}\)^\{2\}\\end\{gathered\}\\right\\\}\}\{\\sum\_\{g,i,t\}c\_\{i,t\}\}\.\\end\{gathered\}\(10\)

#### Terminal ranking loss\.

Eq\.\([10](https://arxiv.org/html/2610.40726#S4.E10)\) fits each kept token to its own centered return, but it contains no comparison between responses\. Reducing that loss on one completion can leaveVϕ​\(si\+,Ti\+\)≤Vϕ​\(si−,Ti−\)V\_\{\\phi\}\(s\_\{i^\{\+\},T\_\{i^\{\+\}\}\}\)\\leq V\_\{\\phi\}\(s\_\{i^\{\-\},T\_\{i^\{\-\}\}\}\)\. The return on the last action equals the outcome reward, so for a mixed pair withRi\+\>Ri−R\_\{i^\{\+\}\}\>R\_\{i^\{\-\}\}the centered targets of the two states differ byRi\+−Ri−R\_\{i^\{\+\}\}\-R\_\{i^\{\-\}\}\. Let𝒫\\mathcal\{P\}be the set of such pairs, we therefore add the terminal ranking loss:

ℒrank​\(ϕ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{rank\}\}\(\\phi\)=1\|𝒫\|∑\(i\+,i−\)∈𝒫\[m\\displaystyle=\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\(i^\{\+\},i^\{\-\}\)\\in\\mathcal\{P\}\}\\bigl\[m\(11\)−Vϕ\(si\+,Ti\+\)\+Vϕ\(si−,Ti−\)\]\+\.\\displaystyle\-V\_\{\\phi\}\(s\_\{i^\{\+\},T\_\{i^\{\+\}\}\}\)\+V\_\{\\phi\}\(s\_\{i^\{\-\},T\_\{i^\{\-\}\}\}\)\\bigr\]\_\{\+\}\.Then critic loss isℒcritic=ℒres\+α​ℒrank\\mathcal\{L\}\_\{\\mathrm\{critic\}\}=\\mathcal\{L\}\_\{\\mathrm\{res\}\}\+\\alpha\\mathcal\{L\}\_\{\\mathrm\{rank\}\}\. We use marginm=0\.1m=0\.1and coefficientα=0\.1\\alpha=0\.1\. Actor and critic optimization begin together at the first step, so Li2SAO requires no critic\-only warmup\.

### 4\.2From Group Sampling to Single Rollout

Group sampling supplies a reliable actor signal while the critic learns to distinguish outcomes\. Once that distinction is reliable, retainingnbn\_\{b\}responses per prompt is unnecessary and delays asynchronous updates\. Li2SAO therefore makes a single irreversible transition fromnbn\_\{b\}tons=1n\_\{s\}=1\. The total number of trajectories per update stays fixed, so the number of prompts increases fromB/nbB/n\_\{b\}toBB\. After the transition, the actor uses GAE in Eq\.\([4](https://arxiv.org/html/2610.40726#S2.E4)\) and the critic resumes fitting absolute returns in Eq\.\([5](https://arxiv.org/html/2610.40726#S2.E5)\)\. The residual transformation and ranking loss become inactive because singleton groups contain no within group comparison\.

#### Readiness criterion\.

We measure critic readiness on mixed groups using the centered terminal outcomeZi=Ri−μgZ\_\{i\}=R\_\{i\}\-\\mu\_\{g\}\. At bootstrap stepkk, the residual explained variance is

ek=1−Vari⁡\[Zi−Vϕ​\(si,Ti\)\]Vari⁡\[Zi\]\.e\_\{k\}=1\-\\frac\{\\operatorname\{Var\}\_\{i\}\\left\[Z\_\{i\}\-V\_\{\\phi\}\(s\_\{i,T\_\{i\}\}\)\\right\]\}\{\\operatorname\{Var\}\_\{i\}\[Z\_\{i\}\]\}\.\(12\)We also compute the fraction of cross outcome pairs ordered correctly:

ak=1\|𝒫\|∑\(i\+,i−\)∈𝒫𝕀\[Vϕ\(si\+,Ti\+\)\>Vϕ\(si−,Ti−\)\]\.a\_\{k\}=\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\(i^\{\+\},i^\{\-\}\)\\in\\mathcal\{P\}\}\\mathbb\{I\}\\bigl\[V\_\{\\phi\}\(s\_\{i^\{\+\},T\_\{i^\{\+\}\}\}\)\>V\_\{\\phi\}\(s\_\{i^\{\-\},T\_\{i^\{\-\}\}\}\)\\bigr\]\.\(13\)Both statistics are aggregated across data parallel workers and smoothed with exponential moving averagese¯k=β​e¯k−1\+\(1−β\)​ek\\bar\{e\}\_\{k\}=\\beta\\bar\{e\}\_\{k\-1\}\+\(1\-\\beta\)e\_\{k\}anda¯k=β​a¯k−1\+\(1−β\)​ak\\bar\{a\}\_\{k\}=\\beta\\bar\{a\}\_\{k\-1\}\+\(1\-\\beta\)a\_\{k\}\. The critic is ready whene¯k\>ηe\\bar\{e\}\_\{k\}\>\\eta\_\{e\}anda¯k\>ηa\\bar\{a\}\_\{k\}\>\\eta\_\{a\}forMMconsecutive steps after a minimum bootstrap durationTminT\_\{\\min\}\. A maximum durationTmaxT\_\{\\max\}provides a fallback when the gate does not open\. Our main configuration usesβ=0\.9\\beta=0\.9,ηe=0\.2\\eta\_\{e\}=0\.2,ηa=0\.65\\eta\_\{a\}=0\.65,M=10M=10,Tmin=60T\_\{\\min\}=60, andTmax=150T\_\{\\max\}=150\. The one way transition prevents noisy batch level fluctuations from returning training to group sampling\.

#### In\-flight switching\.

In in\-flight asynchronous RL system, generation overlaps training, so one job is already prefetched when the gate opens\. Li2SAO finishes that job at group sizenbn\_\{b\}, discards leftover bootstrap groups still held by the rollout worker, and only then sets the group size tons=1n\_\{s\}=1\. The next job contains singleton groups, so every batch has one group size and one advantage estimator\.

### 4\.3Theoretical Analysis

To provide insight into each Li2SAO stage, we characterize its expected update under the policy gradient\. Fix a prompt, a differentiable policyπ=πθ\\pi=\\pi\_\{\\theta\}, a binary terminal rewardRR, and discountγ=1\\gamma=1\. The statests\_\{t\}before model actionata\_\{t\}contains only the preceding history, including tool observations\. Letp=𝔼π​Rp=\\mathbb\{E\}\_\{\\pi\}Rbe the success probability of this prompt, and letVπ​\(s\)V^\{\\pi\}\(s\)be the expected remaining return from statess\. Writeψt=∇θ​log​πθ​\(at∣st\)\\psi\_\{t\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)andS=∑tψtS=\\sum\_\{t\}\\psi\_\{t\}, where the sum runs over model actions\. The first result characterizes the group\-bootstrap signal and its residual critic target\. Regularity conditions and proofs appear in Appendix[A](https://arxiv.org/html/2610.40726#A1)\.

###### Proposition 1\.

Samplen=nb≥2n=n\_\{b\}\\geq 2independent trajectories fromπ\\pi, with0<p<10<p<1\. SetYi=Ri−μgY\_\{i\}=R\_\{i\}\-\\mu\_\{g\}andℳ=\{0<∑iRi<n\}\\mathcal\{M\}=\\\{0<\\sum\_\{i\}R\_\{i\}<n\\\}\. Define the reference group score asg^grp=n−1​∑iSi​Yi/\(σg\+ε\)\\widehat\{g\}\_\{\\mathrm\{grp\}\}=n^\{\-1\}\\sum\_\{i\}S\_\{i\}Y\_\{i\}/\(\\sigma\_\{g\}\+\\varepsilon\), whereε\>0\\varepsilon\>0, then its expectation is a positive rescaling of the success\-probability gradient:

𝔼​g^grp=ωn​\(p\)​∇θp,ωn​\(p\)\>0\.\\mathbb\{E\}\\widehat\{g\}\_\{\\mathrm\{grp\}\}=\\omega\_\{n\}\(p\)\\nabla\_\{\\theta\}p,\\quad\\omega\_\{n\}\(p\)\>0\.\(14\)At a pre\-action statess, the optimal target for the base squared residual loss on mixed groups is

F⋆\(s\)=𝔼\[Yi∣si,t=s,ℳ\]=\(1−1/n\)​\(Vπ​\(s\)−p\)1−pn−1​Vπ​\(s\)−\(1−p\)n−1​\(1−Vπ​\(s\)\)\.\\begin\{gathered\}F^\{\\star\}\(s\)=\\mathbb\{E\}\[Y\_\{i\}\\mid s\_\{i,t\}=s,\\mathcal\{M\}\]\\\\ =\\frac\{\(1\-1/n\)\(V^\{\\pi\}\(s\)\-p\)\}\{1\-p^\{n\-1\}V^\{\\pi\}\(s\)\-\(1\-p\)^\{n\-1\}\(1\-V^\{\\pi\}\(s\)\)\}\.\\end\{gathered\}\(15\)For fixedpp,F⋆​\(s\)F^\{\\star\}\(s\)strictly increases withVπ​\(s\)V^\{\\pi\}\(s\)\.

Thus, this characterization isolates what group centering asks the critic to learn: group\-residual supervision prevents the critic from reducing its regression loss merely by predicting prompt difficulty, and instead preserves response\-dependent distinctions relevant to relative success\. The next proposition shows exact conditional values turn TD residuals and GAE into single\-trajectory estimators of the success\-probability gradient\.

###### Proposition 2\.

Letδtπ=rt\+Vπ​\(st\+1\)−Vπ​\(st\)\\delta\_\{t\}^\{\\pi\}=r\_\{t\}\+V^\{\\pi\}\(s\_\{t\+1\}\)\-V^\{\\pi\}\(s\_\{t\}\)be the temporal\-difference residual of the exact conditional valueVπV^\{\\pi\}\. For anyλ∈\(0,1\]\\lambda\\in\(0,1\], this residual and the fixed\-trace GAE are single\-trajectory estimators of the success\-probability gradient:

∇θp=𝔼π​\[∑tψt​δtπ\]=𝔼π​\[∑tψt​A^tλ​\(Vπ\)\]\.\\nabla\_\{\\theta\}p=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t\}\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\right\]=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)\\right\]\.\(16\)

This proposition specializes the policy\-gradient and GAE identities\([Sutton et al\., 1999](https://arxiv.org/html/2610.40726#bib.bib37);[Schulman et al\., 2016](https://arxiv.org/html/2610.40726#bib.bib20)\)to the model\-action history\. Thus, the identity identifies the ideal target of the switch: atVπV^\{\\pi\}, single\-rollout GAE keeps the group bootstrap’s expected direction\.More theoretical analyses about off\-policy estimation, value calibration, and task structure are in Appendix[A](https://arxiv.org/html/2610.40726#A1)\.

## 5Experiments

Table 1:Pass@kk\(%\) at step 500 on AIME 2024, 2025, and 2026\. Base and SFT are the shared checkpoints before RL: single\-turn reasoning starts from the base model and ReTool from a tool\-use supervised checkpoint\. All rows use 16 samples per problem at temperature 1\.0 and the unbiased estimator of[Chen et al\. \(2021\)](https://arxiv.org/html/2610.40726#bib.bib27)\. Avg\. is the mean over the three years\. The best results are highlighted inBold\.We evaluate Li2SAO on single\-turn mathematical reasoning and multi\-turn agentic tool use\. Our experimental design examines two questions: \(i\) whether group\-relative actor\-critic bootstrapping mitigates the early optimization deficit caused by a cold critic, and \(ii\) whether the subsequent switch to single\-rollout GAE preserves temporal credit assignment and raises the later performance ceiling\.

#### Experimental Setups\.

We compare Li2SAO with GRPO and LiSAO under the training and evaluation protocol of Section[3](https://arxiv.org/html/2610.40726#S3)\. Li2SAO follows the procedure in Section[4](https://arxiv.org/html/2610.40726#S4)\. During the bootstrap, Li2SAO uses 8 responses per prompt, the same group size as GRPO, and begins the actor and critic updates together\. Further details are in Appendix[C](https://arxiv.org/html/2610.40726#A3)\.

Figure 5:Early pass@kkat steps 100 and 200\. Each bar averages AIME 2024, 2025, and 2026\.
### 5\.1Main Results

#### Group\-relative bootstrapping reverses the early deficit\.

Figure[5](https://arxiv.org/html/2610.40726#S5.F5)shows that Li2SAO leads both baselines in average pass@kkat steps 100 and 200 across both task families\. At step 100, average pass@1 reaches 42\.5% on reasoning and 52\.2% on ReTool, exceeding LiSAO by 17\.3 and 6\.8 percentage points and GRPO by 7\.4 and 0\.8 points, respectively\. The advantage at step 200 shows that these early gains persist beyond bootstrapping, supporting its effectiveness without value pretraining\.

#### Single\-rollout GAE raises the later performance ceiling\.

At step 500, Li2SAO achieves the highest average pass@kkfor every evaluatedkkin both settings \(Table[1](https://arxiv.org/html/2610.40726#S5.T1)\)\. Average pass@1 reaches 67\.0% on reasoning and 64\.6% on ReTool, exceeding GRPO by 3\.4 and 2\.8 percentage points and LiSAO by 5\.1 and 1\.4 points, respectively\. Average pass@16 also rises to 86\.7% and 90\.0%, indicating improved coverage under repeated sampling\. Thus, Li2SAO improves both early and final performance while retaining single\-rollout GAE and its temporal credit assignment after bootstrapping\. Figure[7](https://arxiv.org/html/2610.40726#A6.F7)reports the same comparison at each checkpoint and year\.

### 5\.2Additional Experimental Analysis

To further analyze Li2SAO, we report additional experiments in Appendix[G](https://arxiv.org/html/2610.40726#A7), includingfull per\-year results\(Section[G\.1](https://arxiv.org/html/2610.40726#A7.SS1)\),training dynamics\(Section[G\.2](https://arxiv.org/html/2610.40726#A7.SS2)\),critic\-objective ablations\(Section[G\.3](https://arxiv.org/html/2610.40726#A7.SS3)\),wall\-clock efficiency\(Section[G\.4](https://arxiv.org/html/2610.40726#A7.SS4)\), andpolicy staleness\(Section[G\.5](https://arxiv.org/html/2610.40726#A7.SS5)\)\. We also evaluatecoding agentic RLin Appendix[H](https://arxiv.org/html/2610.40726#A8), training larger models on long\-horizon software engineering\. These results demonstrate the effectiveness of Li2SAO\.

## 6Conclusion

This paper studies single\-rollout asynchronous RL without scaled value pretraining and identifies a shared early optimization deficit associated with a cold critic, despite the complementary strengths of GRPO on reasoning and LiSAO on agentic tasks\. Our method, Li2SAO, jointly bootstraps the actor with group\-relative advantages and the critic with group\-centered returns before switching once to single\-rollout GAE\. The policy\-gradient analysis explains the direction of group\-based updates and its recovery by GAE under exact value estimation\. Experiments show that Li2SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling\. These findings support group\-relative bootstrapping as a practical route to effective single\-rollout asynchronous optimization without a pretrained value model\.

## Limitations

Theoretical Scope\.Our analysis is intended to isolate the mechanisms behind the two training stages\. Proposition[1](https://arxiv.org/html/2610.40726#Thmproposition1)shows that group centered supervision removes the prompt difficulty component while preserving the ordering induced by conditional success probabilities\. Proposition[2](https://arxiv.org/html/2610.40726#Thmproposition2)then shows that, with exact conditional values, single rollout GAE recovers the same expected success direction\. These population identities support the method design but do not constitute a convergence guarantee for the complete training procedure\. The implemented critic also uses divergence masking, value clipping, and ranking regularization, so its optimum need not exactly equalF⋆F^\{\\star\}\.

Experimental Scope\.Our experiments use a deliberately controlled setting based on one 4B parameter model family, DAPO Math 17K, binary outcome rewards, and AIME evaluation for both single turn reasoning and ReTool\. This design enables matched comparisons under a common asynchronous pipeline and training protocol, supporting a fine\-grained empirical analysis of algorithmic behavior\. Because the scaled value model used by SAO is not publicly available, our comparison centers on LiSAO, a transparent reproduction without value pretraining\.

## References

- Ahmadianet al\.\(2024\)A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. HookerBack to basics: revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 12248–12267\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. Ponde de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§C\.2](https://arxiv.org/html/2610.40726#A3.SS2.p1.1),[Table 6](https://arxiv.org/html/2610.40726#A7.T6),[§3](https://arxiv.org/html/2610.40726#S3.p1.1),[Table 1](https://arxiv.org/html/2610.40726#S5.T1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p2.1)\.
- Fenget al\.\(2025\)J\. Feng, S\. Huang, X\. Qu, G\. Zhang, Y\. Qin, B\. Zhong, C\. Jiang, J\. Chi, and W\. ZhongReTool: reinforcement learning for strategic tool use in LLMs\.arXiv preprint arXiv:2504\.11536\.Cited by:[§D\.3](https://arxiv.org/html/2610.40726#A4.SS3.p1.1),[§3](https://arxiv.org/html/2610.40726#S3.p1.1)\.
- Fuet al\.\(2025\)W\. Fu, J\. Gao, X\. Shen, C\. Zhu, Z\. Mei, C\. He, S\. Xu, G\. Wei, J\. Mei, J\. Wang, T\. Yang, B\. Yuan, and Y\. WuAReaL: a large\-scale asynchronous reinforcement learning system for language reasoning\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1)\.
- Houet al\.\(2026\)Z\. Hou, Y\. Li, J\. Tang, and Y\. DongSingle\-rollout asynchronous optimization for agentic reinforcement learning\.arXiv preprint arXiv:2607\.07508\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§C\.3](https://arxiv.org/html/2610.40726#A3.SS3.p1.1),[§D\.1](https://arxiv.org/html/2610.40726#A4.SS1.p1.1),[§D\.2](https://arxiv.org/html/2610.40726#A4.SS2.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.40726#S2.SS3.p2.1),[§2\.3](https://arxiv.org/html/2610.40726#S2.SS3.p4.1),[§3](https://arxiv.org/html/2610.40726#S3.p1.1)\.
- Huet al\.\(2024\)J\. Hu, X\. Wu, Z\. Zhu, Xianyu, W\. Wang, D\. Zhang, and Y\. CaoOpenRLHF: an easy\-to\-use, scalable and high\-performance RLHF framework\.arXiv preprint arXiv:2405\.11143\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1)\.
- Huanget al\.\(2026\)L\. J\. Huang, Z\. Zhang, Q\. Hu, S\. Yang, and S\. HanStable asynchrony: variance\-controlled off\-policy RL for LLMs\.arXiv preprint arXiv:2602\.17616\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.40726#S2.SS1.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations,Cited by:[§H\.2](https://arxiv.org/html/2610.40726#A8.SS2.p1.1)\.
- Lianget al\.\(2026\)D\. Liang, L\. Feng, B\. An, and Y\. LiuSAPO: single\-rollout autoregressive policy optimization for agentic reinforcement learning\.arXiv preprint arXiv:2608\.19842\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1)\.
- Mathematical Association of America \(2026\)Mathematical Association of AmericaAmerican Invitational Mathematics Examination\.Note:[https://maa\.org/maa\-invitational\-competitions/](https://maa.org/maa-invitational-competitions/)AIME 2024, AIME 2025, and AIME 2026Cited by:[§3](https://arxiv.org/html/2610.40726#S3.p1.1)\.
- Noukhovitchet al\.\(2025\)M\. Noukhovitch, S\. Huang, S\. Xhonneux, A\. Hosseini, R\. Agarwal, and A\. CourvilleAsynchronous RLHF: faster and more efficient off\-policy RL for language models\.InInternational Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.40726#S2.SS1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1)\.
- Panet al\.\(2025\)J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. ZhangTraining software engineering agents and verifiers with SWE\-Gym\.InProceedings of the 42nd International Conference on Machine Learning,Vol\.267,pp\. 47717–47737\.Cited by:[§H\.2](https://arxiv.org/html/2610.40726#A8.SS2.p1.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. ChintalaPyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§C\.1](https://arxiv.org/html/2610.40726#A3.SS1.p1.1)\.
- Pichéet al\.\(2025\)A\. Piché, E\. Kamalloo, R\. Pardinas, X\. Chen, and D\. BahdanauPipelineRL: faster on\-policy reinforcement learning for long sequence generation\.arXiv preprint arXiv:2509\.19128\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1)\.
- Schulmanet al\.\(2016\)J\. Schulman, P\. Moritz, S\. Levine, M\. I\. Jordan, and P\. AbbeelHigh\-dimensional continuous control using generalized advantage estimation\.InInternational Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1),[§2\.3](https://arxiv.org/html/2610.40726#S2.SS3.p1.1),[§4\.3](https://arxiv.org/html/2610.40726#S4.SS3.p3.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1),[§2\.3](https://arxiv.org/html/2610.40726#S2.SS3.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\.K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§H\.3](https://arxiv.org/html/2610.40726#A8.SS3.p2.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.40726#S2.SS2.p1.1),[§3](https://arxiv.org/html/2610.40726#S3.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: a flexible and efficient RLHF framework\.InProceedings of the Twentieth European Conference on Computer Systems,Rotterdam, Netherlands,pp\. 1279–1297\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.40726#S2.SS1.p1.1)\.
- Shoeybiet al\.\(2019\)M\. Shoeybi, M\. Patwary, R\. Puri, P\. LeGresley, J\. Casper, and B\. CatanzaroMegatron\-LM: training multi\-billion parameter language models using model parallelism\.arXiv preprint arXiv:1909\.08053\.Cited by:[§C\.1](https://arxiv.org/html/2610.40726#A3.SS1.p1.1)\.
- Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. ChristianoLearning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 3008–3021\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. A\. McAllester, S\. P\. Singh, and Y\. MansourPolicy gradient methods for reinforcement learning with function approximation\.InAdvances in Neural Information Processing Systems,Vol\.12\.External Links:[Link](https://proceedings.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html)Cited by:[§4\.3](https://arxiv.org/html/2610.40726#S4.SS3.p3.1)\.
- Wanget al\.\(2025\)X\. Wang, B\. Li, Y\. Song, F\. F\. Xu, X\. Tang, M\. Zhuge, J\. Pan, Y\. Song, B\. Li, J\. Singh, H\. H\. Tran, F\. Li, R\. Ma, M\. Zheng, B\. Qian, Y\. Shao, N\. Muennighoff, Y\. Zhang, B\. Hui, J\. Lin, R\. Brennan, H\. Peng, H\. Ji, and G\. NeubigOpenHands: an open platform for AI software developers as generalist agents\.InInternational Conference on Learning Representations,Cited by:[§H\.1](https://arxiv.org/html/2610.40726#A8.SS1.p1.1)\.
- Wuet al\.\(2025\)B\. Wu, S\. Wang, Y\. Tang, J\. Ding, E\. Helenowski, L\. Tan, T\. Xu, T\. Gowda, Z\. Chen, C\. Zhu, X\. Tang, Y\. Qian, B\. Zhu, and R\. HouLlamaRL: a distributed asynchronous reinforcement learning framework for efficient large\-scale LLM training\.arXiv preprint arXiv:2505\.24034\.Cited by:[§1](https://arxiv.org/html/2610.40726#S1.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§H\.3](https://arxiv.org/html/2610.40726#A8.SS3.p1.1),[§3](https://arxiv.org/html/2610.40726#S3.p1.1)\.
- Yanget al\.\(2025b\)J\. Yang, K\. Lieret, C\. E\. Jimenez, A\. Wettig, K\. Khandpur, Y\. Zhang, B\. Hui, O\. Press, L\. Schmidt, and D\. YangSWE\-smith: scaling data for software engineering agents\.InAdvances in Neural Information Processing Systems,Cited by:[§H\.2](https://arxiv.org/html/2610.40726#A8.SS2.p1.1)\.
- Yanget al\.\(2026\)J\. Yang, Y\. Shi, Z\. Li, Z\. Li, R\. Wang, X\. Zhou, K\. Panaganti, H\. Mi, and L\. LiangStale but stable: staleness\-adaptive trust regions for stabilizing asynchronous reinforcement learning\.arXiv preprint arXiv:2607\.18722\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2023\)Z\. Yao, R\. Y\. Aminabadi, O\. Ruwase, S\. Rajbhandari, X\. Wu, A\. A\. Awan, J\. Rasley, M\. Zhang, C\. Li, C\. Holmes, Z\. Zhou, M\. Wyatt, M\. Smith, L\. Kurilenko, H\. Qin, M\. Tanaka, S\. Che, S\. L\. Song, and Y\. HeDeepSpeed\-Chat: easy, fast and affordable RLHF training of ChatGPT\-like models at all scales\.arXiv preprint arXiv:2308\.01320\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40726#S1.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1),[§C\.3](https://arxiv.org/html/2610.40726#A3.SS3.p1.1),[§H\.3](https://arxiv.org/html/2610.40726#A8.SS3.p2.1),[§3](https://arxiv.org/html/2610.40726#S3.p1.1)\.
- Yuanet al\.\(2025\)Y\. Yuan, Y\. Yue, R\. Zhu, T\. Fan, and L\. YanWhat’s behind PPO’s collapse in long\-CoT? value optimization holds the secret\.arXiv preprint arXiv:2503\.01491\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1)\.
- Yueet al\.\(2025\)Y\. Yue, Y\. Yuan, Q\. Yu, X\. Zuo, R\. Zhu, W\. Xu, J\. Chen, C\. Wang, T\. Fan, Z\. Du, X\. Wei, X\. Yu, G\. Liu, J\. Liu, L\. Liu, H\. Lin, Z\. Lin, B\. Ma, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, R\. Zhang, X\. Liu, M\. Wang, Y\. Wu, and L\. YanVAPO: efficient and reliable reinforcement learning for advanced reasoning tasks\.arXiv preprint arXiv:2504\.05118\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1),[§D\.2](https://arxiv.org/html/2610.40726#A4.SS2.p1.1)\.
- Zhenget al\.\(2025a\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px2.p1.1),[§H\.3](https://arxiv.org/html/2610.40726#A8.SS3.p2.1)\.
- Zhenget al\.\(2025b\)H\. Zheng, J\. Zhao, and B\. ChenProsperity before collapse: how far can off\-policy RL reach with stale data on LLMs?\.arXiv preprint arXiv:2510\.01161\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2610.40726#S2.SS1.p1.1)\.
- Zhenget al\.\(2024\)L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. ShengSGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§C\.1](https://arxiv.org/html/2610.40726#A3.SS1.p1.1)\.
- Zhonget al\.\(2025\)Y\. Zhong, Z\. Zhang, X\. Song, H\. Hu, C\. Jin, B\. Wu, N\. Chen, Y\. Chen, Y\. Zhou, C\. Wan, H\. Zhou, Y\. Jiang, Y\. Zhu, and D\. JiangStreamRL: scalable, heterogeneous, and elastic RL for LLMs with disaggregated stream generation\.arXiv preprint arXiv:2504\.15930\.Cited by:[§1](https://arxiv.org/html/2610.40726#S1.p1.1)\.
- Ziegleret al\.\(2019\)D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. IrvingFine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[Appendix B](https://arxiv.org/html/2610.40726#A2.SS0.SSS0.Px1.p1.1)\.

## Appendix AFrom Group Comparisons to Single\-Rollout Gradients

### A\.1Setup and conditioning conventions

Fix a promptqqand a policy parameterθ\\theta, and writeπ=πθ\\pi=\\pi\_\{\\theta\}\. We use the undiscounted objective withγ=1\\gamma=1\. The indexttcounts model actions\. At each real decision,sts\_\{t\}contains the prompt, the decision index, and the complete history of actions, rewards, and tool observations available beforeata\_\{t\}\. For a fixed horizonHH, writeτ=\(s0,a0,r0,…,sH−1,aH−1,rH−1,sH\)\\tau=\(s\_\{0\},a\_\{0\},r\_\{0\},\\ldots,s\_\{H\-1\},a\_\{H\-1\},r\_\{H\-1\},s\_\{H\}\)\. The natural filtration before each action is

ℱt=σ⁡\(s0,\(aj,rj,sj\+1\)0≤j<t\)\.\\mathcal\{F\}\_\{t\}=\\sigma\\\!\\left\(s\_\{0\},\(a\_\{j\},r\_\{j\},s\_\{j\+1\}\)\_\{0\\leq j<t\}\\right\)\.\(17\)In particular,ata\_\{t\},rtr\_\{t\}, andst\+1s\_\{t\+1\}areℱt\+1\\mathcal\{F\}\_\{t\+1\}\-measurable\. LetL≤HL\\leq Hbe the number of real actions, withL≥1L\\geq 1and\{L≤t\}∈ℱt\\\{L\\leq t\\\}\\in\\mathcal\{F\}\_\{t\}, so thatLLis a bounded stopping time\. After termination, pad the trajectory with an absorbing state, zero rewards, and a dummy action of probability one\. The filtration retains the observed history after termination\.

The conditional distribution of\(st\+1,rt\)\(s\_\{t\+1\},r\_\{t\}\)givenℱt\\mathcal\{F\}\_\{t\}andata\_\{t\}does not depend directly onθ\\theta\. Model actions are discrete, and the probabilitiesπθ​\(a∣s\)\\pi\_\{\\theta\}\(a\\mid s\)are differentiable with locally fixed support\. We assume differentiation can be interchanged with the expectations below and that all displayed expectations are absolutely integrable\. Covariance statements additionally require finite second moments\. All trajectory laws and expectations are conditional on the fixed prompt\.

LetR∈\{0,1\}R\\in\\\{0,1\\\}be the reward observed at termination, and letp=𝔼π​Rp=\\mathbb\{E\}\_\{\\pi\}R\. Thusrt=𝟏\{t=L−1\}Rr\_\{t\}=\\mathbf\{1\}\_\{\\\{t=L\-1\\\}\}R\. Define the return\-to\-goGt=∑k=tH−1rkG\_\{t\}=\\sum\_\{k=t\}^\{H\-1\}r\_\{k\}and the accumulated past rewardR<t=∑k<trkR\_\{<t\}=\\sum\_\{k<t\}r\_\{k\}, so thatR=R<t\+GtR=R\_\{<t\}\+G\_\{t\}\. The state value isVtπ=Vπ​\(st\)=𝔼π​\[Gt∣ℱt\]V\_\{t\}^\{\\pi\}=V^\{\\pi\}\(s\_\{t\}\)=\\mathbb\{E\}\_\{\\pi\}\[G\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\], with zero value after termination\. Its temporal\-difference residual isδtπ=rt\+Vt\+1π−Vtπ\\delta\_\{t\}^\{\\pi\}=r\_\{t\}\+V\_\{t\+1\}^\{\\pi\}\-V\_\{t\}^\{\\pi\}\. For any value functionVVthat is zero at terminal and absorbing states, the generalized advantage estimator with fixedλ∈\(0,1\]\\lambda\\in\(0,1\]is

A^tλ​\(V\)=∑k=tH−1λk−t​\(rk\+V⁡\(sk\+1\)−V⁡\(sk\)\)\.\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\)=\\sum\_\{k=t\}^\{H\-1\}\\lambda^\{k\-t\}\\bigl\(r\_\{k\}\+V\(s\_\{k\+1\}\)\-V\(s\_\{k\}\)\\bigr\)\.\(18\)
Writeψt=∇θ​log​πθ​\(at∣st\)\\psi\_\{t\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)for the policy score andS=∑t=0H−1ψtS=\\sum\_\{t=0\}^\{H\-1\}\\psi\_\{t\}for the trajectory score\. Dummy actions have zero score\. The conditional score has zero mean:

𝔼π\[ψt∣ℱt\]=∑aπθ\(a∣st\)∇θlogπθ\(a∣st\)=∇θ∑aπθ\(a∣st\)=0\.\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)=\\nabla\_\{\\theta\}\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)=0\.\(19\)For anyℱt\\mathcal\{F\}\_\{t\}\-measurable baselineBtB\_\{t\}satisfying𝔼π​\[‖ψt‖​\|Bt\|\]<∞\\mathbb\{E\}\_\{\\pi\}\[\\\|\\psi\_\{t\}\\\|\\,\|B\_\{t\}\|\]<\\infty, the tower property gives

𝔼π​\[ψt​Bt\]=𝔼π​\[𝔼π​\[ψt​Bt∣ℱt\]\]=𝔼π​\[Bt​𝔼π​\[ψt∣ℱt\]\]=0\.\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}B\_\{t\}\]=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}B\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]\\right\]=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[B\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]\\right\]=0\.\(20\)Letℓθ​\(τ\)\\ell\_\{\\theta\}\(\\tau\)denote the trajectory density with respect to a common reference measureν\\nuindependent ofθ\\theta\. Since only the action probabilities depend onθ\\theta, its score is

∇θ​log​ℓθ​\(τ\)=∑t=0H−1∇θ​log​πθ​\(at∣st\)=S\.\\nabla\_\{\\theta\}\\log\\ell\_\{\\theta\}\(\\tau\)=\\sum\_\{t=0\}^\{H\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)=S\.\(21\)Differentiating the expected reward and using𝔼π​S=0\\mathbb\{E\}\_\{\\pi\}S=0yields the likelihood\-ratio identity

∇θp=∫R⁡\(τ\)​∇θℓθ​\(τ\)​ν​\(𝑑τ\)=𝔼π​\[R​S\]=𝔼π​\[\(R−p\)​S\]\.\\nabla\_\{\\theta\}p=\\int R\(\\tau\)\\nabla\_\{\\theta\}\\ell\_\{\\theta\}\(\\tau\)\\,\\nu\(d\\tau\)=\\mathbb\{E\}\_\{\\pi\}\[RS\]=\\mathbb\{E\}\_\{\\pi\}\[\(R\-p\)S\]\.\(22\)The group calculation uses independent trajectories from this policy\. Critic regression conditions on a specified real state before any additional selection based on outcomes\.

### A\.2Proof of Proposition[1](https://arxiv.org/html/2610.40726#Thmproposition1)

#### Finite\-group actor signal\.

The calculation holds for any integrable vectorU=U⁡\(τ\)U=U\(\\tau\)computed from one trajectory, withUi=U⁡\(τi\)U\_\{i\}=U\(\\tau\_\{i\}\)\. LetK=∑iRiK=\\sum\_\{i\}R\_\{i\},K−i=∑j≠iRjK\_\{\-i\}=\\sum\_\{j\\neq i\}R\_\{j\}, andYi=Ri−K/nY\_\{i\}=R\_\{i\}\-K/n\. For a group containingkksuccesses, writedk=σk\+εd\_\{k\}=\\sigma\_\{k\}\+\\varepsilon, where the sample variance is

σk2=k​\(1−k/n\)2\+\(n−k\)​\(k/n\)2n−1=k⁡\(n−k\)n⁡\(n−1\)\.\\sigma\_\{k\}^\{2\}=\\frac\{k\(1\-k/n\)^\{2\}\+\(n\-k\)\(k/n\)^\{2\}\}\{n\-1\}=\\frac\{k\(n\-k\)\}\{n\(n\-1\)\}\.\(23\)Withur=𝔼⁡\[U∣R=r\]u\_\{r\}=\\mathbb\{E\}\[U\\mid R=r\], define

g^​\(U\)=1n​∑iUi​YidK\.\\widehat\{g\}\(U\)=\\frac\{1\}\{n\}\\sum\_\{i\}\\frac\{U\_\{i\}Y\_\{i\}\}\{d\_\{K\}\}\.\(24\)The pair\(Ui,Ri\)\(U\_\{i\},R\_\{i\}\)is independent ofK−iK\_\{\-i\}\. Thus, whenever the conditioning event has positive probability,

𝔼\[Ui∣Ri=r,K=k\]=𝔼\[Ui∣Ri=r,K−i=k−r\]=𝔼\[Ui∣Ri=r\]=ur\.\\mathbb\{E\}\[U\_\{i\}\\mid R\_\{i\}=r,K=k\]=\\mathbb\{E\}\[U\_\{i\}\\mid R\_\{i\}=r,K\_\{\-i\}=k\-r\]=\\mathbb\{E\}\[U\_\{i\}\\mid R\_\{i\}=r\]=u\_\{r\}\.\(25\)For1≤k≤n−11\\leq k\\leq n\-1, the binomial probabilities give

Pr⁡\(Ri=1∣K=k\)=\(n−1k−1\)​pk​\(1−p\)n−k\(nk\)​pk​\(1−p\)n−k=kn\.\\Pr\(R\_\{i\}=1\\mid K=k\)=\\frac\{\\binom\{n\-1\}\{k\-1\}p^\{k\}\(1\-p\)^\{n\-k\}\}\{\\binom\{n\}\{k\}p^\{k\}\(1\-p\)^\{n\-k\}\}=\\frac\{k\}\{n\}\.\(26\)Conditioning first onRiR\_\{i\}therefore yields

𝔼\[∑i:Ri=1Ui\|K=k\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\sum\_\{i:R\_\{i\}=1\}U\_\{i\}\\,\\middle\|\\,K=k\\right\]=∑i=1n𝔼\[𝟏\{Ri=1\}Ui∣K=k\]\\displaystyle=\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\[\\mathbf\{1\}\_\{\\\{R\_\{i\}=1\\\}\}U\_\{i\}\\mid K=k\]\(27\)=∑i=1nPr\(Ri=1∣K=k\)𝔼\[Ui∣Ri=1,K=k\]\\displaystyle=\\sum\_\{i=1\}^\{n\}\\Pr\(R\_\{i\}=1\\mid K=k\)\\mathbb\{E\}\[U\_\{i\}\\mid R\_\{i\}=1,K=k\]=∑i=1nkn​u1=k​u1\.\\displaystyle=\\sum\_\{i=1\}^\{n\}\\frac\{k\}\{n\}u\_\{1\}=ku\_\{1\}\.The unsuccessful trajectories similarly satisfy

𝔼\[∑i:Ri=0Ui\|K=k\]=∑i=1n\(1−kn\)u0=\(n−k\)u0\.\\mathbb\{E\}\\\!\\left\[\\sum\_\{i:R\_\{i\}=0\}U\_\{i\}\\,\\middle\|\\,K=k\\right\]=\\sum\_\{i=1\}^\{n\}\\left\(1\-\\frac\{k\}\{n\}\\right\)u\_\{0\}=\(n\-k\)u\_\{0\}\.\(28\)Within each group, successful trajectories have residual1−k/n1\-k/n, while unsuccessful trajectories have residual−k/n\-k/n\. Consequently,

𝔼​\[g^​\(U\)∣K=k\]\\displaystyle\\mathbb\{E\}\[\\widehat\{g\}\(U\)\\mid K=k\]=1n​dk\[\(1−kn\)𝔼\[∑i:Ri=1Ui\|K=k\]−kn𝔼\[∑i:Ri=0Ui\|K=k\]\]\\displaystyle=\\frac\{1\}\{nd\_\{k\}\}\\left\[\\left\(1\-\\frac\{k\}\{n\}\\right\)\\mathbb\{E\}\\\!\\left\[\\sum\_\{i:R\_\{i\}=1\}U\_\{i\}\\,\\middle\|\\,K=k\\right\]\-\\frac\{k\}\{n\}\\mathbb\{E\}\\\!\\left\[\\sum\_\{i:R\_\{i\}=0\}U\_\{i\}\\,\\middle\|\\,K=k\\right\]\\right\]\(29\)=\(1−k/n\)​k​u1−\(k/n\)​\(n−k\)​u0n​dk\\displaystyle=\\frac\{\(1\-k/n\)ku\_\{1\}\-\(k/n\)\(n\-k\)u\_\{0\}\}\{nd\_\{k\}\}=k⁡\(n−k\)n2​dk​\(u1−u0\)\.\\displaystyle=\\frac\{k\(n\-k\)\}\{n^\{2\}d\_\{k\}\}\(u\_\{1\}\-u\_\{0\}\)\.Fork=0k=0ork=nk=n, everyYiY\_\{i\}is zero, so the group score vanishes\. For the remaining terms, use

k⁡\(n−k\)​\(nk\)=n\!\(k−1\)\!​\(n−k−1\)\!=n⁡\(n−1\)​\(n−2k−1\)\.k\(n\-k\)\\binom\{n\}\{k\}=\\frac\{n\!\}\{\(k\-1\)\!\(n\-k\-1\)\!\}=n\(n\-1\)\\binom\{n\-2\}\{k\-1\}\.\(30\)SinceK∼Binomial⁡\(n,p\)K\\sim\\operatorname\{Binomial\}\(n,p\), total expectation and the substitutionz=k−1z=k\-1give

𝔼​g^​\(U\)\\displaystyle\\mathbb\{E\}\\widehat\{g\}\(U\)=∑k=1n−1\(nk\)​pk​\(1−p\)n−k​k⁡\(n−k\)n2​dk​\(u1−u0\)\\displaystyle=\\sum\_\{k=1\}^\{n\-1\}\\binom\{n\}\{k\}p^\{k\}\(1\-p\)^\{n\-k\}\\frac\{k\(n\-k\)\}\{n^\{2\}d\_\{k\}\}\(u\_\{1\}\-u\_\{0\}\)\(31\)=n−1n​\(u1−u0\)​∑k=1n−1\(n−2k−1\)​pk​\(1−p\)n−kdk\\displaystyle=\\frac\{n\-1\}\{n\}\(u\_\{1\}\-u\_\{0\}\)\\sum\_\{k=1\}^\{n\-1\}\\binom\{n\-2\}\{k\-1\}\\frac\{p^\{k\}\(1\-p\)^\{n\-k\}\}\{d\_\{k\}\}=n−1n​p​\(1−p\)​\(u1−u0\)​∑k=1n−1\(n−2k−1\)​pk−1​\(1−p\)n−k−1dk\\displaystyle=\\frac\{n\-1\}\{n\}p\(1\-p\)\(u\_\{1\}\-u\_\{0\}\)\\sum\_\{k=1\}^\{n\-1\}\\binom\{n\-2\}\{k\-1\}\\frac\{p^\{k\-1\}\(1\-p\)^\{n\-k\-1\}\}\{d\_\{k\}\}=p⁡\(1−p\)​\(u1−u0\)​n−1n​∑z=0n−2\(n−2z\)​pz​\(1−p\)n−2−zdz\+1\.\\displaystyle=p\(1\-p\)\(u\_\{1\}\-u\_\{0\}\)\\,\\frac\{n\-1\}\{n\}\\sum\_\{z=0\}^\{n\-2\}\\binom\{n\-2\}\{z\}\\frac\{p^\{z\}\(1\-p\)^\{n\-2\-z\}\}\{d\_\{z\+1\}\}\.Conditioning on the binary outcome also gives

𝔼⁡\[U⁡\(R−p\)\]=p⁡\(1−p\)​u1\+\(1−p\)​\(−p\)​u0=p⁡\(1−p\)​\(u1−u0\)\.\\mathbb\{E\}\[U\(R\-p\)\]=p\(1\-p\)u\_\{1\}\+\(1\-p\)\(\-p\)u\_\{0\}=p\(1\-p\)\(u\_\{1\}\-u\_\{0\}\)\.\(32\)Define

ωn​\(p\):=n−1n​∑z=0n−2\(n−2z\)​pz​\(1−p\)n−2−zσz\+1\+ε\>0\.\\omega\_\{n\}\(p\):=\\frac\{n\-1\}\{n\}\\sum\_\{z=0\}^\{n\-2\}\\binom\{n\-2\}\{z\}\\frac\{p^\{z\}\(1\-p\)^\{n\-2\-z\}\}\{\\sigma\_\{z\+1\}\+\\varepsilon\}\>0\.\(33\)Every summand is positive because0<p<10<p<1,n≥2n\\geq 2, andε\>0\\varepsilon\>0\. The two expectations therefore satisfy

𝔼​g^​\(U\)=ωn​\(p\)​𝔼​\[U⁡\(R−p\)\]\.\\mathbb\{E\}\\widehat\{g\}\(U\)=\\omega\_\{n\}\(p\)\\mathbb\{E\}\[U\(R\-p\)\]\.\(34\)TakingU=SU=Sgivesg^​\(U\)=g^grp\\widehat\{g\}\(U\)=\\widehat\{g\}\_\{\\mathrm\{grp\}\}, and Eq\. \([22](https://arxiv.org/html/2610.40726#A1.E22)\) yields

𝔼​g^grp=ωn​\(p\)​𝔼π​\[S⁡\(R−p\)\]=ωn​\(p\)​∇θp\.\\mathbb\{E\}\\widehat\{g\}\_\{\\mathrm\{grp\}\}=\\omega\_\{n\}\(p\)\\mathbb\{E\}\_\{\\pi\}\[S\(R\-p\)\]=\\omega\_\{n\}\(p\)\\nabla\_\{\\theta\}p\.\(35\)This proves Eq\. \([14](https://arxiv.org/html/2610.40726#S4.E14)\)\.

#### Residual conditional means and regression risk\.

The critic usesλ=1\\lambda=1for its return target\. For a real decisiont<Lt<L, telescoping and zero terminal bootstrapping give

G^t\\displaystyle\\widehat\{G\}\_\{t\}=Vϕ​\(st\)\+∑k=tL−1\(rk\+Vϕ​\(sk\+1\)−Vϕ​\(sk\)\)\\displaystyle=V\_\{\\phi\}\(s\_\{t\}\)\+\\sum\_\{k=t\}^\{L\-1\}\\bigl\(r\_\{k\}\+V\_\{\\phi\}\(s\_\{k\+1\}\)\-V\_\{\\phi\}\(s\_\{k\}\)\\bigr\)\(36\)=∑k=tL−1rk\+Vϕ​\(st\)\+Vϕ​\(sL\)−Vϕ​\(st\)\\displaystyle=\\sum\_\{k=t\}^\{L\-1\}r\_\{k\}\+V\_\{\\phi\}\(s\_\{t\}\)\+V\_\{\\phi\}\(s\_\{L\}\)\-V\_\{\\phi\}\(s\_\{t\}\)=∑k=tL−1rk=R\.\\displaystyle=\\sum\_\{k=t\}^\{L\-1\}r\_\{k\}=R\.Fix a real states=si,ts=s\_\{i,t\}and putx=Vπ​\(s\)x=V^\{\\pi\}\(s\)\. All conditional identities below are understood almost surely over such states\. Because the reward is terminal,𝔼⁡\[Ri∣s\]=x\\mathbb\{E\}\[R\_\{i\}\\mid s\]=x\. The other trajectories remain independent ofτi\\tau\_\{i\}after conditioning onss, soK−i\|s∼Binomial⁡\(n−1,p\)K\_\{\-i\}\\mid s\\sim\\operatorname\{Binomial\}\(n\-1,p\)andK−iK\_\{\-i\}is conditionally independent ofRiR\_\{i\}\. WritingR¯−i=K−i/\(n−1\)\\overline\{R\}\_\{\-i\}=K\_\{\-i\}/\(n\-1\), we obtain

Yi=Ri−Ri\+K−in=n−1n​Ri−K−in=n−1n​\(Ri−R¯−i\)\.Y\_\{i\}=R\_\{i\}\-\\frac\{R\_\{i\}\+K\_\{\-i\}\}\{n\}=\\frac\{n\-1\}\{n\}R\_\{i\}\-\\frac\{K\_\{\-i\}\}\{n\}=\\frac\{n\-1\}\{n\}\(R\_\{i\}\-\\overline\{R\}\_\{\-i\}\)\.\(37\)It follows that

𝔼⁡\[Yi∣s\]=n−1n​𝔼​\[Ri∣s\]−1n​𝔼​\[K−i∣s\]=n−1n​\(x−p\)\.\\mathbb\{E\}\[Y\_\{i\}\\mid s\]=\\frac\{n\-1\}\{n\}\\mathbb\{E\}\[R\_\{i\}\\mid s\]\-\\frac\{1\}\{n\}\\mathbb\{E\}\[K\_\{\-i\}\\mid s\]=\\frac\{n\-1\}\{n\}\(x\-p\)\.\(38\)Conditional independence makesCov⁡\(Ri,K−i∣s\)=0\\operatorname\{Cov\}\(R\_\{i\},K\_\{\-i\}\\mid s\)=0, hence

Var⁡\(Yi∣s\)\\displaystyle\\operatorname\{Var\}\(Y\_\{i\}\\mid s\)=\(n−1\)2n2​Var⁡\(Ri∣s\)\+1n2​Var⁡\(K−i∣s\)−2​\(n−1\)n2​Cov⁡\(Ri,K−i∣s\)\\displaystyle=\\frac\{\(n\-1\)^\{2\}\}\{n^\{2\}\}\\operatorname\{Var\}\(R\_\{i\}\\mid s\)\+\\frac\{1\}\{n^\{2\}\}\\operatorname\{Var\}\(K\_\{\-i\}\\mid s\)\-\\frac\{2\(n\-1\)\}\{n^\{2\}\}\\operatorname\{Cov\}\(R\_\{i\},K\_\{\-i\}\\mid s\)\(39\)=\(n−1\)2n2​x​\(1−x\)\+n−1n2​p​\(1−p\)\.\\displaystyle=\\frac\{\(n\-1\)^\{2\}\}\{n^\{2\}\}x\(1\-x\)\+\\frac\{n\-1\}\{n^\{2\}\}p\(1\-p\)\.For a scalar predictionff, writem⁡\(s\)=𝔼⁡\[Yi∣s\]m\(s\)=\\mathbb\{E\}\[Y\_\{i\}\\mid s\]\. Since𝔼⁡\[m⁡\(s\)−Yi∣s\]=0\\mathbb\{E\}\[m\(s\)\-Y\_\{i\}\\mid s\]=0, the conditional squared loss decomposes as

𝔼⁡\[\(f−Yi\)2∣s\]\\displaystyle\\mathbb\{E\}\[\(f\-Y\_\{i\}\)^\{2\}\\mid s\]=𝔼⁡\[\(\(f−m⁡\(s\)\)\+\(m⁡\(s\)−Yi\)\)2∣s\]\\displaystyle=\\mathbb\{E\}\[\(\(f\-m\(s\)\)\+\(m\(s\)\-Y\_\{i\}\)\)^\{2\}\\mid s\]\(40\)=\(f−m⁡\(s\)\)2\+2​\(f−m⁡\(s\)\)​𝔼​\[m⁡\(s\)−Yi∣s\]\+𝔼⁡\[\(Yi−m⁡\(s\)\)2∣s\]\\displaystyle=\(f\-m\(s\)\)^\{2\}\+2\(f\-m\(s\)\)\\mathbb\{E\}\[m\(s\)\-Y\_\{i\}\\mid s\]\+\\mathbb\{E\}\[\(Y\_\{i\}\-m\(s\)\)^\{2\}\\mid s\]=\(f−m⁡\(s\)\)2\+Var⁡\(Yi∣s\)\\displaystyle=\(f\-m\(s\)\)^\{2\}\+\\operatorname\{Var\}\(Y\_\{i\}\\mid s\)=\[f−n−1n​\(x−p\)\]2\+\(n−1\)2n2​x​\(1−x\)\+n−1n2​p​\(1−p\)\.\\displaystyle=\\left\[f\-\\frac\{n\-1\}\{n\}\(x\-p\)\\right\]^\{2\}\+\\frac\{\(n\-1\)^\{2\}\}\{n^\{2\}\}x\(1\-x\)\+\\frac\{n\-1\}\{n^\{2\}\}p\(1\-p\)\.

#### Conditioning on mixed groups\.

Letℳ=\{0<K<n\}\\mathcal\{M\}=\\\{0<K<n\\\}\. On\{Ri=1\}\\\{R\_\{i\}=1\\\}, this event requiresK−i<n−1K\_\{\-i\}<n\-1, whereas on\{Ri=0\}\\\{R\_\{i\}=0\\\}it requiresK−i\>0K\_\{\-i\}\>0\. Conditional independence therefore gives the joint probability

Pr⁡\(Ri=1,ℳ∣s\)=Pr⁡\(Ri=1∣s\)​Pr⁡\(K−i<n−1∣s\)=x⁡\(1−pn−1\)\.\\Pr\(R\_\{i\}=1,\\mathcal\{M\}\\mid s\)=\\Pr\(R\_\{i\}=1\\mid s\)\\Pr\(K\_\{\-i\}<n\-1\\mid s\)=x\\bigl\(1\-p^\{n\-1\}\\bigr\)\.\(41\)For the unsuccessful outcome, it gives

Pr⁡\(Ri=0,ℳ∣s\)=Pr⁡\(Ri=0∣s\)​Pr​\(K−i\>0∣s\)=\(1−x\)​\(1−\(1−p\)n−1\)\.\\Pr\(R\_\{i\}=0,\\mathcal\{M\}\\mid s\)=\\Pr\(R\_\{i\}=0\\mid s\)\\Pr\(K\_\{\-i\}\>0\\mid s\)=\(1\-x\)\\bigl\(1\-\(1\-p\)^\{n\-1\}\\bigr\)\.\(42\)These identities also hold atx=0x=0andx=1x=1\. Adding the disjoint events yields

Pr⁡\(ℳ∣s\)\\displaystyle\\Pr\(\\mathcal\{M\}\\mid s\)=Pr⁡\(Ri=1,ℳ∣s\)\+Pr⁡\(Ri=0,ℳ∣s\)\\displaystyle=\\Pr\(R\_\{i\}=1,\\mathcal\{M\}\\mid s\)\+\\Pr\(R\_\{i\}=0,\\mathcal\{M\}\\mid s\)\(43\)=\[1−pn−1\]​x\+\[1−\(1−p\)n−1\]​\(1−x\)\\displaystyle=\[1\-p^\{n\-1\}\]x\+\[1\-\(1\-p\)^\{n\-1\}\]\(1\-x\)=1−x​pn−1−\(1−x\)​\(1−p\)n−1\>0\.\\displaystyle=1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\>0\.Positivity holds for everyx∈\[0,1\]x\\in\[0,1\]because both1−pn−11\-p^\{n\-1\}and1−\(1−p\)n−11\-\(1\-p\)^\{n\-1\}are positive\. Onℳc\\mathcal\{M\}^\{c\}, all rewards coincide, soYi=0Y\_\{i\}=0\. Consequently,

𝔼⁡\[Yi​𝟏ℳ∣s\]=𝔼⁡\[Yi∣s\]−𝔼⁡\[Yi​𝟏ℳc∣s\]=n−1n​\(x−p\)\.\\mathbb\{E\}\[Y\_\{i\}\\mathbf\{1\}\_\{\\mathcal\{M\}\}\\mid s\]=\\mathbb\{E\}\[Y\_\{i\}\\mid s\]\-\\mathbb\{E\}\[Y\_\{i\}\\mathbf\{1\}\_\{\\mathcal\{M\}^\{c\}\}\\mid s\]=\\frac\{n\-1\}\{n\}\(x\-p\)\.\(44\)The conditional mean on mixed groups is therefore

F⋆​\(s\)\\displaystyle F^\{\\star\}\(s\)=𝔼\[Yi∣s,ℳ\]\\displaystyle=\\mathbb\{E\}\[Y\_\{i\}\\mid s,\\mathcal\{M\}\]\(45\)=𝔼⁡\[Yi​𝟏ℳ∣s\]Pr⁡\(ℳ∣s\)\\displaystyle=\\frac\{\\mathbb\{E\}\[Y\_\{i\}\\mathbf\{1\}\_\{\\mathcal\{M\}\}\\mid s\]\}\{\\Pr\(\\mathcal\{M\}\\mid s\)\}=\(n−1\)​\(x−p\)n⁡\[1−x​pn−1−\(1−x\)​\(1−p\)n−1\]\.\\displaystyle=\\frac\{\(n\-1\)\(x\-p\)\}\{n\\bigl\[1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\\bigr\]\}\.Using𝔼\[F⋆\(s\)−Yi∣s,ℳ\]=0\\mathbb\{E\}\[F^\{\\star\}\(s\)\-Y\_\{i\}\\mid s,\\mathcal\{M\}\]=0, the corresponding loss is

𝔼\[\(f−Yi\)2∣s,ℳ\]\\displaystyle\\mathbb\{E\}\[\(f\-Y\_\{i\}\)^\{2\}\\mid s,\\mathcal\{M\}\]=\(f−F⋆\(s\)\)2\+2\(f−F⋆\(s\)\)𝔼\[F⋆\(s\)−Yi∣s,ℳ\]\\displaystyle=\(f\-F^\{\\star\}\(s\)\)^\{2\}\+2\(f\-F^\{\\star\}\(s\)\)\\mathbb\{E\}\[F^\{\\star\}\(s\)\-Y\_\{i\}\\mid s,\\mathcal\{M\}\]\(46\)\+𝔼\[\(F⋆\(s\)−Yi\)2∣s,ℳ\]\\displaystyle\+\\mathbb\{E\}\[\(F^\{\\star\}\(s\)\-Y\_\{i\}\)^\{2\}\\mid s,\\mathcal\{M\}\]=\(f−F⋆​\(s\)\)2\+Var⁡\(Yi∣s,ℳ\)\\displaystyle=\(f\-F^\{\\star\}\(s\)\)^\{2\}\+\\operatorname\{Var\}\(Y\_\{i\}\\mid s,\\mathcal\{M\}\)=\[f−\(n−1\)​\(x−p\)n⁡\[1−x​pn−1−\(1−x\)​\(1−p\)n−1\]\]2\+Var⁡\(Yi∣s,ℳ\)\.\\displaystyle=\\left\[f\-\\frac\{\(n\-1\)\(x\-p\)\}\{n\\bigl\[1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\\bigr\]\}\\right\]^\{2\}\+\\operatorname\{Var\}\(Y\_\{i\}\\mid s,\\mathcal\{M\}\)\.The variance term does not depend onff, so the unique minimizer over scalar predictions isf=F⋆​\(s\)f=F^\{\\star\}\(s\)\. This identifies the population target without assuming that a parameterized critic attains it\.

#### Monotonicity and value recovery\.

Differentiating the explicit fraction yields

dd​x​\[\(n−1\)​\(x−p\)n⁡\[1−x​pn−1−\(1−x\)​\(1−p\)n−1\]\]\\displaystyle\\frac\{d\}\{dx\}\\left\[\\frac\{\(n\-1\)\(x\-p\)\}\{n\\bigl\[1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\\bigr\]\}\\right\]\(47\)=n−1n​1−x​pn−1−\(1−x\)​\(1−p\)n−1−\(x−p\)​\[\(1−p\)n−1−pn−1\]\[1−x​pn−1−\(1−x\)​\(1−p\)n−1\]2\\displaystyle=\\frac\{n\-1\}\{n\}\\,\\frac\{1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\-\(x\-p\)\\bigl\[\(1\-p\)^\{n\-1\}\-p^\{n\-1\}\\bigr\]\}\{\\bigl\[1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\\bigr\]^\{2\}\}=n−1n​1−\(1−p\)n−1\+p⁡\[\(1−p\)n−1−pn−1\]\[1−x​pn−1−\(1−x\)​\(1−p\)n−1\]2\\displaystyle=\\frac\{n\-1\}\{n\}\\,\\frac\{1\-\(1\-p\)^\{n\-1\}\+p\\bigl\[\(1\-p\)^\{n\-1\}\-p^\{n\-1\}\\bigr\]\}\{\\bigl\[1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\\bigr\]^\{2\}\}=\(n−1\)​\[1−pn−\(1−p\)n\]n​\[1−x​pn−1−\(1−x\)​\(1−p\)n−1\]2\>0\.\\displaystyle=\\frac\{\(n\-1\)\\bigl\[1\-p^\{n\}\-\(1\-p\)^\{n\}\\bigr\]\}\{n\\bigl\[1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\\bigr\]^\{2\}\}\>0\.Indeed,pn<pp^\{n\}<pand\(1−p\)n<1−p\(1\-p\)^\{n\}<1\-pforn≥2n\\geq 2, so the numerator is positive\. The denominator is nonzero throughout\[0,1\]\[0,1\]\. The mean value theorem establishes strict ordering inxx, including the endpoints, and completes the proof of Eq\. \([15](https://arxiv.org/html/2610.40726#S4.E15)\)\.

For fixed knownpp, setu=F⋆​\(s\)u=F^\{\\star\}\(s\)\. Clearing the denominator gives

n​u​\[1−x​pn−1−\(1−x\)​\(1−p\)n−1\]=\(n−1\)​\(x−p\)\.nu\\bigl\[1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\\bigr\]=\(n\-1\)\(x\-p\)\.\(48\)Expanding the terms inxxyields

n⁡\[1−\(1−p\)n−1\]​u\+n⁡\[\(1−p\)n−1−pn−1\]​u​x=\(n−1\)​x−\(n−1\)​p\.n\\bigl\[1\-\(1\-p\)^\{n\-1\}\\bigr\]u\+n\\bigl\[\(1\-p\)^\{n\-1\}\-p^\{n\-1\}\\bigr\]ux=\(n\-1\)x\-\(n\-1\)p\.\(49\)Collecting them on the left gives

\[n−1−n⁡\(\(1−p\)n−1−pn−1\)​u\]​x=n⁡\[1−\(1−p\)n−1\]​u\+\(n−1\)​p\.\\left\[n\-1\-n\\bigl\(\(1\-p\)^\{n\-1\}\-p^\{n\-1\}\\bigr\)u\\right\]x=n\\bigl\[1\-\(1\-p\)^\{n\-1\}\\bigr\]u\+\(n\-1\)p\.\(50\)For every attainableu=F⋆​\(s\)u=F^\{\\star\}\(s\), its coefficient is strictly positive:

n−1−n⁡\[\(1−p\)n−1−pn−1\]​F⋆​\(s\)\\displaystyle n\-1\-n\\bigl\[\(1\-p\)^\{n\-1\}\-p^\{n\-1\}\\bigr\]F^\{\\star\}\(s\)\(51\)=\(n−1\)​\[1−\[\(1−p\)n−1−pn−1\]​\(x−p\)1−x​pn−1−\(1−x\)​\(1−p\)n−1\]\\displaystyle=\(n\-1\)\\left\[1\-\\frac\{\\bigl\[\(1\-p\)^\{n\-1\}\-p^\{n\-1\}\\bigr\]\(x\-p\)\}\{1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\}\\right\]=\(n−1\)​\[1−pn−\(1−p\)n\]1−x​pn−1−\(1−x\)​\(1−p\)n−1\>0\.\\displaystyle=\\frac\{\(n\-1\)\\bigl\[1\-p^\{n\}\-\(1\-p\)^\{n\}\\bigr\]\}\{1\-xp^\{n\-1\}\-\(1\-x\)\(1\-p\)^\{n\-1\}\}\>0\.Division therefore recovers the value as

Vπ​\(s\)=n⁡\[1−\(1−p\)n−1\]​u\+\(n−1\)​pn−1−n⁡\[\(1−p\)n−1−pn−1\]​u\.V^\{\\pi\}\(s\)=\\frac\{n\\bigl\[1\-\(1\-p\)^\{n\-1\}\\bigr\]u\+\(n\-1\)p\}\{n\-1\-n\\bigl\[\(1\-p\)^\{n\-1\}\-p^\{n\-1\}\\bigr\]u\}\.\(52\)

### A\.3Proof of Proposition[2](https://arxiv.org/html/2610.40726#Thmproposition2)

#### Outcome martingale and TD residuals\.

DefineMt=𝔼π​\[R∣ℱt\]M\_\{t\}=\\mathbb\{E\}\_\{\\pi\}\[R\\mid\\mathcal\{F\}\_\{t\}\]andDt=Mt\+1−MtD\_\{t\}=M\_\{t\+1\}\-M\_\{t\}\. BecauseR<tR\_\{<t\}isℱt\\mathcal\{F\}\_\{t\}\-measurable,

Mt=𝔼π​\[R<t\+Gt∣ℱt\]=R<t\+𝔼π​\[Gt∣ℱt\]=R<t\+Vtπ\.M\_\{t\}=\\mathbb\{E\}\_\{\\pi\}\[R\_\{<t\}\+G\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=R\_\{<t\}\+\\mathbb\{E\}\_\{\\pi\}\[G\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=R\_\{<t\}\+V\_\{t\}^\{\\pi\}\.\(53\)The initial history contains only the fixed prompt, soM0=pM\_\{0\}=p\. At the terminal horizon, the reward has been observed andVHπ=0V\_\{H\}^\{\\pi\}=0, givingMH=RM\_\{H\}=R\. The tower property shows that\(Mt\)\(M\_\{t\}\)is a martingale:

𝔼π​\[Mt\+1∣ℱt\]=𝔼π​\[𝔼π​\[R∣ℱt\+1\]∣ℱt\]=𝔼π​\[R∣ℱt\]=Mt\.\\mathbb\{E\}\_\{\\pi\}\[M\_\{t\+1\}\\mid\\mathcal\{F\}\_\{t\}\]=\\mathbb\{E\}\_\{\\pi\}\[\\mathbb\{E\}\_\{\\pi\}\[R\\mid\\mathcal\{F\}\_\{t\+1\}\]\\mid\\mathcal\{F\}\_\{t\}\]=\\mathbb\{E\}\_\{\\pi\}\[R\\mid\\mathcal\{F\}\_\{t\}\]=M\_\{t\}\.\(54\)SinceR<t\+1−R<t=rtR\_\{<t\+1\}\-R\_\{<t\}=r\_\{t\}, its increments satisfy

Dt=\(R<t\+1\+Vt\+1π\)−\(R<t\+Vtπ\)=rt\+Vt\+1π−Vtπ=δtπ\.D\_\{t\}=\(R\_\{<t\+1\}\+V\_\{t\+1\}^\{\\pi\}\)\-\(R\_\{<t\}\+V\_\{t\}^\{\\pi\}\)=r\_\{t\}\+V\_\{t\+1\}^\{\\pi\}\-V\_\{t\}^\{\\pi\}=\\delta\_\{t\}^\{\\pi\}\.\(55\)Their conditional means vanish:

𝔼π​\[Dt∣ℱt\]=𝔼π​\[Mt\+1∣ℱt\]−Mt=0\.\\mathbb\{E\}\_\{\\pi\}\[D\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=\\mathbb\{E\}\_\{\\pi\}\[M\_\{t\+1\}\\mid\\mathcal\{F\}\_\{t\}\]\-M\_\{t\}=0\.\(56\)Telescoping over the fixed horizon gives

R−p=MH−M0=∑t=0H−1Dt\.R\-p=M\_\{H\}\-M\_\{0\}=\\sum\_\{t=0\}^\{H\-1\}D\_\{t\}\.\(57\)All increments and policy scores after termination are zero, so the same identities apply to trajectories of different lengths\.

#### Cancellation of past and future increments\.

Substituting Eq\. \([57](https://arxiv.org/html/2610.40726#A1.E57)\) into the score identity yields

∇θp\\displaystyle\\nabla\_\{\\theta\}p=𝔼π​\[\(∑t=0H−1ψt\)​\(∑k=0H−1Dk\)\]\\displaystyle=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\left\(\\sum\_\{t=0\}^\{H\-1\}\\psi\_\{t\}\\right\)\\left\(\\sum\_\{k=0\}^\{H\-1\}D\_\{k\}\\right\)\\right\]\(58\)=∑t=0H−1∑k=0H−1𝔼π​\[ψt​Dk\]\\displaystyle=\\sum\_\{t=0\}^\{H\-1\}\\sum\_\{k=0\}^\{H\-1\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\]=∑t∑k<t𝔼π​\[ψt​Dk\]\+∑t𝔼π​\[ψt​Dt\]\+∑t∑k\>t𝔼π​\[ψt​Dk\]\.\\displaystyle=\\sum\_\{t\}\\sum\_\{k<t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\]\+\\sum\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{t\}\]\+\\sum\_\{t\}\\sum\_\{k\>t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\]\.Ifk<tk<t, thenDkD\_\{k\}isℱk\+1\\mathcal\{F\}\_\{k\+1\}\-measurable andℱk\+1⊆ℱt\\mathcal\{F\}\_\{k\+1\}\\subseteq\\mathcal\{F\}\_\{t\}\. Conditioning onℱt\\mathcal\{F\}\_\{t\}eliminates the policy score:

𝔼π​\[ψt​Dk\]=𝔼π​\[𝔼π​\[ψt​Dk∣ℱt\]\]=𝔼π​\[Dk​𝔼π​\[ψt∣ℱt\]\]=0\.\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\]=\\mathbb\{E\}\_\{\\pi\}\[\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\\mid\\mathcal\{F\}\_\{t\}\]\]=\\mathbb\{E\}\_\{\\pi\}\[D\_\{k\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]\]=0\.\(59\)Ifk\>tk\>t, thenψt\\psi\_\{t\}isℱt\+1\\mathcal\{F\}\_\{t\+1\}\-measurable andℱt\+1⊆ℱk\\mathcal\{F\}\_\{t\+1\}\\subseteq\\mathcal\{F\}\_\{k\}\. Conditioning onℱk\\mathcal\{F\}\_\{k\}instead eliminates the future increment:

𝔼π​\[ψt​Dk\]=𝔼π​\[𝔼π​\[ψt​Dk∣ℱk\]\]=𝔼π​\[ψt​𝔼π​\[Dk∣ℱk\]\]=0\.\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\]=\\mathbb\{E\}\_\{\\pi\}\[\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\\mid\\mathcal\{F\}\_\{k\}\]\]=\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[D\_\{k\}\\mid\\mathcal\{F\}\_\{k\}\]\]=0\.\(60\)Only the matching indices remain, proving the first identity:

∇θp=∑t𝔼π​\[ψt​Dt\]=𝔼π​\[∑tψt​δtπ\]\.\\nabla\_\{\\theta\}p=\\sum\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{t\}\]=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t\}\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\right\]\.\(61\)

#### Generalized advantage estimation\.

For fixed0<λ≤10<\\lambda\\leq 1, unrolling the GAE recursion withA^Hλ=0\\widehat\{A\}\_\{H\}^\{\\lambda\}=0gives

A^tλ​\(Vπ\)\\displaystyle\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)=δtπ\+λ​A^t\+1λ​\(Vπ\)\\displaystyle=\\delta\_\{t\}^\{\\pi\}\+\\lambda\\widehat\{A\}\_\{t\+1\}^\{\\lambda\}\(V^\{\\pi\}\)\(62\)=δtπ\+λ​δt\+1π\+λ2​A^t\+2λ​\(Vπ\)\\displaystyle=\\delta\_\{t\}^\{\\pi\}\+\\lambda\\delta\_\{t\+1\}^\{\\pi\}\+\\lambda^\{2\}\\widehat\{A\}\_\{t\+2\}^\{\\lambda\}\(V^\{\\pi\}\)=∑k=tH−1λk−t​δkπ\.\\displaystyle=\\sum\_\{k=t\}^\{H\-1\}\\lambda^\{k\-t\}\\delta\_\{k\}^\{\\pi\}\.The second line applies whent≤H−2t\\leq H\-2, and the last decision hasA^H−1λ=δH−1π\\widehat\{A\}\_\{H\-1\}^\{\\lambda\}=\\delta\_\{H\-1\}^\{\\pi\}\. Sinceλ\\lambdais deterministic, the cancellation above yields

𝔼π​\[ψt​A^tλ​\(Vπ\)\]\\displaystyle\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)\]=∑k=tH−1λk−t​𝔼π​\[ψt​Dk\]\\displaystyle=\\sum\_\{k=t\}^\{H\-1\}\\lambda^\{k\-t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{k\}\]\(63\)=𝔼π​\[ψt​Dt\]\+∑k=t\+1H−1λk−t​𝔼π​\[ψt​𝔼π​\[Dk∣ℱk\]\]\\displaystyle=\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}D\_\{t\}\]\+\\sum\_\{k=t\+1\}^\{H\-1\}\\lambda^\{k\-t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[D\_\{k\}\\mid\\mathcal\{F\}\_\{k\}\]\]=𝔼π​\[ψt​δtπ\]\.\\displaystyle=\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\]\.Summing over decisions proves

𝔼π​\[∑tψt​A^tλ​\(Vπ\)\]=∑t𝔼π​\[ψt​δtπ\]=∇θp\.\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)\\right\]=\\sum\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\]=\\nabla\_\{\\theta\}p\.\(64\)Forλ=0\\lambda=0, the recursion givesA^t0​\(Vπ\)=δtπ\\widehat\{A\}\_\{t\}^\{0\}\(V^\{\\pi\}\)=\\delta\_\{t\}^\{\\pi\}, so Eq\. \([61](https://arxiv.org/html/2610.40726#A1.E61)\) covers this endpoint directly\. Together with Proposition[1](https://arxiv.org/html/2610.40726#Thmproposition1), the result gives

𝔼​g^grp=ωn​\(p\)​∇θp=ωn​\(p\)​𝔼π​\[∑tψt​A^tλ​\(Vπ\)\]\.\\mathbb\{E\}\\widehat\{g\}\_\{\\mathrm\{grp\}\}=\\omega\_\{n\}\(p\)\\nabla\_\{\\theta\}p=\\omega\_\{n\}\(p\)\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)\\right\]\.\(65\)This identity relates expected gradients and requires exact values for the single\-trajectory estimator\. It does not assert equality of individual samples or of their variances\.

### A\.4Behavior policies and importance weighting

Letβt\(⋅∣ℱt\)\\beta\_\{t\}\(\\cdot\\mid\\mathcal\{F\}\_\{t\}\)be the behavior policy used to sample actionata\_\{t\}\. Any policy\-version information needed to specify this conditional law is included in the history\. The target and behavior laws share the initial prompt and the transition and reward kernels\. Assume the behavior policy assigns positive probability to every action with positive target probability at the histories under consideration\. Behavior probabilities are held fixed when differentiating the target policy\. Define

ρt=πθ​\(at∣st\)βt​\(at∣ℱt\)\.\\rho\_\{t\}=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\beta\_\{t\}\(a\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\)\}\.\(66\)WriteW0:t=∏j=0tρjW\_\{0:t\}=\\prod\_\{j=0\}^\{t\}\\rho\_\{j\}for the prefix importance weight\. Productsρt​ψt\\rho\_\{t\}\\psi\_\{t\}at zero target probability are interpreted as∇θπθ​\(at∣st\)/βt​\(at∣ℱt\)\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)/\\beta\_\{t\}\(a\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\)\.

#### What a single action ratio corrects\.

For any integrable functionb⁡\(ℱt,at\)b\(\\mathcal\{F\}\_\{t\},a\_\{t\}\), conditioning on the sampled action gives

𝔼β​\[ρt​b​\(ℱt,at\)∣ℱt\]=∑aβt​\(a∣ℱt\)​πθ​\(a∣st\)βt​\(a∣ℱt\)​b​\(ℱt,a\)=∑aπθ​\(a∣st\)​b​\(ℱt,a\)\.\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}b\(\\mathcal\{F\}\_\{t\},a\_\{t\}\)\\mid\\mathcal\{F\}\_\{t\}\]=\\sum\_\{a\}\\beta\_\{t\}\(a\\mid\\mathcal\{F\}\_\{t\}\)\\frac\{\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)\}\{\\beta\_\{t\}\(a\\mid\\mathcal\{F\}\_\{t\}\)\}b\(\\mathcal\{F\}\_\{t\},a\)=\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)b\(\\mathcal\{F\}\_\{t\},a\)\.\(67\)Thus the ratio corrects the conditional action distribution\. To distinguish this from correcting the history distribution, define

Qtπ\(s,a\)=𝔼\[rt\+Vt\+1π∣st=s,at=a\]\.Q\_\{t\}^\{\\pi\}\(s,a\)=\\mathbb\{E\}\[r\_\{t\}\+V\_\{t\+1\}^\{\\pi\}\\mid s\_\{t\}=s,a\_\{t\}=a\]\.\(68\)The expectation uses the common transition and reward kernels\. Withψt​\(s,a\)=∇θ​log​πθ​\(a∣s\)\\psi\_\{t\}\(s,a\)=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\\mid s\), let

ft​\(s\)=∑aπθ​\(a∣s\)​ψt​\(s,a\)​\(Qtπ​\(s,a\)−Vtπ​\(s\)\)\.f\_\{t\}\(s\)=\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\)\\psi\_\{t\}\(s,a\)\\bigl\(Q\_\{t\}^\{\\pi\}\(s,a\)\-V\_\{t\}^\{\\pi\}\(s\)\\bigr\)\.\(69\)A second conditioning step gives

𝔼β​\[ρt​ψt​δtπ∣ℱt\]\\displaystyle\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\mid\\mathcal\{F\}\_\{t\}\]=∑aπθ\(a∣st\)ψt\(st,a\)𝔼\[δtπ∣st,at=a\]\\displaystyle=\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)\\psi\_\{t\}\(s\_\{t\},a\)\\mathbb\{E\}\[\\delta\_\{t\}^\{\\pi\}\\mid s\_\{t\},a\_\{t\}=a\]\(70\)=∑aπθ​\(a∣st\)​ψt​\(st,a\)​\(Qtπ​\(st,a\)−Vtπ​\(st\)\)=ft​\(st\)\.\\displaystyle=\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)\\psi\_\{t\}\(s\_\{t\},a\)\\bigl\(Q\_\{t\}^\{\\pi\}\(s\_\{t\},a\)\-V\_\{t\}^\{\\pi\}\(s\_\{t\}\)\\bigr\)=f\_\{t\}\(s\_\{t\}\)\.The same conditional expectation underπ\\pi, without a ratio, is alsoft​\(st\)f\_\{t\}\(s\_\{t\}\)\. Writingdtβd\_\{t\}^\{\\beta\}anddtπd\_\{t\}^\{\\pi\}for the two state distributions therefore yields

𝔼β​\[ρt​ψt​δtπ\]−𝔼π​\[ψt​δtπ\]=∫ft​\(s\)​dtβ​\(𝑑s\)−∫ft​\(s\)​dtπ​\(𝑑s\)=∫ft​\(s\)​\(dtβ−dtπ\)​\(𝑑s\)\.\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\]\-\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\]=\\int f\_\{t\}\(s\)\\,d\_\{t\}^\{\\beta\}\(ds\)\-\\int f\_\{t\}\(s\)\\,d\_\{t\}^\{\\pi\}\(ds\)=\\int f\_\{t\}\(s\)\(d\_\{t\}^\{\\beta\}\-d\_\{t\}^\{\\pi\}\)\(ds\)\.\(71\)

#### Prefix correction and future TD residuals\.

On the history throughata\_\{t\},rtr\_\{t\}, andst\+1s\_\{t\+1\}, cancellation of the common environment kernels gives

d⁡\(Pπ\|ℱt\+1\)d⁡\(Pβ\|ℱt\+1\)=∏j=0tπθ​\(aj∣sj\)βj​\(aj∣ℱj\)=W0:t\.\\frac\{d\(P\_\{\\pi\}\|\_\{\\mathcal\{F\}\_\{t\+1\}\}\)\}\{d\(P\_\{\\beta\}\|\_\{\\mathcal\{F\}\_\{t\+1\}\}\)\}=\\prod\_\{j=0\}^\{t\}\\frac\{\\pi\_\{\\theta\}\(a\_\{j\}\\mid s\_\{j\}\)\}\{\\beta\_\{j\}\(a\_\{j\}\\mid\\mathcal\{F\}\_\{j\}\)\}=W\_\{0:t\}\.\(72\)Sinceψt​δtπ\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}isℱt\+1\\mathcal\{F\}\_\{t\+1\}\-measurable, change of measure yields

𝔼β\[∑tW0:tψtδtπ\]=∑t𝔼π\[ψtδtπ\]=∇θp\.\\mathbb\{E\}\_\{\\beta\}\\\!\\left\[\\sum\_\{t\}W\_\{0:t\}\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\right\]=\\sum\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\]=\\nabla\_\{\\theta\}p\.\(73\)This is a reference identity for prefix weights\. The implemented action ratioρt\\rho\_\{t\}alone does not supply these weights\.

Future TD residuals introduce another discrepancy for GAE\. The Bellman identityVkπ​\(s\)=∑aπθ​\(a∣s\)​Qkπ​\(s,a\)V\_\{k\}^\{\\pi\}\(s\)=\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\)Q\_\{k\}^\{\\pi\}\(s,a\)gives

mkβ:=𝔼β​\[δkπ∣ℱk\]\\displaystyle m\_\{k\}^\{\\beta\}:=\\mathbb\{E\}\_\{\\beta\}\[\\delta\_\{k\}^\{\\pi\}\\mid\\mathcal\{F\}\_\{k\}\]=∑aβk​\(a∣ℱk\)​\(Qkπ​\(sk,a\)−Vkπ​\(sk\)\)\\displaystyle=\\sum\_\{a\}\\beta\_\{k\}\(a\\mid\\mathcal\{F\}\_\{k\}\)\\bigl\(Q\_\{k\}^\{\\pi\}\(s\_\{k\},a\)\-V\_\{k\}^\{\\pi\}\(s\_\{k\}\)\\bigr\)\(74\)=∑aβk​\(a∣ℱk\)​Qkπ​\(sk,a\)−Vkπ​\(sk\)\\displaystyle=\\sum\_\{a\}\\beta\_\{k\}\(a\\mid\\mathcal\{F\}\_\{k\}\)Q\_\{k\}^\{\\pi\}\(s\_\{k\},a\)\-V\_\{k\}^\{\\pi\}\(s\_\{k\}\)=∑a\(βk​\(a∣ℱk\)−πθ​\(a∣sk\)\)​Qkπ​\(sk,a\)\.\\displaystyle=\\sum\_\{a\}\\bigl\(\\beta\_\{k\}\(a\\mid\\mathcal\{F\}\_\{k\}\)\-\\pi\_\{\\theta\}\(a\\mid s\_\{k\}\)\\bigr\)Q\_\{k\}^\{\\pi\}\(s\_\{k\},a\)\.Fork\>tk\>t, the factorρt​ψt\\rho\_\{t\}\\psi\_\{t\}isℱk\\mathcal\{F\}\_\{k\}\-measurable, so

𝔼β​\[ρt​ψt​δkπ\]=𝔼β​\[ρt​ψt​𝔼β​\[δkπ∣ℱk\]\]=𝔼β​\[ρt​ψt​mkβ\]\.\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}\\delta\_\{k\}^\{\\pi\}\]=\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}\\mathbb\{E\}\_\{\\beta\}\[\\delta\_\{k\}^\{\\pi\}\\mid\\mathcal\{F\}\_\{k\}\]\]=\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}m\_\{k\}^\{\\beta\}\]\.\(75\)Expanding GAE and applying Eq\. \([71](https://arxiv.org/html/2610.40726#A1.E71)\) produces

𝔼β​\[∑tρt​ψt​A^tλ​\(Vπ\)\]−∇θp\\displaystyle\\mathbb\{E\}\_\{\\beta\}\\\!\\left\[\\sum\_\{t\}\\rho\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)\\right\]\-\\nabla\_\{\\theta\}p\(76\)=∑t\(𝔼β​\[ρt​ψt​δtπ\]−𝔼π​\[ψt​δtπ\]\)\+∑t<kλk−t​𝔼β​\[ρt​ψt​δkπ\]\\displaystyle=\\sum\_\{t\}\\left\(\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\]\-\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\]\\right\)\+\\sum\_\{t<k\}\\lambda^\{k\-t\}\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}\\delta\_\{k\}^\{\\pi\}\]=∑t∫ft​\(s\)​\(dtβ−dtπ\)​\(ds\)\+∑t<kλk−t​𝔼β​\[ρt​ψt​mkβ\]\.\\displaystyle=\\sum\_\{t\}\\int f\_\{t\}\(s\)\(d\_\{t\}^\{\\beta\}\-d\_\{t\}^\{\\pi\}\)\(ds\)\+\\sum\_\{t<k\}\\lambda^\{k\-t\}\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}m\_\{k\}^\{\\beta\}\]\.The two terms describe differences in the visited states and in subsequent actions, respectively\. Both vanish under on\-policy sampling\.

### A\.5Calibration and the implemented update

#### What residual supervision identifies\.

For two real states of the same prompt, Proposition[1](https://arxiv.org/html/2610.40726#Thmproposition1)implies

sign⁡\(F⋆​\(s′\)−F⋆​\(s\)\)=sign⁡\(Vπ​\(s′\)−Vπ​\(s\)\)\.\\operatorname\{sign\}\(F^\{\\star\}\(s^\{\\prime\}\)\-F^\{\\star\}\(s\)\)=\\operatorname\{sign\}\(V^\{\\pi\}\(s^\{\\prime\}\)\-V^\{\\pi\}\(s\)\)\.\(77\)At the root,F⋆​\(s0\)=0F^\{\\star\}\(s\_\{0\}\)=0for everypp, so this prediction alone does not identify the prompt success probability\. The transformation in Eq\. \([15](https://arxiv.org/html/2610.40726#S4.E15)\) can be nonlinear and need not preserve conditional expectations across stochastic transitions\. Thus ordering alone does not justify substitutingF⋆F^\{\\star\}forVπV^\{\\pi\}in Proposition[2](https://arxiv.org/html/2610.40726#Thmproposition2)\.

For a nonnegative divergence mask or sample weightcc, letℒc​\(f∣s\)=𝔼⁡\[c​𝟏ℳ​\(f−Yi\)2∣s\]\\mathcal\{L\}\_\{c\}\(f\\mid s\)=\\mathbb\{E\}\[c\\mathbf\{1\}\_\{\\mathcal\{M\}\}\(f\-Y\_\{i\}\)^\{2\}\\mid s\]\. Assume its weighted second moment is finite and0<𝔼⁡\[c​𝟏ℳ∣s\]<∞0<\\mathbb\{E\}\[c\\mathbf\{1\}\_\{\\mathcal\{M\}\}\\mid s\]<\\infty\. Expanding the quadratic gives

ℒc​\(f∣s\)=f2​𝔼​\[c​𝟏ℳ∣s\]−2​f​𝔼​\[c​𝟏ℳ​Yi∣s\]\+𝔼⁡\[c​𝟏ℳ​Yi2∣s\]\.\\mathcal\{L\}\_\{c\}\(f\\mid s\)=f^\{2\}\\mathbb\{E\}\[c\\mathbf\{1\}\_\{\\mathcal\{M\}\}\\mid s\]\-2f\\mathbb\{E\}\[c\\mathbf\{1\}\_\{\\mathcal\{M\}\}Y\_\{i\}\\mid s\]\+\\mathbb\{E\}\[c\\mathbf\{1\}\_\{\\mathcal\{M\}\}Y\_\{i\}^\{2\}\\mid s\]\.\(78\)Its leading coefficient is positive, so setting its derivative to zero yields the unique minimizer

Fc⋆​\(s\)=𝔼⁡\[c​𝟏ℳ​Yi∣s\]𝔼⁡\[c​𝟏ℳ∣s\]=𝔼\[cYi∣s,ℳ\]𝔼\[c∣s,ℳ\]\.F\_\{c\}^\{\\star\}\(s\)=\\frac\{\\mathbb\{E\}\[c\\mathbf\{1\}\_\{\\mathcal\{M\}\}Y\_\{i\}\\mid s\]\}\{\\mathbb\{E\}\[c\\mathbf\{1\}\_\{\\mathcal\{M\}\}\\mid s\]\}=\\frac\{\\mathbb\{E\}\[cY\_\{i\}\\mid s,\\mathcal\{M\}\]\}\{\\mathbb\{E\}\[c\\mid s,\\mathcal\{M\}\]\}\.\(79\)The change from the unweighted target is explicit:

Fc⋆​\(s\)−F⋆​\(s\)=𝔼\[cYi∣s,ℳ\]−𝔼\[c∣s,ℳ\]𝔼\[Yi∣s,ℳ\]𝔼\[c∣s,ℳ\]=Cov\(c,Yi∣s,ℳ\)𝔼\[c∣s,ℳ\]\.F\_\{c\}^\{\\star\}\(s\)\-F^\{\\star\}\(s\)=\\frac\{\\mathbb\{E\}\[cY\_\{i\}\\mid s,\\mathcal\{M\}\]\-\\mathbb\{E\}\[c\\mid s,\\mathcal\{M\}\]\\mathbb\{E\}\[Y\_\{i\}\\mid s,\\mathcal\{M\}\]\}\{\\mathbb\{E\}\[c\\mid s,\\mathcal\{M\}\]\}=\\frac\{\\operatorname\{Cov\}\(c,Y\_\{i\}\\mid s,\\mathcal\{M\}\)\}\{\\mathbb\{E\}\[c\\mid s,\\mathcal\{M\}\]\}\.\(80\)Positive weights depending only on the state cancel, whereas masks depending on outcomes can change the target\. Random loss denominators, value clipping, and ranking regularization can further change the objective\. After switching, the base absolute\-return loss instead satisfies

𝔼π​\[\(f−Gt\)2∣st=s\]=\(f−Vπ​\(s\)\)2\+Varπ⁡\(Gt∣st=s\)\.\\mathbb\{E\}\_\{\\pi\}\[\(f\-G\_\{t\}\)^\{2\}\\mid s\_\{t\}=s\]=\(f\-V^\{\\pi\}\(s\)\)^\{2\}\+\\operatorname\{Var\}\_\{\\pi\}\(G\_\{t\}\\mid s\_\{t\}=s\)\.\(81\)This is the calibration required by Proposition[2](https://arxiv.org/html/2610.40726#Thmproposition2)\. Residual explained variance and ranking accuracy do not by themselves certify that the learned critic has attained it\.

#### Approximate critic and advantage bias\.

Fix the critic during the actor update and defineet=Vϕ​\(st\)−Vtπe\_\{t\}=V\_\{\\phi\}\(s\_\{t\}\)\-V\_\{t\}^\{\\pi\}, with zero error at terminal and absorbing states\. Writeδtϕ=rt\+Vϕ​\(st\+1\)−Vϕ​\(st\)\\delta\_\{t\}^\{\\phi\}=r\_\{t\}\+V\_\{\\phi\}\(s\_\{t\+1\}\)\-V\_\{\\phi\}\(s\_\{t\}\)\. Thenδtϕ−δtπ=et\+1−et\\delta\_\{t\}^\{\\phi\}\-\\delta\_\{t\}^\{\\pi\}=e\_\{t\+1\}\-e\_\{t\}\. For0<λ≤10<\\lambda\\leq 1, shifting the summation index gives

A^tλ​\(Vϕ\)−A^tλ​\(Vπ\)\\displaystyle\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\_\{\\phi\}\)\-\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)=∑k=tH−1λk−t​\(ek\+1−ek\)\\displaystyle=\\sum\_\{k=t\}^\{H\-1\}\\lambda^\{k\-t\}\(e\_\{k\+1\}\-e\_\{k\}\)\(82\)=∑j=t\+1Hλj−t−1​ej−∑j=tH−1λj−t​ej\\displaystyle=\\sum\_\{j=t\+1\}^\{H\}\\lambda^\{j\-t\-1\}e\_\{j\}\-\\sum\_\{j=t\}^\{H\-1\}\\lambda^\{j\-t\}e\_\{j\}=−et\+λH−t−1​eH\+∑j=t\+1H−1\(λj−t−1−λj−t\)​ej\\displaystyle=\-e\_\{t\}\+\\lambda^\{H\-t\-1\}e\_\{H\}\+\\sum\_\{j=t\+1\}^\{H\-1\}\\bigl\(\\lambda^\{j\-t\-1\}\-\\lambda^\{j\-t\}\\bigr\)e\_\{j\}=−et\+\(1−λ\)​∑j=t\+1H−1λj−t−1​ej\.\\displaystyle=\-e\_\{t\}\+\(1\-\\lambda\)\\sum\_\{j=t\+1\}^\{H\-1\}\\lambda^\{j\-t\-1\}e\_\{j\}\.The state baselineete\_\{t\}contributes zero expected policy score\. Using Proposition[2](https://arxiv.org/html/2610.40726#Thmproposition2),

𝔼π​\[∑tψt​A^tλ​\(Vϕ\)\]−∇θp\\displaystyle\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\_\{\\phi\}\)\\right\]\-\\nabla\_\{\\theta\}p\(83\)=∑t𝔼π​\[ψt​\(A^tλ​\(Vϕ\)−A^tλ​\(Vπ\)\)\]\\displaystyle=\\sum\_\{t\}\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\psi\_\{t\}\\bigl\(\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\_\{\\phi\}\)\-\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)\\bigr\)\\right\]=−∑t𝔼π\[ψtet\]\+\(1−λ\)∑t<kλk−t−1𝔼π\[ψtek\]\\displaystyle=\-\\sum\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}e\_\{t\}\]\+\(1\-\\lambda\)\\sum\_\{t<k\}\\lambda^\{k\-t\-1\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}e\_\{k\}\]=\(1−λ\)​∑t<kλk−t−1​𝔼π​\[ψt​ek\]\.\\displaystyle=\(1\-\\lambda\)\\sum\_\{t<k\}\\lambda^\{k\-t\-1\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}e\_\{k\}\]\.Forλ=1\\lambda=1, the critic error is only a baseline and the bias vanishes\. For0<λ<10<\\lambda<1, future value errors enter through their correlation with earlier policy scores\. Atλ=0\\lambda=0, the advantage error is directlyet\+1−ete\_\{t\+1\}\-e\_\{t\}, giving bias∑t𝔼π​\[ψt​et\+1\]\\sum\_\{t\}\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}e\_\{t\+1\}\]\.

Under behavior sampling, the action ratio still eliminates the current baseline:

𝔼β​\[ρt​ψt​et\]=𝔼β​\[et​∑aπθ​\(a∣st\)​ψt​\(st,a\)\]=0\.\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}e\_\{t\}\]=\\mathbb\{E\}\_\{\\beta\}\\\!\\left\[e\_\{t\}\\sum\_\{a\}\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)\\psi\_\{t\}\(s\_\{t\},a\)\\right\]=0\.\(84\)The same pathwise expansion therefore gives

𝔼β​\[∑tρt​ψt​\(A^tλ​\(Vϕ\)−A^tλ​\(Vπ\)\)\]=\(1−λ\)​∑t<kλk−t−1​𝔼β​\[ρt​ψt​ek\]\.\\mathbb\{E\}\_\{\\beta\}\\\!\\left\[\\sum\_\{t\}\\rho\_\{t\}\\psi\_\{t\}\\bigl\(\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\_\{\\phi\}\)\-\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V^\{\\pi\}\)\\bigr\)\\right\]=\(1\-\\lambda\)\\sum\_\{t<k\}\\lambda^\{k\-t\-1\}\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}e\_\{k\}\]\.\(85\)These identities concern a fixed critic evaluated on fresh trajectories\. If the critic is fitted on the same trajectories, conditioning only on the decision history need not make its fitted prediction a valid baseline\.

#### Masking, normalization, and adaptive traces\.

For the actor surrogate with stopped advantages, letct∈\[0,1\]c\_\{t\}\\in\[0,1\]be the active multiplier from DIS and policy clipping, away from their nondifferentiable boundaries\. To compare with a fixed normalization, setg¯q=H−1​∇θp\\bar\{g\}\_\{q\}=H^\{\-1\}\\nabla\_\{\\theta\}pand define

X0=1H​∑t<Lρt​ψt​A^tλ​\(Vϕ\)\.X\_\{0\}=\\frac\{1\}\{H\}\\sum\_\{t<L\}\\rho\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\_\{\\phi\}\)\.\(86\)The corresponding score contribution with trajectory length normalization and the realized trace parameterλL\\lambda\_\{L\}is

Ximp=1L​∑t<Lct​ρt​ψt​A^tλL​\(Vϕ\)\.X\_\{\\mathrm\{imp\}\}=\\frac\{1\}\{L\}\\sum\_\{t<L\}c\_\{t\}\\rho\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\_\{L\}\}\(V\_\{\\phi\}\)\.\(87\)Insert the intermediate updatesH−1​∑t<Lct​ρt​ψt​A^tλ​\(Vϕ\)H^\{\-1\}\\sum\_\{t<L\}c\_\{t\}\\rho\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\_\{\\phi\}\)andH−1​∑t<Lct​ρt​ψt​A^tλL​\(Vϕ\)H^\{\-1\}\\sum\_\{t<L\}c\_\{t\}\\rho\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\_\{L\}\}\(V\_\{\\phi\}\)\. Subtracting them and expanding the difference of the two GAE traces yields

Ximp−X0\\displaystyle X\_\{\\mathrm\{imp\}\}\-X\_\{0\}=1H​∑t<L\(ct−1\)​ρt​ψt​A^tλ​\(Vϕ\)\+1H​∑t<Lct​ρt​ψt​∑k=tL−1\(λLk−t−λk−t\)​δkϕ\\displaystyle=\\frac\{1\}\{H\}\\sum\_\{t<L\}\(c\_\{t\}\-1\)\\rho\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\}\(V\_\{\\phi\}\)\+\\frac\{1\}\{H\}\\sum\_\{t<L\}c\_\{t\}\\rho\_\{t\}\\psi\_\{t\}\\sum\_\{k=t\}^\{L\-1\}\(\\lambda\_\{L\}^\{k\-t\}\-\\lambda^\{k\-t\}\)\\delta\_\{k\}^\{\\phi\}\(88\)\+\(1L−1H\)∑t<LctρtψtA^tλL\(Vϕ\)\.\\displaystyle\+\\left\(\\frac\{1\}\{L\}\-\\frac\{1\}\{H\}\\right\)\\sum\_\{t<L\}c\_\{t\}\\rho\_\{t\}\\psi\_\{t\}\\widehat\{A\}\_\{t\}^\{\\lambda\_\{L\}\}\(V\_\{\\phi\}\)\.By adding the behavior error in Eq\. \([76](https://arxiv.org/html/2610.40726#A1.E76)\), the critic error in Eq\. \([85](https://arxiv.org/html/2610.40726#A1.E85)\), and this pathwise difference,

𝔼β​Ximp−g¯q\\displaystyle\\mathbb\{E\}\_\{\\beta\}X\_\{\\mathrm\{imp\}\}\-\\bar\{g\}\_\{q\}=1H​∑t∫ft​\(s\)​\(dtβ−dtπ\)​\(ds\)\+1H​∑t<kλk−t​𝔼β​\[ρt​ψt​mkβ\]\\displaystyle=\\frac\{1\}\{H\}\\sum\_\{t\}\\int f\_\{t\}\(s\)\(d\_\{t\}^\{\\beta\}\-d\_\{t\}^\{\\pi\}\)\(ds\)\+\\frac\{1\}\{H\}\\sum\_\{t<k\}\\lambda^\{k\-t\}\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}m\_\{k\}^\{\\beta\}\]\(89\)\+1−λH∑t<kλk−t−1𝔼β\[ρtψtek\]\+𝔼β\[Ximp−X0\]\.\\displaystyle\+\\frac\{1\-\\lambda\}\{H\}\\sum\_\{t<k\}\\lambda^\{k\-t\-1\}\\mathbb\{E\}\_\{\\beta\}\[\\rho\_\{t\}\\psi\_\{t\}e\_\{k\}\]\+\\mathbb\{E\}\_\{\\beta\}\[X\_\{\\mathrm\{imp\}\}\-X\_\{0\}\]\.The formula separates behavior sampling, critic approximation, and the changes to the surrogate update\. Its expectations use the behavior trajectory law before any additional selection based on outcomes\. An exact critic alone does not make these other terms vanish\. In particular,λL\\lambda\_\{L\}and1/L1/Lcan depend on future actions, so the conditional cancellations in Proposition[2](https://arxiv.org/html/2610.40726#Thmproposition2)do not automatically apply to them\.

### A\.6Single\-turn reasoning and agentic histories

Between two model decisions,st\+1s\_\{t\+1\}includes any intervening tool observations\. The environment contributes no policy score, while its randomness remains in the TD residual\. Using the action value defined above, letAtπ​\(s,a\)=Qtπ​\(s,a\)−Vtπ​\(s\)A\_\{t\}^\{\\pi\}\(s,a\)=Q\_\{t\}^\{\\pi\}\(s,a\)\-V\_\{t\}^\{\\pi\}\(s\)andζt=rt\+Vt\+1π−Qtπ​\(st,at\)\\zeta\_\{t\}=r\_\{t\}\+V\_\{t\+1\}^\{\\pi\}\-Q\_\{t\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\. By definition,

𝔼\[ζt∣st,at\]=𝔼\[rt\+Vt\+1π∣st,at\]−Qtπ\(st,at\)=0\.\\mathbb\{E\}\[\\zeta\_\{t\}\\mid s\_\{t\},a\_\{t\}\]=\\mathbb\{E\}\[r\_\{t\}\+V\_\{t\+1\}^\{\\pi\}\\mid s\_\{t\},a\_\{t\}\]\-Q\_\{t\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)=0\.\(90\)The TD residual decomposes as

δtπ=Qtπ​\(st,at\)−Vtπ​\(st\)\+ζt=Atπ​\(st,at\)\+ζt\.\\delta\_\{t\}^\{\\pi\}=Q\_\{t\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\-V\_\{t\}^\{\\pi\}\(s\_\{t\}\)\+\\zeta\_\{t\}=A\_\{t\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\+\\zeta\_\{t\}\.\(91\)Under the target policy, conditioning first on\(st,at\)\(s\_\{t\},a\_\{t\}\)gives

𝔼π\[ψtδtπ∣st\]=𝔼π\[ψtAtπ\(st,at\)∣st\]\+𝔼π\[ψt𝔼\[ζt∣st,at\]∣st\]=𝔼π\[ψtAtπ\(st,at\)∣st\]\.\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\mid s\_\{t\}\]=\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}A\_\{t\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\\mid s\_\{t\}\]\+\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}\\mathbb\{E\}\[\\zeta\_\{t\}\\mid s\_\{t\},a\_\{t\}\]\\mid s\_\{t\}\]=\\mathbb\{E\}\_\{\\pi\}\[\\psi\_\{t\}A\_\{t\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\\mid s\_\{t\}\]\.\(92\)For finite second moments, the law of total covariance yields

Covπ⁡\(ψt​δtπ∣st\)\\displaystyle\\operatorname\{Cov\}\_\{\\pi\}\(\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\mid s\_\{t\}\)=Covπ\(𝔼\[ψtδtπ∣st,at\]\|st\)\+𝔼π\[Cov\(ψtδtπ∣st,at\)\|st\]\\displaystyle=\\operatorname\{Cov\}\_\{\\pi\}\\\!\\left\(\\mathbb\{E\}\[\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\mid s\_\{t\},a\_\{t\}\]\\,\\middle\|\\,s\_\{t\}\\right\)\+\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\operatorname\{Cov\}\(\\psi\_\{t\}\\delta\_\{t\}^\{\\pi\}\\mid s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{t\}\\right\]\(93\)=Covπ⁡\(ψt​Atπ​\(st,at\)∣st\)\+𝔼π​\[ψt​ψt⊤​Var⁡\(rt\+Vt\+1π∣st,at\)\|st\]\.\\displaystyle=\\operatorname\{Cov\}\_\{\\pi\}\(\\psi\_\{t\}A\_\{t\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\\mid s\_\{t\}\)\+\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\psi\_\{t\}\\psi\_\{t\}^\{\\top\}\\operatorname\{Var\}\(r\_\{t\}\+V\_\{t\+1\}^\{\\pi\}\\mid s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{t\}\\right\]\.Deterministic transitions and rewards make the second term zero\. Stochastic tool responses or rewards can make it nonzero, but tool use alone does not imply stochasticity\. Thus both task types satisfy the same policy\-gradient identities, while the conditional variance and the difficulty of estimating values can differ\. These results do not establish critic convergence, a variance ordering between complete algorithms, or a higher asymptotic reward ceiling\.

## Appendix BRelated Work

#### Synchronous and Asynchronous RL\.

Standard Large Language Model \(LLM\) post\-training collects a complete batch of rollouts under a fixed policy before each update\([Ziegler et al\., 2019](https://arxiv.org/html/2610.40726#bib.bib2);[Stiennon et al\., 2020](https://arxiv.org/html/2610.40726#bib.bib3);[Ouyang et al\., 2022](https://arxiv.org/html/2610.40726#bib.bib4)\)\. Synchronous Reinforcement Learning \(RL\) systems\([Yao et al\., 2023](https://arxiv.org/html/2610.40726#bib.bib5);[Hu et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib6);[Sheng et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib7)\)implement this interleaved generation and training loop, which leaves accelerators idle while waiting for the longest rollout in a batch, especially when reasoning and agentic traces have long\-tailed lengths\. Asynchronous RL systems\([Noukhovitch et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib8);[Fu et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib11);[Piché et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib12)\)therefore decouple generation from learning and train as trajectories arrive, which improves training efficiency\. Recent LLM RL widely adopts Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib16);[DeepSeek\-AI, 2025](https://arxiv.org/html/2610.40726#bib.bib17)\), which estimates advantages by normalizing rewards within a prompt\-level group\. However, this group\-wise sampling does not fit asynchronous training, because an update must wait until every group member finishes and thus recreates a synchronization barrier with larger policy lag\. Recent methods stabilize optimization under this policy lag through off\-policy objective design, variance control, or adaptive trust regions\([Zheng et al\., 2025b](https://arxiv.org/html/2610.40726#bib.bib13);[Huang et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib14);[Yang et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib15)\)\. Single\-rollout Asynchronous Optimization \(SAO\)\([Hou et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib1)\)instead replaces group\-wise sampling with one rollout per prompt, uses token\-level GAE from a learned critic for multi\-turn agentic async RL, and stabilizes training with a scaled pretrained value model\. We reproduce SAO as LiSAO in a lightweight setting without scaled value pretraining, initializing the value model from the policy checkpoint instead\.

#### Group\-Relative and Value\-Based Optimization\.

Critic\-free methods replace a learned value function with sampled baselines\. Beyond GRPO, RLOO uses leave\-one\-out baselines\([Ahmadian et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib18)\), DAPO introduces dynamic sampling and asymmetric clipping\([Yu et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib24)\), and GSPO performs importance weighting and clipping at the sequence level\([Zheng et al\., 2025a](https://arxiv.org/html/2610.40726#bib.bib25)\)\. Their reliance on group statistics reduces critic overhead but limits temporal credit assignment and delays updates until a complete group is available\. Value\-based methods instead learn a critic and estimate advantages with Generalized Advantage Estimation\([Schulman et al\., 2017](https://arxiv.org/html/2610.40726#bib.bib19);[Schulman et al\., 2016](https://arxiv.org/html/2610.40726#bib.bib20)\)\. VC\-PPO and VAPO improve value learning through critic pretraining and decoupled GAE\([Yuan et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib21);[Yue et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib22)\), while SAPO shares an autoregressive backbone between policy and value prediction for single\-rollout optimization\([Liang et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib36)\)\. Unlike methods that remain group\-relative or value\-based throughout training, Li2SAO uses group\-relative actor\-critic bootstrapping, then switches to single\-rollout GAE once the critic is calibrated\.

## Appendix CExperimental Details

### C\.1Experimental Configuration

The experiments are conducted on a Linux server equipped with two Intel Xeon Platinum 8358 CPUs at 2\.60 GHz, 2 TB RAM, and 8 NVIDIA A800\-SXM4\-80GB GPUs\. Our implementation is based on PyTorch\([Paszke et al\., 2019](https://arxiv.org/html/2610.40726#bib.bib29)\)version 2\.11\.0, with CUDA version 12\.9 and Python 3\.12\.3\. Rollouts are generated with SGLang\([Zheng et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib34)\)version 0\.5\.15, and model training uses Megatron\-LM\([Shoeybi et al\., 2019](https://arxiv.org/html/2610.40726#bib.bib35)\)version 0\.16\.0rc0\. We take this stack from the officialslimeDocker image222[https://hub\.docker\.com/r/slimerl/slime](https://hub.docker.com/r/slimerl/slime)and replace the bundledslimecode with our modified implementation\.

### C\.2Datasets and Experimental Settings

Table 2:Task\-specific settings\.
### C\.3Training and Evaluation Details

Both algorithms share the configuration in Table[3](https://arxiv.org/html/2610.40726#A3.T3)\. Algorithm\-specific choices are listed in Table[4](https://arxiv.org/html/2610.40726#A3.T4)\. We do not evaluate during training\. Checkpoints are saved every 100 steps and scored offline with the same reward used in training\. The outcome reward is11when the boxed answer, or otherwise the last integer in the response, is mathematically equivalent to the label, and00otherwise\. Responses that enter the final quarter of the context window additionally receive the overlong penalty of DAPO\([Yu et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib24)\)\. GRPO uses the asymmetric clip of DAPO, while LiSAO uses the token clip and DIS interval of SAO\([Hou et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib1)\)\.

Table 3:Shared training and evaluation configuration\.Table 4:Algorithm\-specific configuration for the GRPO and LiSAO comparison\.

## Appendix DReproduction Details of LiSAO

This section records the choices required to run SAO without its scaled value model\. The comparison in Section[3](https://arxiv.org/html/2610.40726#S3)uses this implementation\.

### D\.1Critic Initialization

SAO relies on a value model produced by scaled value pretraining, and neither that model nor its training pipeline is publicly available\. LiSAO therefore keeps a second copy of the policy parameters and initializes it from the same checkpoint\. The critic output layer is replaced by a scalar value head\. An ablation in SAO shows that freezing multi\-head attention and updating only the MLP blocks regularizes critic optimization, whereas full\-parameter value training yields larger gradients and lower accuracy\([Hou et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib1)\)\. We follow this validated setting: parameters whose names containself\_attentionstay frozen, and training updates the MLP blocks together with the value head\. The actor is held fixed for the first 10 rollout steps while the critic is updated\. After that warmup, each batch applies one policy update andK=2K=2critic updates\. The critic learning rate is5×10−65\\times 10^\{\-6\}, matching the value\-model rate reported for SAO\.

### D\.2Advantage Estimation

The policy advantage uses length\-adaptive GAE withα=1\.5\\alpha=1\.5, so the per\-trajectory coefficient isλ=1−1/\(α​L\)\\lambda=1\-1/\(\\alpha L\)and is clamped to\[0,1\]\[0,1\]\([Yue et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib22);[Hou et al\., 2026](https://arxiv.org/html/2610.40726#bib.bib1)\)\. The discount isγ=1\\gamma=1\. The critic target usesλ=1\\lambda=1\. On multi\-turn ReTool trajectories, GAE is evaluated only on model\-generated tokens, which is the skip\-observation recursion in Eq\.\([6](https://arxiv.org/html/2610.40726#S2.E6)\) and Eq\.\([7](https://arxiv.org/html/2610.40726#S2.E7)\)\. Single\-turn responses consist only of model\-generated tokens, so the same estimator reduces to ordinary GAE\. DIS reads the rollout log probability stored at sampling time and drops tokens whose importance ratio falls outside\[0\.7,6\.0\]\[0\.7,6\.0\]\. The entropy coefficient is00\.

Figure 6:GRPO vs\. LiSAOon single\-turn mathematical reasoning \(left\) and ReTool agentic tool use \(right\)\. Results for each task are reported separately on AIME 2024, 2025, and 2026\. The top row is pass@1 from the shared initialization to step 500\. Each lower row is pass@kkat the labeled checkpoint\.
### D\.3Tasks and In\-flight Updates

Single\-turn training starts from the post\-trained Qwen3\-4B checkpoint and generates text solutions\. ReTool training starts from a tool\-use supervised checkpoint of Qwen3\-4B and generates with a Python interpreter\([Feng et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib23)\)\. In both settings, an unfinished rollout keeps its generated prefix when the trainer publishes a new policy and continues from that prefix\. A single trajectory may therefore contain tokens sampled from more than one policy version\. The token\-level ratio uses the log probability stored for the version that sampled the token\. Held\-out AIME evaluation for the single\-turn models is text\-only\. The ReTool models are evaluated in the same interpreter sandbox used for training\.

## Appendix EFull Comparison of GRPO and LiSAO

Figure[6](https://arxiv.org/html/2610.40726#A4.F6)reports the same comparison separately for each year\. The left three columns are single\-turn mathematical reasoning and the right three are ReTool agentic tool use\. The top row tracks pass@1 from the shared initialization to step 500\. Each lower row plots pass@kkat one checkpoint, withk∈\{1,2,4,8,16\}k\\in\\\{1,2,4,8,16\\\}\.

## Appendix FDetails of Li2SAO

### F\.1Training and Evaluation Configuration

Li2SAO uses the training and evaluation configuration in Table[3](https://arxiv.org/html/2610.40726#A3.T3)\. The clip range, DIS interval, frozen\-attention critic, critic learning rate, andK=2K=2critic updates match LiSAO in Table[4](https://arxiv.org/html/2610.40726#A3.T4)\. Length\-adaptive GAE uses the coefficient in Appendix[D](https://arxiv.org/html/2610.40726#A4)\. Table[5](https://arxiv.org/html/2610.40726#A6.T5)lists the remaining choices\. The trajectory budget stays at 128, so the group sizenb=8n\_\{b\}=8uses 16 prompts per batch andns=1n\_\{s\}=1uses 128\.

Table 5:Li2SAO training configuration\. Shared training choices are in Table[3](https://arxiv.org/html/2610.40726#A3.T3)\. The clip range, DIS interval, critic architecture, critic learning rate, andKKmatch LiSAO in Table[4](https://arxiv.org/html/2610.40726#A3.T4)\.Algorithm 1Algorithmic pseudocode of Li2SAO\.0:Dataset

𝒟\\mathcal\{D\}, policy

πθ\\pi\_\{\\theta\}, critic

VϕV\_\{\\phi\}, budget

BB, sizes

nbn\_\{b\}and

ns=1n\_\{s\}=1, and

\(β,ηe,ηa,M,Tmin,Tmax\)\(\\beta,\\allowbreak\\eta\_\{e\},\\allowbreak\\eta\_\{a\},\\allowbreak M,\\allowbreak T\_\{\\min\},\\allowbreak T\_\{\\max\}\)\.

0:Trained policy

πθ\\pi\_\{\\theta\}\.

Initialize

VϕV\_\{\\phi\}from

πθ\\pi\_\{\\theta\}; set

n←nbn\\leftarrow n\_\{b\},

c←0c\\leftarrow 0,

switched←false\\mathrm\{switched\}\\leftarrow\\mathrm\{false\}
for

k=1,2,…k=1,2,\\ldotsdo

Collect

BBin\-flight trajectories,

nnper prompt; drop over\-lag samples

if

switched=false\\mathrm\{switched\}=\\mathrm\{false\}then

In each group

gg, set

μg,σg\\mu\_\{g\},\\sigma\_\{g\}and

A^i,tactor←\(Ri−μg\)/\(σg\+ε\)\\hat\{A\}\_\{i,t\}^\{\\mathrm\{actor\}\}\\leftarrow\\allowbreak\(R\_\{i\}\-\\mu\_\{g\}\)\\allowbreak/\(\\sigma\_\{g\}\+\\varepsilon\)on model tokens

Update

θ\\thetaby DIS, Eq\.\([8](https://arxiv.org/html/2610.40726#S2.E8)\)

On mixed groups, mask tokens before

did\_\{i\}and set

G~i,t←G^i,t−μg\\widetilde\{G\}\_\{i,t\}\\leftarrow\\allowbreak\\hat\{G\}\_\{i,t\}\-\\mu\_\{g\}
Update

ϕ\\phiby

ℒres\+α​ℒrank\\mathcal\{L\}\_\{\\mathrm\{res\}\}\+\\alpha\\mathcal\{L\}\_\{\\mathrm\{rank\}\}, and update

e¯k,a¯k\\bar\{e\}\_\{k\},\\bar\{a\}\_\{k\}
if

k≥Tmink\\geq T\_\{\\min\},

e¯k\>ηe\\bar\{e\}\_\{k\}\>\\eta\_\{e\}, and

a¯k\>ηa\\bar\{a\}\_\{k\}\>\\eta\_\{a\}then

c←c\+1c\\leftarrow c\+1

else

c←0c\\leftarrow 0

endif

if

c≥Mc\\geq Mor

k≥Tmaxk\\geq T\_\{\\max\}then

Finish the prefetched job; drop leftovers;

n←nsn\\leftarrow n\_\{s\};

switched←true\\mathrm\{switched\}\\leftarrow\\mathrm\{true\}
endif

else

A^t←\\hat\{A\}\_\{t\}\\leftarrowGAE on model tokens; update

θ\\thetaby DIS; fit

VϕV\_\{\\phi\}to

G^t\\hat\{G\}\_\{t\}
endif

endfor

return

πθ\\pi\_\{\\theta\}

### F\.2Algorithmic Pseudocode

Algorithm[1](https://arxiv.org/html/2610.40726#alg1)is the training loop of this procedure onslime\. The critic starts from the same checkpoint as the policy, and the two networks are updated from the first step\. Each step waits for a completed batch of 128 trajectories while the rollout workers continue generating under the current policy\. Trajectories whose policy lag exceeds the shared limit are dropped before the update\. During the bootstrap, those trajectories form groups of eight responses to each prompt\.

### F\.3Execution on the In\-Flight Pipeline

slimebuilds generalized advantage estimates, whitens them across the data parallel group, and stores the corresponding returns before the optimizer step\. Li2SAO is inserted after that computation\. It overwrites actor advantages only when the largest prompt group in the collected batch has more than one response, using the raw outcome rewards of the complete group and writing the score onto tokens that enter the policy loss\. The stored returns are left in place\. Group centering and terminal ranking, when enabled, rewrite the critic targets in a later pass and do not alter the actor advantages\. If a data parallel shard cannot recover every member of a group, that shard keeps the advantage estimate already produced by the trainer\.

Explained variance is reduced across data parallel workers before smoothing\. Each worker sends the count, sum, and sum of squares of its terminal outcomes and values, and the driver computes one variance from those totals\. The driver also owns the readiness counter and the outstanding generation request\. Training of the current batch overlaps collection of the next batch\. When the gate opens, the new group size is retained until that outstanding batch has arrived, and it is installed before the following generation request is sent\. The number of prompts is updated in the same assignment, which keeps 128 trajectories in the batch\. The actor selects its estimator from the group sizes inside the batch, so the delayed installation prevents one optimizer step from seeing both estimators\.

Figure 7:Li2SAO against GRPO and LiSAO\.Results are shown separately for single\-turn mathematical reasoning \(left\) and ReTool agentic tool use \(right\), and for AIME 2024, 2025, and 2026\. The top row is pass@1 from the shared initialization to step 500\. Each lower row is pass@kkat the labeled checkpoint\.Resuming generation first stops the background rollout worker\. That worker is still sampling at the bootstrap group size after the prefetched batch has been taken, so the groups it still holds follow the old sampling rule\. Finished groups that were not consumed, unfinished generations, and the pending sample buffer are discarded, and the worker is then started again\. A subsequent batch therefore cannot combine leftover bootstrap groups with singleton responses\. A new policy is published only after the outstanding generation has been received, so the weight update does not cut through an active rollout\. Tokens already produced keep their prefix and continue under the updated weights, which is the shared in\-flight rule\. Actor updates begin on the first step, so this publication is not delayed by a critic warmup\. The critic runs inside the actor training process with attention frozen, and the value head is trained there together with the MLP blocks\.

## Appendix GAdditional Experimental Analysis

### G\.1Full Comparison

Figure 8:Training accuracyof GRPO, LiSAO, and Li2SAO\. Curves are a two\-sided EMA \(span 10\) of per\-step 0/1 accuracy on the training batches\. The inset shows the first 200 steps\. Gold stars mark the switch, at step 142 on reasoning and step 137 on ReTool\. LiSAO starts at step 10, after critic warmup\.Figure[7](https://arxiv.org/html/2610.40726#A6.F7)places Li2SAO on the same per\-year axes as GRPO and LiSAO, supplementing the averages in Section[5](https://arxiv.org/html/2610.40726#S5)\. Single\-turn mathematical reasoning occupies the left three columns, and ReTool occupies the right three\. The top row follows pass@1 from the shared initialization through step 500\. Each lower row shows pass@kkat one checkpoint, withk∈\{1,2,4,8,16\}k\\in\\\{1,2,4,8,16\\\}\.

#### Single\-turn reasoning\.

Averaged over AIME 2024, 2025, and 2026, Li2SAO leads both baselines at every checkpoint and everykk\. The pass@1 margin is 7\.4 percentage points over GRPO and 17\.3 over LiSAO at step 100, and 10\.4 and 21\.5 points at step 200\. Group\-relative bootstrapping removes the early deficit of LiSAO without value pretraining, and the margin at step 200 shows that the gain remains after the switch to single\-rollout GAE\. Over GRPO the margin is 1\.4 points at step 300 and 3\.4 points at step 500\. Over LiSAO it is 5\.1 points at step 500, and average pass@16 is 86\.7%\. Section[3](https://arxiv.org/html/2610.40726#S3)traces GRPO’s lead on reasoning to a single terminal reward, which already gives a group of solutions its baseline\. The later checkpoints agree with that account: the margin narrows, and Li2SAO still finishes above both baselines\.

#### ReTool\.

Averaged over the three years, Li2SAO leads both baselines at every checkpoint and everykk, with a smaller pass@1 margin\. Over GRPO it is 0\.8 percentage points at step 100, 2\.6 at step 200, and 2\.8 at step 500\. Over LiSAO it is 6\.8 points at step 100 and 1\.4 points at step 500\. Section[3](https://arxiv.org/html/2610.40726#S3)finds that critic noise appears immediately on tool use and lasts longer than on reasoning\. The step\-100 margin over LiSAO is that deficit closed by the bootstrap, and the step\-200 margin shows the same lead after the switch\. Drawing more samples widens the final gap\. At step 500, average pass@16 is 90\.0%, exceeding GRPO by 4\.4 points and LiSAO by 2\.2 points, and it reaches 93\.3% on AIME 2024 and AIME 2026\. Single\-rollout GAE therefore keeps the late advantage of value\-based credit assignment on this task, and the early gap is already closed\.

### G\.2Training Dynamics

Figure[8](https://arxiv.org/html/2610.40726#A7.F8)plots training\-batch accuracy for the runs in Section[5](https://arxiv.org/html/2610.40726#S5)\. Figure[9](https://arxiv.org/html/2610.40726#A7.F9)plots mean response length, with GRPO and LiSAO taken from those runs\. Both series are the two\-sided EMA of span 10 given in the captions\. Actor updates of Li2SAO begin at step 0, together with the critic\. LiSAO enters both plots at step 10, after critic warmup\. Through these training dynamics, we analyze the optimization path at a finer grain than the checkpoint evaluations in Section[5](https://arxiv.org/html/2610.40726#S5), tracing how group\-relative bootstrapping removes the early deficit of Section[3](https://arxiv.org/html/2610.40726#S3)and how the marked switch keeps that repair once the actor moves to single\-rollout GAE\. The length curves then record how many tokens each run spends along training\.

Figure 9:Mean response lengthof GRPO, LiSAO, and Li2SAO\. Curves are a two\-sided EMA \(span 10\) of per\-step mean response length on the training batches\. LiSAO starts at step 10, after critic warmup\.#### Single\-turn reasoning\.

GRPO and Li2SAO stay aligned through the bootstrap, while LiSAO falls behind after about step 50 and is still near 54% at step 100, against about 67% and 68%\. This is the training\-batch form of the early gap in Section[3](https://arxiv.org/html/2610.40726#S3)\. Explained variance is already positive by then, but one terminal reward leaves little structure inside a solution, so the group baseline remains the useful signal and group\-relative advantages keep Li2SAO on the GRPO path\. The switch at step 142 keeps that gain\. Through step 200, Li2SAO stays near 83% with GRPO, while LiSAO is still near 68% and joins the same band only after roughly step 400\. Li2SAO therefore enters single\-rollout GAE from a GRPO\-level policy, the route to the higher final pass@kkin Table[1](https://arxiv.org/html/2610.40726#S5.T1)\.

#### ReTool\.

The deficit of LiSAO appears at once, as in Figure[2](https://arxiv.org/html/2610.40726#S3.F2), because critic noise lasts longer on tool use\. At step 25 it is about 44%, while GRPO and Li2SAO are already near 55%\. Group\-relative bootstrapping closes that immediate deficit\. By the switch at step 137, LiSAO has caught up to about 76%, with the other two near 73%, following the critic recovery in Figure[3](https://arxiv.org/html/2610.40726#S3.F3)\. Once the critic tracks the return, skip\-observation GAE can separate an early plan, a tool call, and a later mistake\. The three curves then stay within about five percentage points and finish near 81%\. The handoff preserves the repaired trajectory, and the remaining updates use the single\-rollout estimator that Section[3](https://arxiv.org/html/2610.40726#S3)links to value\-based credit assignment on tool use\.

#### Response length\.

Figure[9](https://arxiv.org/html/2610.40726#A7.F9)shows how each estimator spends tokens\. On reasoning, LiSAO stays short while the cold critic leaves its updates weak, and it lengthens only as training accuracy recovers\. Li2SAO, using a group\-relative advantage, lengthens earlier than both baselines\. GRPO begins on a similarly short curve and then extends the fastest, becoming the longest trace later in training\. This ordering agrees with Section[3](https://arxiv.org/html/2610.40726#S3)\. A sequence\-level advantage is written on every token and keeps adding length, while one terminal reward gives GAE little structure with which to grow the solution until the critic is useful\. On ReTool the three curves rise together\. LiSAO is short during its early deficit and becomes the longest once the critic tracks the return\. Li2SAO stays with GRPO through the rise and finishes slightly shorter\. Multi\-turn traces keep the tool\-use turns that skip\-observation GAE separates, so the later phase changes credit inside the trajectory while length continues to rise\.

### G\.3Ablations

Table 6:Ablations of Li2SAO\.Each row keeps the group\-relative actor and the readiness switch, and changes one critic target\. The three tables report AIME 2024, 2025, and 2026 separately\. Pass@kkat steps 100, 200, 300, 400, and 500 uses 16 samples per problem at temperature 1\.0 and the unbiased estimator of[Chen et al\. \(2021\)](https://arxiv.org/html/2610.40726#bib.bib27)\.AIME 2024

AIME 2025

AIME 2026

Table[6](https://arxiv.org/html/2610.40726#A7.T6)isolates the two components of the critic objective while keeping the group relative actor, the readiness switch, the trajectory budget, and the in flight pipeline fixed\. Removing the group relative phase recovers LiSAO, while suppressing the switch recovers GRPO, both of which are included in Table[1](https://arxiv.org/html/2610.40726#S5.T1)\. We therefore focus on the design choices that determine whether the critic can reliably inherit the policy update after bootstrapping\.

#### Critic target\.

w/o residual targetfitsVϕV\_\{\\phi\}to the absolute returnG^i,t\\hat\{G\}\_\{i,t\}during bootstrapping\. Because the divergence mask and ranking loss are defined on mixed groups with centered targets, both are disabled in this variant\.w/o ranking lossretains the centered target and divergence mask but removes the pairwise ordering objective\.

#### Residual supervision enables a reliable handoff\.

The complete method consistently outperforms the absolute return variant across benchmark years and interaction regimes\. The gap emerges during bootstrapping and generally persists after the actor switches to critic based advantages\. This pattern indicates that absolute return regression spends critic capacity on prompt dependent difficulty and reward offsets that are irrelevant to the within prompt comparisons used by the initial actor\. Centering the target removes this nuisance variation and the divergence mask excludes groups that provide no comparative signal\. The resulting critic is better aligned with the group relative policy update, which makes its predictions more dependable when it takes over advantage estimation\.

#### Ranking supervision improves discrimination and stability\.

Removing the ranking objective produces a milder but systematic degradation\. The variant can track the full method early in training, yet it more often plateaus or loses ground after the handoff\. Residual regression alone constrains prediction magnitudes but does not directly enforce the ordering between successful and unsuccessful responses\. The ranking objective supplies this missing structure and encourages the critic to preserve the preference relation exposed by mixed outcome groups\. Its benefit across both direct reasoning and tool interaction suggests that the objective improves critic transfer rather than exploiting properties of a particular response format\.

#### The critic objectives are complementary\.

Taken together, the ablations show that the gains do not arise from adding an arbitrary value function to a group based learner\. The residual target aligns the critic with the comparative signal used during bootstrapping, while the ranking objective sharpens that signal into a robust ordering\. Their combination yields the most consistent learning trajectory across tasks and benchmark years, supporting the central premise that a structured critic target is necessary for an effective transition from group relative to critic guided optimization\.

### G\.4Training Efficiency

Table[7](https://arxiv.org/html/2610.40726#A7.T7)reports the wall\-clock time of 500 training steps on 8 A800 GPUs in Appendix[C\.1](https://arxiv.org/html/2610.40726#A3.SS1)\. All rows use the in\-flight pipeline of Section[2\.1](https://arxiv.org/html/2610.40726#S2.SS1), and both device splits draw on this same pool of 8 GPUs\. GRPO keeps eight responses per prompt at every step, LiSAO draws one response per prompt, and Li2SAO starts from the GRPO group size before switching once to one response per prompt\.

Table 7:Wall\-clock timeof 500 training steps for single\-turn mathematical reasoning and ReTool agentic tool use\. Train \+ Rollout is the GPU split between the trainer and the rollout workers\. In LiSAO and Li2SAO, the actor and the critic share the trainer GPUs\. Prompts×\\timesSamples is the number of prompts in each step times the responses drawn for each prompt\.#### Extra rollout GPUs shorten GRPO\.

The two GRPO rows draw the same 16×\\times8 responses, so the difference is the device split\. Moving 2 GPUs from the trainer to the rollout workers reduces reasoning from 51\.5 hours to 36\.4 hours \(29\.3%\) and ReTool from 48\.7 hours to 39\.6 hours \(18\.7%\)\. A group\-relative step forms its advantage only after eight responses to the same prompt are available\. With long\-tailed response lengths, group completion is limited by rollout throughput, and the 2 additional rollout GPUs increase that throughput\.

#### LiSAO and Li2SAO have comparable training costs\.

With 4 trainer GPUs and 4 rollout GPUs, LiSAO completes training in 31\.8 hours on reasoning and 35\.5 hours on ReTool, whereas Li2SAO requires 28\.9 and 33\.1 hours, respectively\. Although Li2SAO begins with group\-relative updates at 16×\\times8, its initial group\-based phase does not increase the total training time relative to the 128×\\times1 configuration used by LiSAO from the outset\. This result follows from the evolution of response length shown in Figure[9](https://arxiv.org/html/2610.40726#A7.F9)\. The group\-relative bootstrapping phase occurs while the mean response length remains low\. By the time responses become longer, Li2SAO has switched to 128×\\times1, so each prompt requires only one of the expensive generations\.

#### Critic training preserves the efficiency advantage over GRPO\.

LiSAO and Li2SAO train the critic jointly with the actor on the same 4 trainer GPUs, whereas GRPO updates only the actor\. Under the 4 \+ 4 split, LiSAO reduces the training time relative to GRPO by 19\.7 hours on reasoning and 13\.2 hours on ReTool\. The corresponding reductions for Li2SAO are 22\.6 and 15\.6 hours, which amount to 43\.9% and 32\.0%, respectively\. This advantage persists when GRPO is allocated the more rollout\-intensive 2 \+ 6 split\. Even under this configuration, LiSAO remains faster by 4\.6 hours on reasoning and 4\.1 hours on ReTool, while Li2SAO remains faster by 7\.5 and 6\.5 hours\. All methods collect 128 trajectories per step, but GRPO can form its group\-relative advantage only after all eight responses to a given prompt become available\. As Figure[9](https://arxiv.org/html/2610.40726#A7.F9)indicates, the resulting group\-completion delay increases as responses become longer\. By contrast, each completed trajectory in the single\-response setting can enter the training batch immediately, which is suited to asynchronous execution\. The scheduling gains therefore outweigh the additional cost of critic optimization\.

### G\.5Staleness Dynamics

Figure 10:Mean policy lagof GRPO, LiSAO, and Li2SAO\. Curves are a two\-sided EMA \(span 10\) of the per\-step mean policy lag\. GRPO is the training\-batch mean\. LiSAO and Li2SAO are the rollout mean, one policy version lower on runs that log both\. LiSAO starts at step 10, after critic warmup\.#### Policy lag\.

Figure[10](https://arxiv.org/html/2610.40726#A7.F10)shows how many policy versions the samples of each step trail the current policy\. Across both tasks, GRPO exhibits the largest lag, LiSAO remains substantially fresher, and Li2SAO moves from the group\-based regime toward LiSAO or below it after the switch\. A group\-relative update cannot use an early response until the other seven responses to the same prompt finish; while it waits, the trainer continues to advance, turning response\-length variation within a group into policy\-version lag\. Single\-response collection removes this slowest\-member barrier, so a completed trajectory can enter the next available batch without waiting for matched responses\. The sharp within\-run decline of Li2SAO after switching supports this mechanism and shows that single\-response sampling, rather than the critic alone, is what reduces policy staleness\.

Figure 11:Off\-policy token ratioof GRPO, LiSAO, and Li2SAO\. Curves are a two\-sided EMA \(span 10\) of the fraction of response tokens generated more than one policy version behind the trainer\. A one\-step lag is counted as on\-policy\. Every curve is the training\-batch ratio\. LiSAO starts at step 10, after critic warmup\.
#### Off\-policy tokens\.

Figure[11](https://arxiv.org/html/2610.40726#A7.F11)applies a token\-level cut to the same staleness\. A response token is off\-policy only when the policy that generated it is more than one version behind the trainer\. All methods are highly off\-policy early in training, but GRPO remains near saturation whereas LiSAO becomes markedly less stale\. Li2SAO again behaves like a group\-based method during bootstrapping and drops sharply after switching, reaching a level comparable to or lower than LiSAO\. This pattern confirms that removing the group\-completion barrier not only shortens waiting but also reduces the fraction of stale tokens actually used for learning\. Together, the two figures show that the single\-response phases of LiSAO and Li2SAO substantially mitigate off\-policy data staleness relative to GRPO\. This conclusion is specifically about policy\-version freshness under the stated threshold; it does not by itself imply that importance weighting eliminates all off\-policy estimation bias\.

## Appendix HCoding Agentic RL

To assess the effectiveness of Li2SAO on more complex agentic tasks and with larger models, we further evaluate the optimization methods on long\-horizon software engineering tasks\.

### H\.1Infrastructure and Agent Environment

Our implementation uses OpenHands\([Wang et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib33)\)999[https://github\.com/OpenHands/OpenHands](https://github.com/OpenHands/OpenHands)as the agent scaffold and a modified version ofslime101010[https://github\.com/THUDM/slime](https://github.com/THUDM/slime)as the asynchronous RL backend\. Each rollout runs in an isolated Docker environment initialized from the instance\-specific repository and base commit\. OpenHands interacts with the repository through its shell and file\-editing tools, while an OpenAI\-compatible adapter routes every model call to the SGLang rollout engines\. Each trajectory is limited to 300 interaction turns and36003600seconds, with a128128K context window and OpenHands context compression enabled so that long multi\-turn traces remain within this budget\. The adapter records the exact generated token IDs, loss masks, and behavior log probabilities for every assistant turn, avoiding a separate re\-tokenization step when the multi\-turn trajectory is converted into training samples\. This preserves alignment between sampled actions and the tokens used for policy optimization across turns\.

The principal systems modification is*agent\-step\-level in\-flight weight synchronization*\. In the originalslime, a weight update aborts an in\-flight generation request and truncates it before any further tokens are sampled under the new weights\. Our modifiedslimecan instead refresh the rollout\-engine weights at agent\-step boundaries while a long\-running OpenHands trajectory is still in flight\. Thus, later agent steps need not wait for the entire trajectory to finish before using a newer policy version\. The behavior log probabilities associated with each generated segment are retained for importance\-ratio computation\.

All coding experiments use eight nodes with eight NVIDIA A800 GPUs per node, for a total of 64 GPUs\. By default, four nodes are allocated to training and the other four to rollout\. The same accelerator allocation and the same asynchronous rollout infrastructure are used for every RL method\. Table[8](https://arxiv.org/html/2610.40726#A8.T8)summarizes the shared configuration\. To scale environment execution, we implement a distributed rollout manager that deploys one sandbox worker on each of the eight nodes\. The manager dispatches OpenHands trajectories with per node concurrency control and backpressure, while each worker launches its sandbox and reward evaluation containers through the local Docker daemon\. This design avoids a centralized container bottleneck and enables concurrent environment execution across all eight nodes\.

Table 8:Shared configuration for coding agentic RL\.
### H\.2Training Data and Reward

We evaluate on SWE\-bench Verified\([Jimenez et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib30)\)111111[https://github\.com/SWE\-bench/SWE\-bench](https://github.com/SWE-bench/SWE-bench)for all coding experiments\. Training uses a mixture of SWE\-Gym\([Pan et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib31)\)121212[https://github\.com/SWE\-Gym/SWE\-Gym](https://github.com/SWE-Gym/SWE-Gym)and SWE\-smith\([Yang et al\., 2025b](https://arxiv.org/html/2610.40726#bib.bib32)\)131313[https://github\.com/SWE\-bench/SWE\-smith](https://github.com/SWE-bench/SWE-smith)\. Each training instance provides a problem statement, a repository snapshot, a base commit, an environment, and tests that determine whether the issue has been resolved\. The agent produces a repository patch through multi\-turn interaction with OpenHands\.

We use the sparse binary reward implemented by the coding pipeline\. Scoring is performed in a fresh container: the repository is reset to the prescribed base commit, the model patch is applied, protected test files are restored, and the instance tests are then executed\. The reward is11only if all fail\-to\-pass tests succeed and all pass\-to\-pass tests remain successful, and00otherwise\. Rollouts that fail before producing a valid interaction trace, for example because the sandbox cannot be created or the model endpoint is unreachable, are marked as aborted and excluded from optimization rather than treated as negative solutions\.

### H\.3Models and Baselines

We train Qwen3\-30B\-A3B\-Instruct\([Yang et al\., 2025a](https://arxiv.org/html/2610.40726#bib.bib26)\)141414[https://huggingface\.co/Qwen/Qwen3\-30B\-A3B\-Instruct\-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507)and Qwen3\-Coder\-30B\-A3B\([Yang et al\., 2025a](https://arxiv.org/html/2610.40726#bib.bib26)\)151515[https://huggingface\.co/Qwen/Qwen3\-Coder\-30B\-A3B\-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct)\. For Qwen3\-30B\-A3B\-Instruct, we explicitly disable its thinking mode during both training and evaluation\. Qwen3\-Coder\-30B\-A3B does not expose a separate thinking mode, so we use its native non\-thinking configuration\. This keeps the interaction protocol comparable and prevents an additional hidden\-reasoning budget from confounding the algorithmic comparison\.

For each model, we compare six settings\.Baseevaluates the unmodified instruction checkpoint without RL\. GRPO uses prompt\-level group\-relative advantages\([Shao et al\., 2024](https://arxiv.org/html/2610.40726#bib.bib16)\)\. DAPO adds dynamic sampling and asymmetric clipping\([Yu et al\., 2025](https://arxiv.org/html/2610.40726#bib.bib24)\)\. GSPO applies sequence\-level importance weighting and clipping\([Zheng et al\., 2025a](https://arxiv.org/html/2610.40726#bib.bib25)\)\. LiSAO uses a learned critic and single\-rollout GAE throughout training\. Li2SAO first performs group\-relative actor\-critic bootstrapping and then switches to single\-rollout GAE according to the procedure in Section[4](https://arxiv.org/html/2610.40726#S4), with the switch window set to\(Tmin,Tmax\)=\(30,60\)\(T\_\{\\min\},T\_\{\\max\}\)=\(30,60\)\. All RL methods follow the shared settings in Table[8](https://arxiv.org/html/2610.40726#A8.T8)and are trained for 200 steps with a global batch of 256 trajectories: group\-based methods use32×832\\times 8, LiSAO uses256×1256\\times 1, and Li2SAO starts from32×832\\times 8and switches to256×1256\\times 1\. Except for their algorithm\-specific advantage estimators, clipping rules, and critic updates, all methods use the same data, agent environment, reward, and total accelerator allocation\. The mathematical experiments are designed to compare algorithms whose advantage estimators differ in kind, so they omit DAPO and GSPO, which remain GRPO variants and still rely on group\-relative advantages\. We include them here because SWE\-bench groups often receive all\-11or all\-00rewards, so DAPO’s dynamic sampling can preserve a nonzero advantage at several times the rollout cost, and because GSPO’s sequence\-level weighting targets the mixture\-of\-experts models trained in this section\.

### H\.4Evaluation Protocol and Results

We evaluate every checkpoint on SWE\-bench Verified with the same OpenHands scaffold and Docker execution protocol used during training\. For each instance, the model must inspect the repository, implement a patch, and validate the change through tool interaction\. We report the official resolved rate, namely the percentage of the 500 verified instances whose fail\-to\-pass tests are fixed without regressions in pass\-to\-pass tests\. Table[9](https://arxiv.org/html/2610.40726#A8.T9)provides the comparison, including the two critic\-target ablations of Li2SAO from Appendix[G\.3](https://arxiv.org/html/2610.40726#A7.SS3)\.

Table 9:SWE\-bench Verified resolved rate\(%\) after coding agentic RL on SWE\-Gym\.#### Li2SAO reaches the highest resolved rate on both models\.

GRPO, DAPO, and GSPO improve both checkpoints and stay close to one another\. DAPO remains comparable to GRPO, and GSPO is the strongest of the three while staying in the same band, so sequence\-level advantages raise the policy and then share a ceiling on these long tool traces\. LiSAO, trained from a cold critic without value pretraining, already joins that band and finishes ahead of GRPO\. Li2SAO is best on both the general instruction model and the code model\. Its margin over LiSAO comes from group\-relative actor\-critic bootstrapping, and its margin over the group\-relative methods comes from the switch to single\-rollout GAE\. The lead narrows on the code checkpoint, while the ranking stays the same\.

#### Both critic targets are required for this handoff\.

Removing the residual target hurts more than removing the ranking loss on both models\. Centering keeps the critic on which patch succeeds, and ranking keeps successful traces above failed ones\. On the instruction model, both ablations still finish above LiSAO and every group\-relative method\. On the code model, both ablations fall back to the band of GSPO and LiSAO, and only the full objective remains ahead\. Group\-relative bootstrapping followed by one switch to single\-rollout GAE raises the resolved rate on long\-horizon software engineering without value pretraining\.