Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RL

# Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RL
Source: [https://arxiv.org/html/2610.11128](https://arxiv.org/html/2610.11128)
Hanyu Wang††thanks:Work done during an internship at Honda Research Institute USA\.Nakul AgarwalAffiliation:Honda Research Institute USAHossein Nourkhiz MahjoubAffiliation:Honda Research Institute USAEhsan Moradi PariAffiliation:Honda Research Institute USAMakoto FukushimaAffiliation:Honda Research Institute JapanJinghui ChenAffiliation:The Pennsylvania State UniversityVaishnav TadiparthiAffiliation:Honda Research Institute USA

###### Abstract

A verified reference solution provides a correct trajectory for training a reasoning model\. Alternatively, a prefix of the reference can guide the model in generating a trajectory of its own\. How much reference guidance should we provide? We study this question through prefix continuation, where the model continues from a reference prefix and keeps the resulting trajectory if it passes verification, falling back to the reference otherwise\. Since both procedures produce correct trajectories, we compare their distributions with the ideal distribution, the model’s own distribution conditioned on successful verification\. For one continuation, we derive the KL divergence in closed form, which, up to a bounded term, decreases with the product of the probability of generating a different correct trajectory and the reference surprisal, the negative log probability of the reference suffix given the prefix\. Since a longer prefix tends to raise the former but lowers the latter, continuation success alone does not determine the preferred amount of guidance\. From this analysis, we learn a prefix selector shared across training questions from continuation outcomes, without estimating success probabilities or additional generation\. The resultingAdaptiveReferenceGuidance \(ARG\) constructs correct trajectories within a fixed generation budget, and we apply it to all\-failure groups in Group Relative Policy Optimization \(GRPO\)\. Experiments on Qwen3\-4B and Qwen3\-8B across five mathematical reasoning benchmarks show that ARG achieves the highest aggregate pass@12 among the evaluated methods with competitive average sampled accuracy\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\)\([Lambert et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib19)\)trains Large Language Models \(LLMs\) using feedback on their own generated responses\([Guo et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib12);[Team et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib46);[Jaech et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib17)\)\. On difficult questions, however, the model may rarely generate a response that reaches the correct answer, leaving little feedback to learn from\. When such a question comes with a reference solution written by humans or a stronger model, a straightforward approach is to train on it directly\([Yan et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib49);[Fu et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib11);[Ma et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib31)\)\. However, the reference may differ substantially from the trajectories the model would produce itself, which can make it a poor learning target\([Setlur et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib41);[Qu et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib36)\)\. Since the model learns more easily from familiar responses\([Ren et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib38)\), another line of work uses the reference as guidance so that the model generates correct trajectories by itself\. Some methods transform the reference into hints\([Zhang et al\., 2026c](https://arxiv.org/html/2610.11128#bib.bib56);[Liao et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib25)\), which requires additional LLM generation and may introduce bias or errors\. Other methods, in contrast, let the model generate a*continuation*from a*reference prefix*, which is an initial segment of the reference\. This approach uses the reference without modification and controls how much guidance the model receives through how much of the reference is revealed in the prefix\.

However, when generating a trajectory following a prefix, deciding how much to reveal involves a trade\-off\. Specifically, using a longer reference prefix, which provides more guidance, makes the continuation more likely to be correct\. Yet the resulting trajectory contains more content the model would not generate itself, making learning harder\. Using a shorter prefix keeps the trajectory closer to the model’s own generation, but increases the chance of failure\. At the two extremes, revealing the entire reference reduces to training on the reference directly, while revealing nothing reduces to standard RLVR, which provides no learning signal on questions the model cannot yet solve\. Balancing the two is therefore essential\. The prefix should reveal just enough for the model to succeed, and leave the rest for the model to generate on its own\.

To analyze this trade\-off, we first ask which trajectories are ideal to learn from\. Ideally, these trajectories should be correct and generated entirely by the model\. We therefore define the distribution of the model’s own correct samples as the*ideal distribution*\. We show that the Kullback\-Leibler \(KL\) divergence from the distribution of generated trajectories used for training to this ideal distribution decreases asdt​Std\_\{t\}S\_\{t\}increases, up to a small bounded term\. Heredtd\_\{t\}is the probability that the model follows the reference’s prefix to generate a successful trajectory, andStS\_\{t\}is the negative log probability of the remaining reference given the prefix\. Intuitively,dtd\_\{t\}reflects how much the prefix helps the model succeed, whileStS\_\{t\}indicates how unfamiliar the reference suffix is\. Moreover,dt​Std\_\{t\}S\_\{t\}captures both sides of the trade\-off\. A longer prefix tends to increasedtd\_\{t\}, since more guidance makes success more likely, but it decreasesStS\_\{t\}, since less of the reference is left\.

As a result, a good prefix should maximizedt​Std\_\{t\}S\_\{t\}\. However,dtd\_\{t\}is unknown, and estimating it for every prefix of every question is impractical\. We avoid this estimation by learning a*prefix selector*, a distribution over a few guidance levels shared across questions\. Since the selector only needs to work well on average, the outcome of each continuation already provides the feedback it needs, without any additional generation\.

These analyses lead to*Adaptive Reference Guidance*\(ARG\)\. Given a question and its reference, ARG samples guidance levels from the prefix selector, generates continuations from the resulting prefixes within a fixed budget, and falls back to the reference when no continuation succeeds\. The same continuations also update the prefix selector, which therefore requires no additional generation\. To examine how the accepted trajectories affect policy learning, we apply ARG to Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib42)\)for groups in which every sampled response fails, resulting in no learning signals\.

Empirically, on Qwen3\-4B and Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib50)\)across five mathematical reasoning benchmarks, ARG achieves the highest aggregatepass​@​12\\mathrm\{pass@12\}of67\.33%67\.33\\%and70\.00%70\.00\\%, respectively\. Moreover, its average sampled accuracy remains competitive, with a small gap to the strongest baseline\. Our contributions are threefold\.

- •We analyze reference\-prefix continuation through the KL divergence to the model’s own correct trajectories, revealing a trade\-off between reference guidance and surprisal\.
- •We derive an unbiased update of the prefix selector from the same continuations, adapting the guidance level across questions and during training without additional generation\.
- •We develop ARG from this analysis and apply it to all\-failure groups in GRPO, achieving the highest aggregatepass​@​12\\mathrm\{pass@12\}across five math benchmarks on two model scales\.

## 2Preliminaries

Reasoning with verifiable rewards\.Letπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)be a language model that generates a trajectoryyyfor a questionxxdrawn from the training distribution𝒟\\mathcal\{D\}\. The trajectory includes the reasoning, the final answer, and the termination token\. A verifier assigns a binary rewardRacc​\(x,y\)∈\{0,1\}R\_\{\\mathrm\{acc\}\}\(x,y\)\\in\\\{0,1\\\}, and we callyy*correct*whenRacc​\(x,y\)=1R\_\{\\mathrm\{acc\}\}\(x,y\)=1\. LetCxC\_\{x\}be the set of correct trajectories andpθ​\(x\)=πθ​\(Cx∣x\)p\_\{\\theta\}\(x\)=\\pi\_\{\\theta\}\(C\_\{x\}\\mid x\)be the success probability\. During training,θold\\theta\_\{\\mathrm\{old\}\}denotes the parameters of the current model, which generates the samples at each training step\. We use natural logarithms throughout\.

Reference\-prefix continuation with fallback\.Suppose that the question comes with a correct referencer∈Cxr\\in C\_\{x\}\. For a prefix length0≤t<\|r\|0\\leq t<\|r\|, we writer=ht⊕atr=h\_\{t\}\\oplus a\_\{t\}, where the reference prefixhth\_\{t\}contains the firsttttokens ofrr, the reference suffixata\_\{t\}contains the remaining tokens, and⊕\\oplusdenotes token concatenation\. The model samples a continuationZ∼πθ\(⋅∣x,ht\)Z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,h\_\{t\}\), which forms the*candidate trajectory*Y~t=ht⊕Z\\widetilde\{Y\}\_\{t\}=h\_\{t\}\\oplus Z\. Att=0t=0, the prefix is empty and the model generates under the original question alone\. We keep the candidate trajectory if it is correct and fall back to the reference otherwise, which gives the accepted trajectory

Y=\{Y~t,Y~t∈Cx,r,Y~t∉Cx,Y∼Qt\.Y=\\begin\{cases\}\\widetilde\{Y\}\_\{t\},&\\widetilde\{Y\}\_\{t\}\\in C\_\{x\},\\\\ r,&\\widetilde\{Y\}\_\{t\}\\notin C\_\{x\},\\end\{cases\}\\qquad Y\\sim Q\_\{t\}\.\(1\)Thus,QtQ\_\{t\}assigns all probability to correct trajectories for every prefix length\. In contrast, training on the reference corresponds to the distributionδr\\delta\_\{r\}, which assigns all probability torr\.

Ideal distribution\.As discussed in Section[1](https://arxiv.org/html/2610.11128#S1), we take the model’s own correct trajectories as the ideal distribution\. Formally, forpθ​\(x\)\>0p\_\{\\theta\}\(x\)\>0, sampling fromπθ\(⋅∣x\)\\pi\_\{\\theta\}\(\\cdot\\mid x\)and keeping only correct trajectories gives

νθ​\(y∣x\)=πθ\(y∣x\)𝟏\{y∈Cx\}pθ​\(x\)\.\\nu\_\{\\theta\}\(y\\mid x\)=\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\\mathbf\{1\}\\\{y\\in C\_\{x\}\\\}\}\{p\_\{\\theta\}\(x\)\}\.\(2\)This distribution also has a direct connection to likelihood learning\([Phan et al\., 2023](https://arxiv.org/html/2610.11128#bib.bib35);[Singh et al\., 2023](https://arxiv.org/html/2610.11128#bib.bib43)\)\. Specifically, for a finite trajectory space and a differentiable policy with positive probabilities,

𝔼Y∼νθ\(⋅∣x\)\[∇θlogπθ\(Y∣x\)\]=∇θlogpθ\(x\)\.\\mathbb\{E\}\_\{Y\\sim\\nu\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\\mid x\)\\right\]=\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(x\)\.\(3\)In other words, learning the likelihood of trajectories drawn fromνθ\\nu\_\{\\theta\}increases the success probability in expectation at the current model\. Finally, we measure how far a distributionQQof correct trajectories is from the ideal distribution by the Kullback\-Leibler \(KL\) divergence

DKL\(Q∥νθ\)=𝔼Y∼Q\[logQ⁡\(Y\)νθ​\(Y∣x\)\]\.D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\_\{\\theta\}\)=\\mathbb\{E\}\_\{Y\\sim Q\}\\left\[\\log\\frac\{Q\(Y\)\}\{\\nu\_\{\\theta\}\(Y\\mid x\)\}\\right\]\.Appendix[A\.2](https://arxiv.org/html/2610.11128#A1.SS2)derives Equation \([3](https://arxiv.org/html/2610.11128#S2.E3)\) and explains this direction of KL\.

## 3Analyzing and Selecting Reference Prefixes

In this section, we analyze how the prefix length affects the accepted trajectory and turn the analysis into a practical way of choosing prefixes\. Section[3\.1](https://arxiv.org/html/2610.11128#S3.SS1)derives the distance from the accepted trajectory to the ideal distribution, which identifies the preferred prefix of a question\. Section[3\.2](https://arxiv.org/html/2610.11128#S3.SS2)explains why this prefix is difficult to find during training, and Section[3\.3](https://arxiv.org/html/2610.11128#S3.SS3)addresses both challenges by learning a prefix selector that performs well on average across questions\.

### 3\.1Distance to the Ideal Distribution

Fix a questionxxwith referencerrand the current modelπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}, and suppress the conditioning onxxand the parameters by writingπ\\pi,CC,p=π⁡\(C\)p=\\pi\(C\), andν\\nu\. We assume a finite trajectory space with0<π⁡\(r\)<10<\\pi\(r\)<1\. For a prefix lengthtt, we define the*successful replacement probability*dtd\_\{t\}and the*reference surprisal*StS\_\{t\}as

dt=Pr⁡\(Y~t∈C,Y~t≠r\),St=−log⁡π⁡\(at∣ht\)\.d\_\{t\}=\\Pr\\bigl\(\\widetilde\{Y\}\_\{t\}\\in C,\\ \\widetilde\{Y\}\_\{t\}\\neq r\\bigr\),\\qquad S\_\{t\}=\-\\log\\pi\(a\_\{t\}\\mid h\_\{t\}\)\.\(4\)Heredtd\_\{t\}excludes an exact copy of the reference, since the accepted trajectory is then stillrr\. In other words,dtd\_\{t\}is the probability that the model replaces the reference suffix with a different correct continuation, andStS\_\{t\}measures how unfamiliar the reference suffix is to the model after the prefix\.

###### Theorem 1\(Distance to the ideal distribution\)\.

Under the setup above, for every prefix lengthtt,

DKL\(Qt∥ν\)=log⁡pπ⁡\(r\)⏟DKL\(δr∥ν\)−dtSt\+\(1−dt\)log\(1−dt\)\.D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)=\\underbrace\{\\log\\frac\{p\}\{\\pi\(r\)\}\}\_\{D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\}\-d\_\{t\}S\_\{t\}\+\(1\-d\_\{t\}\)\\log\(1\-d\_\{t\}\)\.\(5\)Moreover, everyt∗∈arg⁡maxt​dt​Stt^\{\*\}\\in\\arg\\max\_\{t\}d\_\{t\}S\_\{t\}satisfiesDKL\(Qt∗∥ν\)≤mintDKL\(Qt∥ν\)\+1/eD\_\{\\mathrm\{KL\}\}\(Q\_\{t^\{\*\}\}\\\|\\nu\)\\leq\\min\_\{t\}D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)\+1/e\.

Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)expresses the distance betweenQtQ\_\{t\}and the ideal distributionν\\nuin closed form with three terms\. The first term equals that of training on the reference, which does not depend on the prefix\. The third term is bounded within\[−1/e,0\]\[\-1/e,0\]\. Therefore, the prefix affects the distance mainly through the second term,dt​Std\_\{t\}S\_\{t\}\. Intuitively, this product formalizes the trade\-off in choosing the prefix length\. A longer prefix never increasesStS\_\{t\}, since it leaves less of the reference for the continuation to replace\. In contrast, a longer prefix can increasedtd\_\{t\}by making the continuation easier to succeed\. Consequently, both a nearly complete prefix and an empty prefix on a hard question makedt​Std\_\{t\}S\_\{t\}close to zero and leave the distance close toDKL\(δr∥ν\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\), indicating learning from the reference alone\. Moreover, a prefix with a higher success probability of the continuation need not give a smaller distance, as illustrated in Appendix[A\.5](https://arxiv.org/html/2610.11128#A1.SS5)\. Instead, maximizingdt​Std\_\{t\}S\_\{t\}attains the smallest distance up to1/e1/eby the second statement in the theorem, and we callht∗h\_\{t^\{\*\}\}the*preferred prefix*of the question under the current model\. Appendix[A](https://arxiv.org/html/2610.11128#A1)gives the proof\.

### 3\.2Challenges in Finding the Preferred Prefix

Although Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)identifies the preferred prefix, finding it during training is difficult\. The reference surprisalStS\_\{t\}is available for every prefix length from a single forward pass of the model on the reference\. However, the successful replacement probabilitydtd\_\{t\}is unknown, which leads to the following two challenges\.

Challenge 1: Variation across questions and training steps\.Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)concerns a fixed question and a fixed model, anddtd\_\{t\}depends on both\. Therefore, the estimates ofdtd\_\{t\}for one question do not apply to another question\. Furthermore, sincedtd\_\{t\}changes as the model is updated, the estimates obtained at one training step may become outdated at the next\. As a result, the search fort∗t^\{\*\}would have to be repeated for every question and every model update\.

Challenge 2: Limited sampling budget\.Moreover, each such search is expensive\. Identifyingt∗t^\{\*\}comparesdt​Std\_\{t\}S\_\{t\}across prefix lengths and therefore requires an accurate estimate ofdtd\_\{t\}at every prefix length\. Sincedtd\_\{t\}is often small on hard questions, such an estimate requires many continuations even for a single prefix length, and this cost multiplies across prefix lengths\.

### 3\.3Learning a Prefix Selector across Questions

Both challenges arise from seeking the preferred prefix of each question at each training step\. Instead, we learn a way of choosing prefixes that performs well on average over the questions that receive guidance, which addresses both challenges\.

Sharing across questions and training steps\.Averaging over questions requires describing prefixes in the same way for every question, which the prefix length does not achieve for references of different lengths\. Since Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)measures the replaced suffix by its surprisal rather than its length, we describe a prefix through the reference surprisal\. Specifically, for a questionxxwith referencerxr\_\{x\}, letS0,x=−log⁡πθold​\(rx∣x\)S\_\{0,x\}=\-\\log\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(r\_\{x\}\\mid x\)be the surprisal of the entire reference\. A*guidance level*α\\alphain a set𝒜⊂\(0,1\]\\mathcal\{A\}\\subset\(0,1\]selects a prefix lengthtx​\(α\)t\_\{x\}\(\\alpha\)whose reference surprisal is approximatelyα​S0,x\\alpha S\_\{0,x\}\. Thus,α\\alphais the fraction of the reference surprisal left to the continuation, whereα=1\\alpha=1gives the empty prefix\. As with prefix lengths, the levels cannot all be tried within the sampling budget\. Therefore,𝒜\\mathcal\{A\}contains only a few levels, and Section[4\.1](https://arxiv.org/html/2610.11128#S4.SS1)specifies the mapping from levels to prefix lengths\.

We then learn a*prefix selector*ρϕ\\rho\_\{\\phi\}, which is a distribution over𝒜\\mathcal\{A\}with parametersϕ\\phishared across questions\. Let𝒟guide\\mathcal\{D\}\_\{\\mathrm\{guide\}\}denote the distribution of questions that receive guidance at the current training step, which Section[4\.3](https://arxiv.org/html/2610.11128#S4.SS3)specifies for our application\. ForX∼𝒟guideX\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\}andα∼ρϕ\\alpha\\sim\\rho\_\{\\phi\}, we writedX,αd\_\{X,\\alpha\},SX,αS\_\{X,\\alpha\}, andQX,αQ\_\{X,\\alpha\}fordtd\_\{t\},StS\_\{t\}, andQtQ\_\{t\}at the prefix lengthtX​\(α\)t\_\{X\}\(\\alpha\), andνX\\nu\_\{X\}for the ideal distribution ofXX\. The selector maximizes

J⁡\(ϕ\)=𝔼X∼𝒟guide,α∼ρϕ​\[dX,α​SX,αS0,X\],J\(\\phi\)=\\mathbb\{E\}\_\{X\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\},\\,\\alpha\\sim\\rho\_\{\\phi\}\}\\left\[\\frac\{d\_\{X,\\alpha\}S\_\{X,\\alpha\}\}\{S\_\{0,X\}\}\\right\],\(6\)which averagesdt​Std\_\{t\}S\_\{t\}over the questions and the sampled levels in units of the surprisal of the entire reference\. This normalization keeps questions with long or unfamiliar references from dominating the average, and it does not change which prefix maximizesdt​Std\_\{t\}S\_\{t\}within any question\. Since the selector is shared across questions and updated at every training step, continuations on all questions and at all steps inform the same choice, while the selected prefix still adapts to each question through its reference surprisal\. This addresses Challenge 1, and Appendix[B\.3](https://arxiv.org/html/2610.11128#A2.SS3)bounds the sharing cost\.

Learning without estimatingdtd\_\{t\}\.The average objective also avoids the accurate estimates ofdtd\_\{t\}required in Challenge 2\. For a continuation from the prefixhth\_\{t\}, we define the*selector feedback*

Wt=𝟏​\{Y~t∈C,Y~t≠r\}​StS0,W\_\{t\}=\\mathbf\{1\}\\bigl\\\{\\widetilde\{Y\}\_\{t\}\\in C,\\ \\widetilde\{Y\}\_\{t\}\\neq r\\bigr\\\}\\frac\{S\_\{t\}\}\{S\_\{0\}\},\(7\)and writeWX,αW\_\{X,\\alpha\}for the feedback at the prefix lengthtX​\(α\)t\_\{X\}\(\\alpha\)\. In other words, a successful replacement receives the fraction of the reference surprisal left to the continuation, while a failure or an exact copy of the reference receives zero\. Since the indicator has expectationdtd\_\{t\}, the feedback satisfies

𝔼⁡\[Wt\]=dt​StS0\.\\mathbb\{E\}\[W\_\{t\}\]=\\frac\{d\_\{t\}S\_\{t\}\}\{S\_\{0\}\}\.\(8\)Thus, the feedback of a single continuation is an unbiased estimate of its term inJJ\. A single estimate is noisy, and noisy estimates cannot reliably rank the prefix lengths of one question\. However, sinceJJis an average that is linear indX,αd\_\{X,\\alpha\}, the errors of single continuations cancel when pooled over many questions\. Moreover, since𝒜\\mathcal\{A\}contains only a few levels and the selector keeps a positive probability on every level, every level receives feedback from many questions, which addresses Challenge 2\. The following theorem shows thatJJstill measures the distance in Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)and that the selector can be learned from this feedback\.

###### Theorem 2\(Learning the prefix selector from feedback\)\.

Fix the current model, the references, and𝒟guide\\mathcal\{D\}\_\{\\mathrm\{guide\}\}, where every question satisfies the setup of Section[3\.1](https://arxiv.org/html/2610.11128#S3.SS1)\. Letℒ⁡\(ϕ\)\\mathcal\{L\}\(\\phi\)andℒref\\mathcal\{L\}\_\{\\mathrm\{ref\}\}be the expectations ofDKL\(QX,α∥νX\)/S0,XD\_\{\\mathrm\{KL\}\}\(Q\_\{X,\\alpha\}\\\|\\nu\_\{X\}\)/S\_\{0,X\}andDKL\(δrX∥νX\)/S0,XD\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\_\{X\}\}\\\|\\nu\_\{X\}\)/S\_\{0,X\}overX∼𝒟guideX\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\}andα∼ρϕ\\alpha\\sim\\rho\_\{\\phi\}\. Then

J⁡\(ϕ\)≤ℒref−ℒ⁡\(ϕ\)≤J⁡\(ϕ\)\+𝔼X​\[1e​S0,X\]\.J\(\\phi\)\\leq\\mathcal\{L\}\_\{\\mathrm\{ref\}\}\-\\mathcal\{L\}\(\\phi\)\\leq J\(\\phi\)\+\\mathbb\{E\}\_\{X\}\\left\[\\frac\{1\}\{eS\_\{0,X\}\}\\right\]\.\(9\)Moreover, for a differentiable prefix selector with positive probabilities and a baselinebbfixed before sampling,

𝔼⁡\[\(WX,α−b\)​∇ϕ​log⁡ρϕ​\(α\)\]=∇ϕJ​\(ϕ\)\.\\mathbb\{E\}\\left\[\(W\_\{X,\\alpha\}\-b\)\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\)\\right\]=\\nabla\_\{\\phi\}J\(\\phi\)\.\(10\)

The first statement shows that, compared with training on the reference, the selector reduces the average normalized distance by at leastJJ, where each distance is divided by the surprisal of the entire reference\. The gap is at most𝔼X​\[1/\(e​S0,X\)\]\\mathbb\{E\}\_\{X\}\[1/\(eS\_\{0,X\}\)\], which is small when no reference is highly likely under the model\. Therefore, maximizingJJpursues the goal of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)across questions\. The second statement shows that raising the probability of the levels whose feedback exceeds the baselinebbmoves the selector toward a largerJJon average, wherebbintroduces no bias and serves to reduce the variance\. Since these continuations also produce the accepted trajectories, the update requires no additional generation\. Appendix[B](https://arxiv.org/html/2610.11128#A2)proves Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)and extends it to batches, changing models, and randomly sampled levels\.

## 4ARG: Adaptive Reference Guidance

We now build ARG on Theorems[1](https://arxiv.org/html/2610.11128#Thmtheorem1)and[2](https://arxiv.org/html/2610.11128#Thmtheorem2)\. Section[4\.1](https://arxiv.org/html/2610.11128#S4.SS1)describes how ARG generates accepted trajectories for a question, and Section[4\.2](https://arxiv.org/html/2610.11128#S4.SS2)describes how the same continuations update the prefix selector\. Neither step depends on a particular reinforcement learning algorithm\. The algorithm only determines the questions that receive guidance and how the model learns from the accepted trajectories, and Section[4\.3](https://arxiv.org/html/2610.11128#S4.SS3)specifies both for GRPO\.

### 4\.1Generating Accepted Trajectories

Consider a questionxxthat receives guidance, with referencerr\. ARG first scoresrronce under the current modelπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}, which gives the reference surprisalStS\_\{t\}for every prefix length\. Each guidance levelα∈𝒜\\alpha\\in\\mathcal\{A\}then maps to a prefix lengthtx​\(α\)t\_\{x\}\(\\alpha\)\. Specifically,α=1\\alpha=1gives the empty prefix\. Forα<1\\alpha<1, we take the latest prefix length whose reference surprisal is at leastα​S0\\alpha S\_\{0\}and move it to a nearby sentence or formula boundary, so that the prefix ends at a natural break\. Appendix[E\.2](https://arxiv.org/html/2610.11128#A5.SS2)gives the boundary rules\.

The prefix selector assigns a logitϕα\\phi\_\{\\alpha\}to each level and samples from

ρϕ​\(α\)=\(1−εsel\)​exp⁡\(ϕα\)∑α′∈𝒜exp⁡\(ϕα′\)\+εsel\|𝒜\|,\\rho\_\{\\phi\}\(\\alpha\)=\(1\-\\varepsilon\_\{\\mathrm\{sel\}\}\)\\frac\{\\exp\(\\phi\_\{\\alpha\}\)\}\{\\sum\_\{\\alpha^\{\\prime\}\\in\\mathcal\{A\}\}\\exp\(\\phi\_\{\\alpha^\{\\prime\}\}\)\}\+\\frac\{\\varepsilon\_\{\\mathrm\{sel\}\}\}\{\|\\mathcal\{A\}\|\},\(11\)where0<εsel<10<\\varepsilon\_\{\\mathrm\{sel\}\}<1keeps every level observed as the model changes\. Within a fixed budget ofKKcontinuations per question, we independently sampleαj∼ρϕ\\alpha\_\{j\}\\sim\\rho\_\{\\phi\}forj=1,…,Kj=1,\\ldots,K\. For the resulting prefix lengthtj=tx​\(αj\)t\_\{j\}=t\_\{x\}\(\\alpha\_\{j\}\), we sample a continuationZj∼πθold\(⋅∣x,htj\)Z\_\{j\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid x,h\_\{t\_\{j\}\}\)and verify the candidate trajectoryY~j=htj⊕Zj\\widetilde\{Y\}\_\{j\}=h\_\{t\_\{j\}\}\\oplus Z\_\{j\}\.

Every correct candidate trajectory different fromrrcan serve as an accepted trajectory\. If there is none, ARG falls back to the reference\. Therefore, the question always receives a correct trajectory\. When the number of accepted trajectories is limited, as in our application, we prefer candidate trajectories that keep less of the reference\. Specifically, a correct candidate trajectory from the empty prefix keeps none of the reference\. Indeed, it is generated entirely by the model and follows the ideal distribution restricted to trajectories other thanrr\. Therefore, we accept these candidate trajectories first\. Among candidate trajectories from nonempty prefixes, we prefer the shortest prefix, since the surprisal of the prefix,−log⁡πθold​\(ht∣x\)=S0−St\-\\log\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(h\_\{t\}\\mid x\)=S\_\{0\}\-S\_\{t\}, never decreases withtt\. Finally, Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)concerns a single continuation\. For accepted trajectories from several continuations, Appendix[C\.2](https://arxiv.org/html/2610.11128#A3.SS2)gives a sufficient condition under which the distance remains no larger than that of training on the reference\.

### 4\.2Updating the Prefix Selector

Every continuation provides the selector feedbackWj=WtjW\_\{j\}=W\_\{t\_\{j\}\}in Equation \([7](https://arxiv.org/html/2610.11128#S3.E7)\), whether or not its candidate trajectory is accepted\. Since the feedback does not depend on which trajectories are accepted, the second statement of Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)holds under any acceptance rule, while the first statement, like Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1), concerns a single continuation\.

We collect continuations for all questions in a batch while keeping the current model and the prefix selector fixed\. Let𝒰\\mathcal\{U\}index these continuations, with sampled levelsαj\\alpha\_\{j\}and selector feedbackWjW\_\{j\}\. For a nonempty𝒰\\mathcal\{U\}, we update the prefix selector by

ϕ←ϕ\+ηsel​∑j∈𝒰\(Wj−b\)​∇ϕ​log⁡ρϕ​\(αj\),\\phi\\leftarrow\\phi\+\\eta\_\{\\mathrm\{sel\}\}\\sum\_\{j\\in\\mathcal\{U\}\}\(W\_\{j\}\-b\)\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\_\{j\}\),\(12\)whereηsel\>0\\eta\_\{\\mathrm\{sel\}\}\>0is the step size andbbis a running average of past feedback fixed before collection\. By the second statement of Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2), the update follows the gradient ofJJin expectation\. Appendix[B](https://arxiv.org/html/2610.11128#A2)gives the baseline update and derives the expected update as the model and the questions change during training\.

### 4\.3Applying ARG to GRPO

GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib42)\)samples a group ofG=8G=8responses for each question and compares their rewards within the group\. When every response in a group is incorrect, the group provides no signal for improving correctness, since all correctness rewards are identical\. In such an*all\-failure group*, adding a correct trajectory restores the learning signal\. Therefore, we apply ARG to all\-failure groups, usingK=8K=8continuations\. These questions define𝒟guide\\mathcal\{D\}\_\{\\mathrm\{guide\}\}\.

We insert at most four accepted trajectories into each group, which keeps at least four original failures and permits a balanced group, motivated by prior work on intermediate pass rates\([Bae et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib2);[Liu et al\., 2026b](https://arxiv.org/html/2610.11128#bib.bib28)\)\. Following the preference in Section[4\.1](https://arxiv.org/html/2610.11128#S4.SS1), we first accept correct candidate trajectories from the empty prefix\. If fewer than four are accepted, we additionally accept at most one from a nonempty prefix, which limits repeated reference content in the same group\. If no candidate trajectory is accepted, we insert the reference once as fallback\. Each inserted trajectory replaces an original failure at a uniformly chosen position\. We then recompute the group advantagesAiA\_\{i\}from the final correctness and format rewards, as specified in Appendix[E\.3](https://arxiv.org/html/2610.11128#A5.SS3)\.

The model learns each inserted trajectory under the original question through its complete likelihood, including the prefix\. This matches Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1), which measures the complete accepted trajectory under the original question\. Letℐ\\mathcal\{I\}index the inserted trajectories\(xi,yi\)\(x\_\{i\},y\_\{i\}\)and letTTbe the total number of tokens in the loss\. Their contribution to the gradient is

∇θℒinserted=−1T∑i∈ℐAi∇θlogπθ\(yi∣xi\),\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{inserted\}\}=\-\\frac\{1\}\{T\}\\sum\_\{i\\in\\mathcal\{I\}\}A\_\{i\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\\mid x\_\{i\}\),\(13\)where the trajectories, advantages, and token count are fixed during differentiation\. The remaining original responses use the clipped GRPO loss with the same denominator\. Moreover, for binary rewards, Appendix[D](https://arxiv.org/html/2610.11128#A4)shows that insertingY∼QY\\sim Qinto an all\-failure group gives an expected update proportional to the gradient oflogpθ1−pθ−DKL\(Q∥νθ\)\\log\\frac\{p\_\{\\theta\}\}\{1\-p\_\{\\theta\}\}\-D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\_\{\\theta\}\)at the current model\. The gradient of the second term vanishes whenQQis the ideal distribution, in which case the update only increases the odds of success\. This connects the distance in Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)to the GRPO update\.

After collecting continuations, we update the prefix selector by Equation \([12](https://arxiv.org/html/2610.11128#S4.E12)\) and take one policy update on the final groups\. Algorithm[1](https://arxiv.org/html/2610.11128#alg1)summarizes the training procedure\.

## 5Experiments

### 5\.1Experimental Setting

Data and models\.We use DeepMath\-103k with offline reference solutions\([He et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib14)\)as our training set\. We use Qwen3\-4B and Qwen3\-8B with thinking mode disabled\([Yang et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib50)\)as our base LLMs\. For all methods, we evaluate on five mathematical reasoning benchmarks, AIME 2024, AIME 2025, AIME 2026, HMMT February 2025, and HMMT November 2025\([Balunovic et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib3);[Dekoninck et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib8)\)\. Appendix[E](https://arxiv.org/html/2610.11128#A5)gives further details\.

Baselines and evaluation\.We compare against the initial Qwen model, vanilla supervised fine\-tuning \(SFT\), GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib42)\), OPSD\([Zhao et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib58)\), SAGE\([Liao et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib25)\), BREAD\([Zhang et al\., 2026d](https://arxiv.org/html/2610.11128#bib.bib57)\), and LUFFY\([Yan et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib49)\)\. During evaluation, we draw 12 responses per question\.pass​@​12\\mathrm\{pass@12\}measures whether a question has at least one correct response, whileavg​@​12\\mathrm\{avg@12\}measures the fraction of correct responses\. We usepass​@​12\\mathrm\{pass@12\}as the primary metric\.

Table 1:Main experimental results inpass​@​12\\mathrm\{pass@12\}percentages\. Avg\. is the mean over the five benchmarks\. Within each model,boldandunderlinemark the best and second\-best values in each column\.
### 5\.2Results

Table[1](https://arxiv.org/html/2610.11128#S5.T1)evaluates ARG through its application to GRPO\. ARG achieves the highest aggregatepass​@​12\\mathrm\{pass@12\}at both model scales, reaching67\.33%67\.33\\%on Qwen3\-4B and70\.00%70\.00\\%on Qwen3\-8B\. Compared with GRPO, these results improvepass​@​12\\mathrm\{pass@12\}by3\.333\.33and7\.337\.33percentage points, respectively\. ARG also exceeds the reference\-informed baselines in aggregatepass​@​12\\mathrm\{pass@12\}, which use reference solutions as off\-policy responses in LUFFY, as prefixes in BREAD, and as hints in SAGE\([Yan et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib49);[Zhang et al\., 2026d](https://arxiv.org/html/2610.11128#bib.bib57);[Liao et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib25)\)\. Thus, ARG solves the largest fraction of evaluation questions under the 12\-response budget\. Foravg​@​12\\mathrm\{avg@12\}, ARG improves over GRPO on both models and remains competitive with the strongest baseline, as discussed in Appendix[F](https://arxiv.org/html/2610.11128#A6)\. Sections[5\.3](https://arxiv.org/html/2610.11128#S5.SS3)and[5\.4](https://arxiv.org/html/2610.11128#S5.SS4)examine the contributions of the continuations and the prefix selector\.

### 5\.3Ablations

We examine the contribution of the continuations, compare with unguided resampling, and evaluate the surprisal weighting in the selector feedback\.

Reference insertion\.For each all\-failure group, we replace a random failure in the original group with the reference without any extra generation\. This ablation corresponds to training on the reference in Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)\. Table[2\(a\)](https://arxiv.org/html/2610.11128#S5.T2.st1)shows that reference insertion achieves56\.67%56\.67\\%inpass​@​12\\mathrm\{pass@12\}and28\.44%28\.44\\%inavg​@​12\\mathrm\{avg@12\}on Qwen3\-4B, compared with67\.33%67\.33\\%and41\.11%41\.11\\%for ARG\. These results support using the reference to guide continuations and reserving the reference itself for fallback\.

Resampling\.We allocate the same eight continuations to the empty prefix for each all\-failure group\. Correct candidate trajectories replace original failures, and groups with no correct candidate trajectory remain unchanged\. Thus, this comparison evaluates ARG against additional sampling without reference guidance or fallback\. Table[2\(a\)](https://arxiv.org/html/2610.11128#S5.T2.st1)shows that ARG improvespass​@​12\\mathrm\{pass@12\}from60\.00%60\.00\\%to67\.33%67\.33\\%andavg​@​12\\mathrm\{avg@12\}from34\.00%34\.00\\%to41\.11%41\.11\\%on Qwen3\-4B\. During their respective training runs, candidate trajectories are accepted in66\.0%66\.0\\%of all\-failure groups with ARG and only32\.5%32\.5\\%with resampling\. More details are shown in Appendix[G\.2](https://arxiv.org/html/2610.11128#A7.SS2)\.

Without surprisal weighting\.We replace the selector feedback in Equation \([7](https://arxiv.org/html/2610.11128#S3.E7)\) with the successful replacement indicator\. The guidance levels, surprisal\-based prefix construction, selector update rule, and acceptance rule remain unchanged\. Table[2\(b\)](https://arxiv.org/html/2610.11128#S5.T2.st2)shows that removing the weighting lowers both evaluation metrics on both models, withpass​@​12\\mathrm\{pass@12\}decreasing by4\.004\.00and6\.006\.00percentage points on Qwen3\-4B and Qwen3\-8B, respectively\. These results support the surprisal weighting in the selector feedback\. Section[5\.4](https://arxiv.org/html/2610.11128#S5.SS4)examines the selector feedback and prefix selection under ARG, while Appendix[G\.3](https://arxiv.org/html/2610.11128#A7.SS3)compares the prefix selection with that of the ablation\.

\(a\)Sources of accepted trajectories on Qwen3\-4B
\(b\)Surprisal weighting on both models

Table 2:Ablation results averaged over the five benchmarks\. Resampling uses eight continuations from the empty prefix without reference fallback\. The weighting ablation removesSt/S0S\_\{t\}/S\_\{0\}from the selector feedback\.
### 5\.4Training Dynamics

Figure[1](https://arxiv.org/html/2610.11128#S5.F1)examines Qwen3\-8B training dynamics through successful replacement rates, selector feedback, and prefix selection\.

Successful replacement and selector feedback\.Figure[1\(a\)](https://arxiv.org/html/2610.11128#S5.F1.sf1)reports the successful replacement rate for each guidance level\. Each point averages all continuations at that level within a window of 50 training steps\. The levelα=1/4\\alpha=1/4has a higher successful replacement rate thanα=1/2\\alpha=1/2in every window\. However, Figure[1\(b\)](https://arxiv.org/html/2610.11128#S5.F1.sf2)shows that their ordering can change when the successful replacement indicator is weighted by the fractionSt/S0S\_\{t\}/S\_\{0\}of the reference surprisal left to the continuation\. In the final window, for example,α=1/4\\alpha=1/4has lower mean feedback thanα=1/2\\alpha=1/2\(0\.08020\.0802versus0\.09910\.0991\), although its successful replacement rate is higher \(43\.84%43\.84\\%versus23\.65%23\.65\\%\)\. Thus, the feedback favors less reference guidance in this window, consistent with Section[3\.1](https://arxiv.org/html/2610.11128#S3.SS1), where a higher success probability of the continuation need not give a smaller distance\.

\(a\)Successful replacement\(b\)Selector feedback\(c\)Prefix selection
Figure 1:Training dynamics of a Qwen3\-8B ARG run\. Panel \(a\) reports successful replacement rates, panel \(b\) reports mean selector feedbackWtW\_\{t\}, and panel \(c\) shows the selection probabilities\. In panels \(a\) and \(b\), a point at stepsssummarizes all continuations at that level across the past 50 steps, including failures and candidate trajectories not accepted for training\. Panel \(c\) shows the state after each training step without smoothing\.Prefix selection during training\.The selector uses the feedbackWtW\_\{t\}from each continuation to update its selection probabilities\. Figure[1\(c\)](https://arxiv.org/html/2610.11128#S5.F1.sf3)shows how these probabilities change during training\. The selector favorsα=1/4\\alpha=1/4through much of training and increases its probability of selectingα=1/2\\alpha=1/2later, assigning it the highest final probability of49\.0%49\.0\\%\. Since a largerα\\alphaleaves more of the reference surprisal to the continuation, this shift supplies less reference guidance\. Appendix[H](https://arxiv.org/html/2610.11128#A8)provides additional training\-dynamics results\.

## 6Related Work

Learning signals beyond on\-policy rollouts\.LUFFY, SRFT, ReLIFT, and related methods combine reference or teacher trajectories with the model’s own rollouts, through mixed\-policy groups or combined supervised and reinforcement updates\([Yan et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib49);[Fu et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib11);[Ma et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib31);[Zhang et al\., 2026b](https://arxiv.org/html/2610.11128#bib.bib54);[Liu et al\., 2025a](https://arxiv.org/html/2610.11128#bib.bib27);[Dong et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib9);[Lv et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib30)\)\. OPSD and SDPO instead use the model conditioned on solution information or feedback as a teacher for token\-level supervision\([Zhao et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib58);[Hübotter et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib16)\)\. Other methods recover learning signals from failed rollouts, either by repairing them into correct ones\([Li et al\., 2026b](https://arxiv.org/html/2610.11128#bib.bib22);[Yang et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib51);[Zhang et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib55);[Ren et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib39)\)or by reshaping the rewards or advantages of groups with identical rewards\([Chen et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib6);[Nie et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib34);[Nan et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib32);[Le et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib20)\)\.

Reference\-guided generation\.Closer to our work, many methods use the reference only to guide the model’s own generation, either by revealing a prefix of the reference for the model to continue\([Liu et al\., 2026a](https://arxiv.org/html/2610.11128#bib.bib26);[Setlur et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib41);[Qu et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib36);[Zhang et al\., 2026a](https://arxiv.org/html/2610.11128#bib.bib53);[Su et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib44);[Liu et al\., 2025b](https://arxiv.org/html/2610.11128#bib.bib29)\)or by providing hints derived from the reference or its answer\([Liao et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib25);[Zhang et al\., 2026c](https://arxiv.org/html/2610.11128#bib.bib56);[Nath et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib33);[Chen et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib5);[Zhou et al\., 2026a](https://arxiv.org/html/2610.11128#bib.bib59)\)\. The amount of guidance is fixed or scheduled\([Xi et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib47);[Huang et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib15);[Li et al\., 2026a](https://arxiv.org/html/2610.11128#bib.bib21)\), adapted to the observed success of the model\([Zhang et al\., 2026d](https://arxiv.org/html/2610.11128#bib.bib57);[amani et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib1);[Li et al\., 2025b](https://arxiv.org/html/2610.11128#bib.bib24);[Guo et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib13);[Beliaev, 2026](https://arxiv.org/html/2610.11128#bib.bib4)\), or chosen by the gain of a prefix or the transfer of a hint\([Zhou et al\., 2026c](https://arxiv.org/html/2610.11128#bib.bib61);[Xia et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib48)\)\. In contrast, we derive the amount of guidance from the distance between the accepted trajectory and the ideal distribution, which shows that success alone does not determine the preferred prefix, and we learn a prefix selector shared across questions from the continuations themselves\.

Posterior views of learning from correct trajectories\.Training on the model’s own correct samples, as in STaR and ReSTEM, approximates learning from our ideal distribution\([Zelikman et al\., 2022](https://arxiv.org/html/2610.11128#bib.bib52);[Singh et al\., 2023](https://arxiv.org/html/2610.11128#bib.bib43)\), which is also the correct distribution closest to the model in KL and relates to the Bayesian view of KL\-regularized RL\([Dymetman, 2026](https://arxiv.org/html/2610.11128#bib.bib10);[Korbak et al\., 2022](https://arxiv.org/html/2610.11128#bib.bib18)\)\. Latent\-variable methods approximate the posterior over rationales given the correct answer\([Phan et al\., 2023](https://arxiv.org/html/2610.11128#bib.bib35);[Zhou et al\., 2026b](https://arxiv.org/html/2610.11128#bib.bib60)\), and TRICE keeps a stored rationale until a new proposal is correct, similar to our fallback\. Other work analyzes or modifies the function of the success probability that RL maximizes\([Davis & Recht, 2025](https://arxiv.org/html/2610.11128#bib.bib7);[Tajwar et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib45);[Ramasubramanian et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib37)\), or shows that targets unlikely under the model are harder to learn from\([Ren et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib38);[Li et al\., 2025a](https://arxiv.org/html/2610.11128#bib.bib23);[Setlur et al\., 2024](https://arxiv.org/html/2610.11128#bib.bib40)\)\.

## 7Conclusion and Limitations

We study reference\-prefix continuation with fallback through the distance from the accepted trajectory to the ideal distribution\. Up to a small bounded term, this distance decreases withdt​Std\_\{t\}S\_\{t\}, which captures the trade\-off between successful replacement and the reference surprisal\. This analysis leads to ARG, which learns a prefix selector across questions from the continuations themselves\. Applied to all\-failure groups in GRPO, ARG achieves the highest aggregatepass​@​12\\mathrm\{pass@12\}on Qwen3\-4B and Qwen3\-8B across five mathematical benchmarks, with competitive average sampled accuracy\.

Limitations\.ARG is only applied to GRPO, although the method does not depend on a particular reinforcement learning algorithm\. Moreover, the selector chooses among a predefined set of guidance levels, and our experiments cover mathematical reasoning with one model family\. Future work may extend the method to other domains with verifiable rewards\.

## References

- amani et al\. \(2026\)mohammad hossein amani, Aryo Lotfi, Nicolas Baldwin, Samy Bengio, Mehrdad Farajtabar, Emmanuel Abbe, and Robert West\.Rl for reasoning by adaptively revealing rationales\.In C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(eds\.\),*International Conference on Learning Representations*, volume 2026, pp\. 118599–118620, 2026\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2026/file/c073a9d002f3dab99a7af18642d74bdb\-Paper\-Conference\.pdf](https://proceedings.iclr.cc/paper_files/paper/2026/file/c073a9d002f3dab99a7af18642d74bdb-Paper-Conference.pdf)\.
- Bae et al\. \(2026\)Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak\.Online difficulty filtering for reasoning oriented reinforcement learning\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 700–719, 2026\.
- Balunovic et al\. \(2026\)Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev\.Matharena: Evaluating llms on uncontaminated math competitions\.*Advances in Neural Information Processing Systems*, 38, 2026\.
- Beliaev \(2026\)Vladislav Beliaev\.Max out grpo signal: Adaptive trace prefix control for hard reasoning problems\.*arXiv preprint arXiv:2607\.07674*, 2026\.
- Chen et al\. \(2026\)Justin Chen, Xiangyu Peng, Prafulla Kumar Choubey, Kung\-Hsiang Huang, Jiaxin Zhang, Mohit Bansal, and Chien\-Sheng Wu\.Nudging the boundaries of llm reasoning\.In*International Conference on Learning Representations*, volume 2026, pp\. 30315–30335, 2026\.
- Chen et al\. \(2025\)Peter Chen, Xiaopeng Li, Ziniu Li, Xi Chen, and Tianyi Lin\.Stepwise guided policy optimization: Coloring your incorrect reasoning in grpo\.*arXiv preprint arXiv:2505\.11595*, 2025\.
- Davis & Recht \(2025\)Damek Davis and Benjamin Recht\.What is the objective of reasoning with reinforcement learning?*arXiv preprint arXiv:2510\.13651*, 2025\.
- Dekoninck et al\. \(2026\)Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev\.Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms\.*arXiv preprint arXiv:2605\.00674*, 2026\.
- Dong et al\. \(2026\)Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, et al\.Rl\-plus: Countering capability boundary collapse of llms in reinforcement learning with hybrid\-policy optimization\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 43029–43053, 2026\.
- Dymetman \(2026\)Marc Dymetman\.Binary rewards and reinforcement learning: Fundamental challenges\.*arXiv preprint arXiv:2605\.02375*, 2026\.
- Fu et al\. \(2026\)Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao\.Srft: A single\-stage method with supervised and reinforcement fine\-tuning for reasoning\.In*International Conference on Learning Representations*, volume 2026, pp\. 102430–102459, 2026\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- Guo et al\. \(2026\)Yongxin Guo, Wenbo Deng, Zhenglin Cheng, and Xiaoying Tang\.G2rpo\-a: Guided group relative policy optimization with adaptive guidance\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 4525–4539, 2026\.
- He et al\. \(2026\)Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al\.Deepmath\-103k: A large\-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\.In*International Conference on Learning Representations*, volume 2026, pp\. 138306–138322, 2026\.
- Huang et al\. \(2025\)Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov\.Blending supervised and reinforcement fine\-tuning with prefix sampling\.*arXiv preprint arXiv:2507\.01679*, 2025\.
- Hübotter et al\. \(2026\)Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al\.Reinforcement learning via self\-distillation\.*arXiv preprint arXiv:2601\.20802*, 2026\.
- Jaech et al\. \(2024\)Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El\-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al\.Openai o1 system card\.*arXiv preprint arXiv:2412\.16720*, 2024\.
- Korbak et al\. \(2022\)Tomasz Korbak, Ethan Perez, and Christopher Buckley\.Rl with kl penalties is better viewed as bayesian inference\.In*Findings of the Association for Computational Linguistics: EMNLP 2022*, pp\. 1083–1091, 2022\.
- Lambert et al\. \(2024\)Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al\.Tulu 3: Pushing frontiers in open language model post\-training\.*arXiv preprint arXiv:2411\.15124*, 2024\.
- Le et al\. \(2026\)Thanh\-Long V Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang\.No prompt left behind: Exploiting zero\-variance prompts in llm reinforcement learning via entropy\-guided advantage shaping\.In*International Conference on Learning Representations*, volume 2026, pp\. 121956–121982, 2026\.
- Li et al\. \(2026a\)Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, and Jingzhao Zhang\.Questa: Expanding reasoning capacity in llms via question augmentation\.In*International Conference on Learning Representations*, volume 2026, pp\. 144191–144215, 2026a\.
- Li et al\. \(2026b\)Shuozhe Li, Vaishnav Tadiparthi, Kwonjoon Lee, Nakul Agarwal, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Lizhang Chen, Amy Zhang, and Liu Leqi\.Learning robust reasoning through guided adversarial self\-play\.*arXiv preprint arXiv:2602\.00173*, 2026b\.
- Li et al\. \(2025a\)Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran\.Small models struggle to learn from strong reasoners\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pp\. 25366–25394, 2025a\.
- Li et al\. \(2025b\)Ziheng Li, Zexu Sun, Jinman Zhao, Erxue Min, Yongcheng Zeng, Hui Wu, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Xu Chen, et al\.Staying in the sweet spot: Responsive reasoning evolution via capability\-adaptive hint scaffolding\.*arXiv preprint arXiv:2509\.06923*, 2025b\.
- Liao et al\. \(2026\)Baohao Liao, Hanze Dong, Xinxing Xu, Christof Monz, and Jiang Bian\.Self\-hinting language models enhance reinforcement learning\.*arXiv preprint arXiv:2602\.03143*, 2026\.
- Liu et al\. \(2026a\)Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar\.Uft: Unifying supervised and reinforcement fine\-tuning\.*Advances in Neural Information Processing Systems*, 38:101347–101383, 2026a\.
- Liu et al\. \(2025a\)Yihao Liu, Shuocheng Li, Lang Cao, Yuhang Xie, Mengyu Zhou, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang\.Superrl: Reinforcement learning with supervision to boost language model reasoning\.*arXiv preprint arXiv:2506\.01096*, 2025a\.
- Liu et al\. \(2026b\)Zehao Liu, Yuanpu Cao, Jinghui Chen, and Vasant G Honavar\.Restoring the sweet spot: Pass\-rate weighted self\-distillation for llm reasoning\.*arXiv preprint arXiv:2605\.27765*, 2026b\.
- Liu et al\. \(2025b\)Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu\.Ghpo: Adaptive guidance for stable and efficient llm reinforcement learning\.*arXiv preprint arXiv:2507\.10628*, 2025b\.
- Lv et al\. \(2025\)Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, Ning Ding, et al\.Towards a unified view of large language model post\-training\.*arXiv preprint arXiv:2509\.04419*, 2025\.
- Ma et al\. \(2026\)Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, et al\.Learning what reinforcement learning can’t: Interleaved online fine\-tuning for hardest questions\.In*International Conference on Learning Representations*, volume 2026, pp\. 80802–80821, 2026\.
- Nan et al\. \(2025\)Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Dexun Wang, Chunmei Xie, Weiqi Xiong, Xianzhou Zeng, Qixuan Zhou, Yadong Li, et al\.Ngrpo: Negative\-enhanced group relative policy optimization\.*arXiv preprint arXiv:2509\.18851*, 2025\.
- Nath et al\. \(2025\)Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, and Sean Hendryx\.Adaptive guidance accelerates reinforcement learning of reasoning models\.*arXiv preprint arXiv:2506\.13923*, 2025\.
- Nie et al\. \(2026\)Wenhua Nie, Jianan Wu, Junlin Liu, Ziwei Li, Zheng Lin, Zhang Zijian, Yilong Fan, Haoran Zheng, and Jyh\-Shing Roger Jang\.Gradient starvation in binary\-reward grpo: Why group\-mean centering fails and why the simplest fix works\.*arXiv preprint arXiv:2605\.07689*, 2026\.
- Phan et al\. \(2023\)Du Phan, Matthew Douglas Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous\.Training chain\-of\-thought via latent\-variable inference\.*Advances in Neural Information Processing Systems*, 36:72819–72841, 2023\.
- Qu et al\. \(2026\)Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar\.Pope: Learning to reason on hard problems via privileged on\-policy exploration\.*arXiv preprint arXiv:2601\.18779*, 2026\.
- Ramasubramanian et al\. \(2026\)Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar, Guanning Zeng, Qingyang Wu, Zhongzhu Zhou, Chenfeng Xu, Haiwen Feng, Yuda Song, Aarti Singh, et al\.Tail\-likelihood reinforcement learning\.*arXiv preprint arXiv:2609\.02987*, 2026\.
- Ren et al\. \(2024\)Xuan Ren, Biao Wu, and Lingqiao Liu\.I learn better if you speak my language: Understanding the superior performance of fine\-tuning large language models with llm\-generated responses\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 10225–10245, 2024\.
- Ren et al\. \(2026\)Yanwei Ren, Haotian Zhang, Likang Xiao, Xikai Zhang, Jiaxing Huang, Jiayan Qiu, Baosheng Yu, Quan Chen, and Liu Liu\.Recycling failures: Salvaging exploration in rlvr via fine\-grained off\-policy guidance\.*arXiv preprint arXiv:2602\.24110*, 2026\.
- Setlur et al\. \(2024\)Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar\.Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight\-fold\.*Advances in Neural Information Processing Systems*, 37:43000–43031, 2024\.
- Setlur et al\. \(2026\)Amrith Setlur, Zijian Wang, Andrew Cohen, Paria Rashidinejad, and Sang Michael Xie\.Reuse your flops: Scaling rl on hard problems by conditioning on very off\-policy prefixes\.*arXiv preprint arXiv:2601\.18795*, 2026\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Singh et al\. \(2023\)Avi Singh, John D Co\-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al\.Beyond human data: Scaling self\-training for problem\-solving with language models\.*arXiv preprint arXiv:2312\.06585*, 2023\.
- Su et al\. \(2026\)Mingyu Su, Jian Guan, Yuxian Gu, Minlie Huang, and Hongning Wang\.Trust\-region adaptive policy optimization\.In*International Conference on Learning Representations*, volume 2026, pp\. 51806–51828, 2026\.
- Tajwar et al\. \(2026\)Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette\.Maximum likelihood reinforcement learning\.*arXiv preprint arXiv:2602\.02710*, 2026\.
- Team et al\. \(2025\)Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al\.Kimi k1\. 5: Scaling reinforcement learning with llms\.*arXiv preprint arXiv:2501\.12599*, 2025\.
- Xi et al\. \(2024\)Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al\.Training large language models for reasoning through reverse curriculum reinforcement learning\.*arXiv preprint arXiv:2402\.05808*, 2024\.
- Xia et al\. \(2026\)Yu Xia, Canwen Xu, Zhewei Yao, Julian McAuley, and Yuxiong He\.Learning to hint for reinforcement learning\.*arXiv preprint arXiv:2604\.00698*, 2026\.
- Yan et al\. \(2026\)Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang\.Learning to reason under off\-policy guidance\.*Advances in Neural Information Processing Systems*, 38:117157–117186, 2026\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yang et al\. \(2026\)Matthew Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, and Aviral Kumar\.Int: Self\-proposed interventions enable credit assignment in llm reasoning\.In*International Conference on Learning Representations*, volume 2026, pp\. 85054–85091, 2026\.
- Zelikman et al\. \(2022\)Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman\.Star: Bootstrapping reasoning with reasoning\.*Advances in Neural Information Processing Systems*, 35:15476–15488, 2022\.
- Zhang et al\. \(2026a\)Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan\.Stephint: Multi\-level stepwise hints enhance reinforcement learning to reason\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 37846–37864, 2026a\.
- Zhang et al\. \(2026b\)Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou\.On\-policy rl meets off\-policy experts: Harmonizing supervised fine\-tuning and reinforcement learning via dynamic weighting\.In*International Conference on Learning Representations*, volume 2026, pp\. 120693–120726, 2026b\.
- Zhang et al\. \(2025\)Xiaoying Zhang, Yipeng Zhang, Hao Sun, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng\.Critique\-grpo: Advancing llm reasoning with natural language and numerical feedback\.*arXiv preprint arXiv:2506\.03106*, 2025\.
- Zhang et al\. \(2026c\)Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, and Jiaya Jia\.Scaf\-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning\.In*International Conference on Learning Representations*, volume 2026, pp\. 131946–131974, 2026c\.
- Zhang et al\. \(2026d\)Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak\.Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning\.*Advances in Neural Information Processing Systems*, 38:96726–96752, 2026d\.
- Zhao et al\. \(2026\)Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover\.Self\-distilled reasoner: On\-policy self\-distillation for large language models\.*arXiv preprint arXiv:2601\.18734*, 2026\.
- Zhou et al\. \(2026a\)Ruiyang Zhou, Shuozhe Li, Amy Zhang, and Liu Leqi\.Expo: Unlocking hard reasoning with self\-explanation\-guided reinforcement learning\.*Advances in Neural Information Processing Systems*, 38:90989–91016, 2026a\.
- Zhou et al\. \(2026b\)Xiangxin Zhou, Zichen Liu, Haonan Wang, Chao Du, Min Lin, Chongxuan Li, Liang Wang, and Tianyu Pang\.Variational reasoning for language models\.In*International Conference on Learning Representations*, volume 2026, pp\. 154370–154407, 2026b\.
- Zhou et al\. \(2026c\)Yuhang Zhou, Yixin Cao, and Guangnan Ye\.From correctness to utility: Gain\-based prefix evaluation for llm reasoning\.*arXiv preprint arXiv:2606\.07190*, 2026c\.

## Appendix AProofs for the Distance to the Ideal Distribution

This appendix proves Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)and collects its consequences\. Appendix[A\.1](https://arxiv.org/html/2610.11128#A1.SS1)fixes the setup\. Appendix[A\.2](https://arxiv.org/html/2610.11128#A1.SS2)derives the gradient identity in Equation \([3](https://arxiv.org/html/2610.11128#S2.E3)\) and the distance of training on the reference\. Appendix[A\.3](https://arxiv.org/html/2610.11128#A1.SS3)describes the accepted trajectory and the reference surprisal, and then proves Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)\. Appendix[A\.4](https://arxiv.org/html/2610.11128#A1.SS4)gives an equivalent reading of the closed form\. Finally, Appendix[A\.5](https://arxiv.org/html/2610.11128#A1.SS5)illustrates the theorem with a finite example\.

### A\.1Setup

We use the setup of Section[3\.1](https://arxiv.org/html/2610.11128#S3.SS1)\. Fix a questionxx, its correct referencerr, and the current model, and suppress the conditioning onxx\. The trajectory space is finite, andπ\\pidenotes the current policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\. Moreover,CCis the set of correct trajectories,p=π⁡\(C\)p=\\pi\(C\)is the success probability, andν=π\(⋅∣C\)\\nu=\\pi\(\\cdot\\mid C\)is the ideal distribution in Equation \([2](https://arxiv.org/html/2610.11128#S2.E2)\)\. We assume0<π⁡\(r\)<10<\\pi\(r\)<1\. For a token sequencehh, we writeπ⁡\(h\)\\pi\(h\)for the probability that a trajectory sampled fromπ\\pibegins withhh, andπ⁡\(z∣h\)\\pi\(z\\mid h\)for the probability that the model continueshhwithzz\. Since the continuation follows the same policy, including its termination rule, autoregressive factorization givesπ⁡\(h⊕z\)=π⁡\(h\)​π​\(z∣h\)\\pi\(h\\oplus z\)=\\pi\(h\)\\pi\(z\\mid h\)for every complete trajectoryh⊕zh\\oplus z\.

For a prefix length0≤t<\|r\|0\\leq t<\|r\|, the reference splits asr=ht⊕atr=h\_\{t\}\\oplus a\_\{t\}, andQtQ\_\{t\}is the distribution of the accepted trajectory in Equation \([1](https://arxiv.org/html/2610.11128#S2.E1)\)\. Besidesdtd\_\{t\}andStS\_\{t\}in Equation \([4](https://arxiv.org/html/2610.11128#S3.E4)\), the proofs use the probabilities of an exact copy of the reference and of a correct candidate trajectory,

ut=π\(at∣ht\),vt=PrZ∼π\(⋅∣ht\)\(ht⊕Z∈C\)\.u\_\{t\}=\\pi\(a\_\{t\}\\mid h\_\{t\}\),\\qquad v\_\{t\}=\\Pr\_\{Z\\sim\\pi\(\\cdot\\mid h\_\{t\}\)\}\\bigl\(h\_\{t\}\\oplus Z\\in C\\bigr\)\.Here an exact copy means equality of the complete token sequences, and different token sequences can still follow the same mathematical strategy\. By definition,St=−log⁡utS\_\{t\}=\-\\log u\_\{t\}, and in particularS0=−log⁡π⁡\(r\)S\_\{0\}=\-\\log\\pi\(r\)is the reference surprisal of the empty prefix\. Since autoregressive factorization givesut=π⁡\(r\)/π⁡\(ht\)≥π⁡\(r\)\>0u\_\{t\}=\\pi\(r\)/\\pi\(h\_\{t\}\)\\geq\\pi\(r\)\>0, we havedt≤1−ut<1d\_\{t\}\\leq 1\-u\_\{t\}<1, and everyStS\_\{t\}is finite\.

### A\.2The Ideal Distribution

We first justify the choice of the ideal distribution as the learning target in Section[2](https://arxiv.org/html/2610.11128#S2)\. Lemma[A\.1](https://arxiv.org/html/2610.11128#A1.Thmtheorem1)shows that learning the likelihood of its trajectories increases the success probability in expectation\. Lemma[A\.2](https://arxiv.org/html/2610.11128#A1.Thmtheorem2)then gives the distance of training on the reference, which is the first term in Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)\.

###### Lemma A\.1\(Gradient identity\)\.

Suppose thatπθ\\pi\_\{\\theta\}is differentiable inθ\\thetaand assigns positive probability to every trajectory in a neighborhood ofθold\\theta\_\{\\mathrm\{old\}\}\. Writepθ=πθ​\(C\)p\_\{\\theta\}=\\pi\_\{\\theta\}\(C\)\. Then

𝔼Y∼ν​\[∇θ​log​πθ​\(Y\)\|θ=θold\]=∇θ​log​pθ\|θ=θold\.\\mathbb\{E\}\_\{Y\\sim\\nu\}\\Bigl\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\)\\big\|\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\\Bigr\]=\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\\big\|\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\.

###### Proof\.

Since the trajectory space is finite, differentiatingpθ=∑y∈Cπθ​\(y\)p\_\{\\theta\}=\\sum\_\{y\\in C\}\\pi\_\{\\theta\}\(y\)term by term gives

∇θ​log​pθ=1pθ​∑y∈C∇θπθ​\(y\)=∑y∈Cπθ​\(y\)pθ​∇θ​log​πθ​\(y\)\.\\nabla\_\{\\theta\}\\log p\_\{\\theta\}=\\frac\{1\}\{p\_\{\\theta\}\}\\sum\_\{y\\in C\}\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(y\)=\\sum\_\{y\\in C\}\\frac\{\\pi\_\{\\theta\}\(y\)\}\{p\_\{\\theta\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\)\.Atθ=θold\\theta=\\theta\_\{\\mathrm\{old\}\}, the weightπθ​\(y\)/pθ\\pi\_\{\\theta\}\(y\)/p\_\{\\theta\}of eachy∈Cy\\in Cequalsν⁡\(y\)\\nu\(y\)by Equation \([2](https://arxiv.org/html/2610.11128#S2.E2)\), which proves the claim\. ∎

Hereν\\nuis fixed at the current model, and differentiation applies only to the trajectory log likelihood\.

###### Lemma A\.2\(Distance of training on the reference\)\.

Training on the reference has distance

DKL\(δr∥ν\)=logpπ⁡\(r\)=S0\+logp∈\[0,S0\]\.D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)=\\log\\frac\{p\}\{\\pi\(r\)\}=S\_\{0\}\+\\log p\\in\[0,S\_\{0\}\]\.In contrast, ifν\\nuassigns positive probability to a trajectory other thanrr, thenDKL\(ν∥δr\)=∞D\_\{\\mathrm\{KL\}\}\(\\nu\\\|\\delta\_\{r\}\)=\\infty\.

###### Proof\.

Sinceδr\\delta\_\{r\}assigns all probability torrandr∈Cr\\in C, Equation \([2](https://arxiv.org/html/2610.11128#S2.E2)\) givesDKL\(δr∥ν\)=−logν\(r\)=log\(p/π\(r\)\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)=\-\\log\\nu\(r\)=\\log\(p/\\pi\(r\)\)\. SubstitutingS0=−log⁡π⁡\(r\)S\_\{0\}=\-\\log\\pi\(r\)givesS0\+log⁡pS\_\{0\}\+\\log p\. This quantity is nonnegative sinceν⁡\(r\)≤1\\nu\(r\)\\leq 1, and it is at mostS0S\_\{0\}sincep≤1p\\leq 1\. For the reverse direction, a trajectoryy≠ry\\neq rwithν⁡\(y\)\>0\\nu\(y\)\>0hasδr​\(y\)=0\\delta\_\{r\}\(y\)=0, which makes the termν⁡\(y\)​log⁡\(ν⁡\(y\)/δr​\(y\)\)\\nu\(y\)\\log\(\\nu\(y\)/\\delta\_\{r\}\(y\)\)infinite\. ∎

Thus, the direction of KL in Section[2](https://arxiv.org/html/2610.11128#S2)permits a finite comparison with training on the reference\.

### A\.3Proof of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)

The closed form in Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)combines two ingredients\. Lemma[A\.3](https://arxiv.org/html/2610.11128#A1.Thmtheorem3)describes the distribution of the accepted trajectory, and Lemma[A\.4](https://arxiv.org/html/2610.11128#A1.Thmtheorem4)relates the reference surprisal to the probability of the prefix\.

###### Lemma A\.3\(Distribution of the accepted trajectory\)\.

For every prefix lengthtt, we havedt=vt−utd\_\{t\}=v\_\{t\}\-u\_\{t\}andQt​\(r\)=1−dtQ\_\{t\}\(r\)=1\-d\_\{t\}\. Moreover, every trajectoryy≠ry\\neq rwithQt​\(y\)\>0Q\_\{t\}\(y\)\>0is correct and has the formy=ht⊕zy=h\_\{t\}\\oplus z, and it satisfies

Qt​\(y\)=π⁡\(z∣ht\),Qt​\(y\)ν⁡\(y\)=pπ⁡\(ht\)\.Q\_\{t\}\(y\)=\\pi\(z\\mid h\_\{t\}\),\\qquad\\frac\{Q\_\{t\}\(y\)\}\{\\nu\(y\)\}=\\frac\{p\}\{\\pi\(h\_\{t\}\)\}\.These trajectories have total probabilitydtd\_\{t\}underQtQ\_\{t\}\.

###### Proof\.

The candidate trajectoryY~t=ht⊕Z\\widetilde\{Y\}\_\{t\}=h\_\{t\}\\oplus Zequalsrrexactly whenZ=atZ=a\_\{t\}, which has probabilityutu\_\{t\}\. Sincer∈Cr\\in C, the event\{Y~t∈C\}\\\{\\widetilde\{Y\}\_\{t\}\\in C\\\}is the disjoint union of\{Y~t=r\}\\\{\\widetilde\{Y\}\_\{t\}=r\\\}and\{Y~t∈C,Y~t≠r\}\\\{\\widetilde\{Y\}\_\{t\}\\in C,\\ \\widetilde\{Y\}\_\{t\}\\neq r\\\}\. Therefore,vt=ut\+dtv\_\{t\}=u\_\{t\}\+d\_\{t\}\. By Equation \([1](https://arxiv.org/html/2610.11128#S2.E1)\), the accepted trajectory equalsrrwhen the candidate trajectory is an exact copy or is incorrect\. These two events are disjoint and have total probabilityut\+\(1−vt\)=1−dtu\_\{t\}\+\(1\-v\_\{t\}\)=1\-d\_\{t\}\.

Fory≠ry\\neq r, the accepted trajectory equalsyyonly whenY~t=y\\widetilde\{Y\}\_\{t\}=yandy∈Cy\\in C\. Hence,Qt​\(y\)\>0Q\_\{t\}\(y\)\>0requiresy∈Cy\\in Candy=ht⊕zy=h\_\{t\}\\oplus zfor somezz, in which caseQt​\(y\)=π⁡\(z∣ht\)Q\_\{t\}\(y\)=\\pi\(z\\mid h\_\{t\}\)\. Autoregressive factorization givesπ⁡\(z∣ht\)=π⁡\(y\)/π⁡\(ht\)\\pi\(z\\mid h\_\{t\}\)=\\pi\(y\)/\\pi\(h\_\{t\}\), whereπ⁡\(ht\)≥π⁡\(r\)\>0\\pi\(h\_\{t\}\)\\geq\\pi\(r\)\>0\. Sinceν⁡\(y\)=π⁡\(y\)/p\\nu\(y\)=\\pi\(y\)/pfory∈Cy\\in C, the ratioQt​\(y\)/ν​\(y\)Q\_\{t\}\(y\)/\\nu\(y\)equalsp/π⁡\(ht\)p/\\pi\(h\_\{t\}\)\. Finally, these trajectories carry the remaining probability1−Qt​\(r\)=dt1\-Q\_\{t\}\(r\)=d\_\{t\}\. ∎

###### Lemma A\.4\(Monotonicity of the reference surprisal\)\.

For every prefix lengthtt,

S0−St=−log⁡π⁡\(ht\),0≤St≤S0\.S\_\{0\}\-S\_\{t\}=\-\\log\\pi\(h\_\{t\}\),\\qquad 0\\leq S\_\{t\}\\leq S\_\{0\}\.Moreover,StS\_\{t\}is nonincreasing intt\.

###### Proof\.

Autoregressive factorization givesπ⁡\(r\)=π⁡\(ht\)​π​\(at∣ht\)\\pi\(r\)=\\pi\(h\_\{t\}\)\\pi\(a\_\{t\}\\mid h\_\{t\}\)\. Taking negative logarithms givesS0=−log⁡π⁡\(ht\)\+StS\_\{0\}=\-\\log\\pi\(h\_\{t\}\)\+S\_\{t\}\. Sinceπ⁡\(at∣ht\)≤1\\pi\(a\_\{t\}\\mid h\_\{t\}\)\\leq 1andπ⁡\(ht\)≤1\\pi\(h\_\{t\}\)\\leq 1, bothStS\_\{t\}andS0−StS\_\{0\}\-S\_\{t\}are nonnegative\. Fors≤ts\\leq t, every trajectory that begins withhth\_\{t\}also begins withhsh\_\{s\}\. Therefore,π⁡\(ht\)≤π⁡\(hs\)\\pi\(h\_\{t\}\)\\leq\\pi\(h\_\{s\}\), which givesSt=S0\+log⁡π⁡\(ht\)≤SsS\_\{t\}=S\_\{0\}\+\\log\\pi\(h\_\{t\}\)\\leq S\_\{s\}\. ∎

With these two lemmas, the proof of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)reduces to a direct computation\.

###### Theorem1\(Distance to the ideal distribution, restated\)\.

Under the setup in Appendix[A\.1](https://arxiv.org/html/2610.11128#A1.SS1), for every prefix lengthtt,

DKL\(Qt∥ν\)=log⁡pπ⁡\(r\)⏟DKL\(δr∥ν\)−dtSt\+\(1−dt\)log\(1−dt\)\.D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)=\\underbrace\{\\log\\frac\{p\}\{\\pi\(r\)\}\}\_\{D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\}\-d\_\{t\}S\_\{t\}\+\(1\-d\_\{t\}\)\\log\(1\-d\_\{t\}\)\.Moreover, everyt∗∈arg⁡maxt​dt​Stt^\{\*\}\\in\\arg\\max\_\{t\}d\_\{t\}S\_\{t\}satisfiesDKL\(Qt∗∥ν\)≤mintDKL\(Qt∥ν\)\+1/eD\_\{\\mathrm\{KL\}\}\(Q\_\{t^\{\*\}\}\\\|\\nu\)\\leq\\min\_\{t\}D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)\+1/e\.

###### Proof\.

Writed=dtd=d\_\{t\}\. By Lemma[A\.3](https://arxiv.org/html/2610.11128#A1.Thmtheorem3),QtQ\_\{t\}assigns probability1−d1\-dtorr, whose likelihood ratio isQt​\(r\)/ν⁡\(r\)=\(1−d\)​p/π⁡\(r\)Q\_\{t\}\(r\)/\\nu\(r\)=\(1\-d\)p/\\pi\(r\)\. It assigns the remaining probabilityddto trajectories whose likelihood ratio isp/π⁡\(ht\)p/\\pi\(h\_\{t\}\)\. Therefore,

DKL\(Qt∥ν\)=\(1−d\)log\(1−d\)​pπ⁡\(r\)\+dlogpπ⁡\(ht\)\.D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)=\(1\-d\)\\log\\frac\{\(1\-d\)p\}\{\\pi\(r\)\}\+d\\log\\frac\{p\}\{\\pi\(h\_\{t\}\)\}\.Lemma[A\.4](https://arxiv.org/html/2610.11128#A1.Thmtheorem4)giveslog⁡π⁡\(ht\)=log⁡π⁡\(r\)\+St\\log\\pi\(h\_\{t\}\)=\\log\\pi\(r\)\+S\_\{t\}, and henced​log⁡\(p/π⁡\(ht\)\)=d​log⁡\(p/π⁡\(r\)\)−d​Std\\log\(p/\\pi\(h\_\{t\}\)\)=d\\log\(p/\\pi\(r\)\)\-dS\_\{t\}\. Collecting the two terms inlog⁡\(p/π⁡\(r\)\)\\log\(p/\\pi\(r\)\)proves the closed form\. Its first term equalsDKL\(δr∥ν\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)by Lemma[A\.2](https://arxiv.org/html/2610.11128#A1.Thmtheorem2)and does not depend ontt\.

For the second statement, writeDr=DKL\(δr∥ν\)D\_\{r\}=D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)andct=\(1−dt\)​log⁡\(1−dt\)c\_\{t\}=\(1\-d\_\{t\}\)\\log\(1\-d\_\{t\}\)\. Since−a​log⁡a∈\[0,1/e\]\-a\\log a\\in\[0,1/e\]fora∈\[0,1\]a\\in\[0,1\], we havect∈\[−1/e,0\]c\_\{t\}\\in\[\-1/e,0\]\. For every prefix lengthtt, the maximality ofdt∗​St∗d\_\{t^\{\*\}\}S\_\{t^\{\*\}\}gives

DKL\(Qt∗∥ν\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(Q\_\{t^\{\*\}\}\\\|\\nu\)=Dr−dt∗​St∗\+ct∗≤Dr−dt∗​St∗≤Dr−dt​St\\displaystyle=D\_\{r\}\-d\_\{t^\{\*\}\}S\_\{t^\{\*\}\}\+c\_\{t^\{\*\}\}\\leq D\_\{r\}\-d\_\{t^\{\*\}\}S\_\{t^\{\*\}\}\\leq D\_\{r\}\-d\_\{t\}S\_\{t\}=DKL\(Qt∥ν\)−ct≤DKL\(Qt∥ν\)\+1e\.\\displaystyle=D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)\-c\_\{t\}\\leq D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)\+\\frac\{1\}\{e\}\.Taking the minimum over the finitely many prefix lengths completes the proof\. ∎

Sincedt​St≥0d\_\{t\}S\_\{t\}\\geq 0and\(1−dt\)​log⁡\(1−dt\)≤0\(1\-d\_\{t\}\)\\log\(1\-d\_\{t\}\)\\leq 0, the closed form also givesDKL\(Qt∥ν\)≤DKL\(δr∥ν\)D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)\\leq D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)for every prefix lengthtt\. Therefore, a continuation with fallback never moves the distribution of the accepted trajectory farther from the ideal distribution than training on the reference\.

### A\.4Consequences of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)

We now give an equivalent reading of the closed form\. Corollary[A\.5](https://arxiv.org/html/2610.11128#A1.Thmtheorem5)rewrites the distance in terms of the reference content kept in the accepted trajectory\.

###### Corollary A\.5\(Reference content kept in the accepted trajectory\)\.

Leth¯\\bar\{h\}be the reference content kept in the accepted trajectoryY∼QtY\\sim Q\_\{t\}, which isrrwhenY=rY=rand the prefixhth\_\{t\}otherwise\. Then, for every prefix lengthtt,

DKL\(Qt∥ν\)=logp\+\(1−dt\)​S0\+dt​\(S0−St\)⏟𝔼⁡\[−log⁡π⁡\(h¯\)\]\+\(1−dt\)log\(1−dt\)\.D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)=\\log p\+\\underbrace\{\(1\-d\_\{t\}\)S\_\{0\}\+d\_\{t\}\(S\_\{0\}\-S\_\{t\}\)\}\_\{\\mathbb\{E\}\[\-\\log\\pi\(\\bar\{h\}\)\]\}\+\(1\-d\_\{t\}\)\\log\(1\-d\_\{t\}\)\.

###### Proof\.

Lemma[A\.2](https://arxiv.org/html/2610.11128#A1.Thmtheorem2)giveslog⁡\(p/π⁡\(r\)\)=log⁡p\+S0\\log\(p/\\pi\(r\)\)=\\log p\+S\_\{0\}\. Substituting this expression andS0=\(1−dt\)​S0\+dt​S0S\_\{0\}=\(1\-d\_\{t\}\)S\_\{0\}\+d\_\{t\}S\_\{0\}into the closed form of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)gives the display\. By Lemma[A\.3](https://arxiv.org/html/2610.11128#A1.Thmtheorem3),h¯=r\\bar\{h\}=rwith probability1−dt1\-d\_\{t\}andh¯=ht\\bar\{h\}=h\_\{t\}with probabilitydtd\_\{t\}\. Since−log⁡π⁡\(r\)=S0\-\\log\\pi\(r\)=S\_\{0\}and−log⁡π⁡\(ht\)=S0−St\-\\log\\pi\(h\_\{t\}\)=S\_\{0\}\-S\_\{t\}by Lemma[A\.4](https://arxiv.org/html/2610.11128#A1.Thmtheorem4), the middle two terms equal𝔼⁡\[−log⁡π⁡\(h¯\)\]\\mathbb\{E\}\[\-\\log\\pi\(\\bar\{h\}\)\]\. ∎

In other words, apart fromlog⁡p\\log p, which does not depend on the prefix, and a term in\[−1/e,0\]\[\-1/e,0\], the distance to the ideal distribution is the expected surprisal of the reference content that the model has not replaced\. Both extremes of the prefix length keep nearly all of the reference surprisalS0S\_\{0\}\. A prefix covering almost the entire reference keeps it through the prefix, while an empty prefix on a hard question keeps it through fallback\.

### A\.5A Finite Example

This example shows that neither the success probability of the continuation nor the successful replacement probability alone determines which prefix gives the smaller distance\. Consider five complete trajectories whose displayed strings omit a shared deterministic termination token,𝙷⊕𝚛\\mathtt\{H\}\\oplus\\mathtt\{r\},𝙷⊕𝚜\\mathtt\{H\}\\oplus\\mathtt\{s\},𝙷⊕𝚏\\mathtt\{H\}\\oplus\\mathtt\{f\},𝙾⊕𝚝\\mathtt\{O\}\\oplus\\mathtt\{t\}, and𝙾⊕𝚐\\mathtt\{O\}\\oplus\\mathtt\{g\}, with respective probabilities0\.080\.08,0\.010\.01,0\.010\.01,0\.200\.20, and0\.700\.70\. The correct trajectories are𝙷⊕𝚛\\mathtt\{H\}\\oplus\\mathtt\{r\},𝙷⊕𝚜\\mathtt\{H\}\\oplus\\mathtt\{s\}, and𝙾⊕𝚝\\mathtt\{O\}\\oplus\\mathtt\{t\}, and the reference is𝙷⊕𝚛\\mathtt\{H\}\\oplus\\mathtt\{r\}\. The first tokens are𝙷\\mathtt\{H\}and𝙾\\mathtt\{O\}, with probabilities0\.100\.10and0\.900\.90, and the second\-token probabilities are the corresponding conditional probabilities\. These probabilities define a finite autoregressive distribution\.

Training on the reference givesDKL\(δr∥ν\)=1\.287854D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)=1\.287854\. Supplying𝙷\\mathtt\{H\}increasesvtv\_\{t\}while making an exact copy more likely\. In this setting, both the successful replacement probabilitydtd\_\{t\}and the reference surprisalStS\_\{t\}decrease\. Therefore, the empty prefix gives the largerdt​Std\_\{t\}S\_\{t\}and the smaller distance despite the lower success probability of its continuation\.

To isolate the role of the reference surprisal, keep the three probabilities under𝙷\\mathtt\{H\}unchanged and assign probabilities0\.040\.04and0\.860\.86to𝙾⊕𝚝\\mathtt\{O\}\\oplus\\mathtt\{t\}and𝙾⊕𝚐\\mathtt\{O\}\\oplus\\mathtt\{g\}, respectively\. The resulting comparison is

Training on the reference now givesDKL\(δr∥ν\)=0\.485508D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)=0\.485508\. Here supplying𝙷\\mathtt\{H\}increases bothvtv\_\{t\}anddtd\_\{t\}\. However, the decrease in the reference surprisal outweighs the increase indtd\_\{t\}\. Therefore, the empty prefix still gives the largerdt​Std\_\{t\}S\_\{t\}and the smaller distance\. Thus, even the successful replacement probability alone can favor a different prefix from the one preferred by Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)\. Both examples concern a fixed model and illustrate why the choice of prefix should account fordtd\_\{t\}andStS\_\{t\}together\.

## Appendix BProofs for Learning the Prefix Selector

This appendix proves Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)and analyzes the prefix selector during training\. Appendix[B\.1](https://arxiv.org/html/2610.11128#A2.SS1)fixes the setup and derives the properties of the selector feedback, and Appendix[B\.2](https://arxiv.org/html/2610.11128#A2.SS2)proves Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)\. Appendix[B\.3](https://arxiv.org/html/2610.11128#A2.SS3)examines the effects of sharing the selector across questions and of randomizing the guidance level\. Finally, Appendix[B\.4](https://arxiv.org/html/2610.11128#A2.SS4)derives the expected update of the selector in the GRPO application as the model and the questions change, and Appendix[B\.5](https://arxiv.org/html/2610.11128#A2.SS5)bounds the accumulated sampling noise\.

### B\.1Setup and Selector Feedback

Fix the current model, the references, and𝒟guide\\mathcal\{D\}\_\{\\mathrm\{guide\}\}as in Section[3\.3](https://arxiv.org/html/2610.11128#S3.SS3)\. We assume that every question satisfies the setup of Appendix[A\.1](https://arxiv.org/html/2610.11128#A1.SS1), which includes0<πθold​\(rx∣x\)<10<\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(r\_\{x\}\\mid x\)<1and hence0<S0,x<∞0<S\_\{0,x\}<\\infty\. For a questionxx, we writeνx\\nu\_\{x\}for its ideal distribution\. For a guidance levelα\\alpha, we writeQx,αQ\_\{x,\\alpha\},dx,αd\_\{x,\\alpha\}, andSx,αS\_\{x,\\alpha\}for the distribution of the accepted trajectory, the successful replacement probability, and the reference surprisal at the prefix lengthtx​\(α\)t\_\{x\}\(\\alpha\)\. Heretx​\(α\)t\_\{x\}\(\\alpha\)is the actual prefix length, including the boundary adjustment in Section[4\.1](https://arxiv.org/html/2610.11128#S4.SS1)\. Therefore, the nominal levelα\\alphacan differ from the realized ratioSx,α/S0,xS\_\{x,\\alpha\}/S\_\{0,x\}\. We first collect the properties of the selector feedback for a single question, for which we suppressxxas in Appendix[A](https://arxiv.org/html/2610.11128#A1)\.

###### Lemma B\.1\(Selector feedback\)\.

For every prefix lengthtt, the selector feedback satisfiesWt∈\[0,1\]W\_\{t\}\\in\[0,1\]and𝔼⁡\[Wt\]=dt​St/S0\\mathbb\{E\}\[W\_\{t\}\]=d\_\{t\}S\_\{t\}/S\_\{0\}\. Moreover, withct=\(1−dt\)​log⁡\(1−dt\)c\_\{t\}=\(1\-d\_\{t\}\)\\log\(1\-d\_\{t\}\),

DKL\(Qt∥ν\)S0=DKL\(δr∥ν\)S0−𝔼⁡\[Wt\]\+ctS0,ctS0∈\[−1e​S0,0\],\\frac\{D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)\}\{S\_\{0\}\}=\\frac\{D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\}\{S\_\{0\}\}\-\\mathbb\{E\}\[W\_\{t\}\]\+\\frac\{c\_\{t\}\}\{S\_\{0\}\},\\qquad\\frac\{c\_\{t\}\}\{S\_\{0\}\}\\in\\left\[\-\\frac\{1\}\{eS\_\{0\}\},0\\right\],\(14\)and the normalized distances satisfy0≤DKL\(Qt∥ν\)/S0≤DKL\(δr∥ν\)/S0≤10\\leq D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)/S\_\{0\}\\leq D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)/S\_\{0\}\\leq 1\.

###### Proof\.

The indicator in Equation \([7](https://arxiv.org/html/2610.11128#S3.E7)\) has expectationdtd\_\{t\}by Equation \([4](https://arxiv.org/html/2610.11128#S3.E4)\)\. Lemma[A\.4](https://arxiv.org/html/2610.11128#A1.Thmtheorem4)gives0≤St≤S00\\leq S\_\{t\}\\leq S\_\{0\}, and henceWt∈\[0,1\]W\_\{t\}\\in\[0,1\]and𝔼⁡\[Wt\]=dt​St/S0\\mathbb\{E\}\[W\_\{t\}\]=d\_\{t\}S\_\{t\}/S\_\{0\}, which proves Equation \([8](https://arxiv.org/html/2610.11128#S3.E8)\)\. Dividing the closed form of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)byS0S\_\{0\}gives Equation \([14](https://arxiv.org/html/2610.11128#A2.E14)\), and the range ofctc\_\{t\}follows from−a​log⁡a∈\[0,1/e\]\-a\\log a\\in\[0,1/e\]fora∈\[0,1\]a\\in\[0,1\]\. Finally, since𝔼⁡\[Wt\]≥0\\mathbb\{E\}\[W\_\{t\}\]\\geq 0andct≤0c\_\{t\}\\leq 0, Equation \([14](https://arxiv.org/html/2610.11128#A2.E14)\) givesDKL\(Qt∥ν\)≤DKL\(δr∥ν\)D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}\\\|\\nu\)\\leq D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\), and Lemma[A\.2](https://arxiv.org/html/2610.11128#A1.Thmtheorem2)gives0≤DKL\(δr∥ν\)≤S00\\leq D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\\leq S\_\{0\}\. ∎

Thus, a larger expected feedback corresponds to a smaller normalized distance, up to the bounded last term\. Since normalization dividesdt​Std\_\{t\}S\_\{t\}at every prefix of the same reference by the sameS0S\_\{0\}, it preserves the ranking ofdt​Std\_\{t\}S\_\{t\}across these prefixes\. Because of the bounded last term, this ranking need not equal the ranking of the exact distances\. Across questions, normalization expresses each distance relative to the reference surprisal of the empty prefix\.

The ablation without surprisal weighting in Section[5\.3](https://arxiv.org/html/2610.11128#S5.SS3)replacesWtW\_\{t\}with the indicator of successful replacement, whose expectation isdtd\_\{t\}\. Since this indicator still excludes exact copies of the reference, comparing it withWtW\_\{t\}isolates the effect of weighting a successful replacement by the reference surprisal\.

For the rest of this appendix, we write

gx​\(α\)=𝔼⁡\[Wx,α\]=dx,α​Sx,αS0,x∈\[0,1\]g\_\{x\}\(\\alpha\)=\\mathbb\{E\}\[W\_\{x,\\alpha\}\]=\\frac\{d\_\{x,\\alpha\}S\_\{x,\\alpha\}\}\{S\_\{0,x\}\}\\in\[0,1\]for the expected selector feedback of questionxxat levelα\\alpha, where the equality follows from Lemma[B\.1](https://arxiv.org/html/2610.11128#A2.Thmtheorem1)\.

### B\.2Proof of Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)

Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)measures a selector by its average normalized distance and compares it with training on the reference\. Written out, these distances are

ℒ⁡\(ϕ\)=𝔼X∼𝒟guide,α∼ρϕ​\[DKL\(QX,α∥νX\)S0,X\],ℒref=𝔼X∼𝒟guide​\[DKL\(δrX∥νX\)S0,X\]\.\\mathcal\{L\}\(\\phi\)=\\mathbb\{E\}\_\{X\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\},\\,\\alpha\\sim\\rho\_\{\\phi\}\}\\left\[\\frac\{D\_\{\\mathrm\{KL\}\}\(Q\_\{X,\\alpha\}\\\|\\nu\_\{X\}\)\}\{S\_\{0,X\}\}\\right\],\\qquad\\mathcal\{L\}\_\{\\mathrm\{ref\}\}=\\mathbb\{E\}\_\{X\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\}\}\\left\[\\frac\{D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\_\{X\}\}\\\|\\nu\_\{X\}\)\}\{S\_\{0,X\}\}\\right\]\.By Lemma[B\.1](https://arxiv.org/html/2610.11128#A2.Thmtheorem1)andgx​\(α\)∈\[0,1\]g\_\{x\}\(\\alpha\)\\in\[0,1\], the quantitiesℒ⁡\(ϕ\)\\mathcal\{L\}\(\\phi\),ℒref\\mathcal\{L\}\_\{\\mathrm\{ref\}\}, andJ⁡\(ϕ\)J\(\\phi\)lie in\[0,1\]\[0,1\]\.

###### Theorem2\(Learning the prefix selector from feedback, restated\)\.

Fix the current model, the references, and𝒟guide\\mathcal\{D\}\_\{\\mathrm\{guide\}\}, where every question satisfies the setup of Section[3\.1](https://arxiv.org/html/2610.11128#S3.SS1)\. Letℒ⁡\(ϕ\)\\mathcal\{L\}\(\\phi\)andℒref\\mathcal\{L\}\_\{\\mathrm\{ref\}\}be the expectations ofDKL\(QX,α∥νX\)/S0,XD\_\{\\mathrm\{KL\}\}\(Q\_\{X,\\alpha\}\\\|\\nu\_\{X\}\)/S\_\{0,X\}andDKL\(δrX∥νX\)/S0,XD\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\_\{X\}\}\\\|\\nu\_\{X\}\)/S\_\{0,X\}overX∼𝒟guideX\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\}andα∼ρϕ\\alpha\\sim\\rho\_\{\\phi\}\. Then

J⁡\(ϕ\)≤ℒref−ℒ⁡\(ϕ\)≤J⁡\(ϕ\)\+𝔼X​\[1e​S0,X\]\.J\(\\phi\)\\leq\\mathcal\{L\}\_\{\\mathrm\{ref\}\}\-\\mathcal\{L\}\(\\phi\)\\leq J\(\\phi\)\+\\mathbb\{E\}\_\{X\}\\left\[\\frac\{1\}\{eS\_\{0,X\}\}\\right\]\.Moreover, for a differentiable prefix selector with positive probabilities and a baselinebbfixed before sampling,

𝔼⁡\[\(WX,α−b\)​∇ϕ​log⁡ρϕ​\(α\)\]=∇ϕJ​\(ϕ\)\.\\mathbb\{E\}\\left\[\(W\_\{X,\\alpha\}\-b\)\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\)\\right\]=\\nabla\_\{\\phi\}J\(\\phi\)\.

###### Proof\.

For the first statement, averaging Equation \([14](https://arxiv.org/html/2610.11128#A2.E14)\) overX∼𝒟guideX\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\}andα∼ρϕ\\alpha\\sim\\rho\_\{\\phi\}gives

ℒ⁡\(ϕ\)=ℒref−J⁡\(ϕ\)\+ϵ⁡\(ϕ\),−𝔼X​1e​S0,X≤ϵ⁡\(ϕ\)≤0,\\mathcal\{L\}\(\\phi\)=\\mathcal\{L\}\_\{\\mathrm\{ref\}\}\-J\(\\phi\)\+\\epsilon\(\\phi\),\\qquad\-\\mathbb\{E\}\_\{X\}\\frac\{1\}\{eS\_\{0,X\}\}\\leq\\epsilon\(\\phi\)\\leq 0,\(15\)whereϵ⁡\(ϕ\)=𝔼⁡\[cX,α/S0,X\]\\epsilon\(\\phi\)=\\mathbb\{E\}\[c\_\{X,\\alpha\}/S\_\{0,X\}\]andcx,αc\_\{x,\\alpha\}is the last term of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)at the prefix lengthtx​\(α\)t\_\{x\}\(\\alpha\)\. Lemma[B\.1](https://arxiv.org/html/2610.11128#A2.Thmtheorem1)bounds eachcx,α/S0,xc\_\{x,\\alpha\}/S\_\{0,x\}, and rearranging Equation \([15](https://arxiv.org/html/2610.11128#A2.E15)\) gives the first statement\.

For the second statement, the level is sampled independently of the question\. Condition onX=xX=x\. Sinceα∼ρϕ\\alpha\\sim\\rho\_\{\\phi\}and𝔼⁡\[Wx,α∣α\]=gx​\(α\)\\mathbb\{E\}\[W\_\{x,\\alpha\}\\mid\\alpha\]=g\_\{x\}\(\\alpha\),

𝔼⁡\[\(Wx,α−b\)​∇ϕ​log⁡ρϕ​\(α\)∣X=x\]=∑α∈𝒜\(gx​\(α\)−b\)​∇ϕρϕ​\(α\)=∑α∈𝒜gx​\(α\)​∇ϕρϕ​\(α\)\.\\mathbb\{E\}\\left\[\(W\_\{x,\\alpha\}\-b\)\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\)\\mid X=x\\right\]=\\sum\_\{\\alpha\\in\\mathcal\{A\}\}\(g\_\{x\}\(\\alpha\)\-b\)\\nabla\_\{\\phi\}\\rho\_\{\\phi\}\(\\alpha\)=\\sum\_\{\\alpha\\in\\mathcal\{A\}\}g\_\{x\}\(\\alpha\)\\nabla\_\{\\phi\}\\rho\_\{\\phi\}\(\\alpha\)\.The baseline term vanishes since∑α∇ϕρϕ​\(α\)=∇ϕ1=0\\sum\_\{\\alpha\}\\nabla\_\{\\phi\}\\rho\_\{\\phi\}\(\\alpha\)=\\nabla\_\{\\phi\}1=0\. Defineμα=𝔼X∼𝒟guide​\[gX​\(α\)\]∈\[0,1\]\\mu\_\{\\alpha\}=\\mathbb\{E\}\_\{X\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\}\}\[g\_\{X\}\(\\alpha\)\]\\in\[0,1\], which does not depend onϕ\\phi\. Then

J⁡\(ϕ\)=∑α∈𝒜ρϕ​\(α\)​μα,∇ϕJ​\(ϕ\)=∑α∈𝒜μα​∇ϕρϕ​\(α\),J\(\\phi\)=\\sum\_\{\\alpha\\in\\mathcal\{A\}\}\\rho\_\{\\phi\}\(\\alpha\)\\mu\_\{\\alpha\},\\qquad\\nabla\_\{\\phi\}J\(\\phi\)=\\sum\_\{\\alpha\\in\\mathcal\{A\}\}\\mu\_\{\\alpha\}\\nabla\_\{\\phi\}\\rho\_\{\\phi\}\(\\alpha\),where differentiation acts on a finite sum with fixed bounded coefficients\. Averaging the conditional identity overX∼𝒟guideX\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\}\}gives the same expression, which proves the second statement\. ∎

The first statement leads to the following guarantee for a selector that maximizesJJ\.

###### Corollary B\.2\(Maximizing the objective\)\.

IfJ⁡\(ϕ∗\)≥J⁡\(ϕ\)J\(\\phi^\{\*\}\)\\geq J\(\\phi\), thenℒ⁡\(ϕ∗\)≤ℒ⁡\(ϕ\)\+𝔼X​\[1/\(e​S0,X\)\]\\mathcal\{L\}\(\\phi^\{\*\}\)\\leq\\mathcal\{L\}\(\\phi\)\+\\mathbb\{E\}\_\{X\}\[1/\(eS\_\{0,X\}\)\]\.

###### Proof\.

By Equation \([15](https://arxiv.org/html/2610.11128#A2.E15)\),J⁡\(ϕ∗\)≥J⁡\(ϕ\)J\(\\phi^\{\*\}\)\\geq J\(\\phi\)gives

ℒ⁡\(ϕ∗\)≤ℒref−J⁡\(ϕ∗\)≤ℒref−J⁡\(ϕ\)≤ℒ⁡\(ϕ\)\+𝔼X​1e​S0,X\.∎\\mathcal\{L\}\(\\phi^\{\*\}\)\\leq\\mathcal\{L\}\_\{\\mathrm\{ref\}\}\-J\(\\phi^\{\*\}\)\\leq\\mathcal\{L\}\_\{\\mathrm\{ref\}\}\-J\(\\phi\)\\leq\\mathcal\{L\}\(\\phi\)\+\\mathbb\{E\}\_\{X\}\\frac\{1\}\{eS\_\{0,X\}\}\.\\qed

Thus, a larger objective corresponds to a smaller average normalized distance, up to a bounded correction\. Since the correctionϵ⁡\(ϕ\)\\epsilon\(\\phi\)can change with the selector,JJandℒ\\mathcal\{L\}can rank selectors differently within this bound\. The bound is informative only when𝔼X​\[1/S0,X\]\\mathbb\{E\}\_\{X\}\[1/S\_\{0,X\}\]is small, and it can be large when some reference surprisals are close to zero\.

In Section[4\.2](https://arxiv.org/html/2610.11128#S4.SS2), the baseline is a running average of past feedback\. For such a baseline, the same argument applies after conditioning on the history and the current question\. Given this information, the baseline is fixed, the level still followsρϕ\\rho\_\{\\phi\}, and its feedback still has conditional meangx​\(α\)g\_\{x\}\(\\alpha\)\. Moreover, the baseline update in Appendix[B\.4](https://arxiv.org/html/2610.11128#A2.SS4)keepsb∈\[0,1\]b\\in\[0,1\], which ensures integrability\.

### B\.3Sharing and Randomizing Guidance Levels

The prefix selector shares one distribution over guidance levels across questions and samples a level for each continuation\. We now examine both choices\. Proposition[B\.3](https://arxiv.org/html/2610.11128#A2.Thmtheorem3)bounds the cost of sharing within the objectiveJJ\. Proposition[B\.4](https://arxiv.org/html/2610.11128#A2.Thmtheorem4)then compares the distance after randomizing the level with the average over levels inJJ\.

For any distributionρ\\rhoover𝒜\\mathcal\{A\}, writeJ⁡\(ρ\)=∑αρ⁡\(α\)​μαJ\(\\rho\)=\\sum\_\{\\alpha\}\\rho\(\\alpha\)\\mu\_\{\\alpha\}withμα\\mu\_\{\\alpha\}from the proof of Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2), which givesJ⁡\(ρϕ\)=J⁡\(ϕ\)J\(\\rho\_\{\\phi\}\)=J\(\\phi\)\. The best shared distribution has valueJshared∗=maxα⁡μαJ\_\{\\mathrm\{shared\}\}^\{\*\}=\\max\_\{\\alpha\}\\mu\_\{\\alpha\}\. In contrast, an oracle that knowsgxg\_\{x\}for every question and chooses a level for each question separately has valueJindividual∗=𝔼X​maxα​gX​\(α\)J\_\{\\mathrm\{individual\}\}^\{\*\}=\\mathbb\{E\}\_\{X\}\\max\_\{\\alpha\}g\_\{X\}\(\\alpha\)\.

###### Proposition B\.3\(Cost of sharing\)\.

For every distributionρ\\rhoover𝒜\\mathcal\{A\},

Jindividual∗−J⁡\(ρ\)=\(Jindividual∗−Jshared∗\)\+\(Jshared∗−J⁡\(ρ\)\),J\_\{\\mathrm\{individual\}\}^\{\*\}\-J\(\\rho\)=\\bigl\(J\_\{\\mathrm\{individual\}\}^\{\*\}\-J\_\{\\mathrm\{shared\}\}^\{\*\}\\bigr\)\+\\bigl\(J\_\{\\mathrm\{shared\}\}^\{\*\}\-J\(\\rho\)\\bigr\),\(16\)where both terms are nonnegative\. Moreover, the prefix selector in Equation \([11](https://arxiv.org/html/2610.11128#S4.E11)\) satisfies

supϕJ⁡\(ϕ\)=\(1−εsel\)​maxα​μα\+εsel\|𝒜\|​∑αμα\.\\sup\_\{\\phi\}J\(\\phi\)=\(1\-\\varepsilon\_\{\\mathrm\{sel\}\}\)\\max\_\{\\alpha\}\\mu\_\{\\alpha\}\+\\frac\{\\varepsilon\_\{\\mathrm\{sel\}\}\}\{\|\\mathcal\{A\}\|\}\\sum\_\{\\alpha\}\\mu\_\{\\alpha\}\.

###### Proof\.

The decomposition is an identity\. The first term is nonnegative sincemaxα⁡𝔼X​\[gX​\(α\)\]≤𝔼X​maxα​gX​\(α\)\\max\_\{\\alpha\}\\mathbb\{E\}\_\{X\}\[g\_\{X\}\(\\alpha\)\]\\leq\\mathbb\{E\}\_\{X\}\\max\_\{\\alpha\}g\_\{X\}\(\\alpha\)\. The second term is nonnegative sinceJ⁡\(ρ\)J\(\\rho\)is a convex combination of the valuesμα\\mu\_\{\\alpha\}\. For the selector in Equation \([11](https://arxiv.org/html/2610.11128#S4.E11)\), writeqϕq\_\{\\phi\}for its softmax component\. ThenJ⁡\(ϕ\)=\(1−εsel\)​∑αqϕ​\(α\)​μα\+\(εsel/\|𝒜\|\)​∑αμαJ\(\\phi\)=\(1\-\\varepsilon\_\{\\mathrm\{sel\}\}\)\\sum\_\{\\alpha\}q\_\{\\phi\}\(\\alpha\)\\mu\_\{\\alpha\}\+\(\\varepsilon\_\{\\mathrm\{sel\}\}/\|\\mathcal\{A\}\|\)\\sum\_\{\\alpha\}\\mu\_\{\\alpha\}\. The first sum is at mostmaxα⁡μα\\max\_\{\\alpha\}\\mu\_\{\\alpha\}, and it approaches this value asqϕq\_\{\\phi\}concentrates on a maximizing level\. ∎

The first term in Equation \([16](https://arxiv.org/html/2610.11128#A2.E16)\) measures the cost of using the same distribution over levels for every question\. It is zero if one level maximizesgxg\_\{x\}for almost every question\. The second term measures the gap between the current selector and the best shared distribution\. If allμα\\mu\_\{\\alpha\}are equal, every shared distribution has the same objective even when individual questions prefer different levels\. Moreover, the uniform component of the selector has an explicit cost in the current objective, while it keeps every level observed as the model changes\.

Through Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2), the objectiveJJcorresponds to the average of the normalized distance over the selected levels\. However, sampling a level and then making one continuation produces a single distribution of the accepted trajectory, which mixes the distributions for different levels\. For a fixed questionxxand a distributionρ\\rhoover𝒜\\mathcal\{A\}, this mixture isQx,ρ=∑αρ⁡\(α\)​Qx,αQ\_\{x,\\rho\}=\\sum\_\{\\alpha\}\\rho\(\\alpha\)Q\_\{x,\\alpha\}\. LetIxI\_\{x\}be the mutual information between the sampled level and the accepted trajectory, and letℋ\(ρ\)=−∑αρ\(α\)logρ\(α\)\\mathcal\{H\}\(\\rho\)=\-\\sum\_\{\\alpha\}\\rho\(\\alpha\)\\log\\rho\(\\alpha\)be the entropy ofρ\\rho\.

###### Proposition B\.4\(Randomizing the guidance level\)\.

For every questionxxand distributionρ\\rhoover𝒜\\mathcal\{A\},

DKL\(Qx,ρ∥νx\)=∑αρ\(α\)DKL\(Qx,α∥νx\)−Ix,D\_\{\\mathrm\{KL\}\}\(Q\_\{x,\\rho\}\\\|\\nu\_\{x\}\)=\\sum\_\{\\alpha\}\\rho\(\\alpha\)D\_\{\\mathrm\{KL\}\}\(Q\_\{x,\\alpha\}\\\|\\nu\_\{x\}\)\-I\_\{x\},where0≤Ix≤ℋ⁡\(ρ\)0\\leq I\_\{x\}\\leq\\mathcal\{H\}\(\\rho\)\. Consequently, withjx​\(ρ\)=∑αρ⁡\(α\)​gx​\(α\)j\_\{x\}\(\\rho\)=\\sum\_\{\\alpha\}\\rho\(\\alpha\)g\_\{x\}\(\\alpha\),

DKL\(δrx∥νx\)S0,x−jx​\(ρ\)−1/e\+ℋ⁡\(ρ\)S0,x≤DKL\(Qx,ρ∥νx\)S0,x≤DKL\(δrx∥νx\)S0,x−jx​\(ρ\)\.\\frac\{D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\_\{x\}\}\\\|\\nu\_\{x\}\)\}\{S\_\{0,x\}\}\-j\_\{x\}\(\\rho\)\-\\frac\{1/e\+\\mathcal\{H\}\(\\rho\)\}\{S\_\{0,x\}\}\\leq\\frac\{D\_\{\\mathrm\{KL\}\}\(Q\_\{x,\\rho\}\\\|\\nu\_\{x\}\)\}\{S\_\{0,x\}\}\\leq\\frac\{D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\_\{x\}\}\\\|\\nu\_\{x\}\)\}\{S\_\{0,x\}\}\-j\_\{x\}\(\\rho\)\.

###### Proof\.

EveryQx,αQ\_\{x,\\alpha\}has finite distance by Lemma[B\.1](https://arxiv.org/html/2610.11128#A2.Thmtheorem1)\. Splitting the log probability ratio throughQx,ρQ\_\{x,\\rho\}and summing over trajectories with positive probability gives

∑αρ\(α\)DKL\(Qx,α∥νx\)\\displaystyle\\sum\_\{\\alpha\}\\rho\(\\alpha\)D\_\{\\mathrm\{KL\}\}\(Q\_\{x,\\alpha\}\\\|\\nu\_\{x\}\)=∑α,yρ⁡\(α\)​Qx,α​\(y\)​\[log⁡Qx,α​\(y\)Qx,ρ​\(y\)\+log⁡Qx,ρ​\(y\)νx​\(y\)\]\\displaystyle=\\sum\_\{\\alpha,y\}\\rho\(\\alpha\)Q\_\{x,\\alpha\}\(y\)\\left\[\\log\\frac\{Q\_\{x,\\alpha\}\(y\)\}\{Q\_\{x,\\rho\}\(y\)\}\+\\log\\frac\{Q\_\{x,\\rho\}\(y\)\}\{\\nu\_\{x\}\(y\)\}\\right\]=Ix\+DKL\(Qx,ρ∥νx\),\\displaystyle=I\_\{x\}\+D\_\{\\mathrm\{KL\}\}\(Q\_\{x,\\rho\}\\\|\\nu\_\{x\}\),since∑αρ\(α\)DKL\(Qx,α∥Qx,ρ\)\\sum\_\{\\alpha\}\\rho\(\\alpha\)D\_\{\\mathrm\{KL\}\}\(Q\_\{x,\\alpha\}\\\|Q\_\{x,\\rho\}\)equals the mutual informationIxI\_\{x\}\. The mutual information is nonnegative and at most the entropy of the level, which gives0≤Ix≤ℋ⁡\(ρ\)0\\leq I\_\{x\}\\leq\\mathcal\{H\}\(\\rho\)\. For the bounds, divide the identity byS0,xS\_\{0,x\}and apply Equation \([14](https://arxiv.org/html/2610.11128#A2.E14)\) to each level\. The upper bound usesIx≥0I\_\{x\}\\geq 0andcx,α≤0c\_\{x,\\alpha\}\\leq 0, and the lower bound usesIx≤ℋ⁡\(ρ\)I\_\{x\}\\leq\\mathcal\{H\}\(\\rho\)andcx,α≥−1/ec\_\{x,\\alpha\}\\geq\-1/e\. ∎

The mutual information is zero when all distributions with positive weight coincide\. Otherwise, randomizing the level gives a smaller distance than the average over levels\. This construction returns one accepted trajectory from one continuation, and Appendix[C\.2](https://arxiv.org/html/2610.11128#A3.SS2)analyzes acceptance after multiple continuations\.

### B\.4Selector Updates in the GRPO Application

Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)fixes the model, the references, and𝒟guide\\mathcal\{D\}\_\{\\mathrm\{guide\}\}\. We now specify𝒟guide\\mathcal\{D\}\_\{\\mathrm\{guide\}\}and the baseline in the GRPO application of Section[4\.3](https://arxiv.org/html/2610.11128#S4.SS3), and then derive the expected update of the selector for a batch and across training iterations\.

Questions selected for reference guidance\.Fix the current model and a referencerxr\_\{x\}for each question\. Since theGGresponses of a group are sampled independently, the group is an all\-failure group with probabilityfx=\(1−pθold​\(x\)\)Gf\_\{x\}=\(1\-p\_\{\\theta\_\{\\mathrm\{old\}\}\}\(x\)\)^\{G\}\. ForX∼𝒟X\\sim\\mathcal\{D\}, defineζ=𝔼𝒟​\[fX\]\\zeta=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[f\_\{X\}\]\. Whenζ\>0\\zeta\>0, the questions that receive reference guidance follow

𝒟guide​\(x\)=𝒟⁡\(x\)​fxζ\.\\mathcal\{D\}\_\{\\mathrm\{guide\}\}\(x\)=\\frac\{\\mathcal\{D\}\(x\)f\_\{x\}\}\{\\zeta\}\.\(17\)Given the question, the guidance levels and continuations are sampled independently of the group\. Therefore, restricting guidance to questions with an all\-failure group changes their relative weights while preserving𝔼⁡\[Wx,α\]=gx​\(α\)\\mathbb\{E\}\[W\_\{x,\\alpha\}\]=g\_\{x\}\(\\alpha\)for each question and level\. Ifζ=0\\zeta=0, no question receives guidance and the selector update is zero almost surely\.

Baseline and update order\.The selector starts withϕ=0\\phi=0andb=0b=0\. Every term in Equation \([12](https://arxiv.org/html/2610.11128#S4.E12)\) uses the same selector parameters and baseline, which are fixed before the batch is collected\. After a nonempty batch of continuations, the baseline is updated by

b←\(1−β\)​b\+β\|𝒰\|​∑j∈𝒰Wj,b\\leftarrow\(1\-\\beta\)b\+\\frac\{\\beta\}\{\|\\mathcal\{U\}\|\}\\sum\_\{j\\in\\mathcal\{U\}\}W\_\{j\},\(18\)where0<β≤10<\\beta\\leq 1\. Since this update is a convex combination of values in\[0,1\]\[0,1\], induction givesb∈\[0,1\]b\\in\[0,1\]at every iteration\. Centering the updated logits by subtracting their mean leaves the selection probabilities unchanged\. An empty batch of continuations leavesϕ\\phiandbbunchanged\. Table[3](https://arxiv.org/html/2610.11128#A5.T3)specifies the numerical hyperparameters\. Reference surprisals, verification outcomes, and model parameters are fixed during the selector update\. This update precedes the policy update, and both use the continuations generated by the same current model\. All continuations supply feedback, includingKKzero feedback values when no continuation gives a successful replacement and ARG falls back to the reference\.

Expected batch update\.LetΨ\\Psidenote the summed direction in Equation \([12](https://arxiv.org/html/2610.11128#S4.E12)\) before multiplication by the step size,

Ψ=∑j∈𝒰\(Wj−b\)​∇ϕ​log⁡ρϕ​\(αj\),\\Psi=\\sum\_\{j\\in\\mathcal\{U\}\}\(W\_\{j\}\-b\)\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\_\{j\}\),withΨ=0\\Psi=0when𝒰\\mathcal\{U\}is empty\.

###### Proposition B\.5\(Expected batch update\)\.

Consider a batch ofnnquestion positions, each with marginal distribution𝒟\\mathcal\{D\}\. Suppose that the model, the selector, and the baseline are fixed during collection, and that every all\-failure group receivesKKcontinuations whose levels and feedback follow the laws above\. Then

𝔼⁡\[Ψ\]=n​K​ζ​∇ϕJ​\(ϕ\),\\mathbb\{E\}\[\\Psi\]=nK\\zeta\\nabla\_\{\\phi\}J\(\\phi\),\(19\)where the right\-hand side is zero whenζ=0\\zeta=0\.

###### Proof\.

WriteΨ\\Psias a sum over question positionsiiand continuationsℓ=1,…,K\\ell=1,\\ldots,Kof the terms𝟏​\{group​i​is all\-failure\}​\(Wi​ℓ−b\)​∇ϕ​log⁡ρϕ​\(αi​ℓ\)\\mathbf\{1\}\\\{\\text\{group \}i\\text\{ is all\-failure\}\\\}\(W\_\{i\\ell\}\-b\)\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\_\{i\\ell\}\)\. Conditional on the questionxxat positioniiand on its all\-failure event, each term has expectation∑αgx​\(α\)​∇ϕρϕ​\(α\)\\sum\_\{\\alpha\}g\_\{x\}\(\\alpha\)\\nabla\_\{\\phi\}\\rho\_\{\\phi\}\(\\alpha\)by the proof of Theorem[2](https://arxiv.org/html/2610.11128#Thmtheorem2)\. Therefore, positioniicontributes

∑x𝒟⁡\(x\)​fx​K​∑αgx​\(α\)​∇ϕρϕ​\(α\)=K​ζ​∑x𝒟guide​\(x\)​∑αgx​\(α\)​∇ϕρϕ​\(α\)=K​ζ​∇ϕJ​\(ϕ\),\\sum\_\{x\}\\mathcal\{D\}\(x\)f\_\{x\}K\\sum\_\{\\alpha\}g\_\{x\}\(\\alpha\)\\nabla\_\{\\phi\}\\rho\_\{\\phi\}\(\\alpha\)=K\\zeta\\sum\_\{x\}\\mathcal\{D\}\_\{\\mathrm\{guide\}\}\(x\)\\sum\_\{\\alpha\}g\_\{x\}\(\\alpha\)\\nabla\_\{\\phi\}\\rho\_\{\\phi\}\(\\alpha\)=K\\zeta\\nabla\_\{\\phi\}J\(\\phi\),where the first equality uses Equation \([17](https://arxiv.org/html/2610.11128#A2.E17)\)\. Summing over thennpositions by linearity proves the claim\. Whenζ=0\\zeta=0, no group is an all\-failure group almost surely, and henceΨ=0\\Psi=0almost surely\. ∎

Since the proof uses only the marginal laws, it requires no independence between question positions or between continuations\. HoldingJJfixed, the factorsnn,KK, andζ\\zetaonly scale the expected update at a fixed step size\. However, changes in the all\-failure probabilities also change the question weights inJJand hence the update direction\.

Changing models and questions\.Across training iterations, the model, the question distribution, and the selector all change\. Let\(ℱk\)k\(\\mathcal\{F\}\_\{k\}\)\_\{k\}be the increasing filtration of the training history, whereℱk\\mathcal\{F\}\_\{k\}contains the history before iterationkk, including the current model, the selector parametersϕk\\phi\_\{k\}, and the baseline\. For the question positionsXk,1,…,Xk,nX\_\{k,1\},\\ldots,X\_\{k,n\}at iterationkk, define their average conditional marginal distribution by

𝒟k​\(x\)=1n​∑i=1nPr⁡\(Xk,i=x∣ℱk\)\.\\mathcal\{D\}\_\{k\}\(x\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\Pr\(X\_\{k,i\}=x\\mid\\mathcal\{F\}\_\{k\}\)\.With the current all\-failure probabilitiesfk,xf\_\{k,x\}and expected feedbackgk,x​\(α\)g\_\{k,x\}\(\\alpha\), define

ζk=𝔼X∼𝒟k​\[fk,X\],𝒟guide,k​\(x\)=𝒟k​\(x\)​fk,xζk,\\zeta\_\{k\}=\\mathbb\{E\}\_\{X\\sim\\mathcal\{D\}\_\{k\}\}\[f\_\{k,X\}\],\\qquad\\mathcal\{D\}\_\{\\mathrm\{guide\},k\}\(x\)=\\frac\{\\mathcal\{D\}\_\{k\}\(x\)f\_\{k,x\}\}\{\\zeta\_\{k\}\},Jk​\(ϕ\)=𝔼X∼𝒟guide,k​∑αρϕ​\(α\)​gk,X​\(α\),J\_\{k\}\(\\phi\)=\\mathbb\{E\}\_\{X\\sim\\mathcal\{D\}\_\{\\mathrm\{guide\},k\}\}\\sum\_\{\\alpha\}\\rho\_\{\\phi\}\(\\alpha\)g\_\{k,X\}\(\\alpha\),where the last two are defined whenζk\>0\\zeta\_\{k\}\>0\. LetΨk\\Psi\_\{k\}be the summed direction at iterationkk\.

###### Proposition B\.6\(Expected update under changing models and questions\)\.

Suppose that, conditional onℱk\\mathcal\{F\}\_\{k\}, the groups and continuations at iterationkkfollow the laws of Proposition[B\.5](https://arxiv.org/html/2610.11128#A2.Thmtheorem5)under the current model, except that positioniihas the marginal distributionPr\(Xk,i=⋅∣ℱk\)\\Pr\(X\_\{k,i\}=\\cdot\\mid\\mathcal\{F\}\_\{k\}\)\. Then

𝔼⁡\[Ψk∣ℱk\]=n​K​ζk​∇ϕJk​\(ϕk\),\\mathbb\{E\}\[\\Psi\_\{k\}\\mid\\mathcal\{F\}\_\{k\}\]=nK\\zeta\_\{k\}\\nabla\_\{\\phi\}J\_\{k\}\(\\phi\_\{k\}\),\(20\)where the right\-hand side is zero whenζk=0\\zeta\_\{k\}=0\.

###### Proof\.

Apply the proof of Proposition[B\.5](https://arxiv.org/html/2610.11128#A2.Thmtheorem5)conditionally onℱk\\mathcal\{F\}\_\{k\}, with the marginal distribution of positioniigiven byPr\(Xk,i=⋅∣ℱk\)\\Pr\(X\_\{k,i\}=\\cdot\\mid\\mathcal\{F\}\_\{k\}\)\. Since the proof uses only marginal laws, the positions may have different marginal distributions\. Summing their contributions replacesn​𝒟n\\mathcal\{D\}withn​𝒟kn\\mathcal\{D\}\_\{k\}, which gives Equation \([20](https://arxiv.org/html/2610.11128#A2.E20)\)\. ∎

The gradient in Equation \([20](https://arxiv.org/html/2610.11128#A2.E20)\) holds the history\-dependent model, question distribution, and prefix mapping fixed\. Thus, the update follows the current objective in expectation even as training changes these quantities\. The setting includes sampling with replacement from a fixed distribution, for which𝒟k=𝒟\\mathcal\{D\}\_\{k\}=\\mathcal\{D\}, and sampling without replacement from a remaining pool of questions\.

### B\.5Accumulated Sampling Noise

Proposition[B\.6](https://arxiv.org/html/2610.11128#A2.Thmtheorem6)describes the expected update at each iteration\. We now bound how far the accumulated updates deviate from these expectations, starting with the size of each update\.

###### Lemma B\.7\(Bounded update direction\)\.

For the selector in Equation \([11](https://arxiv.org/html/2610.11128#S4.E11)\) and a baseline in\[0,1\]\[0,1\], every term ofΨk\\Psi\_\{k\}has Euclidean norm at most2\\sqrt\{2\}\. Consequently,‖Ψk/\(n​K\)‖2≤2\\\|\\Psi\_\{k\}/\(nK\)\\\|\_\{2\}\\leq\\sqrt\{2\}\.

###### Proof\.

Letqϕq\_\{\\phi\}be the softmax component of Equation \([11](https://arxiv.org/html/2610.11128#S4.E11)\) and𝐞α\\mathbf\{e\}\_\{\\alpha\}the coordinate vector for levelα\\alpha\. Then

∇ϕ​log​ρϕ​\(α\)=\(1−εsel\)​qϕ​\(α\)ρϕ​\(α\)​\(𝐞α−qϕ\)\.\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\)=\\frac\{\(1\-\\varepsilon\_\{\\mathrm\{sel\}\}\)q\_\{\\phi\}\(\\alpha\)\}\{\\rho\_\{\\phi\}\(\\alpha\)\}\\bigl\(\\mathbf\{e\}\_\{\\alpha\}\-q\_\{\\phi\}\\bigr\)\.The prefactor is at most one sinceρϕ​\(α\)≥\(1−εsel\)​qϕ​\(α\)\\rho\_\{\\phi\}\(\\alpha\)\\geq\(1\-\\varepsilon\_\{\\mathrm\{sel\}\}\)q\_\{\\phi\}\(\\alpha\)\. Moreover,‖𝐞α−qϕ‖22=\(1−qϕ​\(α\)\)2\+∑α′≠αqϕ​\(α′\)2≤2​\(1−qϕ​\(α\)\)2≤2\\\|\\mathbf\{e\}\_\{\\alpha\}\-q\_\{\\phi\}\\\|\_\{2\}^\{2\}=\(1\-q\_\{\\phi\}\(\\alpha\)\)^\{2\}\+\\sum\_\{\\alpha^\{\\prime\}\\neq\\alpha\}q\_\{\\phi\}\(\\alpha^\{\\prime\}\)^\{2\}\\leq 2\(1\-q\_\{\\phi\}\(\\alpha\)\)^\{2\}\\leq 2\. SinceWjW\_\{j\}andbbboth lie in\[0,1\]\[0,1\], we have\|Wj−b\|≤1\|W\_\{j\}\-b\|\\leq 1, which bounds each term by2\\sqrt\{2\}\. Finally,Ψk\\Psi\_\{k\}has at mostn​KnKterms, including the case of no continuations\. ∎

Define the centered errors

ξk=Ψkn​K−ζk​∇ϕJk​\(ϕk\)\.\\xi\_\{k\}=\\frac\{\\Psi\_\{k\}\}\{nK\}\-\\zeta\_\{k\}\\nabla\_\{\\phi\}J\_\{k\}\(\\phi\_\{k\}\)\.
###### Proposition B\.8\(Accumulated sampling noise\)\.

Under the conditions of Proposition[B\.6](https://arxiv.org/html/2610.11128#A2.Thmtheorem6)and a baseline in\[0,1\]\[0,1\], for a deterministic numberNiterN\_\{\\mathrm\{iter\}\}of iterations,

𝔼​‖1Niter​∑k=1Niterξk‖22≤2Niter\.\\mathbb\{E\}\\left\\\|\\frac\{1\}\{N\_\{\\mathrm\{iter\}\}\}\\sum\_\{k=1\}^\{N\_\{\\mathrm\{iter\}\}\}\\xi\_\{k\}\\right\\\|\_\{2\}^\{2\}\\leq\\frac\{2\}\{N\_\{\\mathrm\{iter\}\}\}\.\(21\)

###### Proof\.

Proposition[B\.6](https://arxiv.org/html/2610.11128#A2.Thmtheorem6)gives𝔼⁡\[ξk∣ℱk\]=0\\mathbb\{E\}\[\\xi\_\{k\}\\mid\\mathcal\{F\}\_\{k\}\]=0\. The conditional variance identity and Lemma[B\.7](https://arxiv.org/html/2610.11128#A2.Thmtheorem7)then give

𝔼⁡\[‖ξk‖22∣ℱk\]=𝔼⁡\[‖Ψkn​K‖22\|ℱk\]−‖𝔼⁡\[Ψkn​K\|ℱk\]‖22≤2\.\\mathbb\{E\}\\bigl\[\\\|\\xi\_\{k\}\\\|\_\{2\}^\{2\}\\mid\\mathcal\{F\}\_\{k\}\\bigr\]=\\mathbb\{E\}\\\!\\left\[\\left\\\|\\frac\{\\Psi\_\{k\}\}\{nK\}\\right\\\|\_\{2\}^\{2\}\\middle\|\\mathcal\{F\}\_\{k\}\\right\]\-\\left\\\|\\mathbb\{E\}\\\!\\left\[\\frac\{\\Psi\_\{k\}\}\{nK\}\\middle\|\\mathcal\{F\}\_\{k\}\\right\]\\right\\\|\_\{2\}^\{2\}\\leq 2\.Fork<k′k<k^\{\\prime\}, the errorξk\\xi\_\{k\}is measurable with respect toℱk′\\mathcal\{F\}\_\{k^\{\\prime\}\}\. Therefore,𝔼⁡⟨ξk,ξk′⟩=𝔼⁡⟨ξk,𝔼⁡\[ξk′∣ℱk′\]⟩=0\\mathbb\{E\}\\langle\\xi\_\{k\},\\xi\_\{k^\{\\prime\}\}\\rangle=\\mathbb\{E\}\\langle\\xi\_\{k\},\\mathbb\{E\}\[\\xi\_\{k^\{\\prime\}\}\\mid\\mathcal\{F\}\_\{k^\{\\prime\}\}\]\\rangle=0\. Expanding the squared norm of the sum, the cross terms vanish and each of theNiterN\_\{\\mathrm\{iter\}\}diagonal terms is at most22, which proves the bound\. ∎

The baseline update in Equation \([18](https://arxiv.org/html/2610.11128#A2.E18)\) satisfies the condition on the baseline\. Proposition[B\.8](https://arxiv.org/html/2610.11128#A2.Thmtheorem8)controls the noise around the changing expected directions\.

## Appendix CAnalysis of Multiple Continuations

Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)concerns a single continuation, whereas ARG samplesKKcontinuations for each question and accepts candidate trajectories by the rule in Section[4\.1](https://arxiv.org/html/2610.11128#S4.SS1)\. This appendix analyzes the distribution of the accepted trajectory after multiple continuations\. Appendix[C\.1](https://arxiv.org/html/2610.11128#A3.SS1)fixes one prefix and studies how the distance to the ideal distribution changes with the number of continuations\. Appendix[C\.2](https://arxiv.org/html/2610.11128#A3.SS2)gives a sufficient condition under which an arbitrary acceptance rule keeps the distance no larger than that of training on the reference\.

### C\.1Repeated Continuations from a Fixed Prefix

Fix a prefix lengthttand letHtH\_\{t\}be the event that a trajectory begins withhth\_\{t\}\. Since the reference is correct and has positive probability,0<ut≤vt≤10<u\_\{t\}\\leq v\_\{t\}\\leq 1\. For a positive integerkk, drawkkindependent continuations fromhth\_\{t\}\. The*first\-success rule*returns the first correct candidate trajectory, including an exact copy ofrr, and returnsrrif every candidate trajectory is incorrect\. This rule treats an exact copy as a success\. LetQt\(k\)Q\_\{t\}^\{\(k\)\}be the resulting distribution of the accepted trajectory, and letνht=π\(⋅∣C∩Ht\)\\nu\_\{h\_\{t\}\}=\\pi\(\\cdot\\mid C\\cap H\_\{t\}\)be the ideal distribution restricted to trajectories that begin withhth\_\{t\}\.

###### Lemma C\.1\(Distribution after repeated continuations\)\.

Letwk=1−\(1−vt\)kw\_\{k\}=1\-\(1\-v\_\{t\}\)^\{k\}be the probability that at least one candidate trajectory is correct, and defineλk=wk/vt\\lambda\_\{k\}=w\_\{k\}/v\_\{t\}anddt\(k\)=dt​λkd\_\{t\}^\{\(k\)\}=d\_\{t\}\\lambda\_\{k\}\. Then

Qt\(k\)=\(1−wk\)​δr\+wk​νht,Q\_\{t\}^\{\(k\)\}=\(1\-w\_\{k\}\)\\delta\_\{r\}\+w\_\{k\}\\nu\_\{h\_\{t\}\},\(22\)and

DKL\(Qt\(k\)∥ν\)=DKL\(δr∥ν\)−dt\(k\)St\+\(1−dt\(k\)\)log\(1−dt\(k\)\)\+dt\(k\)logλk\.D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(k\)\}\\\|\\nu\)=D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\-d\_\{t\}^\{\(k\)\}S\_\{t\}\+\\bigl\(1\-d\_\{t\}^\{\(k\)\}\\bigr\)\\log\\bigl\(1\-d\_\{t\}^\{\(k\)\}\\bigr\)\+d\_\{t\}^\{\(k\)\}\\log\\lambda\_\{k\}\.

###### Proof\.

A correct trajectoryy=ht⊕zy=h\_\{t\}\\oplus zis returned at continuationjjwith probability\(1−vt\)j−1​π​\(z∣ht\)\(1\-v\_\{t\}\)^\{j\-1\}\\pi\(z\\mid h\_\{t\}\)\. Summing overj=1,…,kj=1,\\ldots,kgivesQt\(k\)​\(y\)=λk​π​\(z∣ht\)Q\_\{t\}^\{\(k\)\}\(y\)=\\lambda\_\{k\}\\pi\(z\\mid h\_\{t\}\)for every correcty≠ry\\neq r\. Sinceνht​\(y\)=π⁡\(z∣ht\)/vt\\nu\_\{h\_\{t\}\}\(y\)=\\pi\(z\\mid h\_\{t\}\)/v\_\{t\}for these trajectories, this probability equalswk​νht​\(y\)w\_\{k\}\\nu\_\{h\_\{t\}\}\(y\)\. The same calculation givesλk​ut\\lambda\_\{k\}u\_\{t\}for an exact copy, and fallback adds\(1−vt\)k\(1\-v\_\{t\}\)^\{k\}\. Therefore,Qt\(k\)​\(r\)=\(1−wk\)\+wk​ut/vt=1−dt\(k\)Q\_\{t\}^\{\(k\)\}\(r\)=\(1\-w\_\{k\}\)\+w\_\{k\}u\_\{t\}/v\_\{t\}=1\-d\_\{t\}^\{\(k\)\}, which proves Equation \([22](https://arxiv.org/html/2610.11128#A3.E22)\)\. For the distance, the trajectories other thanrrhave total probabilitydt\(k\)d\_\{t\}^\{\(k\)\}and likelihood ratioλk​p/π⁡\(ht\)\\lambda\_\{k\}p/\\pi\(h\_\{t\}\)\. Following the proof of Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)with this ratio gives the closed form\. ∎

Compared with the closed form for one continuation, the additional termdt\(k\)​log⁡λkd\_\{t\}^\{\(k\)\}\\log\\lambda\_\{k\}accounts for the larger probability of each trajectory other than the reference after repeated continuations\. Fork=1k=1, we haveλ1=1\\lambda\_\{1\}=1, and the lemma recovers Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)\. The mixture form in Equation \([22](https://arxiv.org/html/2610.11128#A3.E22)\) shows that additional continuations move probability from the reference toνht\\nu\_\{h\_\{t\}\}, which leads to the following limit\.

###### Proposition C\.2\(The limit of a fixed prefix\)\.

The distanceDKL\(Qt\(k\)∥ν\)D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(k\)\}\\\|\\nu\)is nonincreasing inkkand satisfies

−logν\(Ht\)≤DKL\(Qt\(k\)∥ν\)≤−logν\(Ht\)\+\(1−vt\)klogvtut\.\-\\log\\nu\(H\_\{t\}\)\\leq D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(k\)\}\\\|\\nu\)\\leq\-\\log\\nu\(H\_\{t\}\)\+\(1\-v\_\{t\}\)^\{k\}\\log\\frac\{v\_\{t\}\}\{u\_\{t\}\}\.Consequently, it converges to−log⁡ν⁡\(Ht\)\-\\log\\nu\(H\_\{t\}\)ask→∞k\\to\\infty\.

###### Proof\.

Fory∈C∩Hty\\in C\\cap H\_\{t\}, we haveν⁡\(y\)=ν⁡\(Ht\)​νht​\(y\)\\nu\(y\)=\\nu\(H\_\{t\}\)\\nu\_\{h\_\{t\}\}\(y\)\. Hence, every distributionQQsupported onC∩HtC\\cap H\_\{t\}satisfiesDKL\(Q∥ν\)=DKL\(Q∥νht\)−logν\(Ht\)D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\)=D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\_\{h\_\{t\}\}\)\-\\log\\nu\(H\_\{t\}\)\. By Lemma[C\.1](https://arxiv.org/html/2610.11128#A3.Thmtheorem1),Qt\(k\)Q\_\{t\}^\{\(k\)\}is such a distribution\. Therefore, it suffices to studyDKL\(Qt\(k\)∥νht\)D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(k\)\}\\\|\\nu\_\{h\_\{t\}\}\), and the lower bound follows from its nonnegativity\. For the upper bound, convexity of KL applied to Equation \([22](https://arxiv.org/html/2610.11128#A3.E22)\) givesDKL\(Qt\(k\)∥νht\)≤\(1−wk\)DKL\(δr∥νht\)D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(k\)\}\\\|\\nu\_\{h\_\{t\}\}\)\\leq\(1\-w\_\{k\}\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\_\{h\_\{t\}\}\), whereDKL\(δr∥νht\)=−logνht\(r\)=log\(vt/ut\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\_\{h\_\{t\}\}\)=\-\\log\\nu\_\{h\_\{t\}\}\(r\)=\\log\(v\_\{t\}/u\_\{t\}\)and1−wk=\(1−vt\)k1\-w\_\{k\}=\(1\-v\_\{t\}\)^\{k\}\. For monotonicity, takeℓ\>k\\ell\>k\. Ifwk=1w\_\{k\}=1, thenQt\(k\)=Qt\(ℓ\)=νhtQ\_\{t\}^\{\(k\)\}=Q\_\{t\}^\{\(\\ell\)\}=\\nu\_\{h\_\{t\}\}\. Otherwise,a=\(1−wℓ\)/\(1−wk\)∈\[0,1\]a=\(1\-w\_\{\\ell\}\)/\(1\-w\_\{k\}\)\\in\[0,1\]satisfiesQt\(ℓ\)=a​Qt\(k\)\+\(1−a\)​νhtQ\_\{t\}^\{\(\\ell\)\}=aQ\_\{t\}^\{\(k\)\}\+\(1\-a\)\\nu\_\{h\_\{t\}\}, and convexity gives

DKL\(Qt\(ℓ\)∥νht\)≤aDKL\(Qt\(k\)∥νht\)≤DKL\(Qt\(k\)∥νht\)\.D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(\\ell\)\}\\\|\\nu\_\{h\_\{t\}\}\)\\leq aD\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(k\)\}\\\|\\nu\_\{h\_\{t\}\}\)\\leq D\_\{\\mathrm\{KL\}\}\(Q\_\{t\}^\{\(k\)\}\\\|\\nu\_\{h\_\{t\}\}\)\.Finally, the upper bound converges to−log⁡ν⁡\(Ht\)\-\\log\\nu\(H\_\{t\}\)sincevt≥ut\>0v\_\{t\}\\geq u\_\{t\}\>0\. ∎

For the empty prefix,HtH\_\{t\}contains every trajectory\. Therefore, the limiting distribution isν\\nuand the limiting distance is zero\. A nonempty prefix has the same limit exactly whenHtH\_\{t\}contains all correct trajectories with positive probability\. Proposition[C\.2](https://arxiv.org/html/2610.11128#A3.Thmtheorem2)orders the numbers of continuations for each fixed prefix\. However, comparing different prefixes also requires their respectiveutu\_\{t\}andvtv\_\{t\}, which determine both the rate and the limit of the decrease\.

### C\.2A Sufficient Condition for General Acceptance Rules

To cover the acceptance rule of ARG, which excludes exact copies, mixes several prefixes, prefers some candidate trajectories over others, and may accept several of them, we consider a general procedure\. ChooseK≥1K\\geq 1reference prefixesht1,…,htKh\_\{t\_\{1\}\},\\ldots,h\_\{t\_\{K\}\}before generation\. Conditional on this list, draw a continuationZj∼π\(⋅∣htj\)Z\_\{j\}\\sim\\pi\(\\cdot\\mid h\_\{t\_\{j\}\}\)and form the candidate trajectoryY~j=htj⊕Zj\\widetilde\{Y\}\_\{j\}=h\_\{t\_\{j\}\}\\oplus Z\_\{j\}for eachjj\. The continuations may depend on each other, provided that each candidate trajectory has this conditional marginal law given the entire list\. A procedure returnsrror one of the correct candidate trajectories\. If several trajectories are accepted, we consider the distribution obtained by selecting one of them with nonnegative weights that sum to one\. LetQQdenote the resulting distribution of the accepted trajectory, including fallback, and use the convention0​log⁡0=00\\log 0=0\.

###### Proposition C\.3\(Multiple continuations and acceptance\)\.

Under the setup above, definedQ=1−Q⁡\(r\)d\_\{Q\}=1\-Q\(r\)andΛ=∑j=1Kexp⁡\(−Stj\)\\Lambda=\\sum\_\{j=1\}^\{K\}\\exp\(\-S\_\{t\_\{j\}\}\)\. Then

DKL\(Q∥ν\)≤DKL\(δr∥ν\)\+dQlogΛ\+\(1−dQ\)log\(1−dQ\)\.D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\)\\leq D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\+d\_\{Q\}\\log\\Lambda\+\(1\-d\_\{Q\}\)\\log\(1\-d\_\{Q\}\)\.\(23\)In particular,Λ≤1\\Lambda\\leq 1guaranteesDKL\(Q∥ν\)≤DKL\(δr∥ν\)D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\)\\leq D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\. A sufficient condition isminj⁡Stj≥log⁡K\\min\_\{j\}S\_\{t\_\{j\}\}\\geq\\log K\.

###### Proof\.

A trajectoryy≠ry\\neq rcan be selected only if it appears among the candidate trajectories\. Thus,

Q⁡\(y\)≤∑j=1KPr⁡\(Y~j=y\)=∑j=1K𝟏​\{y​begins with​htj\}​π⁡\(y\)π⁡\(htj\)≤π⁡\(y\)​Λπ⁡\(r\),Q\(y\)\\leq\\sum\_\{j=1\}^\{K\}\\Pr\(\\widetilde\{Y\}\_\{j\}=y\)=\\sum\_\{j=1\}^\{K\}\\mathbf\{1\}\\\{y\\text\{ begins with \}h\_\{t\_\{j\}\}\\\}\\frac\{\\pi\(y\)\}\{\\pi\(h\_\{t\_\{j\}\}\)\}\\leq\\frac\{\\pi\(y\)\\Lambda\}\{\\pi\(r\)\},where the last inequality usesπ⁡\(r\)/π⁡\(htj\)=exp⁡\(−Stj\)\\pi\(r\)/\\pi\(h\_\{t\_\{j\}\}\)=\\exp\(\-S\_\{t\_\{j\}\}\)from Lemma[A\.4](https://arxiv.org/html/2610.11128#A1.Thmtheorem4)\. Everyy≠ry\\neq rwithQ⁡\(y\)\>0Q\(y\)\>0is correct and hasπ⁡\(y\)\>0\\pi\(y\)\>0\. Hence,Q⁡\(y\)/ν⁡\(y\)≤p​Λ/π⁡\(r\)Q\(y\)/\\nu\(y\)\\leq p\\Lambda/\\pi\(r\)\. Since these trajectories have total probabilitydQd\_\{Q\}and the reference has probability1−dQ1\-d\_\{Q\}withQ⁡\(r\)/ν⁡\(r\)=\(1−dQ\)​p/π⁡\(r\)Q\(r\)/\\nu\(r\)=\(1\-d\_\{Q\}\)p/\\pi\(r\),

DKL\(Q∥ν\)≤logpπ⁡\(r\)\+\(1−dQ\)log\(1−dQ\)\+dQlogΛ\.D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\)\\leq\\log\\frac\{p\}\{\\pi\(r\)\}\+\(1\-d\_\{Q\}\)\\log\(1\-d\_\{Q\}\)\+d\_\{Q\}\\log\\Lambda\.Lemma[A\.2](https://arxiv.org/html/2610.11128#A1.Thmtheorem2)giveslog\(p/π\(r\)\)=DKL\(δr∥ν\)\\log\(p/\\pi\(r\)\)=D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\), which proves Equation \([23](https://arxiv.org/html/2610.11128#A3.E23)\)\. Since\(1−dQ\)​log⁡\(1−dQ\)≤0\(1\-d\_\{Q\}\)\\log\(1\-d\_\{Q\}\)\\leq 0, anddQ​log⁡Λ≤0d\_\{Q\}\\log\\Lambda\\leq 0whenΛ≤1\\Lambda\\leq 1, the conditionΛ≤1\\Lambda\\leq 1gives the comparison with training on the reference\. Finally,minj⁡Stj≥log⁡K\\min\_\{j\}S\_\{t\_\{j\}\}\\geq\\log KgivesΛ≤K​exp⁡\(−log⁡K\)=1\\Lambda\\leq K\\exp\(\-\\log K\)=1\. ∎

The argument permits arbitrary selection among correct candidate trajectories, including the exclusion of exact copies and the priority rule of ARG\. For one prefix usedKKtimes,Λ=K​exp⁡\(−St\)\\Lambda=K\\exp\(\-S\_\{t\}\), and the bound becomesDKL\(δr∥ν\)−dQ\(St−logK\)\+\(1−dQ\)log\(1−dQ\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\-d\_\{Q\}\(S\_\{t\}\-\\log K\)\+\(1\-d\_\{Q\}\)\\log\(1\-d\_\{Q\}\)\. The termlog⁡K\\log Kaccounts for the possible concentration from selecting among several candidate trajectories\. ForK=1K=1and the acceptance rule in Equation \([1](https://arxiv.org/html/2610.11128#S2.E1)\), we havedQ=dtd\_\{Q\}=d\_\{t\}, and the bound becomes the equality in Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)\.

In ARG, the prefixes come from guidance levels sampled before generation\. The following corollary extends the comparison to this case\.

###### Corollary C\.4\(Guidance levels sampled before generation\)\.

Under the setup of Proposition[C\.3](https://arxiv.org/html/2610.11128#A3.Thmtheorem3), suppose that theKKguidance levels are sampled before generation from any distribution over𝒜K\\mathcal\{A\}^\{K\}, and thatminα∈𝒜⁡Stx​\(α\)≥log⁡K\\min\_\{\\alpha\\in\\mathcal\{A\}\}S\_\{t\_\{x\}\(\\alpha\)\}\\geq\\log K\. Then the distributionQQof the accepted trajectory, averaged over the sampled levels, satisfiesDKL\(Q∥ν\)≤DKL\(δr∥ν\)D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\)\\leq D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\.

###### Proof\.

Conditional on each list of levels, every prefix satisfiesStj≥log⁡KS\_\{t\_\{j\}\}\\geq\\log K\. Therefore, Proposition[C\.3](https://arxiv.org/html/2610.11128#A3.Thmtheorem3)bounds the conditional distance byDKL\(δr∥ν\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\)\. SinceQQis the average of these conditional distributions, convexity of KL in its first argument gives the same bound forQQ\. ∎

The condition in Corollary[C\.4](https://arxiv.org/html/2610.11128#A3.Thmtheorem4)uses the actual prefix lengths after the boundary adjustment and is available from the same reference surprisals used for prefix selection\.

## Appendix DAnalysis of Learning from Inserted Trajectories

The preceding appendices compare distributions of accepted trajectories at a fixed model\. This appendix examines how the inserted trajectories contribute to the policy update in Section[4\.3](https://arxiv.org/html/2610.11128#S4.SS3)\. Appendix[D\.1](https://arxiv.org/html/2610.11128#A4.SS1)derives the gradient of inserted trajectories in Equation \([13](https://arxiv.org/html/2610.11128#S4.E13)\)\. Appendix[D\.2](https://arxiv.org/html/2610.11128#A4.SS2)derives the expected group update under binary rewards for one inserted trajectory, and Appendix[D\.3](https://arxiv.org/html/2610.11128#A4.SS3)extends it to multiple inserted trajectories\. Finally, Appendix[D\.4](https://arxiv.org/html/2610.11128#A4.SS4)relates learning the complete trajectory and learning only the continuation to their respective success events\.

### D\.1The Gradient of Inserted Trajectories

Condition on the realized batch, its masks, and the recomputed group advantages\. Letℐ\\mathcal\{I\}index the inserted trajectories, and letT\>0T\>0be the total number of active response tokens in the batch loss\. For tokenjjof an inserted trajectoryyiy\_\{i\}, letℓi,j​\(θ\)\\ell\_\{i,j\}\(\\theta\)be its log probability given the original question and all preceding tokens of the trajectory\. Inserted trajectories use the clipped objective of Equation \([29](https://arxiv.org/html/2610.11128#A5.E29)\) in Appendix[E\.3](https://arxiv.org/html/2610.11128#A5.SS3)with the stopped current probability in the ratio denominator,

ℒinserted\(θ\)=−1T∑i∈ℐ∑j=1\|yi\|min\{ω~i,j\(θ\)Ai,clip\(ω~i,j\(θ\),0\.8,1\.3\)Ai\},\\mathcal\{L\}\_\{\\mathrm\{inserted\}\}\(\\theta\)=\-\\frac\{1\}\{T\}\\sum\_\{i\\in\\mathcal\{I\}\}\\sum\_\{j=1\}^\{\|y\_\{i\}\|\}\\min\\bigl\\\{\\widetilde\{\\omega\}\_\{i,j\}\(\\theta\)A\_\{i\},\\operatorname\{clip\}\(\\widetilde\{\\omega\}\_\{i,j\}\(\\theta\),0\.8,1\.3\)A\_\{i\}\\bigr\\\},ω~i,j​\(θ\)=exp⁡\(ℓi,j​\(θ\)−stopgrad⁡\(ℓi,j​\(θ\)\)\),\\widetilde\{\\omega\}\_\{i,j\}\(\\theta\)=\\exp\\bigl\(\\ell\_\{i,j\}\(\\theta\)\-\\operatorname\{stopgrad\}\(\\ell\_\{i,j\}\(\\theta\)\)\\bigr\),wherestopgrad\\operatorname\{stopgrad\}preserves a value but assigns zero derivative to it\.

###### Lemma D\.1\(Gradient of inserted trajectories\)\.

Suppose that the advantagesAiA\_\{i\}and the token countTTare held fixed during differentiation and that every token of an inserted trajectory is active\. Then the gradient ofℒinserted\\mathcal\{L\}\_\{\\mathrm\{inserted\}\}is given by Equation \([13](https://arxiv.org/html/2610.11128#S4.E13)\)\.

###### Proof\.

By construction,ω~i,j=1\\widetilde\{\\omega\}\_\{i,j\}=1and∇θω~i,j=∇θℓi,j\\nabla\_\{\\theta\}\\widetilde\{\\omega\}\_\{i,j\}=\\nabla\_\{\\theta\}\\ell\_\{i,j\}\. Since the clipping interval contains one in its interior, both branches of the clipped loss have the same derivative at this ratio\. Therefore, for any fixed real advantageAiA\_\{i\}, the loss of tokenjjhas derivative−Ai∇θℓi,j/T\-A\_\{i\}\\nabla\_\{\\theta\}\\ell\_\{i,j\}/T\. Summing over all positions, including any supplied prefix and the termination token, gives−Ai∇θlogπθ\(yi∣xi\)/T\-A\_\{i\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\\mid x\_\{i\}\)/Tfor each inserted trajectory\. Summing overi∈ℐi\\in\\mathcal\{I\}proves Equation \([13](https://arxiv.org/html/2610.11128#S4.E13)\)\. ∎

Lemma[D\.1](https://arxiv.org/html/2610.11128#A4.Thmtheorem1)identifies the gradient under the prescribed rule of automatic differentiation\. ForAi\>0A\_\{i\}\>0, minimizingℒinserted\\mathcal\{L\}\_\{\\mathrm\{inserted\}\}gives a positively weighted supervised fine\-tuning \(SFT\) contribution\. The advantage specifies its learning weight, while candidate generation and acceptance determine the distribution of the trajectory\.

### D\.2The Group Update with One Inserted Trajectory

We now use binary correctness rewards to separate the contributions of correct and incorrect responses\. This simplification ignores the format reward, which leaves the advantages unchanged when all responses in a group receive the same format reward\. The calculation is at the current model and precedes division by the batch token count\. Fix a questionxxwith set of correct trajectoriesCC, and suppress the conditioning onxxunless shown explicitly\. Assume a finite trajectory space, a differentiable policy with positive trajectory probabilities in a neighborhood ofθold\\theta\_\{\\mathrm\{old\}\}, and0<p=πθold​\(C\)<10<p=\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(C\)<1\. DrawG≥2G\\geq 2independent responses fromπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}and condition on all of them failing verification\. Then replace one uniformly chosen response byY∼QY\\sim Q, whereQQis supported onCC, and whereYYand the replaced position are independent of the original failures\. BothQQand the collected responses are fixed during differentiation\. Set

pθ=πθ\(C\),νθ=πθ\(⋅∣C\),μθ=πθ\(⋅∣Cc\),p\_\{\\theta\}=\\pi\_\{\\theta\}\(C\),\\qquad\\nu\_\{\\theta\}=\\pi\_\{\\theta\}\(\\cdot\\mid C\),\\qquad\\mu\_\{\\theta\}=\\pi\_\{\\theta\}\(\\cdot\\mid C^\{c\}\),and writeν=νθold\\nu=\\nu\_\{\\theta\_\{\\mathrm\{old\}\}\}for the ideal distribution at the current model\.

At the current model, the ratios of the original responses equal one and clipping is inactive\. The inserted trajectory follows Lemma[D\.1](https://arxiv.org/html/2610.11128#A4.Thmtheorem1), and all response tokens are included\. We consider the group ascent direction, which is the sum of the reward\-dependent gradients before division by the batch token count\. With one inserted trajectory, the reward mean is1/G1/Gand the sample standard deviation issR=1/Gs\_\{R\}=1/\\sqrt\{G\}\. Therefore, the inserted trajectory has advantage\(G−1\)/\[G⁡\(sR\+δ\)\]\(G\-1\)/\[G\(s\_\{R\}\+\\delta\)\], and each failure has advantage−1/\[G\(sR\+δ\)\]\-1/\[G\(s\_\{R\}\+\\delta\)\], whereδ\>0\\delta\>0is the stabilizer in Equation \([28](https://arxiv.org/html/2610.11128#A5.E28)\)\. Define the positive coefficient

cG=G−1G⁡\(1/G\+δ\)\.c\_\{G\}=\\frac\{G\-1\}\{G\(1/\\sqrt\{G\}\+\\delta\)\}\.LetF1,…,FG−1F\_\{1\},\\ldots,F\_\{G\-1\}denote the original failures that remain after replacement\. The realized group direction is

g=cG​\[∇θ​log​πθ​\(Y∣x\)−1G−1​∑j=1G−1∇θ​log​πθ​\(Fj∣x\)\]θ=θold\.g=c\_\{G\}\\left\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\\mid x\)\-\\frac\{1\}\{G\-1\}\\sum\_\{j=1\}^\{G\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(F\_\{j\}\\mid x\)\\right\]\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\.\(24\)
###### Proposition D\.2\(Inserted trajectories and the group update\)\.

Under the setup above, letg¯Q=𝔼⁡\[g\]\\bar\{g\}\_\{Q\}=\\mathbb\{E\}\[g\]be the expectation over the original failures, the replaced position, and the inserted trajectory\. Then

g¯Q=cG∇θ\[logpθ1−pθ−DKL\(Q∥νθ\)\]\|θ=θold\.\\bar\{g\}\_\{Q\}=c\_\{G\}\\left\.\\nabla\_\{\\theta\}\\left\[\\log\\frac\{p\_\{\\theta\}\}\{1\-p\_\{\\theta\}\}\-D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\_\{\\theta\}\)\\right\]\\right\|\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\.\(25\)In particular, forQ=νQ=\\nu,

g¯ν=cGp⁡\(1−p\)​∇θpθ\|θ=θold\.\\bar\{g\}\_\{\\nu\}=\\frac\{c\_\{G\}\}\{p\(1\-p\)\}\\left\.\\nabla\_\{\\theta\}p\_\{\\theta\}\\right\|\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\.\(26\)

###### Proof\.

Conditioning independent responses on all of them belonging toCcC^\{c\}gives independent draws fromμθold\\mu\_\{\\theta\_\{\\mathrm\{old\}\}\}, and uniform replacement preserves this distribution for the remaining failures\. Therefore, withF∼μθoldF\\sim\\mu\_\{\\theta\_\{\\mathrm\{old\}\}\},

g¯QcG=\[𝔼Q​∇θ​log⁡πθ​\(Y\)−𝔼μθold​∇θ​log⁡πθ​\(F\)\]\|θ=θold\.\\frac\{\\bar\{g\}\_\{Q\}\}\{c\_\{G\}\}=\\left\.\\left\[\\mathbb\{E\}\_\{Q\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\)\-\\mathbb\{E\}\_\{\\mu\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(F\)\\right\]\\right\|\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\.Applying the argument of Lemma[A\.1](https://arxiv.org/html/2610.11128#A1.Thmtheorem1)to the eventCcC^\{c\}gives𝔼μθold​∇θ​log​πθ​\(F\)=∇θ​log​\(1−pθ\)\\mathbb\{E\}\_\{\\mu\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(F\)=\\nabla\_\{\\theta\}\\log\(1\-p\_\{\\theta\}\)atθold\\theta\_\{\\mathrm\{old\}\}\. For fixedQQ, expandinglog⁡νθ​\(y\)=log⁡πθ​\(y\)−log⁡pθ\\log\\nu\_\{\\theta\}\(y\)=\\log\\pi\_\{\\theta\}\(y\)\-\\log p\_\{\\theta\}gives

∇θDKL\(Q∥νθ\)=−𝔼Q∇θlogπθ\(Y\)\+∇θlogpθ\.\\nabla\_\{\\theta\}D\_\{\\mathrm\{KL\}\}\(Q\\\|\\nu\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{Q\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\)\+\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\.Substituting both expressions proves Equation \([25](https://arxiv.org/html/2610.11128#A4.E25)\)\. WhenQ=νQ=\\nu, the gradient of the KL term vanishes atθold\\theta\_\{\\mathrm\{old\}\}by Lemma[A\.1](https://arxiv.org/html/2610.11128#A1.Thmtheorem1)\. Since∇θ​log​\[pθ/\(1−pθ\)\]=∇θpθ/\[pθ​\(1−pθ\)\]\\nabla\_\{\\theta\}\\log\[p\_\{\\theta\}/\(1\-p\_\{\\theta\}\)\]=\\nabla\_\{\\theta\}p\_\{\\theta\}/\[p\_\{\\theta\}\(1\-p\_\{\\theta\}\)\], this gives Equation \([26](https://arxiv.org/html/2610.11128#A4.E26)\)\. ∎

The ratiopθ/\(1−pθ\)p\_\{\\theta\}/\(1\-p\_\{\\theta\}\)is the odds of passing verification\. The first term in Equation \([25](https://arxiv.org/html/2610.11128#A4.E25)\) favors success over failure\. The second term records which correct trajectories the distributionQQfavors\. For training on the reference,DKL\(δr∥νθ\)=−logνθ\(r\)D\_\{\\mathrm\{KL\}\}\(\\delta\_\{r\}\\\|\\nu\_\{\\theta\}\)=\-\\log\\nu\_\{\\theta\}\(r\), and this term favors the particular reference within the set of correct trajectories\. In contrast, Equation \([26](https://arxiv.org/html/2610.11128#A4.E26)\) shows that inserting trajectories from the ideal distribution gives an expected group update proportional to the gradient of the success probability under the original question\.

For reference fallback,Y=rY=rmakes the positive term in Equation \([24](https://arxiv.org/html/2610.11128#A4.E24)\) a weighted SFT gradient\. For an accepted candidate trajectory, the positive term learns its complete sequence, including the supplied prefix\. In both cases, the original failures supply the negative terms in the same group update\. This combination of learning from correct trajectories with feedback on the model’s own failures connects our update to prior work on integrating supervised and reinforcement fine\-tuning\([Yan et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib49);[Fu et al\., 2026](https://arxiv.org/html/2610.11128#bib.bib11)\)\.

### D\.3Multiple Inserted Trajectories

We keep the preceding setup but insert a random numberN\+N\_\{\+\}of correct trajectories with1≤N\+<G1\\leq N\_\{\+\}<G, where a single reference is inserted when no candidate trajectory is accepted\. The inserted trajectories may depend on each other\. However, they, their numberN\+N\_\{\+\}, and the replaced positions, which are uniformly chosen and distinct, are independent of the original failures\. Conditional onN\+=mN\_\{\+\}=m, letQ^\\widehat\{Q\}andM^\\widehat\{M\}be the empirical distributions of the inserted trajectories and of the remaining failures\. Substituting the advantages\(G−m\)/\[G⁡\(sm\+δ\)\]\(G\-m\)/\[G\(s\_\{m\}\+\\delta\)\]and−m/\[G\(sm\+δ\)\]\-m/\[G\(s\_\{m\}\+\\delta\)\]of the inserted trajectories and the remaining failures gives the group direction

g=κm​\[𝔼Q^​∇θ​log⁡πθ​\(Y\)−𝔼M^​∇θ​log⁡πθ​\(F\)\]θ=θold,g=\\kappa\_\{m\}\\left\[\\mathbb\{E\}\_\{\\widehat\{Q\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\)\-\\mathbb\{E\}\_\{\\widehat\{M\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(F\)\\right\]\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\},where

κm=m⁡\(G−m\)G⁡\(sm\+δ\),sm=m⁡\(G−m\)G⁡\(G−1\)\.\\kappa\_\{m\}=\\frac\{m\(G\-m\)\}\{G\(s\_\{m\}\+\\delta\)\},\\qquad s\_\{m\}=\\sqrt\{\\frac\{m\(G\-m\)\}\{G\(G\-1\)\}\}\.Define the average coefficient and the advantage\-weighted distribution of inserted trajectories by

κ¯=𝔼⁡\[κN\+\],Q\+​\(y\)=𝔼⁡\[κN\+​Q^​\(y\)\]κ¯\.\\bar\{\\kappa\}=\\mathbb\{E\}\[\\kappa\_\{N\_\{\+\}\}\],\\qquad Q\_\{\+\}\(y\)=\\frac\{\\mathbb\{E\}\[\\kappa\_\{N\_\{\+\}\}\\widehat\{Q\}\(y\)\]\}\{\\bar\{\\kappa\}\}\.Since1≤N\+<G1\\leq N\_\{\+\}<G, we haveκ¯\>0\\bar\{\\kappa\}\>0, andQ\+Q\_\{\+\}is a probability distribution onCC\.

###### Proposition D\.3\(Multiple inserted trajectories\)\.

Under the setup above,

𝔼\[g\]=κ¯∇θ\[logpθ1−pθ−DKL\(Q\+∥νθ\)\]\|θ=θold\.\\mathbb\{E\}\[g\]=\\bar\{\\kappa\}\\left\.\\nabla\_\{\\theta\}\\left\[\\log\\frac\{p\_\{\\theta\}\}\{1\-p\_\{\\theta\}\}\-D\_\{\\mathrm\{KL\}\}\(Q\_\{\+\}\\\|\\nu\_\{\\theta\}\)\\right\]\\right\|\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\.For one inserted trajectory,κ1=cG\\kappa\_\{1\}=c\_\{G\}andQ\+=QQ\_\{\+\}=Q, which recovers Proposition[D\.2](https://arxiv.org/html/2610.11128#A4.Thmtheorem2)\.

###### Proof\.

Conditional on the inserted trajectories and their number, uniform replacement preserves the failure distributionμθold\\mu\_\{\\theta\_\{\\mathrm\{old\}\}\}for the remaining failures\. Therefore,𝔼⁡\[κN\+​M^​\(y\)\]=κ¯​μθold​\(y\)\\mathbb\{E\}\[\\kappa\_\{N\_\{\+\}\}\\widehat\{M\}\(y\)\]=\\bar\{\\kappa\}\\mu\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\), and taking expectations in the group direction gives

𝔼⁡\[g\]=κ¯​\[𝔼Q\+​∇θ​log⁡πθ​\(Y\)−𝔼μθold​∇θ​log⁡πθ​\(F\)\]θ=θold\.\\mathbb\{E\}\[g\]=\\bar\{\\kappa\}\\left\[\\mathbb\{E\}\_\{Q\_\{\+\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\)\-\\mathbb\{E\}\_\{\\mu\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(F\)\\right\]\_\{\\theta=\\theta\_\{\\mathrm\{old\}\}\}\.The proof of Proposition[D\.2](https://arxiv.org/html/2610.11128#A4.Thmtheorem2)then applies withcGc\_\{G\}replaced byκ¯\\bar\{\\kappa\}andQQreplaced byQ\+Q\_\{\+\}\. Finally,s1=1/Gs\_\{1\}=1/\\sqrt\{G\}givesκ1=cG\\kappa\_\{1\}=c\_\{G\}\. ∎

The distributionQ\+Q\_\{\+\}weights each realized set of inserted trajectories by its total positive advantageκN\+\\kappa\_\{N\_\{\+\}\}\. Therefore, it accounts for both acceptance and the resulting composition of the group\.

### D\.4Learning the Prefix and the Continuation

The likelihood in Equation \([13](https://arxiv.org/html/2610.11128#S4.E13)\) includes every token of the complete accepted trajectory under the original question\. For a candidate trajectory from a nonempty prefix, the chain rule gives

log⁡πθ​\(ht⊕z∣x\)=log⁡πθ​\(ht∣x\)\+log⁡πθ​\(z∣x,ht\)\.\\log\\pi\_\{\\theta\}\(h\_\{t\}\\oplus z\\mid x\)=\\log\\pi\_\{\\theta\}\(h\_\{t\}\\mid x\)\+\\log\\pi\_\{\\theta\}\(z\\mid x,h\_\{t\}\)\.\(27\)Hence, learning covers both producing the supplied prefix and completing the trajectory, and reference fallback trains the complete reference through Equation \([13](https://arxiv.org/html/2610.11128#S4.E13)\)\. We now connect the two terms in Equation \([27](https://arxiv.org/html/2610.11128#A4.E27)\) to different success events\. Fix a prefix lengthtt, letHtH\_\{t\}be the event that a trajectory begins withhth\_\{t\}, and letνht=πθold\(⋅∣C∩Ht\)\\nu\_\{h\_\{t\}\}=\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid C\\cap H\_\{t\}\)as in Appendix[C\.1](https://arxiv.org/html/2610.11128#A3.SS1)\. This distribution is well defined since the correct reference has positive probability\. LetQZ,t​\(z\)=νht​\(ht⊕z\)Q\_\{Z,t\}\(z\)=\\nu\_\{h\_\{t\}\}\(h\_\{t\}\\oplus z\)be the distribution of its continuations\. Hereπθ​\(C∣Ht\)\\pi\_\{\\theta\}\(C\\mid H\_\{t\}\)is the probability that a continuation fromhth\_\{t\}gives a correct candidate trajectory, which equalsvtv\_\{t\}atθold\\theta\_\{\\mathrm\{old\}\}\.

###### Proposition D\.4\(Learning the prefix and the continuation\)\.

Under the assumptions of Lemma[A\.1](https://arxiv.org/html/2610.11128#A1.Thmtheorem1), atθ=θold\\theta=\\theta\_\{\\mathrm\{old\}\},

𝔼Y∼νht​∇θ​log​πθ​\(Y\)=∇θ​log​πθ​\(C∩Ht\),𝔼Z∼QZ,t​∇θ​log​πθ​\(Z∣ht\)=∇θ​log​πθ​\(C∣Ht\)\.\\mathbb\{E\}\_\{Y\\sim\\nu\_\{h\_\{t\}\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\)=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(C\\cap H\_\{t\}\),\\qquad\\mathbb\{E\}\_\{Z\\sim Q\_\{Z,t\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Z\\mid h\_\{t\}\)=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(C\\mid H\_\{t\}\)\.Moreover, the distributionQt\(k\)Q\_\{t\}^\{\(k\)\}of the accepted trajectory afterkkcontinuations in Lemma[C\.1](https://arxiv.org/html/2610.11128#A3.Thmtheorem1)satisfies

𝔼Y∼Qt\(k\)​∇θ​log​πθ​\(Y\)=\(1−wk\)​∇θ​log​πθ​\(r\)\+wk​∇θ​log​πθ​\(C∩Ht\)\.\\mathbb\{E\}\_\{Y\\sim Q\_\{t\}^\{\(k\)\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Y\)=\(1\-w\_\{k\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(r\)\+w\_\{k\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(C\\cap H\_\{t\}\)\.

###### Proof\.

The first identity follows from Lemma[A\.1](https://arxiv.org/html/2610.11128#A1.Thmtheorem1)withCCreplaced byC∩HtC\\cap H\_\{t\}\. For the second, Equation \([27](https://arxiv.org/html/2610.11128#A4.E27)\) gives∇θ​log​πθ​\(ht⊕Z\)=∇θ​log​πθ​\(ht\)\+∇θ​log​πθ​\(Z∣ht\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(h\_\{t\}\\oplus Z\)=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(h\_\{t\}\)\+\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(Z\\mid h\_\{t\}\), andπθ​\(C∩Ht\)=πθ​\(ht\)​πθ​\(C∣Ht\)\\pi\_\{\\theta\}\(C\\cap H\_\{t\}\)=\\pi\_\{\\theta\}\(h\_\{t\}\)\\pi\_\{\\theta\}\(C\\mid H\_\{t\}\)\. Subtracting∇θ​log​πθ​\(ht\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(h\_\{t\}\)from both sides of the first identity gives the second\. Finally, the mixture form in Equation \([22](https://arxiv.org/html/2610.11128#A3.E22)\), whose weights are fixed atθold\\theta\_\{\\mathrm\{old\}\}, and the first identity give the last statement\. ∎

Underνht\\nu\_\{h\_\{t\}\}, the expected gradient of the complete trajectory addresses the joint event of producing the prefix and passing verification\. In contrast, learning only the continuation addresses verification given an externally supplied prefix\. The two coincide for the empty prefix\. The joint event covers the correct trajectories that begin with this prefix, while the overall success eventCCalso includes correct trajectories that begin differently\. Finally, the mixture identity in Proposition[D\.4](https://arxiv.org/html/2610.11128#A4.Thmtheorem4)shows how fallback and successful continuations contribute to the expected gradient of the accepted trajectory before any weighting\.

## Appendix EImplementation and Evaluation Details

We first specify the training procedure, reference boundaries, and policy loss\. We then give the training configurations, question prompt, and evaluation protocol\. Together, these details connect the single\-continuation analysis in Theorem[1](https://arxiv.org/html/2610.11128#Thmtheorem1)to the complete training procedure\.

### E\.1Training Procedure

Training uses 11,068 screened DeepMath examples with matched offline DeepSeek\-R1 reference solutions\([Guo et al\., 2025](https://arxiv.org/html/2610.11128#bib.bib12)\)\. We start from DeepMath\-103K, keep problems with difficulty at least 5\.0, and drop problems whose gold answer is not a single short expression, is a multiple\-choice letter, or cannot be verified by our checker when written inside`\\boxed\{\}`\. From the remaining problems, we randomly sample 16,000 in proportion to difficulty and let the base Qwen3\-4B attempt each one eight times under the training prompt and sampling settings\. Problems solved in all eight attempts are removed\. We also drop the few problems whose reference solution does not end in the gold answer\. Table[3](https://arxiv.org/html/2610.11128#A5.T3)gives the ARG configuration\. Algorithm[1](https://arxiv.org/html/2610.11128#alg1)separates collection under a fixed policy from updates of the selector and policy\. All continuations, including failures and candidate trajectories not selected for the policy update, provide selector feedback\.

In the algorithm,𝒰x\\mathcal\{U\}\_\{x\}containsKKdistinct batch\-wide continuation indices for questionxx\. Candidate quantities are computed for everyj∈𝒰xj\\in\\mathcal\{U\}\_\{x\}\. The operationAccept\\operatorname\{Accept\}returns the candidate indices selected by the rule in Section[4\.3](https://arxiv.org/html/2610.11128#S4.SS3)\.

Algorithm 1One ARG training update with GRPO1\.InputBatch

ℬ\\mathcal\{B\}with references, policy

θ\\theta, selector

ϕ\\phi, baseline

bb\.

2\.

θold←θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\theta,

𝒰←∅\\mathcal\{U\}\\leftarrow\\varnothing
3\.

𝐲x=\(yx,i\)i=1G\\mathbf\{y\}\_\{x\}=\(y\_\{x,i\}\)\_\{i=1\}^\{G\},

yx,i∼iidπθold\(⋅∣x\)y\_\{x,i\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid x\)for

x∈ℬx\\in\\mathcal\{B\}
4\.for

x∈ℬx\\in\\mathcal\{B\}with

∑i=1GRacc​\(x,yx,i\)=0\\sum\_\{i=1\}^\{G\}R\_\{\\mathrm\{acc\}\}\(x,y\_\{x,i\}\)=0do

5\.

St←−log⁡πθold​\(at∣x,ht\)S\_\{t\}\\leftarrow\-\\log\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid x,h\_\{t\}\)for

0≤t<\|r\|0\\leq t<\|r\|,

𝒥x←∅\\mathcal\{J\}\_\{x\}\\leftarrow\\varnothing
6\.if

0<S0<∞0<S\_\{0\}<\\inftythen

7\.

αj​∼iid​ρϕ\\alpha\_\{j\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\rho\_\{\\phi\},

tj←tx​\(αj\)t\_\{j\}\\leftarrow t\_\{x\}\(\\alpha\_\{j\}\)
8\.

Zj∼πθold\(⋅∣x,htj\)Z\_\{j\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid x,h\_\{t\_\{j\}\}\),

Y~j←htj⊕Zj\\widetilde\{Y\}\_\{j\}\\leftarrow h\_\{t\_\{j\}\}\\oplus Z\_\{j\}
9\.

Wj←𝟏​\{Y~j∈Cx,Y~j≠r\}​Stj/S0W\_\{j\}\\leftarrow\\mathbf\{1\}\\\{\\widetilde\{Y\}\_\{j\}\\in C\_\{x\},\\ \\widetilde\{Y\}\_\{j\}\\neq r\\\}\\,S\_\{t\_\{j\}\}/S\_\{0\}
10\.

𝒰←𝒰∪𝒰x\\mathcal\{U\}\\leftarrow\\mathcal\{U\}\\cup\\mathcal\{U\}\_\{x\},

𝒥x←Accept⁡\(\(Y~j,tj\)j∈𝒰x,r\)\\mathcal\{J\}\_\{x\}\\leftarrow\\operatorname\{Accept\}\(\(\\widetilde\{Y\}\_\{j\},t\_\{j\}\)\_\{j\\in\\mathcal\{U\}\_\{x\}\},r\)
11\.endif

12\.Replace rows in

𝐲x\\mathbf\{y\}\_\{x\}by

\(Y~j\)j∈𝒥x\(\\widetilde\{Y\}\_\{j\}\)\_\{j\\in\\mathcal\{J\}\_\{x\}\}, or by

rrif

𝒥x=∅\\mathcal\{J\}\_\{x\}=\\varnothing\.

13\.endfor

14\.Recompute final rewards

RiR\_\{i\}, then

Ai←\(Ri−R¯\)/\(sR\+δ\)A\_\{i\}\\leftarrow\(R\_\{i\}\-\\bar\{R\}\)/\(s\_\{R\}\+\\delta\)\.

15\.if

𝒰≠∅\\mathcal\{U\}\\neq\\varnothingthen

16\.

ϕ←ϕ\+ηsel​∑j∈𝒰\(Wj−b\)​∇ϕ​log⁡ρϕ​\(αj\)\\phi\\leftarrow\\phi\+\\eta\_\{\\mathrm\{sel\}\}\\sum\_\{j\\in\\mathcal\{U\}\}\(W\_\{j\}\-b\)\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(\\alpha\_\{j\}\)
17\.

b←\(1−β\)​b\+β\|𝒰\|​∑j∈𝒰Wjb\\leftarrow\(1\-\\beta\)b\+\\frac\{\\beta\}\{\|\\mathcal\{U\}\|\}\\sum\_\{j\\in\\mathcal\{U\}\}W\_\{j\}
18\.endif

19\.Update

θ\\thetausing Equation \([13](https://arxiv.org/html/2610.11128#S4.E13)\) and the remaining\-response loss in Appendix[E\.3](https://arxiv.org/html/2610.11128#A5.SS3)\.

Table 3:ARG configuration for both model sizes\.
### E\.2Prefix Boundaries

The levelα=1\\alpha=1always uses the empty prefix\. Forα<1\\alpha<1, we take the latest prefix length whose reference surprisal is at leastα​S0\\alpha S\_\{0\}and, if this length is positive, move it to the first valid sentence or formula boundary at or after it\. If no such boundary exists, we use the latest earlier valid boundary, or the empty prefix if there is none\. A valid boundary falls on a token boundary, closes all mathematical delimiters, and leaves a nontrivial suffix containing the final boxed answer\. The selector feedback uses the reference surprisalStS\_\{t\}at this adjusted prefix length, and the prefix counts toward the maximum trajectory length\.

### E\.3Policy Loss

Final rewardsRiR\_\{i\}combine binary verifier accuracy with0\.20\.2times a binary format reward indicating an extracted boxed answer\. Thus, a correct response receives1\.21\.2, while an incorrect response receives0\.20\.2or zero according to its format\. We recompute advantages within each final group as

Ai=Ri−R¯sR\+δ,A\_\{i\}=\\frac\{R\_\{i\}\-\\bar\{R\}\}\{s\_\{R\}\+\\delta\},\(28\)whereR¯\\bar\{R\}andsRs\_\{R\}are the mean and sample standard deviation of the group’s final rewards, andδ=10−6\\delta=10^\{\-6\}ensures numerical stability\. All\-equal rewards give exactly zero advantages\.

For an ordinary responseyiy\_\{i\}, letyi,jy\_\{i,j\}be tokenjjand define

ωi,j​\(θ\)=πθ​\(yi,j∣xi,yi,<j\)πθold​\(yi,j∣xi,yi,<j\)\.\\omega\_\{i,j\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,j\}\\mid x\_\{i\},y\_\{i,<j\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,j\}\\mid x\_\{i\},y\_\{i,<j\}\)\}\.Let𝒪\\mathcal\{O\}index the original responses that remain in their groups, and letTTbe the total number of active response tokens in the final batch\. Their loss contribution uses the asymmetric clipping interval\[0\.8,1\.3\]\[0\.8,1\.3\],

ℒoriginal\(θ\)=−1T∑i∈𝒪∑j=1\|yi\|min\{ωi,j\(θ\)Ai,clip\(ωi,j\(θ\),0\.8,1\.3\)Ai\}\.\\mathcal\{L\}\_\{\\mathrm\{original\}\}\(\\theta\)=\-\\frac\{1\}\{T\}\\sum\_\{i\\in\\mathcal\{O\}\}\\sum\_\{j=1\}^\{\|y\_\{i\}\|\}\\min\\big\\\{\\omega\_\{i,j\}\(\\theta\)A\_\{i\},\\operatorname\{clip\}\(\\omega\_\{i,j\}\(\\theta\),0\.8,1\.3\)A\_\{i\}\\big\\\}\.\(29\)Inserted trajectories use the stopped current probability in the ratio denominator, as derived in Appendix[D](https://arxiv.org/html/2610.11128#A4)\. With unit token weights over each complete inserted trajectory, this gives Equation \([13](https://arxiv.org/html/2610.11128#S4.E13)\)\. Loss\-active tokens include supplied prefixes and termination tokens when present, and both contributions share the same batch token denominator\.

### E\.4Training Configurations and Question Prompt

The GRPO, SAGE, BREAD, LUFFY, and ARG runs share the optimizer and student rollout settings in Table[4](https://arxiv.org/html/2610.11128#A5.T4)\. The resampling and weighting ablations use the same settings\. SFT learns from the offline references\. For OPSD, we follow[Zhao et al\. \(2026\)](https://arxiv.org/html/2610.11128#bib.bib58)in using one student response per question, a training generation limit of 1024 tokens, and temperature1\.11\.1\. Table[4](https://arxiv.org/html/2610.11128#A5.T4)specifies the remaining settings used in our experiments\. All listed runs use 32 questions per update and one optimization epoch per collected batch\. Their data and parameter\-initialization seeds are zero, and the learning rate is constant after warmup\.

Table 4:Recorded configurations for both model sizes\. GRPO\-based runs here comprise GRPO, SAGE, BREAD, LUFFY, ARG, resampling, and the weighting ablation\. The reference\-insertion ablation is specified by its replacement rule in Section[5\.3](https://arxiv.org/html/2610.11128#S5.SS3)\.The ordinary student question prompt and the evaluation prompt contain a single user message with the question, a blank line, and the instruction below\. They use no system message or few\-shot example\. Auxiliary teacher or reference conditioning is specific to each training method\. The Qwen3 chat template is applied with thinking mode disabled, which appends an empty thinking block before response generation\. The rendered question prompt is shown schematically below\.

```
<|im_start|>user
<question>

Please reason step by step, and put your final answer within \boxed{}.
<|im_end|>
<|im_start|>assistant
<think>

</think>
```

Here<question\>denotes the question text, and the line break before<\|im\_end\|\>is added for display\. For ARG generation, the selected reference prefix supplies the beginning of the candidate trajectory\. Evaluation supplies no reference prefix\.

### E\.5Checkpoint and Evaluation Records

Table[5](https://arxiv.org/html/2610.11128#A5.T5)lists the reported checkpoints\. GRPO, SAGE, BREAD, LUFFY, ARG, resampling, and the weighting ablations use the fixed budget of 200 updates\. Reference insertion also reports its step\-200 checkpoint\. For SFT and OPSD, we select the reported checkpoints by comparing meanpass​@​12\\mathrm\{pass@12\}andavg​@​12\\mathrm\{avg@12\}on the five evaluation benchmarks\.

Table 5:Reported checkpoint steps for both model sizes\. Base uses the released parameters\. SFT and OPSD use benchmark\-selected checkpoints, and the remaining trained entries report step 200\.All 20 reported model and method entries use the same 150 evaluation questions, comprising 30 questions from each of the five benchmarks\. The benchmark files, answer extractor, and verifier implementation match across entries\. Table[6](https://arxiv.org/html/2610.11128#A5.T6)gives the common decoding settings\. Each question contributes 12 scored responses\.

Table 6:Evaluation configuration shared by all reported entries\.The scorer extracts the final balanced boxed answer\. A response without a boxed answer is incorrect\. Extracted answers are checked with Math\-Verify 0\.9\.0 and a normalized string comparison\. For each question,pass​@​12\\mathrm\{pass@12\}records whether at least one of its 12 responses is correct, whileavg​@​12\\mathrm\{avg@12\}is the fraction of those responses that are correct\. Both metrics are averaged over questions and reported as percentages\. Since the benchmarks have equal sizes, the mean over benchmarks also equals the mean over all 150 questions\.

## Appendix FAdditional Main Results

Table[7](https://arxiv.org/html/2610.11128#A6.T7)reportsavg​@​12\\mathrm\{avg@12\}for the same checkpoints and the same 12 scored responses per question as Table[1](https://arxiv.org/html/2610.11128#S5.T1), following the scoring protocol in Appendix[E\.5](https://arxiv.org/html/2610.11128#A5.SS5)\.

Table 7:Main experimental results inavg​@​12\\mathrm\{avg@12\}percentages\. Avg\. is the mean over the five benchmarks\. Within each model,boldandunderlinemark the best and second\-best values in each column\.ARG improves aggregateavg​@​12\\mathrm\{avg@12\}over GRPO by2\.672\.67and7\.457\.45percentage points on Qwen3\-4B and Qwen3\-8B, respectively, and achieves the second\-highest aggregateavg​@​12\\mathrm\{avg@12\}on both models\. The difference between the two metrics is consistent with where ARG intervenes\. Specifically, ARG modifies the GRPO update only for all\-failure groups, where allGGresponses to a training question are incorrect, and leaves the update for all other questions unchanged\. Therefore, the additional learning signal concentrates on questions that the model currently fails to solve\. When a question changes from unsolved to solved by a few of the 12 responses,pass​@​12\\mathrm\{pass@12\}counts the entire question, whereasavg​@​12\\mathrm\{avg@12\}counts only the fraction of correct responses\. As a result, the effect of ARG appears most clearly inpass​@​12\\mathrm\{pass@12\}, which measures the fraction of questions that the model solves within the sampling budget\.

## Appendix GAdditional Ablation Results

We provide benchmark\-level results, then examine how the ablations change the accepted trajectories and the prefix selector during training\. Reference insertion and resampling use Qwen3\-4B, while the surprisal\-weighting comparison covers both model sizes\.

### G\.1Evaluation Results

Tables[8](https://arxiv.org/html/2610.11128#A7.T8)and[9](https://arxiv.org/html/2610.11128#A7.T9)reportpass​@​12\\mathrm\{pass@12\}andavg​@​12\\mathrm\{avg@12\}separately for all five benchmarks\. All entries use the checkpoint after 200 training steps and the same evaluation protocol as the main results\. Each question contributes 12 scored responses, as specified in Appendix[E\.5](https://arxiv.org/html/2610.11128#A5.SS5)\.

Table 8:Benchmark\-level ablation results inpass​@​12\\mathrm\{pass@12\}percentages\. Avg\. is the mean over benchmarks\. Here w/o weighting denotes ARG without surprisal weighting\.Table 9:Benchmark\-level ablation results inavg​@​12\\mathrm\{avg@12\}percentages, computed from the same scored responses as Table[8](https://arxiv.org/html/2610.11128#A7.T8)\. Avg\. is the mean over benchmarks\.For the ablations in Table[9](https://arxiv.org/html/2610.11128#A7.T9), ARG has higher average sampled accuracy on every benchmark\.

### G\.2Accepted Candidate Trajectories during Training

To measure how often continuations supply the trajectories used for training, we count all\-failure groups that accept at least one candidate trajectory\. Reference fallback is excluded from this count\. Figure[2](https://arxiv.org/html/2610.11128#A7.F2)reports the proportion in consecutive windows of 50 training steps, using the total group counts within each window\. Each curve describes the all\-failure questions encountered in its own training run\.

\(a\)Qwen3\-4B resampling\(b\)Qwen3\-8B surprisal weighting
Figure 2:Proportion of all\-failure groups that accept candidate trajectories\. A point at stepssaggregates stepss−49s\-49throughss\. Reference fallback is excluded\. The curves come from their respective training runs\.On Qwen3\-4B, the proportion increases from59\.4%59\.4\\%to71\.8%71\.8\\%for ARG across the first and last windows, while resampling decreases from37\.6%37\.6\\%to26\.4%26\.4\\%\. Over the full runs, ARG accepts candidate trajectories in 318 of 482 all\-failure groups and uses reference fallback in the remaining 164\. Resampling accepts candidate trajectories in 145 of 446 all\-failure groups, leaving the remaining 301 unchanged\. Resampling retains all correct candidate trajectories\. Both methods use eight continuations per all\-failure group, and ARG also uses reference guidance and fallback\.

The weighting ablation gives a different comparison\. On Qwen3\-8B, it accepts candidate trajectories in64\.8%64\.8\\%of all\-failure groups, compared with60\.2%60\.2\\%for ARG, despite its lower evaluation scores\. On Qwen3\-4B, these proportions are similar at65\.9%65\.9\\%and66\.0%66\.0\\%\. Thus, the proportion of groups that accept candidate trajectories alone does not explain the evaluation results\.

### G\.3Surprisal Weighting and Prefix Selection

The weighting ablation uses the successful replacement indicator as feedback, while retaining the same guidance levels, prefix construction, selector hyperparameters, and acceptance rule\. Figure[3](https://arxiv.org/html/2610.11128#A7.F3)compares the resulting selection probabilities\. On Qwen3\-4B, both selectors favorα=1/4\\alpha=1/4for most of training\. On Qwen3\-8B, the unweighted selector favorsα=1/4\\alpha=1/4throughout training, whereas the probabilities under ARG change more over time\. The final probabilities under ARG are49\.0%49\.0\\%forα=1/2\\alpha=1/2and34\.6%34\.6\\%forα=1/4\\alpha=1/4, compared with6\.8%6\.8\\%and86\.4%86\.4\\%without weighting\.

\(a\)Qwen3\-4B ARG\(b\)Qwen3\-4B w/o weighting\(c\)Qwen3\-8B ARG\(d\)Qwen3\-8B w/o weighting
Figure 3:Selection probabilities of the prefix selector with and without surprisal weighting\. Curves show the state after each training step without smoothing\. Steps with no selector update retain the previous probabilities\. The levelα=1\\alpha=1uses an empty prefix, while smaller levels leave less of the reference surprisal to the continuation\.

## Appendix HTraining Dynamics

We supplement Section[5\.4](https://arxiv.org/html/2610.11128#S5.SS4)with the corresponding analysis on Qwen3\-4B and the sources of accepted trajectories for both models\. Comparisons with the ablated methods appear in Appendix[G](https://arxiv.org/html/2610.11128#A7)\.

### H\.1Successful Replacement, Feedback, and Prefix Selection

Figure[4](https://arxiv.org/html/2610.11128#A8.F4)reports successful replacement rates, mean selector feedback, and prefix selection on Qwen3\-4B\. As in the Qwen3\-8B plots, each point in the first two panels uses all continuations sampled at that level within a window of 50 training steps\. Successful replacement requires a correct candidate trajectory different from the reference\. We computeWtW\_\{t\}with the actualSt/S0S\_\{t\}/S\_\{0\}after prefix\-boundary adjustment, including zero feedback from failures and exact reference copies\. The third panel shows the selection probabilities after each training step without smoothing\.

\(a\)Successful replacement\(b\)Selector feedback\(c\)Prefix selection
Figure 4:Training dynamics of a Qwen3\-4B ARG run\. In panels \(a\) and \(b\), a point at stepssaverages the continuations at each level from stepss−49s\-49throughss\. Panel \(c\) shows the selection probabilities after each training step without smoothing\. The colors and corresponding axis ranges match Figure[1](https://arxiv.org/html/2610.11128#S5.F1)\.The levelα=1/4\\alpha=1/4has the highest successful replacement rate in each window\. Weighting reduces its advantage overα=1/2\\alpha=1/2, and their mean\-feedback ordering varies across windows\. Over the full run, the mean feedback is0\.07540\.0754forα=1/4\\alpha=1/4and0\.05880\.0588forα=1/2\\alpha=1/2\. These observations accompany the persistent preference forα=1/4\\alpha=1/4shown in Figure[4\(c\)](https://arxiv.org/html/2610.11128#A8.F4.sf3)\.

The windowed statistics summarize continuations collected as the policy, the all\-failure questions, and the selection probabilities change\. Since the selector updates sequentially from these continuations, its state reflects their order and the preceding updates\. Steps with no selector update retain the previous probabilities\. This gives a view of the feedback observed during training alongside the resulting selection history\.

### H\.2Sources of Accepted Trajectories

Figure[5](https://arxiv.org/html/2610.11128#A8.F5)records which trajectories enter policy learning\. We call a candidate trajectory guided if it comes from a nonempty prefix and unguided if it comes from the empty prefix\. Each all\-failure group belongs to one of four categories, according to whether it accepts only guided candidate trajectories, only unguided ones, both types, or the reference as fallback\. Thus, the figure describes groups, with each group counted once even when it accepts multiple trajectories\.

\(a\)Qwen3\-4B\(b\)Qwen3\-8B
Figure 5:Sources of accepted trajectories among all\-failure groups in consecutive windows of 50 training steps\. The mixed category contains groups that accept both guided and unguided candidate trajectories\. Fallback supplies the complete reference when no candidate trajectory is accepted\. Counts below the windows give the number of all\-failure groups\.Candidate trajectories are accepted in66\.0%66\.0\\%and60\.2%60\.2\\%of all\-failure groups for Qwen3\-4B and Qwen3\-8B, respectively\. The remaining groups use reference fallback\. Most groups that accept candidate trajectories use guided ones, while a smaller fraction accept unguided ones\. Fallback remains present throughout training, and its proportion varies across windows as the all\-failure questions change\. These records complement the selection curves by showing the trajectories used for the policy updates\.