Residual Advantage：面向可验证奖励强化学习的学生相对教师指导

# Residual Advantage:Student-Relative Teacher Guidancefor RL with Verifiable Rewards
Source: [https://arxiv.org/html/2610.11519](https://arxiv.org/html/2610.11519)
Xiaobing ChenAffiliation:Harbin Engineering UniversityAffiliation:TencentAffiliation:Harbin Institute of TechnologyEmail:[xbchen@stu\.hit\.edu\.cn](mailto:)

###### Abstract

Reinforcement learning with verifiable rewards \(RLVR\) and on\-policy distillation \(OPD\) have become two main paradigms for post\-training reasoning models\. RLVR gives each response a single outcome label, leaving the steps inside it without separate credit\. OPD provides token\-level guidance at student\-visited prefixes, but its pointwise signal does not directly reflect the pattern of teacher–student disagreement across the vocabulary\. Dense, unbounded log\-ratio supervision can amplify the teacher’s influence, yet a strong solver is not necessarily a suitable guide when the student’s solution paths depart from the teacher’s\. We propose Residual Advantage \(RA\), which treats the teacher–student probability residual as a bounded one\-step reward, subtracts the corresponding state value under the student policy to form a standard advantage, and centers the result within each response before adding it to the verifier advantage\. The guidance term has zero mean within each response, so the verifier advantage remains the response’s mean label and the teacher only redistributes credit among the steps within it\. Co\-RAfurther updates a teacher LoRA with verifier advantages on the same scored student batch and uses the updated teacher in the next iteration’s residual, adapting guidance to the student’s attempts\. With Qwen3\-1\.7B\-Base and Qwen3\-4B\-Base students and a Qwen3\-8B teacher,RAcombined with GRPO or REINFORCE\+\+ improves the underlying sequence\-advantage algorithm in all 24 comparisons on three mathematical benchmarks, raising macro Avg@8 by 1\.7–3\.6 points and Pass@8 by 3\.9–6\.3 points\. Both combinations surpass teacher\-only OPD, and Co\-RAadds a further 1\.0–1\.5 Avg@8 points\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\) trains a language model on its own sampled solutions, scored by a checker of the final answer\([Shao et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib29);[DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib5)\)\. GRPO and REINFORCE\+\+ turn this terminal reward into one advantage per response\([Shao et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib29);[Hu, 2025](https://arxiv.org/html/2610.11519#bib.bib13)\), so every step receives the same label even when the response mixes useful and mistaken steps\. Learning is also confined to the successful paths the student can sample, and under practical budgets much of the gain reflects sharpening of what the model can already do\([Yue et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib38);[Huang et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib14)\)\.

On\-policy distillation \(OPD\) instead supplies a teacher’s next\-token distribution at every prefix the student visits\([Agarwal et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib1);[Lu & Thinking Machines Lab, 2025](https://arxiv.org/html/2610.11519#bib.bib22)\)\. Its label, the log ratiolog⁡\(q⁡\(a\)/p⁡\(a\)\)\\log\(q\(a\)/p\(a\)\)or one coordinate of a top\-kkor full\-vocabulary divergence, says how far apart teacher and student are at that token but does not directly reflect where the teacher’s extra mass sits; a full\-vocabulary objective sees this only implicitly, through the softmax coupling of the coordinates\. The log\-ratio label is moreover unbounded, growing aslog⁡\(1/q⁡\(a\)\)\\log\(1/q\(a\)\)at a token the teacher assigns little probability, so on such tokens the teacher’s signal can become disproportionately strong\. At the same time, a strong solver is not necessarily a good teacher for the current student, least of all on prefixes where the student has left the teacher’s solution paths\([Li et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib18);[Tiapkin et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib31)\)\. The question is how teacher guidance should enter RLVR so that the student draws on the teacher’s knowledge while the response\-level label stays tied to outcome feedback and room for exploration remains\.

Figure 1:Local teacher guidance redistributes the verifier’s response\-level credit\.\(a\)At one prefix, the rewardQd​\(s,v\)=qs​\(v\)−ps​\(v\)Q\_\{d\}\(s,v\)=q\_\{s\}\(v\)\-p\_\{s\}\(v\)over the student’s likeliest tokens, its valueVdp​\(s\)V\_\{d\}^\{p\}\(s\)under the student policy, and the advantageAsd​\(a\)A\_\{s\}^\{d\}\(a\)of the sampled token, here the student’s third choice and the teacher’s first\.\(b\)Labels along one response, with blue and coral stems for positive and negative guidance termsδt\\delta\_\{t\}; each label isAVA^\{V\}plus a centered guidance term, so the labels’ weighted mean isAVA^\{V\}\.To this end, we propose Residual Advantage \(RA\)\.RAtreats the probability differenced=q−pd=q\-pas a bounded one\-step reward\. At each visited prefix it subtracts the reward’s expectation under the student’s own policy, that is, the policy’s value under this reward,

Ad​\(a\)=d⁡\(a\)−⟨p,d⟩=𝔼v∼p​\[\(q⁡\(a\)−q⁡\(v\)\)−\(p⁡\(a\)−p⁡\(v\)\)\],A^\{d\}\(a\)=d\(a\)\-\\langle p,d\\rangle=\\mathbb\{E\}\_\{v\\sim p\}\\big\[\(q\(a\)\-q\(v\)\)\-\(p\(a\)\-p\(v\)\)\\big\],This design reflects how the teacher’s disagreement with the student is distributed\. It asks whether, relative to the student, the teacher favors the sampled token over the other candidates the student would consider, each weighted by the student’s probability of choosing it\. Even with the same residual at the sampled token, teachers that distribute the rest of the vocabulary differently can therefore give local advantages of opposite signs \(Appendix[A\.4](https://arxiv.org/html/2610.11519#A1.SS4)\)\.RAthen centers the local advantages within the response and adds them to the verifier advantage,At=AV\+λ⁡\(Atd−A¯d\)A\_\{t\}=A^\{V\}\+\\lambda\(A\_\{t\}^\{d\}\-\\overline\{A\}^\{d\}\), whereA¯d\\overline\{A\}^\{d\}is the response mean andλ\\lambdathe guidance weight\. This gives the guidance term zero mean within every response, so the verifier advantage remains the response’s mean label and the teacher only redistributes credit among the steps within it\. The guidance is thus student\-relative twice over, as an advantage under the student’s own next\-token distribution and as a deviation within the student’s own response\.

The appeal of the construction is that it is both elegant and natural\. The local advantage is a single inner product,Ad​\(a\)=⟨ea−p,q−p⟩A^\{d\}\(a\)=\\langle e\_\{a\}\-p,\\,q\-p\\rangle, and three properties follow from it\. It is the rate at which the squared distance12​∥p−q∥22\\tfrac\{1\}\{2\}\\lVert p\-q\\rVert\_\{2\}^\{2\}falls when the student moves probability towardaa\. Its expected update at a prefix is the exact logit gradient of that distance, so the student’s own probability of a token scales how strongly the teacher can move it\. And it is bounded by the student–teacher distance times the probability the student leaves to other tokens, so guidance fades on its own as the two converge\. Scaled in this way by the student’s own probability, it also stays small wherever the student is already sure, so the teacher mainly adjusts the choices the student is still weighing instead of imposing its preference on those the student has settled, and it does not inflate small disagreements into large labels\. The construction requires no schedule, threshold, or learned component\.

Co\-RAextends the construction to a teacher that learns from the student’s attempts\. Each iteration scores a student batch, fixes its labels, and updates a teacher LoRA with the base estimator’s verifier advantages while the student receives the guided label, much as actor and critic share a rollout batch in PPO\([Schulman et al\., 2017](https://arxiv.org/html/2610.11519#bib.bib28)\)\. The updated teacher supplies the distribution in the next iteration’s residual, with the same advantage construction and weight asRA\. Training on the student’s own rollouts with the verifier’s advantages lets the teacher adapt to the reasoning along the student’s solution paths, with the aim of reducing the bias a frozen teacher carries on trajectories it would not have generated itself\. Across two students, two estimators, and three mathematical benchmarks,RAimproves all 24 three\-seed benchmark means of its base estimator, both combinations surpass teacher\-only OPD, and Co\-RAraises accuracy further with mixed changes in coverage\.

## 2Residual Advantage

#### Setting\.

A studentpθp\_\{\\theta\}generates a responseai,1:Lia\_\{i,1:L\_\{i\}\}to promptxix\_\{i\}\. At the visited prefixξi​t=\(xi,ai,<t\)\\xi\_\{it\}=\(x\_\{i\},a\_\{i,<t\}\), letpi​tp\_\{it\}be the rollout student’s next\-token distribution andqi​tq\_\{it\}the teacher’s distribution at the same prefix over the same aligned vocabulary\. We write⟨p,f⟩=∑vp⁡\(v\)​f​\(v\)\\langle p,f\\rangle=\\sum\_\{v\}p\(v\)f\(v\), leteae\_\{a\}be the one\-hot vector of tokenaa, and drop the indexiiwhen a single response is meant\. The base estimator converts a terminal verifier reward into one advantageAiVA\_\{i\}^\{V\}per response, by GRPO’s within\-group normalization\([Shao et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib29)\)or by REINFORCE\+\+’s batch statistics\([Hu, 2025](https://arxiv.org/html/2610.11519#bib.bib13)\), and the actor maximizes the clipped PPO surrogate\([Schulman et al\., 2017](https://arxiv.org/html/2610.11519#bib.bib28)\)with importance ratioπθ​\(a∣ξ\)/p⁡\(a∣ξ\)\\pi\_\{\\theta\}\(a\\mid\\xi\)/p\(a\\mid\\xi\), the actor’s own ratio, distinct from the teacher–student ratioq/pq/p\.RAchanges only the per\-token label that enters this surrogate\. Every teacher\-derived quantity is computed at the rollout policy and held fixed, so no gradient flows through it\.

### 2\.1Student\-relative residual advantage

At prefixss, we define the one\-step reward of tokenvvas the teacher–student probability*residual*,

Rs​\(v\)=ds​\(v\)=q⁡\(v∣s\)−p⁡\(v∣s\)\.R\_\{s\}\(v\)=d\_\{s\}\(v\)=q\(v\\mid s\)\-p\(v\\mid s\)\.\(1\)Under the student policy, the state value of this reward isVdp​\(s\)=𝔼v∼p​Rs​\(v\)=⟨ps,ds⟩V\_\{d\}^\{p\}\(s\)=\\mathbb\{E\}\_\{v\\sim p\}R\_\{s\}\(v\)=\\langle p\_\{s\},d\_\{s\}\\rangle, its action value is the immediate rewardQd​\(s,a\)=ds​\(a\)Q\_\{d\}\(s,a\)=d\_\{s\}\(a\), and the token advantage is

Asd​\(a\)=Qd​\(s,a\)−Vdp​\(s\)=ds​\(a\)−⟨ps,ds⟩=⟨ea−ps,ds⟩\.\\boxed\{\\,A\_\{s\}^\{d\}\(a\)=Q\_\{d\}\(s,a\)\-V\_\{d\}^\{p\}\(s\)=d\_\{s\}\(a\)\-\\langle p\_\{s\},d\_\{s\}\\rangle=\\langle e\_\{a\}\-p\_\{s\},\\,d\_\{s\}\\rangle\.\\,\}\(2\)The reward is immediate, so nothing is propagated to earlier positions; credit across positions comes from the verifier advantage\. SubtractingVdpV\_\{d\}^\{p\}is the REINFORCE baseline for this reward, computed exactly\([Williams, 1992](https://arxiv.org/html/2610.11519#bib.bib33);[Sutton et al\., 2000](https://arxiv.org/html/2610.11519#bib.bib30);[Foerster et al\., 2018](https://arxiv.org/html/2610.11519#bib.bib7)\), and we call it the*student value baseline*\. The gapds​\(a\)d\_\{s\}\(a\)at the sampled token alone does not determine the sign \(Appendix[A\.4](https://arxiv.org/html/2610.11519#A1.SS4)\)\.

###### Proposition 1\(Three properties of the local advantage\)

At a fixed prefix, whose index we drop, letL⁡\(p,q\)=12​∥p−q∥22L\(p,q\)=\\tfrac\{1\}\{2\}\\lVert p\-q\\rVert\_\{2\}^\{2\}, letp=softmax⁡\(z\)p=\\operatorname\{softmax\}\(z\), and letψa=ea−p=∇z​log​p​\(a\)\\psi\_\{a\}=e\_\{a\}\-p=\\nabla\_\{z\}\\log p\(a\)be the score of tokenaain logit coordinates\.\(i\)Ad​\(a\)=⟨ψa,d⟩A^\{d\}\(a\)=\\langle\\psi\_\{a\},d\\rangleis the first\-order decrease ofLLalong the feasible movep↦\(1−η\)​p\+η​eap\\mapsto\(1\-\\eta\)p\+\\eta e\_\{a\}; the exact change is−η​Ad​\(a\)\+η22​∥ψa∥22\-\\eta A^\{d\}\(a\)\+\\tfrac\{\\eta^\{2\}\}\{2\}\\lVert\\psi\_\{a\}\\rVert\_\{2\}^\{2\}\.\(ii\)𝔼a∼p​ψa=0\\mathbb\{E\}\_\{a\\sim p\}\\psi\_\{a\}=0and𝔼a∼p​ψa​ψa⊤=Jp=Diag⁡\(p\)−p​p⊤\\mathbb\{E\}\_\{a\\sim p\}\\psi\_\{a\}\\psi\_\{a\}^\{\\top\}=J\_\{p\}=\\operatorname\{Diag\}\(p\)\-pp^\{\\top\}, the Jacobian of the softmax\. Hence, withddheld fixed,𝔼a∼p​Ad​\(a\)=0\\mathbb\{E\}\_\{a\\sim p\}A^\{d\}\(a\)=0and

𝔼a∼p​\[Ad​\(a\)​ψa\]=Jp​d=−∇zL​\(p,q\),−∂L∂zb=p⁡\(b\)​Ad​\(b\)\.\\mathbb\{E\}\_\{a\\sim p\}\\big\[A^\{d\}\(a\)\\,\\psi\_\{a\}\\big\]=J\_\{p\}d=\-\\nabla\_\{z\}L\(p,q\),\\qquad\-\\frac\{\\partial L\}\{\\partial z\_\{b\}\}=p\(b\)\\,A^\{d\}\(b\)\.\(3\)\(iii\)\|Ad​\(a\)\|≤∥ψa∥2​∥d∥2≤2​\(1−p⁡\(a\)\)​∥d∥2\|A^\{d\}\(a\)\|\\leq\\lVert\\psi\_\{a\}\\rVert\_\{2\}\\lVert d\\rVert\_\{2\}\\leq\\sqrt\{2\}\\,\(1\-p\(a\)\)\\lVert d\\rVert\_\{2\}, and−12≤Ad​\(a\)≤2\-\\tfrac\{1\}\{2\}\\leq A^\{d\}\(a\)\\leq 2\.

Part \(i\) says what the local advantage measures\. Part \(ii\) says what it does on average\. The update descends the squared distance to the teacher, and logitbbmoves byp⁡\(b\)​Ad​\(b\)p\(b\)A^\{d\}\(b\), so the student’s own probability of a token scales how far the teacher can move it\. Since𝔼a∼p​ψa=0\\mathbb\{E\}\_\{a\\sim p\}\\psi\_\{a\}=0, the student value baseline is applied automatically in expectation at a single prefix; its role appears only when local advantages are compared across prefixes \(Appendix[A\.7](https://arxiv.org/html/2610.11519#A1.SS7)\)\. Part \(iii\) is where the residual parts from the log ratio, which estimates reverse\-KL descent without bias but has no bound\. The residual’s bound gives the guidance bounds of Section[2\.2](https://arxiv.org/html/2610.11519#S2.SS2)\.

Separating the sampled token makes this scaling explicit,

Ad\(a\)=\(1−p\(a\)\)\[d\(a\)−𝔼v∼p\(⋅∣v≠a\)d\(v\)\],A^\{d\}\(a\)=\\big\(1\-p\(a\)\\big\)\\Big\[d\(a\)\-\\mathbb\{E\}\_\{v\\sim p\(\\cdot\\mid v\\neq a\)\}d\(v\)\\Big\],\(4\)the teacher’s preference foraaover the student’s other choices, weighted by the probability the student still gives them\. The local advantage vanishes on a token the student has settled, whatever the teacher says, and by part \(ii\) logitbbmoves in expectation byp​\(b\)​\(1−p​\(b\)\)p\(b\)\(1\-p\(b\)\)times a bounded contrast, so the teacher acts most where the student is undecided\. None of this is imposed by hand\. The relative comparison, the confidence scaling, the zero mean, and the bound are all read off one inner product\.

### 2\.2Combining with verifier feedback

Letct≥0c\_\{t\}\\geq 0be the actor’s relative weights on the response’s positions, summing to one \(ct=1/Mc\_\{t\}=1/Mfor token\-mean reduction overMMpositions\), and letA¯d=∑tct​Atd\\overline\{A\}^\{d\}=\\sum\_\{t\}c\_\{t\}A\_\{t\}^\{d\}be the response mean of the local advantages\.RAadds the centered local advantage, scaled by the guidance weightλ\\lambda, to the verifier advantage,

δt=λ\(Atd−A¯d\),At=AV\+δt,λ≥0\.\\boxed\{\\,\\delta\_\{t\}=\\lambda\\,\(A\_\{t\}^\{d\}\-\\overline\{A\}^\{d\}\),\\qquad A\_\{t\}=A^\{V\}\+\\delta\_\{t\},\\qquad\\lambda\\geq 0\.\\,\}\(5\)For every response,

∑tct​At=AV,δt−δu=λ⁡\(Atd−Aud\),\\sum\_\{t\}c\_\{t\}A\_\{t\}=A^\{V\},\\qquad\\delta\_\{t\}\-\\delta\_\{u\}=\\lambda\\,\(A\_\{t\}^\{d\}\-A\_\{u\}^\{d\}\),\(6\)so the verifier advantage is exactly the response’s mean label and every difference between two decisions is preserved\. Under the weighted inner product⟨u,v⟩c=∑tct​ut​vt\\langle u,v\\rangle\_\{c\}=\\sum\_\{t\}c\_\{t\}u\_\{t\}v\_\{t\}, the constant verifier label and the centered guidance term are orthogonal, and the label splits as

∥𝐀∥2,c2=\(AV\)2\+λ2​Vart∼c⁡\(Atd\)\.\\lVert\\mathbf\{A\}\\rVert\_\{2,c\}^\{2\}=\(A^\{V\}\)^\{2\}\+\\lambda^\{2\}\\operatorname\{Var\}\_\{t\\sim c\}\(A\_\{t\}^\{d\}\)\.\(7\)The verifier decides how good the response is as a whole, and the teacher decides how that credit is shared among its steps\. This division concerns the label\. Multiplied by different score vectors, a zero\-mean guidance term need not give a gradient orthogonal to the verifier’s, and on a single token it can outweigh it \(Appendices[A\.3](https://arxiv.org/html/2610.11519#A1.SS3)and[A\.7](https://arxiv.org/html/2610.11519#A1.SS7)\)\. Centering is the orthogonal projection onto zero\-mean labels\. Without it, a nonzero response mean in the teacher label would be indistinguishable from a change ofAVA^\{V\}, since both multiply the same mean score direction∑tct​ψt\\sum\_\{t\}c\_\{t\}\\psi\_\{t\}\.

#### Bounded by the distance, fading on its own\.

By Proposition[1](https://arxiv.org/html/2610.11519#Thmproposition1)\(iii\),

\|δt\|≤52​λ​\(1−ct\),∑tct​\|δt\|≤∥δ∥2,c≤2​λ​\(∑tct​∥pt−qt∥22\)1/2\.\|\\delta\_\{t\}\|\\leq\\tfrac\{5\}\{2\}\\lambda\(1\-c\_\{t\}\),\\qquad\\sum\_\{t\}c\_\{t\}\|\\delta\_\{t\}\|\\leq\\lVert\\delta\\rVert\_\{2,c\}\\leq\\sqrt\{2\}\\lambda\\Big\(\\sum\_\{t\}c\_\{t\}\\lVert p\_\{t\}\-q\_\{t\}\\rVert\_\{2\}^\{2\}\\Big\)^\{1/2\}\.\(8\)The first bound limits each guidance term\. The second bounds the guidance amplitude by the student–teacher distance on the visited prefixes \(Appendix[A\.6](https://arxiv.org/html/2610.11519#A1.SS6)\), so teacher guidance fades as the two converge, at a fixed weight and without a schedule; Appendix[C\.1](https://arxiv.org/html/2610.11519#A3.SS1)shows the mean absolute guidance declining over a training run\. The log ratio also vanishes at exact agreement but admits no such bound, since it diverges at a token of vanishing student probability while∥p−q∥2→0\\lVert p\-q\\rVert\_\{2\}\\to 0\.

Figure 2:One iteration ofRAand Co\-RA\.Three schematic responses, only the middle one correct, receive verifier advantagesAiVA\_\{i\}^\{V\}\.RAadds a centered guidance term to every token label\. Token strips show one cell per position, blue forAt\>0A\_\{t\}\>0and rose forAt<0A\_\{t\}<0, darker for larger\|At\|\|A\_\{t\}\|; the last cell is the response mean, which centering pins toAiVA\_\{i\}^\{V\}\. Insets show \(a\) the student value baseline at one prefix and \(b\) response centering\. In Co\-RA, a LoRA adapter on the frozen teacher is trained with the verifier advantages of the same scored batch and supplies the next iteration’s residual\.

### 2\.3Co\-RAteaches the teacher on the student’s attempts

Co\-RAfollows a score\-then\-update cycle\. In every iteration the student generates a batch, the verifier scores it, and the current student and teacher produce the actor labels\. These labels are fixed before the student and a LoRA adapterqϕq\_\{\\phi\}\([Hu et al\., 2022](https://arxiv.org/html/2610.11519#bib.bib12)\)on the frozen teacherq0q\_\{0\}are updated on the same batch\. Both updates use the run’s own GRPO or REINFORCE\+\+ advantages, the adapter receivingAiVA\_\{i\}^\{V\}and the student the combined label, so the teacher is adapted at the prefixes this student visits according to how its continuations turned out\. The aim is for the teacher to become familiar with the reasoning along the student’s solution paths and to take in the verifier’s signal there, reducing its bias on the student’s trajectories so that its guidance becomes more accurate for this student\. With batch token\-mean reduction \(Appendix[B\.8](https://arxiv.org/html/2610.11519#A2.SS8)\), the adapter’s gradient at the start of each update is

∇ϕ𝒥T\|ϕk=1Z​∑iAiV​∑t∈𝒯i∇ϕ​log​qϕ,i​t​\(ai​t\)\|ϕk,Z=∑i\|𝒯i\|,\\left\.\\nabla\_\{\\phi\}\\mathcal\{J\}\_\{T\}\\right\|\_\{\\phi\_\{k\}\}=\\frac\{1\}\{Z\}\\sum\_\{i\}A\_\{i\}^\{V\}\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\}\\left\.\\nabla\_\{\\phi\}\\log q\_\{\\phi,it\}\(a\_\{it\}\)\\right\|\_\{\\phi\_\{k\}\},\\qquad Z=\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|,\(9\)where𝒯i\\mathcal\{T\}\_\{i\}is the set of token positions of responseii\. This is the base estimator’s clipped update applied to the teacher LoRA on student\-generated data\.

At iterationkkthe adapted teacherqϕkq\_\{\\phi\_\{k\}\}supplies the residualdt\(k\)=qϕk,t−pk,td\_\{t\}^\{\(k\)\}=q\_\{\\phi\_\{k\},t\}\-p\_\{k,t\}against the rollout studentpkp\_\{k\}, and the student uses the same construction and weightλ=10\\lambda=10asRA,

AtCo\-RA=AV\+λ⁡\(Atd,\(k\)−A¯d,\(k\)\),Atd,\(k\)=dt\(k\)​\(at\)−⟨pk,t,dt\(k\)⟩,A\_\{t\}^\{\\text\{Co\-\{RA\}\}\}=A^\{V\}\+\\lambda\\bigl\(A\_\{t\}^\{d,\(k\)\}\-\\overline\{A\}^\{d,\(k\)\}\\bigr\),\\qquad A\_\{t\}^\{d,\(k\)\}=d\_\{t\}^\{\(k\)\}\(a\_\{t\}\)\-\\langle p\_\{k,t\},d\_\{t\}^\{\(k\)\}\\rangle,\(10\)whereA¯d,\(k\)=∑tct​Atd,\(k\)\\overline\{A\}^\{d,\(k\)\}=\\sum\_\{t\}c\_\{t\}A\_\{t\}^\{d,\(k\)\}\. After the updates,qϕk\+1q\_\{\\phi\_\{k\+1\}\}supplies the residual in the next iteration\. The adapter starts at zero, so the first student update coincides withRA\. At each scored batch the response\-mean identity and the distance bound hold with the current teacherqϕkq\_\{\\phi\_\{k\}\}\(Appendix[A\.9](https://arxiv.org/html/2610.11519#A1.SS9)\)\.

## 3Experiments

Table 1:Residual guidance improves both RLVR estimators on every benchmark and metric, and the adapted teacher raises accuracy further\.Three\-seed means in %\. Student rows are untrained, rows marked \+ addRAor Co\-RAto the estimator above, bold marks the best per column and student, and the last row is the Qwen3\-8B teacher; per\-seed scores and SD are in Appendix[D](https://arxiv.org/html/2610.11519#A4)\.#### Setup\.

Students are Qwen3\-1\.7B\-Base and Qwen3\-4B\-Base; the teacher is the post\-trained Qwen3\-8B model in non\-thinking mode, which shares their vocabulary\([Yang et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib35)\)\. Training uses DAPO\-Math\-17K\([Yu et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib36)\)for two epochs with 128 prompts per rollout batch, eight responses per prompt for GRPO runs and one for OPD and REINFORCE\+\+ runs, temperature1\.01\.0, and a 12,000\-token cap\. Verifier runs reward answer correctness, linearly discounted for correct responses beyond 8,000 tokens \(Appendix[B\.3](https://arxiv.org/html/2610.11519#A2.SS3)\), andRAis added to the base estimator at the reference weightλ=10\\lambda=10, chosen on a validation split of training prompts \(Appendix[B\.5](https://arxiv.org/html/2610.11519#A2.SS5)\)\. Teacher\-only OPD minimizes the reverse KL divergenceKL\(pt∥qt\)\\operatorname\{KL\}\(p\_\{t\}\\\|q\_\{t\}\)over the full vocabulary at every visited prefix\([Agarwal et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib1);[Lu & Thinking Machines Lab, 2025](https://arxiv.org/html/2610.11519#bib.bib22)\), the same teacher informationRAuses\. Evaluation samples eight completions per problem at temperature0\.80\.8with the same cap on MATH\-500\([Lightman et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib20)\), AIME24, and AIME25 and reports Avg@8 and Pass@8 \(Table[1](https://arxiv.org/html/2610.11519#S3.T1)\) from unshaped answer correctness; macro scores weight the three benchmarks equally\. EachRArun adds one teacher forward pass per rollout, and Co\-RAalso trains the adapter \(Appendix[B\.8](https://arxiv.org/html/2610.11519#A2.SS8)\)\.

#### RAimproves every matched comparison\.

Table[1](https://arxiv.org/html/2610.11519#S3.T1)contains 24 comparisons ofRAwith its base estimator \(two students, two estimators, three benchmarks, two metrics\), andRAimproves all 24 three\-seed means\. The macro gains are1\.741\.74–3\.583\.58Avg@8 and3\.893\.89–6\.336\.33Pass@8 points \(Table[10](https://arxiv.org/html/2610.11519#A2.T10)\)\.

#### The guidance transfers across estimators\.

The gains are larger for REINFORCE\+\+, narrowing its macro Avg@8 deficit to GRPO at both sizes\. GRPO keeps the higher completion accuracy after guidance, but REINFORCE\+\+ withRAhas the higher macro Pass@8 at both sizes, a trade between accuracy and coverage that a single score would hide\.

#### Verifier plus teacher improves on teacher\-only distillation\.

BothRAcombinations exceed teacher\-only OPD in macro Avg@8 and Pass@8 at both sizes, by1\.801\.80–2\.962\.96and2\.732\.73–3\.903\.90points \(Table[10](https://arxiv.org/html/2610.11519#A2.T10)\)\. The one exception is AIME25 Avg@8 at 1\.7B, where OPD’s5\.975\.97exceeds the5\.005\.00ofRA\+REINFORCE\+\+, although bothRAruns have the higher Pass@8 there\. In training, OPD also starts more slowly than REINFORCE\+\+ and stays belowRA\+REINFORCE\+\+ over most of the shared budget \(Appendix[C](https://arxiv.org/html/2610.11519#A3)\)\. At the same budget, OPD followed by REINFORCE\+\+ reaches28\.3928\.39and REINFORCE\+\+ with an added reverse\-KL loss26\.1426\.14, both belowRA\+REINFORCE\+\+ \(Appendix[D\.5](https://arxiv.org/html/2610.11519#A4.SS5)\), and sampling the log ratio at the generated token, with the same expected update at each prefix, drops OPD to24\.8024\.80at 1\.7B, below verifier\-only training \(Appendix[D\.6](https://arxiv.org/html/2610.11519#A4.SS6)\)\.

#### Teaching the teacher on student rollouts raises accuracy in every tested setting\.

Co\-RAadds1\.031\.03–1\.451\.45macro Avg@8 points to the fourRAruns, and Avg@8 rises in all twelve benchmark means\. Coverage changes are mixed: Pass@8 rises in four benchmark means, is unchanged in two, and falls in six \(Table[10](https://arxiv.org/html/2610.11519#A2.T10)\)\. Co\-RA\+GRPO is the most accurate run for both students\. Yet each adapted teacher scores0\.80\.8–2\.32\.3macro Avg@8 and0\.70\.7–3\.53\.5Pass@8 points below the frozen teacher \(Appendix[D\.9](https://arxiv.org/html/2610.11519#A4.SS9)\), while the students it guides improve: it is a better guide, not a better solver\.

#### The margin over the untrained student persists up to Pass@512\.

FromK=8K=8toK=512K=512for the 1\.7B REINFORCE\+\+ runs \(Appendix[D\.7](https://arxiv.org/html/2610.11519#A4.SS7)\), the macro margin ofRAover the untrained student grows from13\.413\.4to16\.916\.9points, and its margins over REINFORCE\+\+ and OPD from5\.85\.8to10\.410\.4and from3\.93\.9to6\.86\.8, while that of verifier\-only training narrows from7\.67\.6to6\.66\.6, as[Yue et al\. \(2025\)](https://arxiv.org/html/2610.11519#bib.bib38)report for RLVR\. Within the tested budgetRAthus raises the coverage the student can reach\.

## 4Analysis

### 4\.1Ablation Study

We remove the student value baseline and response centering in turn on Qwen3\-1\.7B\-Base with REINFORCE\+\+ at the unchanged weightλ=10\\lambda=10and compare every control with fullRA\. Since removing either operation also changes the guidance amplitude, every control is also run at weights55and2020\(Table[2](https://arxiv.org/html/2610.11519#S4.T2), Appendix[D\.4](https://arxiv.org/html/2610.11519#A4.SS4)\), so that no conclusion rests on a single amplitude\.

#### Student value baseline\.

Removing the student value baseline lowers both macro scores, by2\.092\.09Avg@8 and3\.583\.58Pass@8 points at the reference weight and similarly at weights55and2020\(Table[2](https://arxiv.org/html/2610.11519#S4.T2)\)\. The label then carries the sampled residualdt​\(at\)d\_\{t\}\(a\_\{t\}\), which still contains the state value⟨pt,dt⟩\\langle p\_\{t\},d\_\{t\}\\rangle\. This value is fixed before the token is chosen, so it rewards whatever token is sampled at a prefix where the teacher concentrates on the student’s likely tokens, and response centering then spreads this prefix\-level effect over the other positions \(Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4)\)\. Removing both operations costs1\.731\.73Avg@8 and2\.772\.77Pass@8 points; the state value then also enters the response mean, shifting it by−0\.32\-0\.32on average and beyond0\.50\.5on a quarter of the offline responses \(Table[16](https://arxiv.org/html/2610.11519#A3.T16)\)\.

Table 2:Every residual variant improves on verifier\-only training at every tested weight, and fullRAstays above every control at every matched weight\.Qwen3\-1\.7B\-Base with REINFORCE\+\+; three\-seed macro means in %\. The weight isλ\\lambda,γx\\gamma\_\{x\},γraw\\gamma\_\{\\rm raw\}, orα\\alphaas marked, with1010the reference weight; per\-seed values and sample SD are in Table[23](https://arxiv.org/html/2610.11519#A4.T23)\.
#### Response centering\.

Restoring response centering at the same weight adds0\.540\.54Avg@8 and0\.460\.46Pass@8 points, a smaller gain that holds at all three weights \(Table[2](https://arxiv.org/html/2610.11519#S4.T2)\)\. Without centering, the response’s mean label isAV\+λ​A¯dA^\{V\}\+\\lambda\\overline\{A\}^\{d\}\. On the 256 offline responses of Appendix[C\.2](https://arxiv.org/html/2610.11519#A3.SS2), the local advantages average to3×10−43\\times 10^\{\-4\}per token, so the student value baseline leaves no systematic part, and on responses longer than1,0001\{,\}000tokens the shift has an SD below0\.030\.03, where centering changes almost nothing \(Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4)\)\. On shorter responses, where few tokens set the average, the SD is0\.320\.32, and8%8\\%of all responses carry a shift beyond0\.50\.5, the scale of a verifier advantage \(Table[16](https://arxiv.org/html/2610.11519#A3.T16)\)\. Without centering these responses receive a response\-level label the verifier did not assign, which can even reverse the sign of the verifier’s judgment; centering removes this shift deterministically for every response \(Equation[6](https://arxiv.org/html/2610.11519#S2.E6)\), so the response\-level label no longer depends on how the guidance happens to average along a response\. Across seeds, fullRAalso varies less than the uncentered control in macro Avg@8 at every weight \(Table[23](https://arxiv.org/html/2610.11519#A4.T23)\)\.

#### Guidance weight\.

All tested weights improve both macro scores over verifier\-only training \(Table[2](https://arxiv.org/html/2610.11519#S4.T2)\)\. Halving the reference weight lowers both means; doubling it leaves them nearly unchanged \(29\.2929\.29vs\.29\.2529\.25Avg@8\) but widens the seed SD from0\.250\.25to1\.681\.68, as in every variant \(Table[23](https://arxiv.org/html/2610.11519#A4.T23)\)\.

### 4\.2Probability residual versus log ratio

Table 3:The probability residual with both baselines is the best construction, and no log\-ratio variant reaches it at any swept weight\.Three\-seed macro means in % with SD; weights marked†were chosen on the validation split \(Appendix[B\.5](https://arxiv.org/html/2610.11519#A2.SS5)\), and Tables[23](https://arxiv.org/html/2610.11519#A4.T23)and[24](https://arxiv.org/html/2610.11519#A4.T24)give all swept weights\.#### Log ratio\.

Table[3](https://arxiv.org/html/2610.11519#S4.T3)replaces the residual by the log ratio at the sampled token, raw or with both operations\. The raw log ratio helps only at small weights, peaking atβ=0\.2\\beta=0\.2, and falls below verifier\-only training fromβ=1\.2\\beta=1\.2on; its conditional mean at a prefix is−KL\(pt∥qt\)\-\\operatorname\{KL\}\(p\_\{t\}\\\|q\_\{t\}\)\(Appendix[A\.7](https://arxiv.org/html/2610.11519#A1.SS7)\), so it shifts the mean label in proportion toβ\\beta\. With both operations, clipped to\[−5,5\]\[\-5,5\]or not, the log ratio stays above verifier\-only training at every swept weight, and the probability residual is better still, by1\.211\.21Avg@8 and1\.701\.70Pass@8 points over the best log variant\.

#### Direct loss\.

A full\-vocabulary squared\-distance loss, whose gradient at each visited prefix equals the expected update of the uncentered residual label \(Appendix[A\.2](https://arxiv.org/html/2610.11519#A1.SS2)\), beats verifier\-only training but stays belowRAand the uncentered control at every weight \(Table[2](https://arxiv.org/html/2610.11519#S4.T2)\)\. The gap lies not in the expected teacher gradient, which the sampled label matches \(cosine0\.990\.99with its analytic value\), but in how the update is shared with the verifier \(Appendix[C\.3](https://arxiv.org/html/2610.11519#A3.SS3)\)\. BecauseRA’s teacher term acts on the tokens the verifier scores,64%64\\%of its gradient mass raises correct or lowers incorrect responses, against49%49\\%for the direct loss, and after clipping and Adam the teacher takes0\.280\.28of the update against0\.440\.44, keepingRA’s update closer to the verifier’s direction \(cosine0\.930\.93against0\.800\.80\)\.

### 4\.3Where the guidance lands

By Equation[4](https://arxiv.org/html/2610.11519#S2.E4)and Proposition[1](https://arxiv.org/html/2610.11519#Thmproposition1)\(ii\), the teacher moves logitbbin expectation byp​\(b\)​\(1−p​\(b\)\)p\(b\)\(1\-p\(b\)\)times a bounded contrast\. The student\-centered log ratio factorizes the same way, but its contrast is unbounded in both directions \(Appendix[A\.3](https://arxiv.org/html/2610.11519#A1.SS3)\)\. Table[15](https://arxiv.org/html/2610.11519#A3.T15)\(Appendix[C\.2](https://arxiv.org/html/2610.11519#A3.SS2)\) describes the start of training, bucketing the tokens of 256 offline responses of the untrained 1\.7B student by the student’s probability of the sampled token\. Confident tokens, withp⁡\(at\)≥0\.9p\(a\_\{t\}\)\\geq 0\.9, are 61% of all tokens and receive under 1% of either label, as the factor1−p⁡\(a\)1\-p\(a\)predicts\. The residual puts 59% of its guidance on the 12% of tokens with0\.1≤p⁡\(at\)<0\.50\.1\\leq p\(a\_\{t\}\)<0\.5, five times their share, where the student is choosing among live alternatives\. The centered log ratio puts 50% of its total on the 4% of tokens the teacher assigns less than10−410^\{\-4\}, where its label diverges\. The residual thus weights the forks of the student’s own reasoning, consistent with the finding that a few high\-entropy tokens carry most of the useful RL signal\([Wang et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib32)\), whereas the log ratio puts it on tokens that one side considers nearly impossible\.

### 4\.4More guidance is not always better

RAassigns the teacher a restrained role\. It labels only the tokens the student samples, so where the teacher clearly disagrees, its preferred token receives no supervision of its own unless the student samples it\. A natural alternative is denser, more direct supervision that prescribes the student’s actions rather than only evaluating them; we examine it through a minimal extension of the sampled route, a teacher\-peak branch\. At each position where the teacher’s top tokenvt⋆=arg⁡maxv​qt​\(v\)v\_\{t\}^\{\\star\}=\\arg\\max\_\{v\}q\_\{t\}\(v\)differs from the student’s top token and from the sampled token, which may itself be the student’s top token, the branch addsvt⋆v\_\{t\}^\{\\star\}as a pseudo\-sample\. The added token receives the label a sampled token there would receive, with the verifier part clipped at zero,\(\[AV\]\+\+λ⁡\(ht​\(vt⋆\)−h¯\)\)​log⁡πθ​\(vt⋆\)\\big\(\[A^\{V\}\]\_\{\+\}\+\\lambda\(h\_\{t\}\(v\_\{t\}^\{\\star\}\)\-\\bar\{h\}\)\\big\)\\log\\pi\_\{\\theta\}\(v\_\{t\}^\{\\star\}\), whereht​\(v\)=Atd​\(v\)h\_\{t\}\(v\)=A\_\{t\}^\{d\}\(v\)andh¯\\bar\{h\}is the response mean of the sampled route\. The sampled route and its centering are unchanged; the branch supervises the teacher’s top token as if the student had also sampled it, and its verifier part can only raise a response’s total label \(Appendix[A\.8](https://arxiv.org/html/2610.11519#A1.SS8)\)\.

Since the value of additional guidance may depend on the training objective, we add the same branch to three base methods\. Sampled OPD \(Appendix[D\.6](https://arxiv.org/html/2610.11519#A4.SS6)\) learns from the teacher alone; there the added token receives the sampled\-OPD labellog⁡qt​\(vt⋆\)−log⁡pt​\(vt⋆\)\\log q\_\{t\}\(v\_\{t\}^\{\\star\}\)\-\\log p\_\{t\}\(v\_\{t\}^\{\\star\}\), which makes sampled OPD a top\-kkvariant\. REINFORCE\+\+ learns from the verifier alone, so the branch is its only source of teacher signal\.RA\+REINFORCE\+\+ already receives a teacher label at every sampled token\.

Table 4:The same teacher\-peak branch slightly helps sampled OPD but lowers both students trained with the verifier\.Change in three\-seed macro score when the branch is added, Qwen3\-1\.7B\-Base, in points\. The weightλ\\lambdamultiplies the branch’s teacher part and, withRA, also the sampled guidance\. Absolute, benchmark, and per\-seed scores are in Appendix[D\.3](https://arxiv.org/html/2610.11519#A4.SS3)\.Under sampled OPD \(Table[4](https://arxiv.org/html/2610.11519#S4.T4)\) the branch raises macro Avg@8 from24\.8024\.80to25\.1025\.10and the Avg@8 of every seed\. Under REINFORCE\+\+, whereλ\\lambdascales only the branch, it lowers macro Avg@8 by0\.380\.38–1\.761\.76points and Pass@8 by0\.460\.46–2\.112\.11points, a loss that grows with the branch weight\. Added toRA, it lowers macro Avg@8 by2\.052\.05–2\.162\.16points and Pass@8 by3\.413\.41–3\.913\.91points, and at every weight each seed falls below the same seed ofRA, whereas doublingRA’s own weight does not \(Table[2](https://arxiv.org/html/2610.11519#S4.T2)\)\.

Because the branch alters several factors at once, it does not isolate a mechanism, and we do not pursue one here\. It does answer the question it was designed for: stronger and denser teacher supervision does not by itself yield better performance\. A student trained only to imitate the teacher benefits slightly from the teacher’s top token, whereas both students trained with the verifier degrade, increasingly so with the branch weight under REINFORCE\+\+\. One conjecture is that steering the student directly toward the teacher’s choice crowds out the exploration on which RL relies\.RAinstead scales its expected update on each token by the student’s probability of that token and leaves the choice to the student’s interaction with the verifier; amplifying this gated signal is harmless\. This restraint may be part of what makes its guidance effective, and it leaves an open question for RLVR: how a dense, critic\-like signal such as a teacher’s should be shaped and enter RL training\.

## 5Related Work

#### Distilling on the student’s own samples\.

GKD studies divergence choices on student\-generated sequences\([Agarwal et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib1)\), MiniLLM optimizes reverse KL with log\-ratio returns\([Gu et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib10)\), sampled OPD uses the log ratio at the generated token as a dense reward\([Lu & Thinking Machines Lab, 2025](https://arxiv.org/html/2610.11519#bib.bib22)\), and OPD\+ derives divergence\-dependent corrections\([Zhao et al\., 2026](https://arxiv.org/html/2610.11519#bib.bib39)\)\.RAreplaces the log ratio by the bounded residual, judges the chosen token against the student’s own alternatives, and removes the teacher’s response mean\. What distinguishesRAis its restraint: the teacher acts not as a distillation target, as in these works, but as a critic\-like signal within RL\.

#### Combining distillation with verifiable rewards\.

LUFFY mixes off\-policy teacher traces into RLVR\([Yan et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib34)\)\. OPDVR gates the sampled log ratio so that token reward signs agree with the verifier\([Lin et al\., 2026](https://arxiv.org/html/2610.11519#bib.bib21)\), whereasRAkeeps the verifier advantage as the response mean and allows local sign changes\.[Li et al\. \(2026a\)](https://arxiv.org/html/2610.11519#bib.bib16)favor OPD followed by RL over joint combinations; at our matched budget both schedules stay belowRA\(Appendix[D\.5](https://arxiv.org/html/2610.11519#A4.SS5)\)\.

#### Baselines and token\-level credit\.

A state\-value baseline leaves the expected policy gradient unchanged\([Williams, 1992](https://arxiv.org/html/2610.11519#bib.bib33);[Sutton et al\., 2000](https://arxiv.org/html/2610.11519#bib.bib30)\), and the variance\-optimal baseline also accounts for the score norm\([Greensmith et al\., 2004](https://arxiv.org/html/2610.11519#bib.bib9)\)\. GRPO, REINFORCE\+\+, RLOO, and ReMax use group, batch, leave\-one\-out, and greedy baselines\([Shao et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib29);[Hu, 2025](https://arxiv.org/html/2610.11519#bib.bib13);[Ahmadian et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib2);[Li et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib19)\), and COMA marginalizes an action under the policy\([Foerster et al\., 2018](https://arxiv.org/html/2610.11519#bib.bib7)\);RAuses an exact policy expectation as the baseline of its local reward\. Process supervision uses finer labels\([Lightman et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib20)\), VinePPO extra sampling\([Kazemnejad et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib15)\), and implicit PRMs and PRIME the log ratio of an outcome\-trained model to a reference\([Yuan et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib37);[Cui et al\., 2025](https://arxiv.org/html/2610.11519#bib.bib4)\)\. VPD refines a feedback\-conditioned teacher with outcome supervision and distills it on student rollouts\([Li et al\., 2026b](https://arxiv.org/html/2610.11519#bib.bib17)\); Co\-RAinstead trains a teacher LoRA with the base estimator’s clipped update on the same scored student batch\.

## 6Conclusion

Residual Advantage judges the teacher–student probability difference against the student’s own alternatives and centers it within each response, so the verifier sets a response’s mean label and the teacher allocates credit among its steps\. The guidance term decreases both as the student approaches the teacher and as the student becomes confident in its own choice, an elegant property that helpsRApreserve exploration in RL\. Across two students, two estimators, and three benchmarks,RAimproves every matched three\-seed mean, both combinations surpass teacher\-only OPD, and Co\-RAimproves accuracy further while its adapted teacher becomes a worse solver than the frozen one\. Up to Pass@512 the margin ofRAover the untrained student grows while that of verifier\-only training narrows\. Removing either baseline, replacing the residual by the log ratio, or adding direct supervision of the teacher’s top token lowers performance\.

### AI use statement

Generative AI tools were used throughout this work\. In this work, we used generative AI tools to help develop the theoretical framework, formulate mathematical claims, provide ingredients for and assist in the writing of proofs, propose and refine hypotheses, design and give feedback on the research methodology and experiments, implement methods, assist with translation, and interpret results\. We have not used generative AI tools to generate synthetic data sets; cleaning or reformatting datasets and qualitative or thematic data analysis are not applicable to this work, which uses public datasets unmodified and reports quantitative results only\. Additionally, we used generative AI tools to brainstorm and identify research gaps, source information and identify, summarize, and analyze related literature, suggest experimental parameters, create and edit software code, create and modify figures and tables, suggest the structure of the paper, draft and edit parts of the text to improve readability, format references, and propose the title\. We have reviewed all AI\-assisted work: every derivation and proof was rederived and checked by the authors, all cited works were verified against their original sources, all code was reviewed by the authors and every reported experiment was run and evaluated under the authors’ control, and every number in the paper was checked against the recorded results\. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI\.

### Acknowledgments

Part of this work was done at Tencent\. We thank Tencent for providing the computational resources used in this work\.

## References

- Agarwal et al\. \(2024\)Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem\.On\-policy distillation of language models: Learning from self\-generated mistakes\.In*International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=3zKtaqxLhW](https://openreview.net/forum?id=3zKtaqxLhW)\.
- Ahmadian et al\. \(2024\)Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker\.Back to basics: Revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 12248–12267, 2024\.
- Basu et al\. \(1998\)Ayanendranath Basu, Ian R\. Harris, Nils L\. Hjort, and M\. C\. Jones\.Robust and efficient estimation by minimising a density power divergence\.*Biometrika*, 85\(3\):549–559, 1998\.
- Cui et al\. \(2025\)Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, et al\.Process reinforcement through implicit rewards\.*arXiv preprint arXiv:2502\.01456*, 2025\.
- DeepSeek\-AI et al\. \(2025\)DeepSeek\-AI et al\.DeepSeek\-R1: Incentivizing reasoning capability in LLMs via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.URL[https://arxiv\.org/abs/2501\.12948](https://arxiv.org/abs/2501.12948)\.
- Duchi et al\. \(2008\)John Duchi, Shai Shalev\-Shwartz, Yoram Singer, and Tushar Chandra\.Efficient projections onto theℓ1\\ell\_\{1\}\-ball for learning in high dimensions\.In*Proceedings of the 25th International Conference on Machine Learning*, pp\. 272–279, 2008\.
- Foerster et al\. \(2018\)Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson\.Counterfactual multi\-agent policy gradients\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 32, 2018\.
- Gao et al\. \(2025\)Shiping Gao, Fanqi Wan, Jiajian Guo, Xiaojun Quan, and Qifan Wang\.Advantage\-guided distillation for preference alignment in small language models\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=xsx3Fpo3UD](https://openreview.net/forum?id=xsx3Fpo3UD)\.
- Greensmith et al\. \(2004\)Evan Greensmith, Peter L\. Bartlett, and Jonathan Baxter\.Variance reduction techniques for gradient estimates in reinforcement learning\.*Journal of Machine Learning Research*, 5:1471–1530, 2004\.
- Gu et al\. \(2024\)Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang\.MiniLLM: Knowledge distillation of large language models\.In*International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=5h0qf7IBZZ](https://openreview.net/forum?id=5h0qf7IBZZ)\.
- Hampel \(1974\)Frank R\. Hampel\.The influence curve and its role in robust estimation\.*Journal of the American Statistical Association*, 69\(346\):383–393, 1974\.
- Hu et al\. \(2022\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations*, 2022\.URL[https://openreview\.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9)\.
- Hu \(2025\)Jian Hu\.REINFORCE\+\+: A simple and efficient approach for aligning large language models, 2025\.URL[https://arxiv\.org/abs/2501\.03262v1](https://arxiv.org/abs/2501.03262v1)\.Version 1, 4 January 2025\.
- Huang et al\. \(2025\)Audrey Huang, Adam Block, Dylan J\. Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T\. Ash, and Akshay Krishnamurthy\.Self\-improvement in language models: The sharpening mechanism\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=yNN1xYMgxG](https://openreview.net/forum?id=yNN1xYMgxG)\.
- Kazemnejad et al\. \(2025\)Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux\.VinePPO: Refining credit assignment in RL training of LLMs\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267, pp\. 29557–29590, 2025\.
- Li et al\. \(2026a\)Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie, Chen Zhao, and Xi Ye\.Sequential beats joint: On the interplay between on\-policy distillation and RLVR, 2026a\.URL[https://arxiv\.org/abs/2609\.04108v2](https://arxiv.org/abs/2609.04108v2)\.Version 2, 4 September 2026\.
- Li et al\. \(2026b\)Yang Li, Erik Nijkamp, Semih Yavuz, and Shafiq Rayhan Joty\.Learning from language feedback via variational policy distillation, 2026b\.URL[https://arxiv\.org/abs/2605\.15113v1](https://arxiv.org/abs/2605.15113v1)\.Version 1, 14 May 2026\.
- Li et al\. \(2025\)Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran\.Small models struggle to learn from strong reasoners\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pp\. 25366–25394, 2025\.
- Li et al\. \(2024\)Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi\-Quan Luo\.ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235, pp\. 29128–29163, 2024\.
- Lightman et al\. \(2024\)Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.In*International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=v8L0pN6EOi](https://openreview.net/forum?id=v8L0pN6EOi)\.
- Lin et al\. \(2026\)Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, and Gao Huang\.On\-policy distillation with verifiable reward, 2026\.URL[https://arxiv\.org/abs/2608\.24696v1](https://arxiv.org/abs/2608.24696v1)\.Version 1, 25 August 2026\.
- Lu & Thinking Machines Lab \(2025\)Kevin Lu and Thinking Machines Lab\.On\-policy distillation\.Thinking Machines Lab: Connectionism,[https://thinkingmachines\.ai/blog/on\-policy\-distillation/](https://thinkingmachines.ai/blog/on-policy-distillation/), 2025\.
- Martins & Astudillo \(2016\)André F\. T\. Martins and Ramón Fernandez Astudillo\.From softmax to sparsemax: A sparse model of attention and multi\-label classification\.In*Proceedings of the 33rd International Conference on Machine Learning*, volume 48, pp\. 1614–1623, 2016\.
- Michelot \(1986\)Christian Michelot\.A finite algorithm for finding the projection of a point onto the canonical simplex ofℝn\\mathbb\{R\}^\{n\}\.*Journal of Optimization Theory and Applications*, 50\(1\):195–200, 1986\.
- Peng et al\. \(2019\)Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine\.Advantage\-weighted regression: Simple and scalable off\-policy reinforcement learning\.*arXiv preprint arXiv:1910\.00177*, 2019\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D\. Manning, Stefano Ermon, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems*, volume 36, pp\. 53728–53741\. Curran Associates, Inc\., 2023\.doi:10\.52202/075280\-2338\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf)\.
- Rafailov et al\. \(2024\)Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn\.FromrrtoQ∗Q^\{\*\}: Your language model is secretly a Q\-function\.In*First Conference on Language Modeling*, 2024\.URL[https://arxiv\.org/abs/2404\.12358](https://arxiv.org/abs/2404.12358)\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.URL[https://arxiv\.org/abs/1707\.06347](https://arxiv.org/abs/1707.06347)\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.URL[https://arxiv\.org/abs/2402\.03300](https://arxiv.org/abs/2402.03300)\.
- Sutton et al\. \(2000\)Richard S\. Sutton, David McAllester, Satinder Singh, and Yishay Mansour\.Policy gradient methods for reinforcement learning with function approximation\.In*Advances in Neural Information Processing Systems*, volume 12, pp\. 1057–1063, 2000\.
- Tiapkin et al\. \(2025\)Daniil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Ramé, and Mathieu Blondel\.On teacher hacking in language model distillation\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267, pp\. 59552–59569, 2025\.
- Wang et al\. \(2025\)Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin\.Beyond the 80/20 rule: High\-entropy minority tokens drive effective reinforcement learning for LLM reasoning\.*arXiv preprint arXiv:2506\.01939*, 2025\.
- Williams \(1992\)Ronald J\. Williams\.Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.*Machine Learning*, 8\(3–4\):229–256, 1992\.doi:10\.1007/BF00992696\.
- Yan et al\. \(2025\)Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang\.Learning to reason under off\-policy guidance\.In*Advances in Neural Information Processing Systems*, volume 38, 2025\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Yu et al\. \(2025\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al\.DAPO: An open\-source LLM reinforcement learning system at scale\.In*Advances in Neural Information Processing Systems*, volume 38, 2025\.URL[https://arxiv\.org/abs/2503\.14476](https://arxiv.org/abs/2503.14476)\.
- Yuan et al\. \(2025\)Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng\.Free process rewards without process labels\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267, pp\. 73511–73525, 2025\.
- Yue et al\. \(2025\)Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang\.Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?In*Advances in Neural Information Processing Systems*, volume 38, 2025\.
- Zhao et al\. \(2026\)Hanyang Zhao, Haoxian Chen, Han Lin, Genta Indra Winata, David Yao, and Wenpin Tang\.OPD\+: Rethinking the advantage design for on\-policy distillation, 2026\.URL[https://arxiv\.org/abs/2606\.01039v1](https://arxiv.org/abs/2606.01039v1)\.Version 1, 31 May 2026\.

## Appendix AProofs and update identities

Throughout,p,qp,qare probability vectors on the same finite support with at least two coordinates,d=q−pd=q\-p,Ad​\(a\)=⟨ea−p,d⟩A^\{d\}\(a\)=\\langle e\_\{a\}\-p,d\\rangle, andλ≥0\\lambda\\geq 0\. Labels are held fixed during differentiation, expectations over actions use the sameppthat defines the label, and scores are evaluated at the rollout policy\. Probability identities also hold on the boundary of the simplex; logit identities use positive probabilities\. The constructed examples are exact computations on stipulated vectors\.

### A\.1Proof of Proposition[1](https://arxiv.org/html/2610.11519#Thmproposition1)

#### Part \(i\)\.

LetL⁡\(p,q\)=12​∥p−q∥22L\(p,q\)=\\frac\{1\}\{2\}\\lVert p\-q\\rVert\_\{2\}^\{2\}\. The pathpη=\(1−η\)​p\+η​eap\_\{\\eta\}=\(1\-\\eta\)p\+\\eta e\_\{a\}stays in the simplex for0≤η≤10\\leq\\eta\\leq 1, andpη−q=−d\+η⁡\(ea−p\)p\_\{\\eta\}\-q=\-d\+\\eta\(e\_\{a\}\-p\), so

L⁡\(pη,q\)−L⁡\(p,q\)=−η⁡⟨ea−p,d⟩\+η22​∥ea−p∥22=−η​Ad​\(a\)\+η22​∥ea−p∥22\.L\(p\_\{\\eta\},q\)\-L\(p,q\)=\-\\eta\\langle e\_\{a\}\-p,d\\rangle\+\\frac\{\\eta^\{2\}\}\{2\}\\lVert e\_\{a\}\-p\\rVert\_\{2\}^\{2\}=\-\\eta A^\{d\}\(a\)\+\\frac\{\\eta^\{2\}\}\{2\}\\lVert e\_\{a\}\-p\\rVert\_\{2\}^\{2\}\.\(11\)ThusAd​\(a\)A^\{d\}\(a\)is the rate of improvement along a probability move towardaa, and ifAd​\(a\)\>0A^\{d\}\(a\)\>0every feasibleη<2​Ad​\(a\)/∥ea−p∥22\\eta<2A^\{d\}\(a\)/\\lVert e\_\{a\}\-p\\rVert\_\{2\}^\{2\}strictly reducesLL\.

#### Part \(ii\)\.

Forp=softmax⁡\(z\)p=\\operatorname\{softmax\}\(z\), writeJp=Diag⁡\(p\)−p​p⊤J\_\{p\}=\\operatorname\{Diag\}\(p\)\-pp^\{\\top\}\. The student mean of the label is∑ap⁡\(a\)​Ad​\(a\)=⟨p,d⟩−⟨p,d⟩=0\\sum\_\{a\}p\(a\)A^\{d\}\(a\)=\\langle p,d\\rangle\-\\langle p,d\\rangle=0, and since∇z​log​p​\(a\)=ea−p\\nabla\_\{z\}\\log p\(a\)=e\_\{a\}\-p,

𝔼A∼p​\[Ad​\(A\)​\(eA−p\)\]=p⊙Ad=Jp​d=−∇zL​\(p,q\),−∂L∂za=p⁡\(a\)​Ad​\(a\)\.\\mathbb\{E\}\_\{A\\sim p\}\[A^\{d\}\(A\)\(e\_\{A\}\-p\)\]=p\\odot A^\{d\}=J\_\{p\}d=\-\\nabla\_\{z\}L\(p,q\),\\qquad\-\\frac\{\\partial L\}\{\\partial z\_\{a\}\}=p\(a\)A^\{d\}\(a\)\.\(12\)HereJp=𝔼a∼p​\[\(ea−p\)​\(ea−p\)⊤\]J\_\{p\}=\\mathbb\{E\}\_\{a\\sim p\}\[\(e\_\{a\}\-p\)\(e\_\{a\}\-p\)^\{\\top\}\]is the Jacobian of the softmax and the Fisher information of the categorical distribution in logit coordinates, so the logit gradient isJpJ\_\{p\}applied to the probability\-space gradient∇pL=−d\\nabla\_\{p\}L=\-d\. The same computation shows that any fixed labelwwhas expected updateJp​wJ\_\{p\}wand that subtracting an action\-independent baseline⟨p,w⟩\\langle p,w\\rangleleaves it unchanged, because𝔼a∼p​\[ea−p\]=0\\mathbb\{E\}\_\{a\\sim p\}\[e\_\{a\}\-p\]=0; in particularddandAdA^\{d\}have the same expected update at a fixed prefix\.

#### Part \(iii\)\.

Cauchy–Schwarz gives\|⟨ea−p,d⟩\|≤∥ea−p∥2​∥d∥2\|\\langle e\_\{a\}\-p,d\\rangle\|\\leq\\lVert e\_\{a\}\-p\\rVert\_\{2\}\\lVert d\\rVert\_\{2\}with∥ea−p∥22=\(1−p⁡\(a\)\)2\+∑v≠ap​\(v\)2≤2​\(1−p⁡\(a\)\)2\\lVert e\_\{a\}\-p\\rVert\_\{2\}^\{2\}=\(1\-p\(a\)\)^\{2\}\+\\sum\_\{v\\neq a\}p\(v\)^\{2\}\\leq 2\(1\-p\(a\)\)^\{2\}, since∑v≠ap​\(v\)2≤\(1−p⁡\(a\)\)2\\sum\_\{v\\neq a\}p\(v\)^\{2\}\\leq\(1\-p\(a\)\)^\{2\}; equality holds when the student’s remaining mass sits on one token\. SeparatingaainAd​\(a\)=d⁡\(a\)−∑vp⁡\(v\)​d​\(v\)A^\{d\}\(a\)=d\(a\)\-\\sum\_\{v\}p\(v\)d\(v\)gives Equation[4](https://arxiv.org/html/2610.11519#S2.E4)\. The range\[−12,2\]\[\-\\tfrac\{1\}\{2\},2\]is established in Appendix[A\.6](https://arxiv.org/html/2610.11519#A1.SS6)\.

#### Other divergences\.

For anyℓ\\elldifferentiable near the interior pointpp, defineAℓ​\(a\)=−⟨ea−p,∇pℓ⟩A^\{\\ell\}\(a\)=\-\\langle e\_\{a\}\-p,\\nabla\_\{p\}\\ell\\rangle, the negative directional derivative ofℓ\\ellalongea−pe\_\{a\}\-p\. This label has zero student mean, and

𝔼a∼p​\[Aℓ​\(a\)​\(ea−p\)\]=−Jp​∇pℓ=−∇zℓ​\(softmax⁡\(z\)\)\\mathbb\{E\}\_\{a\\sim p\}\\big\[A^\{\\ell\}\(a\)\(e\_\{a\}\-p\)\\big\]=\-J\_\{p\}\\nabla\_\{p\}\\ell=\-\\nabla\_\{z\}\\,\\ell\(\\operatorname\{softmax\}\(z\)\)\(13\)by the chain rule\. Forℓ=L\\ell=Lthis recoversAL=AdA^\{L\}=A^\{d\}\. Forℓ=KL\(p∥q\)\\ell=\\operatorname\{KL\}\(p\\\|q\),∇pℓ=log⁡\(p/q\)\+𝟏\\nabla\_\{p\}\\ell=\\log\(p/q\)\+\\mathbf\{1\}, the constant is annihilated by⟨ea−p,⋅⟩\\langle e\_\{a\}\-p,\\cdot\\rangle, andAℓ​\(a\)=log⁡q⁡\(a\)p⁡\(a\)−⟨p,log⁡qp⟩A^\{\\ell\}\(a\)=\\log\\frac\{q\(a\)\}\{p\(a\)\}\-\\langle p,\\log\\frac\{q\}\{p\}\\rangleis the student\-centered log ratio\. The quadratic remainder in Equation[11](https://arxiv.org/html/2610.11519#A1.E11)is specific toLL\.

The identity concerns expectations\. A single sampled ascent vectorga=Ad​\(a\)​\(ea−p\)g\_\{a\}=A^\{d\}\(a\)\(e\_\{a\}\-p\)can point againstJp​dJ\_\{p\}d, as in Table[5](https://arxiv.org/html/2610.11519#A1.T5), where a small logit step alonggag\_\{a\}increasesLL\.

Table 5:A sampled update that opposes the expected update\.The probability vectors and sampled action are inputs; the remaining entries follow by substitution\.
#### The log ratio as an implicit advantage\.

Suppose the teacher is a KL\-regularized improvement of the student at this prefix,q=p​exp⁡\(A⋆/β\)/Zq=p\\exp\(A^\{\\star\}/\\beta\)/Zwith score vectorA⋆A^\{\\star\},β\>0\\beta\>0, andZ=⟨p,exp⁡\(A⋆/β\)⟩Z=\\langle p,\\exp\(A^\{\\star\}/\\beta\)\\rangle\. Then

log⁡q⁡\(a\)p⁡\(a\)−⟨p,log⁡qp⟩\\displaystyle\\log\\frac\{q\(a\)\}\{p\(a\)\}\-\\left\\langle p,\\log\\frac\{q\}\{p\}\\right\\rangle=A⋆​\(a\)−⟨p,A⋆⟩β,\\displaystyle=\\frac\{A^\{\\star\}\(a\)\-\\langle p,A^\{\\star\}\\rangle\}\{\\beta\},d⁡\(a\)\\displaystyle d\(a\)=p⁡\(a\)​\[eA⋆​\(a\)/βZ−1\]≥−p⁡\(a\),\\displaystyle=p\(a\)\\left\[\\frac\{e^\{A^\{\\star\}\(a\)/\\beta\}\}\{Z\}\-1\\right\]\\geq\-p\(a\),\(14\)so the student\-centered log ratio recovers the score contrast exactly, while the residual passes it through a student\-weighted saturating map\. For fixedA⋆A^\{\\star\}, expanding atβ−1=0\\beta^\{\-1\}=0gives

exp⁡\(A⋆/β\)Z\\displaystyle\\frac\{\\exp\(A^\{\\star\}/\\beta\)\}\{Z\}=𝟏\+A⋆−⟨p,A⋆⟩​𝟏β\+O⁡\(β−2\),\\displaystyle=\\mathbf\{1\}\+\\frac\{A^\{\\star\}\-\\langle p,A^\{\\star\}\\rangle\\mathbf\{1\}\}\{\\beta\}\+O\(\\beta^\{\-2\}\),d\\displaystyle d=Jp​A⋆β\+O⁡\(β−2\)\.\\displaystyle=\\frac\{J\_\{p\}A^\{\\star\}\}\{\\beta\}\+O\(\\beta^\{\-2\}\)\.\(15\)This is the implicit\-advantage model behind on\-policy distillation and preference optimization\([Rafailov et al\., 2023](https://arxiv.org/html/2610.11519#bib.bib26);[Rafailov et al\., 2024](https://arxiv.org/html/2610.11519#bib.bib27);[Peng et al\., 2019](https://arxiv.org/html/2610.11519#bib.bib25)\), in which a teacher one ideal KL step ahead of the student carries a score contrast in its log ratio\. A frozen solver need not be such a step for the current student, andRAbuilds its local advantage from the probability residual instead\.

### A\.2A local objective and its target

###### Proposition 2\(Local quadratic surrogate\)

Fix a prefix, the teacherqq, a finite verifier action\-value vectorQQ, andλ\>0\\lambda\>0\. At the rollout policy, the expected logit update of the local labelAV\+λ​Ad​\(a\)A^\{V\}\+\\lambda A^\{d\}\(a\), before response centering, is∇zF​\(p\)\\nabla\_\{z\}F\(p\)for

F⁡\(p\)=⟨p,Q⟩−λ2​∥p−q∥22\.F\(p\)=\\langle p,Q\\rangle\-\\frac\{\\lambda\}\{2\}\\lVert p\-q\\rVert\_\{2\}^\{2\}\.\(16\)On the closed simplex, this strictly concave function has the unique maximizer

p⋆=ΠΔ​\(q\+Q/λ\),p⋆​\(v\)=\[q⁡\(v\)\+Q⁡\(v\)/λ−τ\]\+,p^\{\\star\}=\\Pi\_\{\\Delta\}\(q\+Q/\\lambda\),\\qquad p^\{\\star\}\(v\)=\[q\(v\)\+Q\(v\)/\\lambda\-\\tau\]\_\{\+\},\(17\)whereτ\\taumakes the entries sum to one\. At fixedQQandλ\\lambda, the mapq↦p⋆q\\mapsto p^\{\\star\}is nonexpansive inℓ2\\ell\_\{2\}\.

#### Proof\.

LetQ\(v\)=𝔼\[AV∣ξ,at=v\]Q\(v\)=\\mathbb\{E\}\[A^\{V\}\\mid\\xi,a\_\{t\}=v\]\. Conditioning on the sampled action gives𝔼⁡\[AV​\(ea−p\)\]=Jp​Q\\mathbb\{E\}\[A^\{V\}\(e\_\{a\}\-p\)\]=J\_\{p\}Q, and with Equation[12](https://arxiv.org/html/2610.11519#A1.E12),

𝔼⁡\[\(AV\+λ​Ad​\(a\)\)​\(ea−p\)\]=Jp​\(Q\+λ​d\),\\mathbb\{E\}\\big\[\(A^\{V\}\+\\lambda A^\{d\}\(a\)\)\(e\_\{a\}\-p\)\\big\]=J\_\{p\}\(Q\+\\lambda d\),\(18\)which is∇zF​\(p\)=Jp​∇pF\\nabla\_\{z\}F\(p\)=J\_\{p\}\\nabla\_\{p\}Fbecause∇pF=Q\+λ​d\\nabla\_\{p\}F=Q\+\\lambda d\. Completing the square,

F⁡\(p\)=−λ2​∥p−\(q\+Q/λ\)∥22\+⟨q,Q⟩\+∥Q∥222​λ,F\(p\)=\-\\frac\{\\lambda\}\{2\}\\Big\\lVert p\-\\big\(q\+Q/\\lambda\\big\)\\Big\\rVert\_\{2\}^\{2\}\+\\langle q,Q\\rangle\+\\frac\{\\lVert Q\\rVert\_\{2\}^\{2\}\}\{2\\lambda\},\(19\)so forλ\>0\\lambda\>0the unique maximizer on the simplex is the Euclidean projection ofq\+Q/λq\+Q/\\lambda, which has the threshold form of Equation[17](https://arxiv.org/html/2610.11519#A1.E17)\([Michelot, 1986](https://arxiv.org/html/2610.11519#bib.bib24);[Duchi et al\., 2008](https://arxiv.org/html/2610.11519#bib.bib6)\)and coincides with sparsemax\([Martins & Astudillo, 2016](https://arxiv.org/html/2610.11519#bib.bib23)\)\. Projection onto a closed convex set is nonexpansive, which gives the last claim\. Ifλ=0\\lambda=0the objective is linear and any distribution on the maximizers ofQQis optimal\.

#### Support\.

p⋆​\(v\)=0p^\{\\star\}\(v\)=0if and only ifq⁡\(v\)\+Q⁡\(v\)/λ≤τq\(v\)\+Q\(v\)/\\lambda\\leq\\tau\. A token outside the teacher’s support is retained exactly whenQ⁡\(v\)\>λ​τQ\(v\)\>\\lambda\\tau, a teacher\-favored token is dropped exactly whenQ⁡\(v\)≤−λ⁡\(q⁡\(v\)−τ\)Q\(v\)\\leq\-\\lambda\(q\(v\)\-\\tau\), andp⋆=qp^\{\\star\}=qwhenQ≡0Q\\equiv 0\.

#### Log ratio\.

For positivep,qp,qandβ\>0\\beta\>0, the labelAV\+β​log⁡\(q/p\)A^\{V\}\+\\beta\\log\(q/p\)has expected update∇z\[⟨p,Q⟩−βKL\(p∥q\)\]\\nabla\_\{z\}\[\\langle p,Q\\rangle\-\\beta\\operatorname\{KL\}\(p\\\|q\)\], becauseJp​𝟏=0J\_\{p\}\\mathbf\{1\}=0\. Withpβ⋆=q​exp⁡\(Q/β\)/Zp\_\{\\beta\}^\{\\star\}=q\\exp\(Q/\\beta\)/ZandZ=⟨q,exp⁡\(Q/β\)⟩Z=\\langle q,\\exp\(Q/\\beta\)\\rangle,

⟨p,Q⟩−βKL\(p∥q\)=βlogZ−βKL\(p∥pβ⋆\),\\langle p,Q\\rangle\-\\beta\\operatorname\{KL\}\(p\\\|q\)=\\beta\\log Z\-\\beta\\operatorname\{KL\}\(p\\\|p\_\{\\beta\}^\{\\star\}\),\(20\)sopβ⋆p\_\{\\beta\}^\{\\star\}is the unique maximizer\. Ifqqhas zero coordinates the statement holds on its support, and finite KL excludes mass outside it\. Both maximizers describe a local optimum at one prefix; which tokens a finite training run discovers is a separate question\.

#### Scope\.

The vectorQQis frozen at the rollout policy; the surrogate does not differentiate its dependence on the policy or on the visited prefixes\. Under the assumptions of Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4), each position’s expected teacher term carries the factor1−ct1\-c\_\{t\}of Equation[36](https://arxiv.org/html/2610.11519#A1.E36), so the sum of local objectives uses the effective weightλ⁡\(1−ct\)\\lambda\(1\-c\_\{t\}\)at positiontt\. The random lengths and batch normalizerZZof the actual protocol fall outside these hypotheses \(Appendix[A\.7](https://arxiv.org/html/2610.11519#A1.SS7)\)\. The directL2L\_\{2\}control appliesα​Jp​d\\alpha J\_\{p\}dat each visited prefix through a differentiable loss and matches the uncentered local expectation atα=λ\\alpha=\\lambda; response centering, gradient noise, and optimizer state still separate the two training updates\.

### A\.3Local probability scaling

###### Proposition 3\(Local logit influence\)

Fix a prefix, positive distributionsp,qp,q, a finite verifier action\-value vectorQQ, and nonnegative weightsλ,β\\lambda,\\beta, before response centering\.\(i\)On logit coordinatebb, the expected residual update isλ​p​\(b\)​Ad​\(b\)\\lambda p\(b\)A^\{d\}\(b\), with

−12​λ​p​\(b\)≤λ​p​\(b\)​Ad​\(b\)≤2​λ​p​\(b\)​\(1−p⁡\(b\)\)2≤827​λ,\-\\tfrac\{1\}\{2\}\\lambda p\(b\)\\leq\\lambda p\(b\)A^\{d\}\(b\)\\leq 2\\lambda p\(b\)\(1\-p\(b\)\)^\{2\}\\leq\\tfrac\{8\}\{27\}\\lambda,and the verifier contribution isp⁡\(b\)​CQ​\(b\)p\(b\)C\_\{Q\}\(b\), whereCQ​\(b\)=Q⁡\(b\)−⟨p,Q⟩C\_\{Q\}\(b\)=Q\(b\)\-\\langle p,Q\\rangle\.\(ii\)The raw or student\-centered log\-ratio label givesβp\(b\)\[log\(q\(b\)/p\(b\)\)\+KL\(p∥q\)\]\\beta p\(b\)\[\\log\(q\(b\)/p\(b\)\)\+\\operatorname\{KL\}\(p\\\|q\)\]on that coordinate\. For fixed positiveqqandp⁡\(b\)→0p\(b\)\\to 0with the other probabilities proportionally rescaled, this isβ​p​\(b\)​\[log⁡\(1/p⁡\(b\)\)\+O⁡\(1\)\]\\beta p\(b\)\[\\log\(1/p\(b\)\)\+O\(1\)\]\.

#### Proof\.

Coordinatebbof Equation[12](https://arxiv.org/html/2610.11519#A1.E12)isp⁡\(b\)​Ad​\(b\)p\(b\)A^\{d\}\(b\), and Equation[28](https://arxiv.org/html/2610.11519#A1.E28)bounds the label; the function2​x​\(1−x\)22x\(1\-x\)^\{2\}on\[0,1\]\[0,1\]has derivative2​\(1−x\)​\(1−3​x\)2\(1\-x\)\(1\-3x\)and maximum8/278/27atx=1/3x=1/3\. CoordinatebbofJp​QJ\_\{p\}Qisp⁡\(b\)​CQ​\(b\)p\(b\)C\_\{Q\}\(b\)\. For the log ratio, coordinatebbofJp​ℓJ\_\{p\}\\ellisp⁡\(b\)​\(ℓ⁡\(b\)−⟨p,ℓ⟩\)p\(b\)\(\\ell\(b\)\-\\langle p,\\ell\\rangle\)with⟨p,ℓ⟩=−KL\(p∥q\)\\langle p,\\ell\\rangle=\-\\operatorname\{KL\}\(p\\\|q\), which stays finite in the stated limit, so the bracket equalslog⁡\(1/p⁡\(b\)\)\+O⁡\(1\)\\log\(1/p\(b\)\)\+O\(1\)\. Relative to a nonzero verifier coordinate its magnitude isβ​\|log⁡\(1/p⁡\(b\)\)\+O⁡\(1\)\|/\|CQ​\(b\)\|\\beta\|\\log\(1/p\(b\)\)\+O\(1\)\|/\|C\_\{Q\}\(b\)\|, unbounded for fixedQQ, whereas\|Ad​\(b\)\|≤2\|A^\{d\}\(b\)\|\\leq 2\.

#### Relative magnitude\.

Both the verifier and residual terms areO⁡\(p⁡\(b\)\)O\(p\(b\)\), but equal order bounds neither their ratio nor the sign of their sum\. Atp=\(\.99,\.01\)p=\(\.99,\.01\),q=\(\.01,\.99\)q=\(\.01,\.99\),Q=\(0,−\.01\)Q=\(0,\-\.01\), andλ=10\\lambda=10, the second coordinate receives\+0\.19404\+0\.19404from the residual and−0\.000099\-0\.000099from the verifier, so the teacher reverses the net direction\. A sufficient condition for a positive combined update on coordinatebbisCQ​\(b\)\>λ/2C\_\{Q\}\(b\)\>\\lambda/2, sinceCQ​\(b\)\+λ​Ad​\(b\)≥CQ​\(b\)−λ/2C\_\{Q\}\(b\)\+\\lambda A^\{d\}\(b\)\\geq C\_\{Q\}\(b\)\-\\lambda/2; atλ=10\\lambda=10the threshold is55, and under the fixed\-weight composition of Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4)the effective weight isλ⁡\(1−ct\)\\lambda\(1\-c\_\{t\}\)\. These are statements about the expected logit update at the rollout policy, and they say nothing about entropy, exploration, or a finite parameter step\.

Response centering is a different operation from the student value baseline\. Applied to the student\-centered log label it gives the same1−ct1\-c\_\{t\}factor as Equation[36](https://arxiv.org/html/2610.11519#A1.E36); applied to the raw log label it can introduce cross\-position terms, because the raw conditional mean−KL\(pt∥qt\)\-\\operatorname\{KL\}\(p\_\{t\}\\\|q\_\{t\}\)depends on earlier actions \(Appendix[A\.7](https://arxiv.org/html/2610.11519#A1.SS7)\)\.

### A\.4The residual and the log ratio as ends of one family

For positive distributions and1<α≤21<\\alpha\\leq 2, define

Dα​\(p,q\)=∑v\[pvα−qvα−α​qvα−1​\(pv−qv\)\]α⁡\(α−1\),D\_\{\\alpha\}\(p,q\)=\\frac\{\\sum\_\{v\}\[p\_\{v\}^\{\\alpha\}\-q\_\{v\}^\{\\alpha\}\-\\alpha q\_\{v\}^\{\\alpha\-1\}\(p\_\{v\}\-q\_\{v\}\)\]\}\{\\alpha\(\\alpha\-1\)\},\(21\)the separable Bregman divergence generated byxα/\[α⁡\(α−1\)\]x^\{\\alpha\}/\[\\alpha\(\\alpha\-1\)\]\. Its negative student\-probability derivative is the label

wα​\(v\)=q​\(v\)α−1−p​\(v\)α−1α−1,1<α≤2,w\_\{\\alpha\}\(v\)=\\frac\{q\(v\)^\{\\alpha\-1\}\-p\(v\)^\{\\alpha\-1\}\}\{\\alpha\-1\},\\qquad 1<\\alpha\\leq 2,\(22\)whose limitw1=log⁡\(q/p\)w\_\{1\}=\\log\(q/p\)is the OPD label and whose endpointw2=q−pw\_\{2\}=q\-pis the residual, withD1=KL\(p∥q\)D\_\{1\}=\\operatorname\{KL\}\(p\\\|q\)andD2=12​∥p−q∥2D\_\{2\}=\\frac\{1\}\{2\}\\lVert p\-q\\rVert^\{2\}\. By Equation[13](https://arxiv.org/html/2610.11519#A1.E13),

−dd​η​Dα​\(p\+η⁡\(ea−p\),q\)\|η=0\\displaystyle\-\\left\.\\frac\{d\}\{d\\eta\}D\_\{\\alpha\}\(p\+\\eta\(e\_\{a\}\-p\),q\)\\right\|\_\{\\eta=0\}=wα​\(a\)−⟨p,wα⟩,\\displaystyle=w\_\{\\alpha\}\(a\)\-\\langle p,w\_\{\\alpha\}\\rangle,𝔼p​\[\(wα​\(A\)−𝔼p​wα\)​\(eA−p\)\]\\displaystyle\\mathbb\{E\}\_\{p\}\[\(w\_\{\\alpha\}\(A\)\-\\mathbb\{E\}\_\{p\}w\_\{\\alpha\}\)\(e\_\{A\}\-p\)\]=−∇zDα,\\displaystyle=\-\\nabla\_\{z\}D\_\{\\alpha\},\(23\)so the path and expected\-gradient readings of Proposition[1](https://arxiv.org/html/2610.11519#Thmproposition1)hold throughout the family\. The teacher sensitivity∂wα​\(v\)/∂q⁡\(v\)=q​\(v\)α−2\\partial w\_\{\\alpha\}\(v\)/\\partial q\(v\)=q\(v\)^\{\\alpha\-2\}is unbounded at a teacher\-rare token for everyα<2\\alpha<2and constant at the quadratic endpoint, the only member of the family with bounded teacher influence\.

Atp=\(1−ϵ,ϵ\)p=\(1\-\\epsilon,\\epsilon\)andq=\(ϵ,1−ϵ\)q=\(\\epsilon,1\-\\epsilon\), setD=1−2​ϵD=1\-2\\epsilonandL=log⁡\[\(1−ϵ\)/ϵ\]L=\\log\[\(1\-\\epsilon\)/\\epsilon\]\. The student\-centered labels of the confidently wrong mode are−2​ϵ​D\-2\\epsilon Dfor the residual and−2​ϵ​L\-2\\epsilon Lfor the log ratio; the sampled gradients have norms2​2​ϵ2​D2\\sqrt\{2\}\\epsilon^\{2\}Dand2​2​ϵ2​L2\\sqrt\{2\}\\epsilon^\{2\}L, and the expected gradients2​2​ϵ​\(1−ϵ\)​D2\\sqrt\{2\}\\epsilon\(1\-\\epsilon\)Dand2​2​ϵ​\(1−ϵ\)​L2\\sqrt\{2\}\\epsilon\(1\-\\epsilon\)L\. SinceD/L→0D/L\\to 0, the residual pushes a confidently wrong mode more gently than the log ratio\. These statements precede response centering, which can transfer a label from another position into a confident one\.

#### A sampled reward and its local advantage\.

Figure[3](https://arxiv.org/html/2610.11519#A1.F3)holds the student’s distribution and the teacher’s probability of the sampled token fixed while changing the teacher’s alternatives\. Both teachers give the same sampled log ratio and the same residual, but different student values yield local advantages of opposite signs\. This is the usual distinction betweenQQandQ−VQ\-V; in expectation, sampled OPD still follows the negative full\-vocabulary reverse\-KL gradient\.

Figure 3:The same sampled reward can yield local advantages of opposite signs\.Both teachers assign the sampled token probability0\.280\.28against the student’s0\.300\.30, giving the same log ratio \(−0\.069\-0\.069\) and residual \(−0\.02\-0\.02\)\. Their residual vectors have the same squared norm,0\.03980\.0398, but the student values differ\. The resultingAd​\(2\)A^\{d\}\(2\)is\+0\.021\+0\.021for teacher I and−0\.063\-0\.063for teacher II\.Table[6](https://arxiv.org/html/2610.11519#A1.T6)lists the inputs and derived quantities of the two teachers in Figure[3](https://arxiv.org/html/2610.11519#A1.F3)\.

Table 6:The two teachers of Figure[3](https://arxiv.org/html/2610.11519#A1.F3)\.The student distribution and sampled action are shared\. The last row pairs the two teachers as positions of one response with equal position weights and unit guidance weight\.

### A\.5Label centering and gradient variance

Forψ⁡\(A\)=∇θ​log​p​\(A\)\\psi\(A\)=\\nabla\_\{\\theta\}\\log p\(A\)and an action\-independent scalarν\\nu, letgν=\(d⁡\(A\)−ν\)​ψ​\(A\)g\_\{\\nu\}=\(d\(A\)\-\\nu\)\\psi\(A\)\. Its mean does not depend onν\\nu, and completing the square gives

ν∗=𝔼p​\[d⁡\(A\)​‖ψ⁡\(A\)‖2\]𝔼p​‖ψ⁡\(A\)‖2,tr⁡Var⁡\(gν\)=tr⁡Var⁡\(gν∗\)\+𝔼p​‖ψ‖2​\(ν−ν∗\)2\.\\nu\_\{\*\}=\\frac\{\\mathbb\{E\}\_\{p\}\[d\(A\)\\\|\\psi\(A\)\\\|^\{2\}\]\}\{\\mathbb\{E\}\_\{p\}\\\|\\psi\(A\)\\\|^\{2\}\},\\qquad\\operatorname\{tr\}\\operatorname\{Var\}\(g\_\{\\nu\}\)=\\operatorname\{tr\}\\operatorname\{Var\}\(g\_\{\\nu\_\{\*\}\}\)\+\\mathbb\{E\}\_\{p\}\\\|\\psi\\\|^\{2\}\(\\nu\-\\nu\_\{\*\}\)^\{2\}\.\(24\)Thusb=𝔼p​db=\\mathbb\{E\}\_\{p\}dminimizes𝔼p​\(d−ν\)2\\mathbb\{E\}\_\{p\}\(d\-\\nu\)^\{2\}, whereas the gradient\-variance optimum weights by score norm and may differ\. For binary logits, writep=\(x,1−x\)p=\(x,1\-x\)andd=\(u,−u\)d=\(u,\-u\)\. Thenb=\(2​x−1\)​ub=\(2x\-1\)u,ν∗=−b\\nu\_\{\*\}=\-b, and

tr⁡Var⁡\(g0\)=2​x​\(1−x\)​b2,tr⁡Var⁡\(gb\)=8​x​\(1−x\)​b2\.\\operatorname\{tr\}\\operatorname\{Var\}\(g\_\{0\}\)=2x\(1\-x\)b^\{2\},\\qquad\\operatorname\{tr\}\\operatorname\{Var\}\(g\_\{b\}\)=8x\(1\-x\)b^\{2\}\.\(25\)At a single prefix the student value baseline therefore has four times the raw trace variance whenever the latter is nonzero\. It is chosen for the conditional zero mean that Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4)requires, and that property has this variance cost\.

Table 7:A confident student\.The sampled and expected student\-centered logit updates are computed from the stipulated probability vectors\.In Table[7](https://arxiv.org/html/2610.11519#A1.T7), the rare action supplies most of the expected update despite its low sampling probability\.

### A\.6Sensitivity, range, and the guidance bounds

The label is linear inqq, so at fixedp,ap,areplacingqqbyq\+eq\+echanges it byΔ​Ad​\(a\)=⟨ea−p,e⟩\\Delta A^\{d\}\(a\)=\\langle e\_\{a\}\-p,e\\rangleand

\|Δ​Ad​\(a\)\|≤∥ea−p∥2​∥e∥2=1−2​p​\(a\)\+∥p∥22​∥e∥2≤2​∥e∥2\.\|\\Delta A^\{d\}\(a\)\|\\leq\\lVert e\_\{a\}\-p\\rVert\_\{2\}\\lVert e\\rVert\_\{2\}=\\sqrt\{1\-2p\(a\)\+\\lVert p\\rVert\_\{2\}^\{2\}\}\\,\\lVert e\\rVert\_\{2\}\\leq\\sqrt\{2\}\\lVert e\\rVert\_\{2\}\.\(26\)Moving teacher massϵ\\epsilonfrom an off\-action tokenuuto another off\-action tokenvvchanges the label byϵ⁡\[p⁡\(u\)−p⁡\(v\)\]\\epsilon\[p\(u\)\-p\(v\)\], a distinction the sampled log ratio cannot make, since its teacher sensitivity is∂log⁡\(q/p\)/∂q=1/q\\partial\\log\(q/p\)/\\partial q=1/q\. Bounded sensitivity is the bounded\-influence property of the quadratic comparison\([Hampel, 1974](https://arxiv.org/html/2610.11519#bib.bib11);[Basu et al\., 1998](https://arxiv.org/html/2610.11519#bib.bib3)\), and Appendix[A\.4](https://arxiv.org/html/2610.11519#A1.SS4)places it among the alternatives\.

For the range, setx=p⁡\(a\)x=p\(a\)andy=maxv≠a⁡p⁡\(v\)y=\\max\_\{v\\neq a\}p\(v\)\. The affine functionAd​\(a\)=q⁡\(a\)−⟨p,q⟩−x\+∥p∥22A^\{d\}\(a\)=q\(a\)\-\\langle p,q\\rangle\-x\+\\lVert p\\rVert\_\{2\}^\{2\}is maximized overqqatq=eaq=e\_\{a\}and minimized at an off\-action coordinate with student massyy, so

Ad​\(a\)\\displaystyle A^\{d\}\(a\)≤\(1−x\)2\+∑v≠ap​\(v\)2≤2​\(1−x\)2,\\displaystyle\\leq\(1\-x\)^\{2\}\+\\sum\_\{v\\neq a\}p\(v\)^\{2\}\\leq 2\(1\-x\)^\{2\},\(27\)Ad​\(a\)\\displaystyle A^\{d\}\(a\)≥∥p∥22−x−y≥\(x−12\)2\+\(y−12\)2−12≥−12\.\\displaystyle\\geq\\lVert p\\rVert\_\{2\}^\{2\}\-x\-y\\geq\(x\-\\tfrac\{1\}\{2\}\)^\{2\}\+\(y\-\\tfrac\{1\}\{2\}\)^\{2\}\-\\tfrac\{1\}\{2\}\\geq\-\\tfrac\{1\}\{2\}\.\(28\)The lower endpoint is attained byp=\(1/2,1/2\)p=\(1/2,1/2\),q=\(0,1\)q=\(0,1\),a=1a=1, and the supremum22is approached byp=\(ϵ,1−ϵ\)p=\(\\epsilon,1\-\\epsilon\),q=\(1,0\)q=\(1,0\),a=1a=1asϵ↓0\\epsilon\\downarrow 0\. For any realized nonnegative weights with∑tct=1\\sum\_\{t\}c\_\{t\}=1, the guidance termδt=λ​∑u≠tcu​\(Atd−Aud\)\\delta\_\{t\}=\\lambda\\sum\_\{u\\neq t\}c\_\{u\}\(A\_\{t\}^\{d\}\-A\_\{u\}^\{d\}\)therefore satisfies

∑tct​δt=0,\|δt\|≤52​λ​\(1−ct\)≤52​λ,\\sum\_\{t\}c\_\{t\}\\delta\_\{t\}=0,\\qquad\|\\delta\_\{t\}\|\\leq\\frac\{5\}\{2\}\\lambda\(1\-c\_\{t\}\)\\leq\\frac\{5\}\{2\}\\lambda,\(29\)for every realized response, whatever determined its weights\. With fixed weights and a perturbation only at positionjj,∥Δ​δ∥2,c=λ​cj​\(1−cj\)​\|Δ​Ajd\|\\lVert\\Delta\\delta\\rVert\_\{2,c\}=\\lambda\\sqrt\{c\_\{j\}\(1\-c\_\{j\}\)\}\|\\Delta A\_\{j\}^\{d\}\|, where∥v∥2,c2=∑tct​vt2\\lVert v\\rVert\_\{2,c\}^\{2\}=\\sum\_\{t\}c\_\{t\}v\_\{t\}^\{2\}\.

#### Vanishing with distributional agreement\.

Cauchy–Schwarz also bounds each local advantage by the full residual,\|Atd\|≤2​∥dt∥2\|A\_\{t\}^\{d\}\|\\leq\\sqrt\{2\}\\lVert d\_\{t\}\\rVert\_\{2\}, so for any realized response and its normalized weights,

\|δt\|\\displaystyle\|\\delta\_\{t\}\|≤2​λ​\[\(1−ct\)​∥dt∥2\+∑u≠tcu​∥du∥2\]\\displaystyle\\leq\\sqrt\{2\}\\lambda\\left\[\(1\-c\_\{t\}\)\\lVert d\_\{t\}\\rVert\_\{2\}\+\\sum\_\{u\\neq t\}c\_\{u\}\\lVert d\_\{u\}\\rVert\_\{2\}\\right\]≤2​2​λ​\(1−ct\)​maxu​∥du∥2\.\\displaystyle\\leq 2\\sqrt\{2\}\\lambda\(1\-c\_\{t\}\)\\max\_\{u\}\\lVert d\_\{u\}\\rVert\_\{2\}\.\(30\)For the mean absolute amplitude, weighted Cauchy–Schwarz andVarc⁡\(Ad\)≤∑tct​\(Atd\)2\\operatorname\{Var\}\_\{c\}\(A^\{d\}\)\\leq\\sum\_\{t\}c\_\{t\}\(A\_\{t\}^\{d\}\)^\{2\}give the sharper aggregate bound

∑tct​\|δt\|≤∥δ∥2,c=λ​Varc⁡\(Ad\)≤2​λ​\(∑tct​∥dt∥22\)1/2\.\\sum\_\{t\}c\_\{t\}\|\\delta\_\{t\}\|\\leq\\lVert\\delta\\rVert\_\{2,c\}=\\lambda\\sqrt\{\\operatorname\{Var\}\_\{c\}\(A^\{d\}\)\}\\leq\\sqrt\{2\}\\lambda\\left\(\\sum\_\{t\}c\_\{t\}\\lVert d\_\{t\}\\rVert\_\{2\}^\{2\}\\right\)^\{1/2\}\.\(31\)At a fixed guidance weight, the guidance term therefore vanishes uniformly when student and teacher agree uniformly at the scored prefixes\. The implication runs one way\. Small guidance can also arise when centering cancels large, nearly equal local advantages, and neither bound says how agreement or guidance evolve during training\.

The log ratio also vanishes at exact agreement, but it has no uniform bound of this kind in probability distance\. Forp=\(1−ϵ,ϵ\)p=\(1\-\\epsilon,\\epsilon\)andq=\(1−ϵ2,ϵ2\)q=\(1\-\\epsilon^\{2\},\\epsilon^\{2\}\),∥p−q∥2→0\\lVert p\-q\\rVert\_\{2\}\\to 0while the rare\-action student\-centered log label\(1−ϵ\)​log⁡\[ϵ/\(1\+ϵ\)\]→−∞\(1\-\\epsilon\)\\log\[\\epsilon/\(1\+\\epsilon\)\]\\to\-\\infty, and pairing this position with an agreement position at equal weights gives an unbounded response\-centered log label as well\.

#### Outcome and guidance components\.

WithPc=I−𝟏​c⊤P\_\{c\}=I\-\\mathbf\{1\}c^\{\\top\}, the actor label has the vector form

𝐀=AV​𝟏\+λ​Pc​𝐀d,c⊤​Pc=0\.\\mathbf\{A\}=A^\{V\}\\mathbf\{1\}\+\\lambda P\_\{c\}\\mathbf\{A\}^\{d\},\\qquad c^\{\\top\}P\_\{c\}=0\.\(32\)Under⟨u,v⟩c=∑tct​ut​vt\\langle u,v\\rangle\_\{c\}=\\sum\_\{t\}c\_\{t\}u\_\{t\}v\_\{t\}, the two components are orthogonal because⟨𝟏,Pc​𝐀d⟩c=c⊤​Pc​𝐀d=0\\langle\\mathbf\{1\},P\_\{c\}\\mathbf\{A\}^\{d\}\\rangle\_\{c\}=c^\{\\top\}P\_\{c\}\\mathbf\{A\}^\{d\}=0, and expanding the squared weighted norm gives

∥𝐀∥2,c2\\displaystyle\\lVert\\mathbf\{A\}\\rVert\_\{2,c\}^\{2\}=\(AV\)2\+λ2​Vart∼c⁡\(Atd\),\\displaystyle=\(A^\{V\}\)^\{2\}\+\\lambda^\{2\}\\operatorname\{Var\}\_\{t\\sim c\}\(A\_\{t\}^\{d\}\),Vart∼c⁡\(Atd\)\\displaystyle\\operatorname\{Var\}\_\{t\\sim c\}\(A\_\{t\}^\{d\}\)=∑tct​\(Atd−A¯d\)2\.\\displaystyle=\\sum\_\{t\}c\_\{t\}\(A\_\{t\}^\{d\}\-\\overline\{A\}^\{d\}\)^\{2\}\.\(33\)Teacher guidance adds variation to the actor label while preserving its response mean\. Orthogonality of the label components does not extend to their parameter gradients \(Appendix[A\.7](https://arxiv.org/html/2610.11519#A1.SS7)\)\.

Table 8:Further quantities for the two teachers\.The teachers are those of Figure[3](https://arxiv.org/html/2610.11519#A1.F3)\. The final column pairs each teacher’s position with an agreement position at equal weightsc=\(1/2,1/2\)c=\(1/2,1/2\), keeping the symbolic weightλ\\lambda\.The two teachers differ in both the expected logit update and the response\-centered guidance term \(Table[8](https://arxiv.org/html/2610.11519#A1.T8)\)\. Pairing each teacher’s position with an agreement position isolates its own contribution\.

### A\.7Comparing local advantages across a response

At one prefix the student value baseline leaves the expected update unchanged \(Proposition[1](https://arxiv.org/html/2610.11519#Thmproposition1)\(ii\)\)\. Its role appears when local advantages from different prefixes are combined by centering\. Withℱt\\mathcal\{F\}\_\{t\}the prompt and actions before positionttandψt=∇θ​log​pt​\(at\)\\psi\_\{t\}=\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\_\{t\}\), the local advantage has conditional zero mean,𝔼⁡\[Atd∣ℱt\]=0\\mathbb\{E\}\[A\_\{t\}^\{d\}\\mid\\mathcal\{F\}\_\{t\}\]=0, whereas the raw residual has conditional mean⟨pt,dt⟩\\langle p\_\{t\},d\_\{t\}\\rangle, the level of disagreement at that prefix\.

###### Proposition 4\(Response composition\)

Under on\-policy sampling with a fixed finite horizon and trajectory\-independent token weights,

𝔼⁡\[∑tct​δt​ψt\]=λ​∑tct​\(1−ct\)​𝔼​\[Atd​ψt\]\.\\mathbb\{E\}\\Big\[\\sum\_\{t\}c\_\{t\}\\delta\_\{t\}\\psi\_\{t\}\\Big\]=\\lambda\\sum\_\{t\}c\_\{t\}\(1\-c\_\{t\}\)\\,\\mathbb\{E\}\\big\[A\_\{t\}^\{d\}\\psi\_\{t\}\\big\]\.\(34\)

Local advantages and scores are both martingale differences under the student’s sampling, so every cross\-position term vanishes\. Without the student value baseline this cancellation fails, and the disagreement levels of other prefixes enter each token’s update through centering\. The identity needs the stated assumptions, and the response\-mean identity of Equation[6](https://arxiv.org/html/2610.11519#S2.E6)needs none of them\.

#### What the identity says about centering\.

With token\-mean weightsct=1/Mc\_\{t\}=1/M, the factor1−ct1\-c\_\{t\}is within10−310^\{\-3\}of one for a response of a thousand tokens or more\. Under the hypotheses of Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4), centering therefore leaves the expected update almost unchanged, and the response mean it removes,A¯d\\overline\{A\}^\{d\}, has zero expectation and fluctuation of orderM−1/2M^\{\-1/2\}\. Appendix[C\.2](https://arxiv.org/html/2610.11519#A3.SS2)measuresA¯d\\overline\{A\}^\{d\}on the offline responses\. The local advantages average to3×10−43\\times 10^\{\-4\}per token, so the student value baseline holds in practice and the response mean carries no systematic part, and on responses longer than1,0001\{,\}000tokensλ​A¯d\\lambda\\overline\{A\}^\{d\}has an SD below0\.030\.03, as the hypotheses predict\. The fluctuation is not uniformly small, however\. On shorter responses, where few tokens set the average, its SD is0\.320\.32, and8%8\\%of all responses carry\|λ​A¯d\|\>0\.5\|\\lambda\\overline\{A\}^\{d\}\|\>0\.5, the scale of a verifier advantage\. Without centering these responses receive a response\-level label that the verifier did not assign, and Equation[6](https://arxiv.org/html/2610.11519#S2.E6)removes it for every response\. The hypotheses of Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4)thus describe the long responses, which carry most of the tokens, and the small gain of centering in Section[4\.1](https://arxiv.org/html/2610.11519#S4.SS1), about half a point at every weight, is what acting on the rest is worth\.

For a realized response, the teacher gradient at the rollout policy isGd=λ​∑tct​\(Atd−A¯d\)​ψt=λ​Covt∼c⁡\(Atd,ψt\)G\_\{d\}=\\lambda\\sum\_\{t\}c\_\{t\}\(A\_\{t\}^\{d\}\-\\overline\{A\}^\{d\}\)\\psi\_\{t\}=\\lambda\\,\\operatorname\{Cov\}\_\{t\\sim c\}\(A\_\{t\}^\{d\},\\psi\_\{t\}\)and the verifier’s isGV=AV​∑tct​ψtG\_\{V\}=A^\{V\}\\sum\_\{t\}c\_\{t\}\\psi\_\{t\}\. The verifier moves the response along its mean score direction and the teacher along the covariance between local advantages and scores\. The two label components are orthogonal, but these two parameter directions need not be, so the mean identity fixes the teacher’s share of the label and not of the update\.

#### Proof of Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4)\.

Letℱt\\mathcal\{F\}\_\{t\}contain the prompt and actions preceding positiontt, and letψt=∇θ​log​pθ​\(At∣ℱt\)\\psi\_\{t\}=\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(A\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\)at the rollout parameters\. The score identity and the student value baseline give𝔼⁡\[ψt∣ℱt\]=0\\mathbb\{E\}\[\\psi\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=0and𝔼⁡\[Atd∣ℱt\]=0\\mathbb\{E\}\[A\_\{t\}^\{d\}\\mid\\mathcal\{F\}\_\{t\}\]=0\. Expanding the centered estimator,

∑tct​δt​ψt=λ​∑tct​\(1−ct\)​Atd​ψt−λ​∑t∑u≠tct​cu​Aud​ψt\.\\sum\_\{t\}c\_\{t\}\\delta\_\{t\}\\psi\_\{t\}=\\lambda\\sum\_\{t\}c\_\{t\}\(1\-c\_\{t\}\)A\_\{t\}^\{d\}\\psi\_\{t\}\-\\lambda\\sum\_\{t\}\\sum\_\{u\\neq t\}c\_\{t\}c\_\{u\}A\_\{u\}^\{d\}\\psi\_\{t\}\.\(35\)Foru<tu<t,AudA\_\{u\}^\{d\}isℱt\\mathcal\{F\}\_\{t\}\-measurable, so its product withψt\\psi\_\{t\}has zero expectation\. Foru\>tu\>t,ψt\\psi\_\{t\}isℱu\\mathcal\{F\}\_\{u\}\-measurable, and conditioning onℱu\\mathcal\{F\}\_\{u\}annihilatesAudA\_\{u\}^\{d\}\. Thus

𝔼⁡\[∑tct​δt​ψt\]=λ​∑tct​\(1−ct\)​𝔼​\[Atd​ψt\],\\mathbb\{E\}\\\!\\left\[\\sum\_\{t\}c\_\{t\}\\delta\_\{t\}\\psi\_\{t\}\\right\]=\\lambda\\sum\_\{t\}c\_\{t\}\(1\-c\_\{t\}\)\\mathbb\{E\}\[A\_\{t\}^\{d\}\\psi\_\{t\}\],\(36\)which is Equation[34](https://arxiv.org/html/2610.11519#A1.E34)\. Each conditional term is the negative gradient of the local squared distance through that prefix’s logits with the prefix held fixed; the identity does not differentiate the distribution of visited prefixes\. Without the student value baseline the caseu\>tu\>tfails, because a later raw residual has conditional mean⟨pu,du⟩\\langle p\_\{u\},d\_\{u\}\\rangle, and centering then couples positionttto the disagreement levels of prefixes not yet visited\.

The same proof applies to the student\-centered log labelℓtp=ℓt​\(At\)−⟨pt,ℓt⟩\\ell\_\{t\}^\{p\}=\\ell\_\{t\}\(A\_\{t\}\)\-\\langle p\_\{t\},\\ell\_\{t\}\\rangle,

𝔼⁡\[∑tct​β​\(ℓtp−ℓ¯p\)​ψt\]=β​∑tct​\(1−ct\)​𝔼​\[ℓtp​ψt\],\\mathbb\{E\}\\left\[\\sum\_\{t\}c\_\{t\}\\beta\(\\ell\_\{t\}^\{p\}\-\\bar\{\\ell\}^\{p\}\)\\psi\_\{t\}\\right\]=\\beta\\sum\_\{t\}c\_\{t\}\(1\-c\_\{t\}\)\\mathbb\{E\}\[\\ell\_\{t\}^\{p\}\\psi\_\{t\}\],\(37\)so response centering changes the expected update even though the student value baseline alone does not\. For the rawℓt\\ell\_\{t\}the cross terms can survive because𝔼\[ℓt∣ℱt\]=−KL\(pt∥qt\)\\mathbb\{E\}\[\\ell\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=\-\\operatorname\{KL\}\(p\_\{t\}\\\|q\_\{t\}\)\. For example, letp1=q1=\(1/2,1/2\)p\_\{1\}=q\_\{1\}=\(1/2,1/2\)andp2=\(1/2,1/2\)p\_\{2\}=\(1/2,1/2\)after either first action, withq2=\(\.9,\.1\)q\_\{2\}=\(\.9,\.1\)after the first action,q2=p2q\_\{2\}=p\_\{2\}after the second, andc1=c2=1/2c\_\{1\}=c\_\{2\}=1/2\. The independent first\-action logit then receives the expected update−βlog\(\.6\)/16≠0\-\\beta\\log\(\.6\)/16\\neq 0from response\-centered raw log guidance although its own raw log label is zero, whereas the centered log label gives it zero expected update\.

### A\.8The teacher\-peak branch

Let𝒯i\\mathcal\{T\}\_\{i\}contain the positions of responseii; every one of them receives the guidance term\. At each visited prefix, both routes use

hi​t​\(v\)=di​t​\(v\)−⟨pi​t,di​t⟩=Ai​td​\(v\),di​t=qi​t−pi​t\.h\_\{it\}\(v\)=d\_\{it\}\(v\)\-\\langle p\_\{it\},d\_\{it\}\\rangle=A\_\{it\}^\{d\}\(v\),\\qquad d\_\{it\}=q\_\{it\}\-p\_\{it\}\.The sampled label remainsAi​t=AiV\+λ⁡\(hi​t​\(ai​t\)−h¯i\)A\_\{it\}=A\_\{i\}^\{V\}\+\\lambda\(h\_\{it\}\(a\_\{it\}\)\-\\bar\{h\}\_\{i\}\), whereh¯i=∑t∈𝒯ici​t​hi​t​\(ai​t\)\\bar\{h\}\_\{i\}=\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\}c\_\{it\}h\_\{it\}\(a\_\{it\}\)uses exactly the sampledRAweights\. Withvi​t⋆=arg⁡maxv​qi​t​\(v\)v\_\{it\}^\{\\star\}=\\arg\\max\_\{v\}q\_\{it\}\(v\),si​t=arg⁡maxv​pi​t​\(v\)s\_\{it\}=\\arg\\max\_\{v\}p\_\{it\}\(v\), andℬi=\{t∈𝒯i:vi​t⋆≠si​t,vi​t⋆≠ai​t\}\\mathcal\{B\}\_\{i\}=\\\{t\\in\\mathcal\{T\}\_\{i\}:v\_\{it\}^\{\\star\}\\neq s\_\{it\},\\ v\_\{it\}^\{\\star\}\\neq a\_\{it\}\\\}, the branch adds the teacher’s peak as a pseudo\-sample carrying the label a sampled token at that position would receive, with the verifier part clipped at zero:

𝒥branch​\(θ\)\\displaystyle\\mathcal\{J\}\_\{\\rm branch\}\(\\theta\)=1Z​∑i∑t∈𝒯iAi​t​log⁡πθ​\(ai​t∣ξi​t\)\\displaystyle=\\frac\{1\}\{Z\}\\sum\_\{i\}\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\}A\_\{it\}\\log\\pi\_\{\\theta\}\(a\_\{it\}\\mid\\xi\_\{it\}\)\+1Z∑i∑t∈ℬi\(\[AiV\]\+\+λ\(hi​t\(vi​t⋆\)−h¯i\)\)logπθ\(vi​t⋆∣ξi​t\),\\displaystyle\\quad\+\\frac\{1\}\{Z\}\\sum\_\{i\}\\sum\_\{t\\in\\mathcal\{B\}\_\{i\}\}\\big\(\[A\_\{i\}^\{V\}\]\_\{\+\}\+\\lambda\(h\_\{it\}\(v\_\{it\}^\{\\star\}\)\-\\bar\{h\}\_\{i\}\)\\big\)\\log\\pi\_\{\\theta\}\(v\_\{it\}^\{\\star\}\\mid\\xi\_\{it\}\),\(38\)with\[x\]\+=max⁡\(x,0\)\[x\]\_\{\+\}=\\max\(x,0\)andZ=∑i\|𝒯i\|Z=\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|\. The sampled route is computed first, and the added entries enter neitherh¯i\\bar\{h\}\_\{i\}nor the normalizer, so the sampled labels are those ofRAand the sampled mean is unchanged for every response,

∑t∈𝒯ici​t​Ai​t=AiV,∑t∈𝒯ici​t​λ​\(hi​t​\(ai​t\)−h¯i\)=0\.\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\}c\_\{it\}A\_\{it\}=A\_\{i\}^\{V\},\\qquad\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\}c\_\{it\}\\lambda\(h\_\{it\}\(a\_\{it\}\)\-\\bar\{h\}\_\{i\}\)=0\.\(39\)The added labels have no zero\-mean constraint\. They sum over responseiito\|ℬi\|​\[AiV\]\+\+λ​∑t∈ℬi\(hi​t​\(vi​t⋆\)−h¯i\)\|\\mathcal\{B\}\_\{i\}\|\[A\_\{i\}^\{V\}\]\_\{\+\}\+\\lambda\\sum\_\{t\\in\\mathcal\{B\}\_\{i\}\}\(h\_\{it\}\(v\_\{it\}^\{\\star\}\)\-\\bar\{h\}\_\{i\}\), whose verifier part is never negative, so the branch raises the total label of every response with a positive verifier advantage and leaves the verifier part of the others unchanged\. The teacher part pushes toward the peak apart from the shift byh¯i\\bar\{h\}\_\{i\}, as the next proposition shows\.

###### Proposition 5\(The teacher’s peak under the residual advantage\)

Fix a prefix with positivep,qp,qon a common coordinate support and a teacher peakb∈arg⁡maxv⁡q⁡\(v\)b\\in\\arg\\max\_\{v\}q\(v\)over that support\. Then∥p∥22−p⁡\(b\)≤Ad​\(b\)≤2​\(1−p⁡\(b\)\)2\\lVert p\\rVert\_\{2\}^\{2\}\-p\(b\)\\leq A^\{d\}\(b\)\\leq 2\(1\-p\(b\)\)^\{2\}\. HenceAd​\(b\)≥−p⁡\(b\)​\(1−p⁡\(b\)\)≥−14A^\{d\}\(b\)\\geq\-p\(b\)\(1\-p\(b\)\)\\geq\-\\tfrac\{1\}\{4\}, the constant−14\-\\tfrac\{1\}\{4\}is sharp, andAd​\(b\)≥0A^\{d\}\(b\)\\geq 0wheneverp⁡\(b\)≤∥p∥22p\(b\)\\leq\\lVert p\\rVert\_\{2\}^\{2\}\.

#### Proof\.

WriteAd​\(b\)=⟨eb−p,q−p⟩=\[q⁡\(b\)−⟨p,q⟩\]−\[p⁡\(b\)−∥p∥22\]A^\{d\}\(b\)=\\langle e\_\{b\}\-p,q\-p\\rangle=\[q\(b\)\-\\langle p,q\\rangle\]\-\[p\(b\)\-\\lVert p\\rVert\_\{2\}^\{2\}\]\. Sinceq⁡\(b\)≥q⁡\(u\)q\(b\)\\geq q\(u\)for everyuu,⟨p,q⟩≤q⁡\(b\)\\langle p,q\\rangle\\leq q\(b\), which gives the lower bound; with∥p∥22≥p​\(b\)2\\lVert p\\rVert\_\{2\}^\{2\}\\geq p\(b\)^\{2\}it yieldsAd​\(b\)≥−p⁡\(b\)​\(1−p⁡\(b\)\)≥−14A^\{d\}\(b\)\\geq\-p\(b\)\(1\-p\(b\)\)\\geq\-\\tfrac\{1\}\{4\}, andp⁡\(b\)≤∥p∥22p\(b\)\\leq\\lVert p\\rVert\_\{2\}^\{2\}makes the lower bound nonnegative\. For the upper bound,q⁡\(b\)−⟨p,q⟩=∑u≠bp⁡\(u\)​\(q⁡\(b\)−q⁡\(u\)\)≤\(1−p⁡\(b\)\)​q​\(b\)q\(b\)\-\\langle p,q\\rangle=\\sum\_\{u\\neq b\}p\(u\)\(q\(b\)\-q\(u\)\)\\leq\(1\-p\(b\)\)q\(b\)and∥p∥22−p⁡\(b\)≤p​\(b\)2\+\(1−p⁡\(b\)\)2−p⁡\(b\)=\(1−p⁡\(b\)\)​\(1−2​p​\(b\)\)\\lVert p\\rVert\_\{2\}^\{2\}\-p\(b\)\\leq p\(b\)^\{2\}\+\(1\-p\(b\)\)^\{2\}\-p\(b\)=\(1\-p\(b\)\)\(1\-2p\(b\)\), soAd​\(b\)≤\(1−p⁡\(b\)\)​\(q⁡\(b\)\+1−2​p​\(b\)\)≤2​\(1−p⁡\(b\)\)2A^\{d\}\(b\)\\leq\(1\-p\(b\)\)\(q\(b\)\+1\-2p\(b\)\)\\leq 2\(1\-p\(b\)\)^\{2\}\. For sharpness putp⁡\(b\)=12p\(b\)=\\tfrac\{1\}\{2\}, spread the remaining mass uniformly overnnother tokens, and letqqbe uniform over thesen\+1n\+1tokens; thenq⁡\(b\)=⟨p,q⟩q\(b\)=\\langle p,q\\rangleandAd​\(b\)=−14\+14​nA^\{d\}\(b\)=\-\\tfrac\{1\}\{4\}\+\\tfrac\{1\}\{4n\}\.

The peak’s local advantage is therefore negative only when the student already gives the peak more probability than its collision probability∥p∥22=𝔼u∼p​\[p⁡\(u\)\]\\lVert p\\rVert\_\{2\}^\{2\}=\\mathbb\{E\}\_\{u\\sim p\}\[p\(u\)\], and never by more than14\\tfrac\{1\}\{4\}\. Withh¯i∈\[−12,2\]\\bar\{h\}\_\{i\}\\in\[\-\\tfrac\{1\}\{2\},2\], a convex combination of local advantages \(Proposition[1](https://arxiv.org/html/2610.11519#Thmproposition1)\(iii\)\), each added label lies in\[\[AiV\]\+−94​λ,\[AiV\]\+\+52​λ\]\\big\[\[A\_\{i\}^\{V\}\]\_\{\+\}\-\\tfrac\{9\}\{4\}\\lambda,\\ \[A\_\{i\}^\{V\}\]\_\{\+\}\+\\tfrac\{5\}\{2\}\\lambda\\big\]\.

### A\.9An adapted teacher in the residual advantage

At iterationkk, Co\-RAscores the current student batch with the pre\-update teacherqϕkq\_\{\\phi\_\{k\}\}and holds these scores fixed during the student update\. Its residual isdt\(k\)=qϕk,t−pk,td\_\{t\}^\{\(k\)\}=q\_\{\\phi\_\{k\},t\}\-p\_\{k,t\}, its local advantage isAtd,\(k\)=⟨eat−pk,t,dt\(k\)⟩A\_\{t\}^\{d,\(k\)\}=\\langle e\_\{a\_\{t\}\}\-p\_\{k,t\},d\_\{t\}^\{\(k\)\}\\rangle, and with the same normalized response weightsctc\_\{t\}its guidance term isδt\(k\)=λ⁡\(Atd,\(k\)−∑ucu​Aud,\(k\)\)\\delta\_\{t\}^\{\(k\)\}=\\lambda\(A\_\{t\}^\{d,\(k\)\}\-\\sum\_\{u\}c\_\{u\}A\_\{u\}^\{d,\(k\)\}\)\. The per\-response results of Appendix[A\.6](https://arxiv.org/html/2610.11519#A1.SS6)apply directly:

∑tct​\(AV\+δt\(k\)\)\\displaystyle\\sum\_\{t\}c\_\{t\}\\bigl\(A^\{V\}\+\\delta\_\{t\}^\{\(k\)\}\\bigr\)=AV,\\displaystyle=A^\{V\},\|δt\(k\)\|\\displaystyle\|\\delta\_\{t\}^\{\(k\)\}\|≤52​λ​\(1−ct\),\\displaystyle\\leq\\tfrac\{5\}\{2\}\\lambda\(1\-c\_\{t\}\),\|δt\(k\)\|\\displaystyle\|\\delta\_\{t\}^\{\(k\)\}\|≤2​2​λ​\(1−ct\)​maxu​∥qϕk,u−pk,u∥2\.\\displaystyle\\leq 2\\sqrt\{2\}\\lambda\(1\-c\_\{t\}\)\\max\_\{u\}\\lVert q\_\{\\phi\_\{k\},u\}\-p\_\{k,u\}\\rVert\_\{2\}\.\(40\)Teacher adaptation thus preserves the mean identity, bounded guidance, and uniform vanishing with distributional agreement at each scoring step, and the fixed\-prefix expected\-gradient identity holds withq=qϕkq=q\_\{\\phi\_\{k\}\}\.

The teacher LoRA is trained on the same student\-generated trajectories with their verifier advantages \(Appendix[B\.8](https://arxiv.org/html/2610.11519#A2.SS8)\)\. At the start of its clipped update the snapshot ratio is one, so

∇ϕ𝒥T\|ϕk=1Z​∑iAiV​∑t∈𝒯i∇ϕ​log​qϕ,i​t​\(ai​t\)\|ϕk,Z=∑i\|𝒯i\|,\\left\.\\nabla\_\{\\phi\}\\mathcal\{J\}\_\{T\}\\right\|\_\{\\phi\_\{k\}\}=\\frac\{1\}\{Z\}\\sum\_\{i\}A\_\{i\}^\{V\}\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\}\\left\.\\nabla\_\{\\phi\}\\log q\_\{\\phi,it\}\(a\_\{it\}\)\\right\|\_\{\\phi\_\{k\}\},\\qquad Z=\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|,\(41\)withAiVA\_\{i\}^\{V\}held fixed\. At a fixed prefix a sampled term has logit gradientAV​\(ea−qϕk\)A^\{V\}\(e\_\{a\}\-q\_\{\\phi\_\{k\}\}\), and ifA^\(v\)=𝔼\[AV∣ξ,a=v\]\\widehat\{A\}\(v\)=\\mathbb\{E\}\[A^\{V\}\\mid\\xi,a=v\]under the student’s continuations, its student\-policy expectation ispk⊙A^−qϕk​⟨pk,A^⟩p\_\{k\}\\odot\\widehat\{A\}\-q\_\{\\phi\_\{k\}\}\\langle p\_\{k\},\\widehat\{A\}\\rangle\. The trajectories come frompkp\_\{k\}and no importance correction toqϕkq\_\{\\phi\_\{k\}\}is applied, so this is a student\-on\-policy gradient for the teacher\. Shared parameters, the LoRA parameterization, and clipping determine the actual teacher update\.

The updatedqϕk\+1q\_\{\\phi\_\{k\+1\}\}enters the next iteration through the same residual\. At a fixed student distribution, replacing a teacher byq\+Δ​qq\+\\Delta qchanges the local advantage by⟨ea−p,Δ​q⟩\\langle e\_\{a\}\-p,\\Delta q\\rangle, as in Equation[26](https://arxiv.org/html/2610.11519#A1.E26), so teacher adaptation changes the existing guidance without adding another student signal\. Both distributions and the visited prefixes move across iterations, and the identities above do not by themselves give monotone distance reduction, policy improvement, or declining guidance during training\.

## Appendix BExperimental Protocol and Comparison Definitions

### B\.1Training and evaluation configuration

Table 9:Training and evaluation configuration\.Lengths count response tokens\. The correct\-only linear length penalty applies to verifier runs; all runs share the response cap\. Evaluation uses unshaped answer correctness\.SettingConfigurationStudentsQwen3\-1\.7B\-Base and Qwen3\-4B\-BaseTeacher backbone / inference modeQwen3\-8B \(post\-trained\) / non\-thinkingTraining data / prompt countDAPO\-Math\-17K / 17,398Validation prompts drawn from the training file / problems used256 / 64Validation samples per problem / temperature8 / 1\.0Training epochs / prompts per rollout batch2 / 128Incomplete final batch / rollout batches per epochDropped / 135Responses per prompt8 \(GRPO runs\) / 1 \(OPD and REINFORCE\+\+ runs\)Training temperature / response cap1\.0 / 12,000Verifier full\-reward threshold / penalty window8,000 / 4,000RAweightλ\\lambda\(reference configuration\)10RAweight sweep \(1\.7B, REINFORCE\+\+\)5,10,205,\\ 10,\\ 20Co\-RAresidual weightλ\\lambda10No\-baseline / raw residual / no\-centering weights \(1\.7B\)γx=10\\gamma\_\{x\}=10/γraw=10\\gamma\_\{\\rm raw\}=10/λ=10\\lambda=10Residual\-control and directL2L\_\{2\}weight sweeps \(1\.7B\)5,10,205,\\ 10,\\ 20eachRaw logβ\\betaand centered logγlog\\gamma\_\{\\log\}sweeps \(1\.7B\)0\.05,0\.2,0\.6,1\.2,20\.05,\\ 0\.2,\\ 0\.6,\\ 1\.2,\\ 2Clipped centered log ratio: clipcc/ weight sweep55/0\.2,0\.6,1\.20\.2,\\ 0\.6,\\ 1\.2Reportedβ\\beta/γlog\\gamma\_\{\\log\}/ clippedγlog\\gamma\_\{\\log\}/α\\alpha0\.20\.2/0\.60\.6/0\.60\.6/1010Weight selectionValidation split \(Appendix[B\.5](https://arxiv.org/html/2610.11519#A2.SS5)\)Teacher\-only OPD objective / weightFull\-vocabulary reverse KL /11Reverse\-KL loss on REINFORCE\+\+: sweep / reported \(Appendix[D\.5](https://arxiv.org/html/2610.11519#A4.SS5)\)0\.05,0\.2,0\.60\.05,\\ 0\.2,\\ 0\.6/0\.20\.2Sampled OPD \(Appendix[D\.6](https://arxiv.org/html/2610.11519#A4.SS6)\) label / weightlog⁡\[qt​\(at\)/pt​\(at\)\]\\log\[q\_\{t\}\(a\_\{t\}\)/p\_\{t\}\(a\_\{t\}\)\]/11Evaluation temperature / response cap0\.8 / 12,000Evaluation completions per problem8MATH\-500 / AIME24 / AIME25 problems500 / 30 / 30MATH\-500 Avg@8 / Pass@8 increments \(pp\)0\.025 / 0\.2AIME Avg@8 / Pass@8 increments \(pp\)100/240100/240/100/30100/30Table 10:Macro differences among runs\.Differences of three\-seed macro means from Tables[1](https://arxiv.org/html/2610.11519#S3.T1),[19](https://arxiv.org/html/2610.11519#A4.T19), and[4](https://arxiv.org/html/2610.11519#S4.T4), in percentage points\. Each mean and difference is computed from raw counts before rounding; subtracting the displayed macro columns can differ by0\.010\.01\.
### B\.2Probability and actor conventions

Table 11:Probability, actor, and reporting settings\.These settings supplement Table[9](https://arxiv.org/html/2610.11519#A2.T9)\.Student and teacher are evaluated at the same visited prefix on the same aligned output coordinates\. The aligned width is the width of the probability tensor, which exceeds the tokenizer’s nominal size \(Table[11](https://arxiv.org/html/2610.11519#A2.T11)\); the local advantage sums over all aligned coordinates\.

Rollouts use temperature one and full probability support\. Labels are computed at the rollout distribution and held fixed through the student PPO update\. The actor loss covers every token of each response, and the guidance term is applied at every one of these positions, with no token excluded; the centering weights follow the actor’s relative token measure\.

Probabilities, the student value baseline, and actor labels are accumulated at the precision in Table[11](https://arxiv.org/html/2610.11519#A2.T11), and no further whitening is applied toAV\+δA^\{V\}\+\\deltaafter the base estimator has normalized its advantages\. The sampled routes use token\-mean reduction\. WithZZthe batch’s total token count, responseii’s sampled teacher gradient is\(Mi/Z\)​∑tci​t​δi​t​ψi​t\(M\_\{i\}/Z\)\\sum\_\{t\}c\_\{it\}\\delta\_\{it\}\\psi\_\{it\}, so the normalized response gradient analyzed in the text carries the trajectory\-dependent batch weightMi/ZM\_\{i\}/Z, which lies outside the fixed\-weight expectation of Proposition[4](https://arxiv.org/html/2610.11519#Thmproposition4)\. The teacher\-peak branch adds its term with the sameZZ\. The PPO update and student optimizer follow Table[11](https://arxiv.org/html/2610.11519#A2.T11)\.

### B\.3Training reward and advantage construction

Table 12:Verifier advantage normalization settings\.Normalization follows reward shaping, with no reward\-side KL\.Training uses the batch, sampling, and length settings of Table[9](https://arxiv.org/html/2610.11519#A2.T9)\. For runs with verifier feedback, letyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}be answer correctness andLiL\_\{i\}the number of response tokens\. The reward passed to the base estimator is

Ri=yi​clip⁡\(12000−Li4000,0,1\)\.R\_\{i\}=y\_\{i\}\\,\\operatorname\{clip\}\\\!\\left\(\\frac\{12000\-L\_\{i\}\}\{4000\},0,1\\right\)\.\(42\)A correct response receives full reward up to the threshold in Table[9](https://arxiv.org/html/2610.11519#A2.T9), then loses reward linearly until the cap; incorrect responses receive zero\. Shaping occurs before advantage normalization, so a zero raw reward can yield a negative response advantage\. Teacher\-only OPD uses neither this reward nor its soft length penalty and keeps the same 12,000\-token hard cap\.

GRPO normalizes the shaped returns within each prompt group by their mean and sample standard deviation, with the group size and denominator stabilizer of Table[12](https://arxiv.org/html/2610.11519#A2.T12)\. Groups with identical returns stay in the batch with zero advantages, and a group of correct responses can still receive different advantages through the length shaping\.

REINFORCE\+\+ uses terminal returns and the discount and token measure in Table[12](https://arxiv.org/html/2610.11519#A2.T12)\. With no reward\-side KL, the response advantage is

R¯tok\\displaystyle\\bar\{R\}\_\{\\rm tok\}=∑i\|𝒯i\|​Ri∑i\|𝒯i\|,\\displaystyle=\\frac\{\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|R\_\{i\}\}\{\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|\},vtok\\displaystyle v\_\{\\rm tok\}=∑i\|𝒯i\|​\(Ri−R¯tok\)2∑i\|𝒯i\|,\\displaystyle=\\frac\{\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|\(R\_\{i\}\-\\bar\{R\}\_\{\\rm tok\}\)^\{2\}\}\{\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|\},AiV,R\\displaystyle A\_\{i\}^\{V,\\mathrm\{R\}\}=Ri−R¯tokmax⁡\{vtok,10−8\}\.\\displaystyle=\\frac\{R\_\{i\}\-\\bar\{R\}\_\{\\rm tok\}\}\{\\sqrt\{\\max\\\{v\_\{\\rm tok\},10^\{\-8\}\\\}\}\}\.\(43\)Here\|𝒯i\|\|\\mathcal\{T\}\_\{i\}\|counts the tokens of responseii, so the normalization is token\-weighted when response lengths differ, and its variance is computed from the shaped returns\.

### B\.4Evaluation measures

Evaluation uses the checkpoint at the end of the second epoch\. MATH\-500 has 500 problems and each AIME set 30\. For every problem, eight completions are sampled at temperature0\.80\.8with the 12,000\-token cap\. Withyj​ky\_\{jk\}the correctness of completionkkof problemjj, each seed’s scores are

Avg​@​8\\displaystyle\\mathrm\{Avg@8\}=1008​N​∑j=1N∑k=18yj​k,\\displaystyle=\\frac\{100\}\{8N\}\\sum\_\{j=1\}^\{N\}\\sum\_\{k=1\}^\{8\}y\_\{jk\},Pass​@​8\\displaystyle\\mathrm\{Pass@8\}=100N∑j=1N\[∑k=18yj​k\>0\]\.\\displaystyle=\\frac\{100\}\{N\}\\sum\_\{j=1\}^\{N\}\\mathbf\{1\}\\\!\\left\[\\sum\_\{k=1\}^\{8\}y\_\{jk\}\>0\\right\]\.\(44\)Both metrics use unshaped correctness\. Avg@8 is completion accuracy, and Pass@8 is the fraction of problems solved at least once in the eight sampled completions, with no extrapolation to a larger budget\. Macro scores weight the three benchmark percentages equally within each seed and metric, and the main tables report the mean over three seeds\. Appendix[D](https://arxiv.org/html/2610.11519#A4)gives per\-seed macro scores and their sample SD, which uses denominator3−13\-1and is neither a standard error nor a confidence interval\.

Single\-seed score increments follow from these counts \(Table[9](https://arxiv.org/html/2610.11519#A2.T9)\), and three\-seed averages permit finer values\. Means and SD are computed from per\-seed scores and shown to two decimals, so recomputing them from displayed values can differ in the last decimal\.

### B\.5Validation split and weight selection

Before training, 256 prompts are drawn at random from the DAPO\-Math\-17K file with a fixed seed, and 64 of them form the validation set\. Validation samples eight responses per problem at the training temperature of1\.01\.0and scores mean accuracy over the samples\. The guidance weightλ\\lambdaand the control weights of Table[9](https://arxiv.org/html/2610.11519#A2.T9)were chosen on this split, and the evaluation benchmarks of Section[3](https://arxiv.org/html/2610.11519#S3)played no part in any selection\. The validation prompts are not removed from the training file, so they select weights among training\-distribution prompts and never among evaluation problems\. The same 256 prompts supply the offline responses of Appendix[C\.2](https://arxiv.org/html/2610.11519#A3.SS2)\. The sweeps of Tables[23](https://arxiv.org/html/2610.11519#A4.T23)and[24](https://arxiv.org/html/2610.11519#A4.T24)evaluate every swept weight on the benchmarks after the fact, so they show how the selected weights compare with their neighbors and not how they were chosen\.

### B\.6Guidance construction

Table 13:Residual guidance with a fixed or adapted teacher\.Each method uses one teacher distribution, the student value baseline, and response centering\. The weight isλ=10\\lambda=10in both reference configurations\.Table[13](https://arxiv.org/html/2610.11519#A2.T13)compares the teacher distributions ofRAand Co\-RA\.RAkeepsq0q\_\{0\}fixed; Co\-RAuses the current adapter snapshotqϕkq\_\{\\phi\_\{k\}\}in the same residual, with the same weightλ=10\\lambda=10and centering weights\. The combined label is fixed during each scored batch’s update, and the updated adapter is used in the next iteration\. Teacher\-only OPD minimizes the exact per\-token reverse KL divergenceKL\(pt∥qt\)\\operatorname\{KL\}\(p\_\{t\}\\\|q\_\{t\}\), summed over the full aligned vocabulary at every visited prefix of the student’s responses, with the teacher held fixed and gradients through all of the student’s probabilities\. It uses no verifier feedback or length shaping\. The sampled variant of Appendix[D\.6](https://arxiv.org/html/2610.11519#A4.SS6)replaces this loss by the policy\-gradient labellog⁡\[qt​\(at\)/pt​\(at\)\]\\log\[q\_\{t\}\(a\_\{t\}\)/p\_\{t\}\(a\_\{t\}\)\]at the sampled token\([Lu & Thinking Machines Lab, 2025](https://arxiv.org/html/2610.11519#bib.bib22)\)and keeps every other setting\.

#### Control weights\.

The controls are not amplitude\-calibrated\. The residual controls \(without the student value baseline, without response centering, and the raw residual\) use the reference weight1010, and each of them and the direct\-L2L\_\{2\}loss is also run at55and2020, the weights of the sweep of fullRA\. The log\-ratio controls are swept over the weights in Table[9](https://arxiv.org/html/2610.11519#A2.T9), and Table[3](https://arxiv.org/html/2610.11519#S4.T3)reports each at the weight selected on the validation split \(Appendix[B\.5](https://arxiv.org/html/2610.11519#A2.SS5)\); Tables[23](https://arxiv.org/html/2610.11519#A4.T23)and[24](https://arxiv.org/html/2610.11519#A4.T24)list every swept weight on the benchmarks\. No weight, includingλ\\lambdaof fullRA, was selected on the evaluation benchmarks\. Each weight multiplies its signal directly and stays fixed during training\.

The raw log ratio addsβ​ℓi​t\\beta\\ell\_\{it\}, withℓi​t=log⁡\[qi​t​\(ai​t\)/pi​t​\(ai​t\)\]\\ell\_\{it\}=\\log\[q\_\{it\}\(a\_\{it\}\)/p\_\{it\}\(a\_\{it\}\)\], to the REINFORCE\+\+ verifier advantage without either operation\. The raw residual addsγraw​di​t​\(ai​t\)\\gamma\_\{\\rm raw\}d\_\{it\}\(a\_\{it\}\)without either operation, and the uncentered control retainsAtdA\_\{t\}^\{d\}\(Equation[46](https://arxiv.org/html/2610.11519#A2.E46)\)\. DirectL2L\_\{2\}is a loss control; atα=λ\\alpha=\\lambdaand a fixed prefix its gradient at the rollout policy matches the expected residual\-advantage gradient before response centering \(Appendix[A\.2](https://arxiv.org/html/2610.11519#A1.SS2)\), while response centering and batch weights lie outside this identity\.

#### Teacher\-peak branch\.

At each visited prefix letvt⋆=arg⁡maxv​qt​\(v\)v\_\{t\}^\{\\star\}=\\arg\\max\_\{v\}q\_\{t\}\(v\)andst=arg⁡maxv​pt​\(v\)s\_\{t\}=\\arg\\max\_\{v\}p\_\{t\}\(v\), with both maxima taken over the aligned support\. The branch addsvt⋆v\_\{t\}^\{\\star\}exactly whenvt⋆≠stv\_\{t\}^\{\\star\}\\neq s\_\{t\}andvt⋆≠atv\_\{t\}^\{\\star\}\\neq a\_\{t\}; the sampled token may equalsts\_\{t\}\. WithRA, the sampled label remainsAt=AV\+λ⁡\(ht​\(at\)−h¯\)A\_\{t\}=A^\{V\}\+\\lambda\(h\_\{t\}\(a\_\{t\}\)\-\\bar\{h\}\), withht​\(v\)=dt​\(v\)−⟨pt,dt⟩h\_\{t\}\(v\)=d\_\{t\}\(v\)\-\\langle p\_\{t\},d\_\{t\}\\rangleandh¯\\bar\{h\}computed first from the sampled entries, and the added token receives\[AV\]\+\+λ⁡\(ht​\(vt⋆\)−h¯\)\[A^\{V\}\]\_\{\+\}\+\\lambda\(h\_\{t\}\(v\_\{t\}^\{\\star\}\)\-\\bar\{h\}\)with\[x\]\+=max⁡\(x,0\)\[x\]\_\{\+\}=\\max\(x,0\), the label a sampled token at that position would receive with its verifier part clipped at zero\. Both routes use the same residualdt=qt−ptd\_\{t\}=q\_\{t\}\-p\_\{t\}and weightλ\\lambda, andZZcounts only sampled tokens, so the added entries neither dilute the sampled route nor introduce a separate weight\. Equation[38](https://arxiv.org/html/2610.11519#A1.E38)gives the batch objective\. With REINFORCE\+\+ the sampled label isAVA^\{V\}, and the added token receives the same\[AV\]\+\+λ⁡\(ht​\(vt⋆\)−h¯\)\[A^\{V\}\]\_\{\+\}\+\\lambda\(h\_\{t\}\(v\_\{t\}^\{\\star\}\)\-\\bar\{h\}\), withh¯\\bar\{h\}computed from the sampled tokens, soλ\\lambdamultiplies only the branch’s teacher part\. Sampled OPD has no verifier term; there the added token receives the sampled\-OPD labellog⁡qt​\(vt⋆\)−log⁡pt​\(vt⋆\)\\log q\_\{t\}\(v\_\{t\}^\{\\star\}\)\-\\log p\_\{t\}\(v\_\{t\}^\{\\star\}\), so each triggered position is supervised at two tokens, a top\-kkvariant of the sampled estimate\.

The REINFORCE\+\+ runs take one student optimizer step per rollout batch\. At the rollout policy the ratio is one and clipping is inactive, so the log\-probability objective of Equation[38](https://arxiv.org/html/2610.11519#A1.E38)and the ratio surrogate have equal gradients and give the same single update; they differ once the parameters move\.

### B\.7Label and direct\-loss controls

Every control below keeps the REINFORCE\+\+ verifier advantage and changes only how the teacher enters, through the actor label or, for the direct loss, through the objective; Table[9](https://arxiv.org/html/2610.11519#A2.T9)lists the weights\.

#### Raw log ratio\.

To test the role ofRA’s advantage construction, we replace its guidance term with the raw sampled log ratio while retaining the REINFORCE\+\+ sequence advantage:

Atadd=AV\+β​ℓt,ℓt=log⁡qt​\(at\)pt​\(at\)\.A\_\{t\}^\{\\mathrm\{add\}\}=A^\{V\}\+\\beta\\,\\ell\_\{t\},\\qquad\\ell\_\{t\}=\\log\\frac\{q\_\{t\}\(a\_\{t\}\)\}\{p\_\{t\}\(a\_\{t\}\)\}\.The weightβ\\betais swept over five values and held fixed within each run of the 1\.7B student \(Table[24](https://arxiv.org/html/2610.11519#A4.T24); Appendix[B\.6](https://arxiv.org/html/2610.11519#A2.SS6)\)\.

Adding the raw log ratio improves both macro scores over plain REINFORCE\+\+ atβ≤0\.6\\beta\\leq 0\.6and lowers both atβ≥1\.2\\beta\\geq 1\.2, whereas the completeRAguidance term improves both at every tested weight \(Tables[23](https://arxiv.org/html/2610.11519#A4.T23)and[24](https://arxiv.org/html/2610.11519#A4.T24)\)\. At its best weight,β=0\.2\\beta=0\.2, it remains1\.971\.97Avg@8 and2\.702\.70Pass@8 points below fullRA\. With verifier feedback present in both guided runs, this comparison favors the completeRAconstruction over the raw log ratio\. The following controls examine its components and a direct\-loss alternative\.

#### Alternative advantage constructions\.

The centered log ratio uses the conventional action\-minus\-value construction with the log\-ratio reward, followed by the same response centering\. The control without the student value baseline retains the sampled residual and response centering but drops the expectation over the student’s alternatives\. WritingLt​\(v\)=log⁡\[qt​\(v\)/pt​\(v\)\]L\_\{t\}\(v\)=\\log\[q\_\{t\}\(v\)/p\_\{t\}\(v\)\]andPc=I−𝟏​c⊤P\_\{c\}=I\-\\mathbf\{1\}c^\{\\top\}for response centering, their actor labels are

𝐀clog\\displaystyle\\mathbf\{A\}^\{\\rm clog\}=AV​𝟏\+γlog​Pc​\[Lt​\(at\)−⟨pt,Lt⟩\]t,\\displaystyle=A^\{V\}\\mathbf\{1\}\+\\gamma\_\{\\log\}P\_\{c\}\[L\_\{t\}\(a\_\{t\}\)\-\\langle p\_\{t\},L\_\{t\}\\rangle\]\_\{t\},\(45\)𝐀noV\\displaystyle\\mathbf\{A\}^\{\\rm noV\}=AV​𝟏\+γx​Pc​\[dt​\(at\)\]t\.\\displaystyle=A^\{V\}\\mathbf\{1\}\+\\gamma\_\{x\}P\_\{c\}\[d\_\{t\}\(a\_\{t\}\)\]\_\{t\}\.The clipped centered log ratio replacesLtL\_\{t\}in𝐀clog\\mathbf\{A\}^\{\\rm clog\}byclip⁡\(Lt,−c,c\)\\operatorname\{clip\}\(L\_\{t\},\-c,c\)withc=5c=5\. These guidance terms are held fixed, as inRA, and their weights stay fixed during training:γx=10\\gamma\_\{x\}=10, andγlog\\gamma\_\{\\log\}is swept for both log\-ratio controls \(Appendix[B\.6](https://arxiv.org/html/2610.11519#A2.SS6)\)\. These controls ask how the local reward and its student value baseline contribute once teacher guidance is centered across a response\.

#### Raw residual and removal of response centering\.

Two controls on the 1\.7B student with REINFORCE\+\+ use the same probability residual asRA:

Atraw=AV\+γraw​dt​\(at\),AtnoRC=AV\+λ​Atd\.A\_\{t\}^\{\\rm raw\}=A^\{V\}\+\\gamma\_\{\\rm raw\}d\_\{t\}\(a\_\{t\}\),\\qquad A\_\{t\}^\{\\rm noRC\}=A^\{V\}\+\\lambda A\_\{t\}^\{d\}\.\(46\)The raw residual removes both the student value baseline and response centering\. Its weightγraw=10\\gamma\_\{\\rm raw\}=10equals the referenceλ\\lambdaand multipliesdt​\(at\)d\_\{t\}\(a\_\{t\}\)directly\. The second control retains the student value baseline and removes only response centering, at the unchanged weightλ=10\\lambda=10\. All three residual controls are also run at weights55and2020\(Table[23](https://arxiv.org/html/2610.11519#A4.T23)\)\. The comparison fixes the weight while changing the centering operation rather than matching the resulting signal amplitude\. All teacher terms are held fixed and use the same actor reduction asRA\. The raw residual differs from the no\-student\-value\-baseline control in Equation[45](https://arxiv.org/html/2610.11519#A2.E45), which retains response centering\.

Without response centering, the weighted response mean of the actor label isAV\+γraw​∑tct​dt​\(at\)A^\{V\}\+\\gamma\_\{\\rm raw\}\\sum\_\{t\}c\_\{t\}d\_\{t\}\(a\_\{t\}\)for the raw residual andAV\+λ​A¯dA^\{V\}\+\\lambda\\overline\{A\}^\{d\}for the uncentered control, whereasRApreservesAVA^\{V\}exactly\. The student value baseline compares actions at a prefix; response centering removes the realized teacher mean across positions\. Table[2](https://arxiv.org/html/2610.11519#S4.T2)summarizes these two controls across the weight sweep, and Table[18](https://arxiv.org/html/2610.11519#A4.T18)gives the individual runs\.

#### The same local geometry through a direct loss\.

The direct\-L2L\_\{2\}control adds a differentiable penalty to the verifier objective:

𝒥direct​\(θ\)\\displaystyle\\mathcal\{J\}\_\{\\rm direct\}\(\\theta\)=𝒥PPO​\(AV\)−α2​Z​∑i∑t∈𝒯iteach∑v∈𝒱\(πθ​\(v∣ξi​t\)−qi​t​\(v\)\)2\.\\displaystyle=\\mathcal\{J\}\_\{\\rm PPO\}\(A^\{V\}\)\-\\frac\{\\alpha\}\{2Z\}\\sum\_\{i\}\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}^\{\\rm teach\}\}\\sum\_\{v\\in\\mathcal\{V\}\}\\bigl\(\\pi\_\{\\theta\}\(v\\mid\\xi\_\{it\}\)\-q\_\{it\}\(v\)\\bigr\)^\{2\}\.\(47\)HereZZis the token\-reduction denominator\. The teacher distributions and visited prefixes are fixed, and gradients pass through the current student probabilities\. The vocabulary sum is not divided byKK\. We runα∈\{5,10,20\}\\alpha\\in\\\{5,10,20\\\}, the weights of theλ\\lambdasweep, without amplitude calibration;α=10\\alpha=10gives the highest macro Avg@8 and is reported in Table[3](https://arxiv.org/html/2610.11519#S4.T3)\. At a fixed prefix, the logit gradient at the rollout policy isα​Jp​d\\alpha J\_\{p\}d, matching the expected residual\-advantage gradient at this common weight before response centering \(Proposition[1](https://arxiv.org/html/2610.11519#Thmproposition1)\(ii\); Appendix[A\.2](https://arxiv.org/html/2610.11519#A1.SS2)\)\. Response centering, trajectory\-dependent reduction weights, and sampled gradient noise still distinguish the two complete training updates\. The REINFORCE\+\+ comparison uses one optimizer step per rollout batch, for which surrogate clipping is inactive at the rollout policy\.

### B\.8Adapter objective and update order

Table 14:Adapter configuration and update schedule\.The teacher backboneq0q\_\{0\}is frozen\. The adapter and its separate optimizer follow Table[14](https://arxiv.org/html/2610.11519#A2.T14); the adapter targets the attention and MLP projections, and its effective update starts at zero\.

At iterationkk, the studentpkp\_\{k\}generates a batch and the current adapter snapshotqϕkq\_\{\\phi\_\{k\}\}scores it before either model is updated\. The verifier advantageAiV,ℬA\_\{i\}^\{V,\\mathcal\{B\}\}comes from the run’s base estimatorℬ∈\{GRPO,REINFORCE\+\+\}\\mathcal\{B\}\\in\\\{\\mathrm\{GRPO\},\\mathrm\{REINFORCE\+\+\}\\\}, and the adapter applies that estimator’s clipped surrogate to the same scored batch:

𝒥T​\(ϕ\)\\displaystyle\\mathcal\{J\}\_\{T\}\(\\phi\)=1Z​∑i∑t∈𝒯imin⁡\(ρi​tT​AiV,ℬ,clip⁡\(ρi​tT,1−ϵclip,1\+ϵclip\)​AiV,ℬ\),\\displaystyle=\\frac\{1\}\{Z\}\\sum\_\{i\}\\sum\_\{t\\in\\mathcal\{T\}\_\{i\}\}\\min\\\!\\big\(\\rho^\{T\}\_\{it\}A\_\{i\}^\{V,\\mathcal\{B\}\},\\ \\operatorname\{clip\}\(\\rho^\{T\}\_\{it\},1\-\\epsilon\_\{\\rm clip\},1\+\\epsilon\_\{\\rm clip\}\)A\_\{i\}^\{V,\\mathcal\{B\}\}\\big\),ρi​tT\\displaystyle\\rho^\{T\}\_\{it\}=qϕ,i​t​\(ai​t\)qϕk,i​t​\(ai​t\),Z=∑i\|𝒯i\|,\\displaystyle=\\frac\{q\_\{\\phi,it\}\(a\_\{it\}\)\}\{q\_\{\\phi\_\{k\},it\}\(a\_\{it\}\)\},\\qquad Z=\\sum\_\{i\}\|\\mathcal\{T\}\_\{i\}\|,with clipping widthϵclip\\epsilon\_\{\\rm clip\}from Table[14](https://arxiv.org/html/2610.11519#A2.T14), the student’s response tokens and batch token\-mean reduction, andqϕkq\_\{\\phi\_\{k\}\}the adapter snapshot that scored this iteration’s responses\. The snapshot ratio limits how farqϕq\_\{\\phi\}moves during its update and does not reweight the student’s samples toward the teacher’s distribution\. At the start of the update every ratio is one and the gradient is Equation[9](https://arxiv.org/html/2610.11519#S2.E9)\. The adapter receives the verifier advantage without the guidance term, optimizer and adapter states persist across iterations, and no adapter KL regularizer or label clipping is used\.

Every iteration follows the same order\. The student generates a batch; the current snapshots and the verifier score it; the labels and rollout probabilities are frozen; the student and the teacher LoRA are updated on that batch, in either order, since their targets are fixed and their parameters are separate\. The updatedqϕk\+1q\_\{\\phi\_\{k\+1\}\}scores the next iteration, whose residual isqϕk\+1−pk\+1q\_\{\\phi\_\{k\+1\}\}\-p\_\{k\+1\}with the same weight, student value baseline, and response centering \(Table[13](https://arxiv.org/html/2610.11519#A2.T13)\)\. The backboneq0q\_\{0\}stays frozen and the adapter state is carried forward\.

### B\.9Computation and evidence scope

All runs train for two epochs on the 17,398\-prompt DAPO\-Math\-17K file\. With 128 prompts per rollout batch and the final incomplete batch dropped, each epoch processes 135 batches, or 17,280 prompt occurrences, so the two epochs comprise 270 rollout batches and 34,560 prompt occurrences\. The scheduled generation budgets are therefore 276,480 responses for GRPO runs and 34,560 for OPD and REINFORCE\+\+ runs, cumulative counts of student generations on which Appendix[C](https://arxiv.org/html/2610.11519#A3)reports training dynamics\.RAadds a teacher forward pass and the full\-vocabulary inner product; Co\-RAscores with the current adapted teacher and additionally trains its LoRA on every rollout batch \(Table[14](https://arxiv.org/html/2610.11519#A2.T14)\)\. The method needs neither counterfactual continuations nor a separately fitted value head\. Wall\-clock and peak\-memory measurements are not reported\.

## Appendix CTraining dynamics

Figure[4](https://arxiv.org/html/2610.11519#A3.F4)reports logged training accuracy for the Qwen3\-4B\-Base student across the five frozen\-teacher and verifier runs under the two\-epoch DAPO\-Math\-17K protocol of Table[9](https://arxiv.org/html/2610.11519#A2.T9)\. The horizontal coordinate counts scheduled student generations: each processed prompt contributes eight responses in GRPO runs and one in OPD and REINFORCE\+\+ runs\. Teacher scoring and updates, retries, filtering, and evaluation lie outside this count\.

Figure 4:Training accuracy of Qwen3\-4B\-Base on DAPO\-Math\-17K\.\(A\) Full two\-epoch response budgets: 276,480 generations for GRPO runs and 34,560 for the one\-response runs\. \(B\) The first 34,560 generations, the shaded region in \(A\), show the early trajectories at the same response budget\. Curves show logged training accuracy; Co\-RAis omitted\.RAmaintains higher training accuracy than its verifier\-only estimator over most of training\. Among the one\-response runs, OPD rises more slowly than REINFORCE\+\+ at first and later exceeds it, while staying belowRA\+REINFORCE\+\+ over most of the shared response budget, as the expanded early window shows\. Teacher\-only OPD trains without verifier feedback and without the length shaping of the verifier runs, so its trajectory reflects a different recipe and not only a different label\.

Training accuracy measures correctness on the training prompts under the rollout policy, whereas Table[1](https://arxiv.org/html/2610.11519#S3.T1)evaluates held\-out benchmarks at a different sampling temperature, so the sizes of the two gains need not agree\.

### C\.1Guidance magnitude and response length

\(a\) Teacher guidance

\(b\) Response length by correctness

Figure 5:Training diagnostics for the 4B student withRA\+REINFORCE\+\+\.The guidance weight is fixed atλ=10\\lambda=10, and the horizontal axis counts rollout batches\. \(a\) Recorded mean absolute guidance term\|δt\|\|\\delta\_\{t\}\|, averaged over the tokens of each batch\. \(b\) Mean response length separated by verifier correctness\. The curves are from a single training seed\.At the fixed weightλ=10\\lambda=10, the recorded mean absolute guidance term falls from about0\.430\.43to about0\.300\.30over training \(Figure[5](https://arxiv.org/html/2610.11519#A3.F5)a\)\. The teacher signal weakens in this run without any weight reduction, as the distance bound of Equation[8](https://arxiv.org/html/2610.11519#S2.E8)allows, while still contributing to the actor labels at the end of training\. The quantity is a token average over responses whose lengths change during training, so it is read together with panel \(b\)\.

Correct\-response mean length rises but stays below about 4,000 tokens\. Incorrect\-response mean length rises more sharply and later fluctuates mostly between 9,000 and 12,000 tokens, close to the cap; the shaped reward penalizes length only for correct responses, so nothing in the reward opposes the growth of incorrect ones\. Correct responses in this run therefore stay short relative to the cap, and failed responses dominate the token count, which is where a per\-token modulation of a shared negative label has the most to act on\.RAsupplies local guidance at student\-visited prefixes without complete teacher trajectories; length alone does not measure exploration\.

### C\.2Where the guidance lands

Table[15](https://arxiv.org/html/2610.11519#A3.T15)is computed offline rather than logged during training\. The untrained Qwen3\-1\.7B\-Base student generates one response for each of the 256 validation prompts of Appendix[B\.5](https://arxiv.org/html/2610.11519#A2.SS5), under the training sampling settings of Table[9](https://arxiv.org/html/2610.11519#A2.T9)\(vLLM, temperature1\.01\.0, full support, 12,000\-token cap\), and the student and teacher distributions at every visited prefix are then recomputed in float32 over the full aligned vocabulary, giving 176,628 tokens\. Two labels are computed on these same responses, fullRA\(λ=10\\lambda=10\) and the centered log\-ratio control \(γlog=0\.6\\gamma\_\{\\log\}=0\.6, Table[3](https://arxiv.org/html/2610.11519#S4.T3)\), and the tokens are bucketed by the student’s probability of the sampled token\. For each bucket the table reports the share of tokens and the share of the summed absolute guidance term,\|δt\|\|\\delta\_\{t\}\|forRAand the centered log\-ratio term of Equation[45](https://arxiv.org/html/2610.11519#A2.E45)for the control; the ratio of the two shares is the bucket’s enrichment\. Because both labels are evaluated on the same tokens, the two columns share one set of token shares; the statistic depends only on the two distributions and the sampled tokens, not on correctness\. The last row is the overlapping set of tokens with teacher probability below10−410^\{\-4\}\. Both labels apply the student value baseline and response centering, so the two columns differ only in the reward\. The residual concentrates on tokens where the student is uncertain, whereas the log ratio concentrates on tokens that one side considers nearly impossible, above all on tokens the teacher assigns less than10−410^\{\-4\}, where its label diverges \(Proposition[3](https://arxiv.org/html/2610.11519#Thmproposition3)\); Section[4\.3](https://arxiv.org/html/2610.11519#S4.SS3)discusses the pattern\.

Table 15:Where the guidance lands\.Share of tokens and of the summed absolute guidance term per bucket of the student’s probability of the sampled token, for fullRAand the centered log\-ratio control, both computed offline on the same 256 responses of the untrained 1\.7B student, with the ratio of the two shares in parentheses\. The last row overlaps the others\.#### The response mean of the local advantages\.

Table[16](https://arxiv.org/html/2610.11519#A3.T16)reports, on the same 256 responses, the mean over a response’s tokens of the local advantage,A¯d\\overline\{A\}^\{d\}, and of the raw residualdt​\(at\)d\_\{t\}\(a\_\{t\}\), both scaled byλ=10\\lambda=10\. Without response centering, the first is what the teacher adds to the response’s mean label \(Equation[5](https://arxiv.org/html/2610.11519#S2.E5)\) and the second is the corresponding shift of the raw residual control \(Equation[46](https://arxiv.org/html/2610.11519#A2.E46)\)\. Across tokens the local advantage averages to3×10−43\\times 10^\{\-4\}, so the student value baseline holds in practice; its response mean is a fluctuation whose spread falls with length, from an SD of0\.320\.32below1,0001\{,\}000tokens to0\.030\.03between1,0001\{,\}000and4,0004\{,\}000, and8%8\\%of all responses lie beyond0\.50\.5in absolute value, most of them positive \(the 95th percentile is\+0\.54\+0\.54and the 5th−0\.29\-0\.29\)\. The raw residual shifts every length group by a negative amount,−0\.32\-0\.32on average, with26%26\\%of responses beyond0\.50\.5\. Only three of the 256 responses are correct, so no split by correctness is reported\. Section[4\.1](https://arxiv.org/html/2610.11519#S4.SS1)uses both facts\.

Table 16:Response mean of the guidance without centering\.Mean and SD across responses ofλ​A¯d\\lambda\\overline\{A\}^\{d\}\(local advantage, with the student value baseline\) and ofλ​dt​\(at\)¯\\lambda\\,\\overline\{d\_\{t\}\(a\_\{t\}\)\}\(raw residual\), on the 256 offline responses of Table[15](https://arxiv.org/html/2610.11519#A3.T15), withλ=10\\lambda=10;MMis the number of tokens andnnthe number of responses in each group\.

### C\.3How the sampled label and the direct loss share the update

The directL2L\_\{2\}loss of Section[4\.2](https://arxiv.org/html/2610.11519#S4.SS2)and the uncentered residual label have the same expected teacher gradient at each visited prefix \(Appendix[A\.2](https://arxiv.org/html/2610.11519#A1.SS2)\), yet the loss trails both residual labels at every swept weight \(Table[2](https://arxiv.org/html/2610.11519#S4.T2)\)\. Table[17](https://arxiv.org/html/2610.11519#A3.T17)compares the three runs at the reference weight1010on Qwen3\-1\.7B\-Base with REINFORCE\+\+, over their three seeds\. On each logged batch,gTg\_\{T\}andgVg\_\{V\}are the gradients of the teacher term and of the verifier term\. The conditional noise istr⁡Cov⁡\(gT\)/∥𝔼​gT∥2\\operatorname\{tr\}\\operatorname\{Cov\}\(g\_\{T\}\)/\\lVert\\mathbb\{E\}g\_\{T\}\\rVert^\{2\}over the sampled tokens at the visited prefixes\. The shared\-path term comparescos⁡\(gT,gV\)\\cos\(g\_\{T\},g\_\{V\}\)on the sampled tokens with the same cosine when the teacher term is evaluated on independently resampled tokens\. A unit of teacher\-gradient mass is outcome\-compatible when it raises a correct response or lowers an incorrect one\. The updateuuis the parameter step after global gradient clipping and Adam,uVu\_\{V\}the step the verifier term alone would take,sTs\_\{T\}the teacher’s share ofuu, and the next\-batch probe evaluates, on the following batch, the gain in the verifier objective produced by each step\.

Table 17:The sampled label reproduces the direct loss’s expected teacher gradient but shares the update with the verifier\.Qwen3\-1\.7B\-Base with REINFORCE\+\+ at weight1010; aggregates over three seeds\.StatisticFullRAUncenteredDirectL2L\_\{2\}Cosine of sampled and analyticgTg\_\{T\}0\.9940\.9921\.000Conditional noise ofgTg\_\{T\}0\.851\.080cos⁡\(gT,gV\)\\cos\(g\_\{T\},g\_\{V\}\), sampled tokens\+0\.18\+0\.18\+0\.13\+0\.130\.000\.00cos⁡\(gT,gV\)\\cos\(g\_\{T\},g\_\{V\}\), resampled tokens\+0\.02\+0\.020\.000\.00–Outcome\-compatible teacher mass64%60%49%Batches withcos⁡\(gT,gV\)<0\\cos\(g\_\{T\},g\_\{V\}\)<026%33%50%RMS∥gT∥/∥gV∥\\lVert g\_\{T\}\\rVert/\\lVert g\_\{V\}\\rVert0\.580\.620\.43Global clip rate39%43%25%Teacher share of the updatesTs\_\{T\}0\.280\.310\.44cos⁡\(u,uV\)\\cos\(u,u\_\{V\}\)0\.930\.910\.80Verifier retention0\.940\.910\.80Next\-batch verifier gain1\.13×\\times1\.08×\\times1\.00×\\timesPaired win rate againstL2L\_\{2\}70%63%–Averaged over 64 resampled tokens, the sampled teacher gradient matches its analytic expectation \(cosine0\.9940\.994forRA, norm ratio1\.0011\.001\), so the gap does not lie in the expected teacher gradient\. The sampled gradient is noisier; for the uncentered label, most of its extra second moment relative to the direct loss is the estimator’s own noise \(7575–90%90\\%\), and1515–25%25\\%comes from sharing the verifier’s sampled path:cos⁡\(gT,gV\)\\cos\(g\_\{T\},g\_\{V\}\)is\+0\.18\+0\.18forRAon the sampled tokens and\+0\.02\+0\.02on resampled ones, whereas the direct loss, which acts on every token, is uncorrelated with the verifier\. As a result,64%64\\%ofRA’s teacher\-gradient mass is outcome\-compatible against49%49\\%for the direct loss, mainly because more of it lowers incorrect responses \(46%46\\%against35%35\\%\)\. The direct loss has the smaller gradient \(0\.430\.43of the verifier’s, against0\.580\.58\), but it is deterministic given the prefixes, and after clipping and Adam it takes the larger share of the update,0\.440\.44against0\.280\.28\. Over training this share rises from0\.360\.36to0\.480\.48for the direct loss and falls from0\.300\.30to0\.250\.25forRA, and the cosine of the update with the verifier’s step falls from0\.880\.88to0\.740\.74for the direct loss whileRAstays at0\.920\.92–0\.940\.94\. On the next batch,RA’s step raises the verifier objective1\.101\.10–1\.161\.16times as much as the direct loss’s step and wins6666–74%74\\%of paired comparisons\. The orderingRA, uncentered label, direct loss holds on every seed for the shared\-path term, the outcome\-compatible mass, the teacher share, and the cosine with the verifier’s step\. The divergence between the uncentered label and the direct loss grows with the weight \(update cosine0\.970\.97,0\.910\.91, and0\.830\.83at55,1010, and2020\), as does the margin ofRAover the direct loss in Table[2](https://arxiv.org/html/2610.11519#S4.T2), and it lies mostly on tokens with0\.1≤p⁡\(at\)<0\.50\.1\\leq p\(a\_\{t\}\)<0\.5\(55%55\\%\), the forks of Section[4\.3](https://arxiv.org/html/2610.11519#S4.SS3)\. Response centering corrects the uncentered label mainly on responses shorter than1,0001\{,\}000tokens \(79%79\\%of the correction\), in line with Section[4\.1](https://arxiv.org/html/2610.11519#S4.SS1)\.

## Appendix DThree\-seed results

Each method is evaluated over three training seeds under the protocol of Appendix[B](https://arxiv.org/html/2610.11519#A2)\. Table[18](https://arxiv.org/html/2610.11519#A4.T18)first averages the three benchmarks equally within each seed, then reports the mean and sample standard deviation \(SD\) of the three macro scores; SD uses denominator3−13\-1and describes variation across runs rather than a standard error or confidence interval\. S1–S3 identify the three runs\.

Table 18:Macro performance across three training seeds\.Scores in %;±\\pmdenotes sample SD across the three run\-level macro scores\. The middle block lists the 1\.7B REINFORCE\+\+ controls of Section[4](https://arxiv.org/html/2610.11519#S4), whose reference is theRA\+REINFORCE\+\+ run of the first block; log\-ratio weights are in parentheses\.MethodAvg@8Pass@8S1S2S3Mean±\\pmSDS1S2S3Mean±\\pmSDQwen3\-1\.7B\-BaseOPD27\.6127\.1727\.5827\.45±\\pm0\.2541\.6040\.6941\.6041\.30±\\pm0\.53REINFORCE\+\+26\.1224\.9125\.9725\.67±\\pm0\.6641\.2037\.9339\.1839\.44±\\pm1\.65RA\+REINFORCE\+\+29\.3628\.9629\.4329\.25±\\pm0\.2546\.1143\.1146\.3845\.20±\\pm1\.81Co\-RA\+REINFORCE\+\+30\.1330\.6531\.3330\.70±\\pm0\.6144\.2947\.3647\.4946\.38±\\pm1\.81GRPO27\.9027\.3827\.1927\.49±\\pm0\.3740\.5640\.2240\.4240\.40±\\pm0\.17RA\+GRPO29\.9429\.4930\.1329\.86±\\pm0\.3344\.4244\.2245\.6044\.75±\\pm0\.74Co\-RA\+GRPO30\.2831\.9031\.0931\.09±\\pm0\.8143\.0445\.5344\.2944\.29±\\pm1\.24Qwen3\-1\.7B\-Base, controls with REINFORCE\+\+ \(Section[4](https://arxiv.org/html/2610.11519#S4)\)w/o student value baseline27\.2227\.4426\.8227\.16±\\pm0\.3141\.6043\.1140\.1641\.62±\\pm1\.48w/o response centering28\.8028\.3129\.0128\.71±\\pm0\.3642\.7845\.2046\.2444\.74±\\pm1\.78Raw residual \(w/o both\)27\.8927\.6327\.0427\.52±\\pm0\.4342\.7143\.0441\.5342\.43±\\pm0\.79Clipped centered log ratio \(c=5c\{=\}5,0\.60\.6\)28\.2827\.7328\.1228\.04±\\pm0\.2844\.0942\.9143\.5143\.50±\\pm0\.59Centered log ratio \(0\.60\.6\)28\.1827\.4527\.9327\.85±\\pm0\.3742\.9142\.9843\.3143\.07±\\pm0\.21Raw log ratio \(0\.20\.2\)27\.5126\.8527\.4727\.28±\\pm0\.3742\.7141\.7343\.0442\.50±\\pm0\.68DirectL2L\_\{2\}loss \(α=10\\alpha\{=\}10\)28\.2628\.1327\.6728\.02±\\pm0\.3144\.0243\.3142\.9843\.44±\\pm0\.53RAwithλ=5\\lambda\{=\}528\.6229\.0028\.8928\.84±\\pm0\.2045\.2744\.4245\.1344\.94±\\pm0\.45RAwithλ=20\\lambda\{=\}2029\.6730\.7527\.4529\.29±\\pm1\.6845\.3348\.0942\.4445\.29±\\pm2\.82Qwen3\-4B\-BaseOPD34\.6034\.2534\.9734\.61±\\pm0\.3648\.1147\.9151\.5149\.18±\\pm2\.02REINFORCE\+\+34\.1933\.7333\.6733\.86±\\pm0\.2847\.9147\.6444\.5146\.69±\\pm1\.89RA\+REINFORCE\+\+37\.3136\.8137\.2537\.12±\\pm0\.2753\.1652\.8953\.0253\.02±\\pm0\.13Co\-RA\+REINFORCE\+\+38\.5338\.1438\.5138\.39±\\pm0\.2250\.9350\.6050\.8050\.78±\\pm0\.17GRPO35\.5535\.9036\.0635\.83±\\pm0\.2647\.9148\.1148\.0448\.02±\\pm0\.10RA\+GRPO37\.2937\.5637\.8737\.57±\\pm0\.2950\.7350\.9354\.0751\.91±\\pm1\.87Co\-RA\+GRPO38\.3038\.7838\.7338\.60±\\pm0\.2652\.8953\.0953\.0253\.00±\\pm0\.10### D\.1Benchmark\-level results for the controls

Table[19](https://arxiv.org/html/2610.11519#A4.T19)lists benchmark\-level three\-seed means for the control runs of Section[4](https://arxiv.org/html/2610.11519#S4), with the labels of Tables[2](https://arxiv.org/html/2610.11519#S4.T2)and[3](https://arxiv.org/html/2610.11519#S4.T3)\. The reference runs repeat Table[1](https://arxiv.org/html/2610.11519#S3.T1); the teacher\-peak branch runs of Section[4\.4](https://arxiv.org/html/2610.11519#S4.SS4)are in Appendix[D\.3](https://arxiv.org/html/2610.11519#A4.SS3)\. Macro means and sample SD use per\-seed macro scores, so recomputing them from the displayed benchmark values can differ in the last decimal\.

Each control uses the same weight across its three seeds\. The log\-ratio and no\-baseline weights multiply the complete response\-centered guidance term in Equation[45](https://arxiv.org/html/2610.11519#A2.E45); the direct\-L2L\_\{2\}weight scales the quadratic auxiliary loss in Equation[47](https://arxiv.org/html/2610.11519#A2.E47); the raw residual and the raw log ratio add the uncentered sampled reward at the weightsγraw=10\\gamma\_\{\\rm raw\}=10andβ=0\.2\\beta=0\.2\(Equation[46](https://arxiv.org/html/2610.11519#A2.E46); Appendix[B\.7](https://arxiv.org/html/2610.11519#A2.SS7)\)\. Log\-ratio and direct\-L2L\_\{2\}controls appear at the weights reported in Table[3](https://arxiv.org/html/2610.11519#S4.T3)\.

Table 19:Benchmark\-level results for the control runs of Section[4](https://arxiv.org/html/2610.11519#S4)\.Qwen3\-1\.7B\-Base with REINFORCE\+\+; three\-seed means in %\. The macro columns repeat Tables[2](https://arxiv.org/html/2610.11519#S4.T2)and[3](https://arxiv.org/html/2610.11519#S4.T3); sample SD are reported in Table[18](https://arxiv.org/html/2610.11519#A4.T18)\.VariantMATH\-500AIME24AIME25MacroAvg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8Verifier only \(REINFORCE\+\+\)65\.7683\.877\.7820\.003\.4714\.4425\.6739\.44Teacher only \(OPD\)67\.7885\.008\.6123\.335\.9715\.5627\.4541\.30FullRA\(λ=10\\lambda\{=\}10\)69\.6985\.6013\.0630\.005\.0020\.0029\.2545\.20Probability residual, baselines removedw/o student value baseline67\.1784\.879\.7224\.444\.5815\.5627\.1641\.62w/o response centering69\.0385\.3312\.3628\.894\.7220\.0028\.7144\.74Raw residual \(w/o both\)67\.8385\.0710\.4225\.564\.3116\.6727\.5242\.43Log ratio in place of the probability residualClipped centered log ratio \(c=5c\{=\}5\)68\.4386\.0710\.9725\.564\.7218\.8928\.0443\.50Centered log ratio68\.1385\.8710\.6925\.564\.7217\.7827\.8543\.07Raw log ratio67\.5385\.2710\.0024\.444\.3117\.7827\.2842\.50Direct lossDirectL2L\_\{2\}loss68\.5085\.8710\.8326\.674\.7217\.7828\.0243\.44Guidance weightRAwithλ=5\\lambda\{=\}568\.7385\.9312\.3628\.895\.4220\.0028\.8444\.94RAwithλ=20\\lambda\{=\}2068\.9885\.8713\.0631\.115\.8318\.8929\.2945\.29
### D\.2Per\-seed benchmark results for the controls

Table[20](https://arxiv.org/html/2610.11519#A4.T20)gives per\-seed benchmark results for seven controls: no student value baseline, centered log ratio, and clipped centered log ratio \(Equation[45](https://arxiv.org/html/2610.11519#A2.E45)\), no response centering and the raw residual \(Equation[46](https://arxiv.org/html/2610.11519#A2.E46)\), the raw log ratio \(Appendix[B\.7](https://arxiv.org/html/2610.11519#A2.SS7)\), and directL2L\_\{2\}\(Equation[47](https://arxiv.org/html/2610.11519#A2.E47)\)\. Each cell gives Avg@8 / Pass@8 in percent; macro scores weight the three benchmarks equally\. The weight sweeps follow in Tables[23](https://arxiv.org/html/2610.11519#A4.T23)and[24](https://arxiv.org/html/2610.11519#A4.T24)\.

Table 20:Per\-seed benchmark results for seven controls\.Qwen3\-1\.7B\-Base with REINFORCE\+\+; entries are Avg@8 / Pass@8 in %\. Weights: no student value baselineγx=10\\gamma\_\{x\}=10, no response centeringλ=10\\lambda=10, raw residualγraw=10\\gamma\_\{\\rm raw\}=10, clipped \(c=5c=5\) and centered log ratioγlog=0\.6\\gamma\_\{\\log\}=0\.6, raw log ratioβ=0\.2\\beta=0\.2, and directL2L\_\{2\}α=10\\alpha=10\.
### D\.3Teacher\-peak branch results

Tables[21](https://arxiv.org/html/2610.11519#A4.T21)and[22](https://arxiv.org/html/2610.11519#A4.T22)list benchmark\-level means and per\-seed macro scores for the branch runs of Section[4\.4](https://arxiv.org/html/2610.11519#S4.SS4); the base runs are in Tables[27](https://arxiv.org/html/2610.11519#A4.T27),[18](https://arxiv.org/html/2610.11519#A4.T18), and[23](https://arxiv.org/html/2610.11519#A4.T23)\. At every weight, each seed ofRAwith the branch falls below the same seed ofRA\. With REINFORCE\+\+, each seed with the branch falls below the same verifier\-only seed at the weights55and1010, and two of the three seeds do at2020\.

Table 21:Benchmark\-level results for the teacher\-peak branch\.Qwen3\-1\.7B\-Base; three\-seed means in %, with sample SD for the macro scores\.Table 22:Per\-seed macro scores for the teacher\-peak branch\.Entries are macro Avg@8 / Pass@8 in %\.
### D\.4Weight sweeps

Tables[23](https://arxiv.org/html/2610.11519#A4.T23)and[24](https://arxiv.org/html/2610.11519#A4.T24)give three runs at each tested weight for Qwen3\-1\.7B\-Base with REINFORCE\+\+\. Theλ=10\\lambda=10row repeats the reference runs of Table[1](https://arxiv.org/html/2610.11519#S3.T1), and the other rows come from the control runs of Tables[2](https://arxiv.org/html/2610.11519#S4.T2)and[3](https://arxiv.org/html/2610.11519#S4.T3)\. Columns S1–S3 give each run’s macro Avg@8 / Pass@8 in percent, and the last two columns give three\-seed means with sample SD\. The guidance weightλ\\lambdamultiplies the complete guidance term in Equation[5](https://arxiv.org/html/2610.11519#S2.E5); the control weights are defined in Appendix[B\.7](https://arxiv.org/html/2610.11519#A2.SS7)\. Table[2](https://arxiv.org/html/2610.11519#S4.T2)repeats the three\-seed means of Table[23](https://arxiv.org/html/2610.11519#A4.T23), and the verifier\-only reference is in Table[18](https://arxiv.org/html/2610.11519#A4.T18)\. Table[25](https://arxiv.org/html/2610.11519#A4.T25)breaks every residual run down by benchmark and seed\.

Table 23:Weight sweeps with the probability residual\.Qwen3\-1\.7B\-Base with REINFORCE\+\+; S1–S3 entries are macro Avg@8 / Pass@8 in %\. The shaded row is the reference configuration of Table[1](https://arxiv.org/html/2610.11519#S3.T1), and†marks the direct\-L2L\_\{2\}weight selected on the validation split and reported in Table[3](https://arxiv.org/html/2610.11519#S4.T3)\. The weight isλ\\lambdafor fullRAand the uncentered control,γx\\gamma\_\{x\}andγraw\\gamma\_\{\\rm raw\}for the two controls without the student value baseline, andα\\alphafor directL2L\_\{2\}\.Table 24:Weight sweeps with the log ratio\.Qwen3\-1\.7B\-Base with REINFORCE\+\+; S1–S3 entries are macro Avg@8 / Pass@8 in %\. For each control,†marks the weight selected on the validation split \(Appendix[B\.5](https://arxiv.org/html/2610.11519#A2.SS5)\) and reported in Table[3](https://arxiv.org/html/2610.11519#S4.T3)\.Table 25:Per\-seed benchmark results for the residual weight sweeps\.Qwen3\-1\.7B\-Base with REINFORCE\+\+; entries are Avg@8 / Pass@8 in %\. The shaded rows are the reference runs of Table[1](https://arxiv.org/html/2610.11519#S3.T1)\. The weight isλ\\lambdafor fullRAand the uncentered control,γx\\gamma\_\{x\}for the control without the student value baseline, andγraw\\gamma\_\{\\rm raw\}for the raw residual\. Macro values average the three benchmark percentages within each run\.
### D\.5Distillation combined with RL: sequential and joint

[Li et al\. \(2026a\)](https://arxiv.org/html/2610.11519#bib.bib16)find that OPD followed by RL attains higher accuracy than the joint combinations they test\. We run this schedule on Qwen3\-1\.7B\-Base\. REINFORCE\+\+ continues for one epoch from the one\-epoch checkpoint of full\-vocabulary OPD, so the run processes the same 34,560 student responses asRA\+REINFORCE\+\+, with every other setting as in Table[9](https://arxiv.org/html/2610.11519#A2.T9)\. Table[26](https://arxiv.org/html/2610.11519#A4.T26)compares it with its two stages and withRA\+REINFORCE\+\+\. The sequential run improves on both stages, reaching28\.3928\.39macro Avg@8 and43\.6143\.61Pass@8, and stays0\.860\.86and1\.591\.59points belowRA\+REINFORCE\+\+, which spends the same budget in one stage\. It is ahead ofRAon AIME25 Avg@8 and behind on the other five benchmark scores, and everyRAseed exceeds every sequential seed in macro Avg@8, whereas the Pass@8 seeds overlap\.

The joint schedule adds the full\-vocabulary reverse KL of OPD, weighted byβ\\beta, to the REINFORCE\+\+ loss at every visited prefix, with every other setting as in Table[9](https://arxiv.org/html/2610.11519#A2.T9)\. Atβ=0\.05\\beta=0\.05,0\.20\.2, and0\.60\.6it reaches25\.9425\.94,26\.1426\.14, and25\.8225\.82macro Avg@8 and39\.7139\.71,39\.9439\.94, and39\.5339\.53Pass@8\. Table[26](https://arxiv.org/html/2610.11519#A4.T26)reportsβ=0\.2\\beta=0\.2, the weight selected on the validation split \(Appendix[B\.5](https://arxiv.org/html/2610.11519#A2.SS5)\):0\.470\.47and0\.500\.50points above verifier\-only training, with every seed above the same verifier\-only seed in macro Avg@8, but3\.113\.11and5\.265\.26points belowRA\+REINFORCE\+\+ and below the sequential schedule, in line with[Li et al\. \(2026a\)](https://arxiv.org/html/2610.11519#bib.bib16)\.

Table 26:Sequential and joint combinations of OPD and REINFORCE\+\+ againstRA\+REINFORCE\+\+ at a matched response budget\.Qwen3\-1\.7B\-Base; three\-seed means in %, with sample SD for the macro scores and the per\-seed scores of the two combinations below; the reverse\-KL weight isβ=0\.2\\beta=0\.2\. The other rows repeat Table[1](https://arxiv.org/html/2610.11519#S3.T1); bold marks the best mean per column\.
### D\.6Full\-vocabulary and sampled OPD

The teacher\-only OPD runs of Table[1](https://arxiv.org/html/2610.11519#S3.T1)minimize the reverse KL divergence over the full vocabulary at every visited prefix \(Appendix[B\.6](https://arxiv.org/html/2610.11519#A2.SS6)\), which uses the same teacher information asRA\.[Lu & Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.11519#bib.bib22)instead use the log ratio at the sampled token as a policy\-gradient label\. Table[27](https://arxiv.org/html/2610.11519#A4.T27)runs this sampled variant on Qwen3\-1\.7B\-Base with every other setting unchanged\. In expectation over the sampled token the two give the same update at each prefix \(Appendix[A\.1](https://arxiv.org/html/2610.11519#A1.SS1)\), so they differ mainly in the variance of the estimate\. The sampled variant reaches24\.8024\.80macro Avg@8 and38\.6038\.60Pass@8,2\.652\.65and2\.702\.70points below full\-vocabulary OPD and0\.870\.87and0\.840\.84points below verifier\-only REINFORCE\+\+, and it is lower than full\-vocabulary OPD on every benchmark score\. A single sampled update can point against the expected one \(Table[5](https://arxiv.org/html/2610.11519#A1.T5)\), and in this setting the variance of the sampled estimate costs2\.652\.65macro Avg@8 points\.

Table 27:Full\-vocabulary and sampled OPD\.Qwen3\-1\.7B\-Base; three\-seed means in %, with sample SD for the macro scores and the per\-seed scores of the sampled run below\. The other rows repeat Table[1](https://arxiv.org/html/2610.11519#S3.T1)\.
### D\.7Pass@K up to 512

Table[28](https://arxiv.org/html/2610.11519#A4.T28)and Figure[6](https://arxiv.org/html/2610.11519#A4.F6)give the macro Pass@K of the 1\.7B REINFORCE\+\+ runs forK∈\{8,16,…,512\}K\\in\\\{8,16,\\dots,512\\\}completions per problem, sampled under the evaluation settings of Table[1](https://arxiv.org/html/2610.11519#S3.T1); theK=8K=8column reproduces Table[1](https://arxiv.org/html/2610.11519#S3.T1)\. Pass@K is the share of problems solved at least once among theKKcompletions, averaged over the three benchmarks and then over three seeds\. Every run gains withKKand the increments shrink, to1\.11\.1,1\.31\.3, and1\.31\.3points at the last doubling for the untrained student, REINFORCE\+\+, and OPD, against2\.92\.9forRA; the margins over the untrained student atK=8K=8andK=512K=512are7\.67\.6and6\.66\.6points for REINFORCE\+\+,9\.59\.5and10\.210\.2for OPD, and13\.413\.4and16\.916\.9forRA\.

Table 28:Macro Pass@K up toK=512K=512, Qwen3\-1\.7B\-Base with REINFORCE\+\+\.Three\-seed means in %, with the increment from the previousKKin gray\.Figure 6:The margin over the untrained student does not close up to Pass@512\.Macro Pass@K of Table[28](https://arxiv.org/html/2610.11519#A4.T28)and each trained run’s margin over the untrained student, in points\.
### D\.8Untrained students and the teacher

Table[29](https://arxiv.org/html/2610.11519#A4.T29)evaluates the two untrained students and the teacher under the evaluation protocol of Appendix[B](https://arxiv.org/html/2610.11519#A2): the same prompt template, eight completions per problem at temperature0\.80\.8, and the 12,000\-token cap, with the teacher in non\-thinking mode\. Here±\\pmis the sample SD across three evaluation runs with different sampling seeds, not across training seeds\. Verifier\-only training lifts macro Avg@8 by 12 to 15 points over the untrained students;RAadds 1\.7 to 3\.6 points and Co\-RA2\.8 to 5\.0 over the corresponding estimator\. The most accurate 4B run, Co\-RA\+GRPO, ends 5\.4 macro Avg@8 points below the teacher, 1\.6 on MATH\-500 and about 7 on each AIME set, and teacher\-only OPD reaches 62% and 79% of the teacher’s macro Avg@8 at the two sizes\. Every trained run exceeds the untrained student’s Pass@8 on every benchmark, and Appendix[D\.7](https://arxiv.org/html/2610.11519#A4.SS7)follows the 1\.7B REINFORCE\+\+ runs to Pass@512\.

Table 29:Untrained students and the teacher under the evaluation protocol\.Qwen3\-1\.7B\-Base and Qwen3\-4B\-Base before any training, and the Qwen3\-8B teacher in non\-thinking mode\. Scores in %;±\\pmis the sample SD across three evaluation runs\.
### D\.9The adapted teacher under the evaluation protocol

Table[30](https://arxiv.org/html/2610.11519#A4.T30)evaluates the teacher LoRA at the end of each Co\-RArun, applied to the frozen Qwen3\-8B teacher in non\-thinking mode, under the protocol of Table[29](https://arxiv.org/html/2610.11519#A4.T29); each adapted row averages the three adapters of the run’s three training seeds, evaluated once each, and±\\pmis the sample SD across them\. Every adapted teacher scores below the frozen teacher, by0\.80\.8–2\.32\.3macro Avg@8 and0\.70\.7–3\.53\.5Pass@8 points, and the drop is larger for the adapters trained with the 1\.7B student than for those trained with the 4B student\. The adapted teacher is a better guide for the student that trained it \(Table[1](https://arxiv.org/html/2610.11519#S3.T1)\) and a worse solver on its own, in line with the distinction between a strong solver and a good teacher drawn in Section 1\. Table[31](https://arxiv.org/html/2610.11519#A4.T31)lists the macro scores of each adapter\.

Table 30:The adapted teacher is a worse solver than the frozen teacher\.Qwen3\-8B in non\-thinking mode with the Co\-RAadapter of each setting, under the evaluation protocol; scores in %\. The frozen row repeats Table[29](https://arxiv.org/html/2610.11519#A4.T29), where±\\pmis the SD across three evaluation runs; in the adapted rows±\\pmis the sample SD across the three seeds’ adapters\.Table 31:Per\-seed macro results of the adapted teachers\.Macro Avg@8 / Pass@8 in % of the adapter from each Co\-RAtraining seed\.