SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation

# Sparse Probing and Outcome Calibration for On-Policy Distillation
Source: [https://arxiv.org/html/2608.04419](https://arxiv.org/html/2608.04419)
Zikun Qu1, Min Zhang2, Mingze Kong1, Zhiwei Shang1, Yikun Ban3, Shuang Qiu4,Zhongxiang Dai1∗ 1The Chinese University of Hong Kong, Shenzhen,2East China Normal University, 3Beihang University,4City University of Hong Kong

###### Abstract

On\-policy distillation \(OPD\) provides dense teacher supervision on student\-generated trajectories, but standard reverse\-KL training can assign insufficient probability to other plausible continuations\. Teacher entropy alone does not reveal whether uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail, nor whether the student already represents those candidates well\. Moreover, local teacher probabilities may not predict downstream success\. We introduce*SparseProbing andOutcome\-calibratedTargets*OPD \(SPOT\), which addresses two coupled decisions,*where to probe*and*what to distill*, through an acquisition–exploration–exploitation procedure\. During acquisition, a position\-level score combines normalized teacher entropy, the probability mass captured by a small top\-kkcandidate set, and student–teacher mismatch to allocate a limited probing budget\. During exploration,SPOTevaluates teacher\-proposed candidates through verifier\-scored student continuations\. During exploitation, these outcomes produce a closed\-form, KL\-regularized target that favors candidates with better downstream outcomes while remaining anchored to the teacher distribution\. Extensive experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness ofSPOTin improving reasoning performance while balancing solution quality and coverage\.

## 1Introduction

Transferring the reasoning capabilities of large language models to smaller student models is an important goal in LLM post\-training\. Supervised fine\-tuning and off\-policy distillation leverage expert\- or teacher\-generated trajectories, but train the student on contexts that differ from those induced by its own predictions, leading to exposure bias and compounding errors at inference\(Bengioet al\.,[2015](https://arxiv.org/html/2608.04419#bib.bib22); Ranzatoet al\.,[2016](https://arxiv.org/html/2608.04419#bib.bib23)\)\. On\-policy reinforcement learning \(RL\) instead optimizes student\-generated rollouts, but reasoning\-oriented RL typically relies on sequence\-level or terminal verifier rewards, providing limited fine\-grained credit assignment\(Shaoet al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib6); Guoet al\.,[2025](https://arxiv.org/html/2608.04419#bib.bib21)\)\. On\-policy distillation \(OPD\) combines these advantages: it trains on student\-generated prefixes while retaining dense token\-level teacher feedback\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib5); Lu and Lab,[2025](https://arxiv.org/html/2608.04419#bib.bib25)\)\.

Yet standard OPD typically minimizes reverse KL, whose mode\-seeking behavior favors the teacher’s dominant continuation but can assign insufficient probability to other plausible continuations, potentially limiting solution coverage\. EOPD addresses this limitation by using teacher entropy as a trigger: at high\-entropy positions, it augments reverse\-KL training with a top\-kkapproximation of forward KL to preserve plausible local alternatives\(Minka,[2005](https://arxiv.org/html/2608.04419#bib.bib39); Jinet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib1)\)\. However, high entropy alone does not reveal whether the teacher’s uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail\. Nor does it reveal whether the student already assigns sufficient probability to and similarly ranks those plausible tokens\. Evaluating an alternative next token requires rolling out a student continuation and checking its final outcome, making such probing costly\. An entropy threshold can flag uncertain positions, but by itself cannot prioritize where a limited probing budget will be most useful\. The first challenge is therefore to allocate that budget to positions where the teacher assigns substantial probability to a small set of alternatives that the student does not yet represent well\.

Selecting a candidate position does not yet determine*what to distill*\. On student\-generated prefixes, the teacher’s local next\-token probabilities need not predict downstream success: a token assigned higher probability by the teacher may yield an unsuccessful continuation under the current student policy, whereas a lower\-probability alternative may yield a successful one\(Liet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib4); Houet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib28)\)\. We therefore treat the teacher distribution as a proposal prior over candidate branches, rather than as definitive evidence of downstream success, and calibrate it using the outcomes of student continuations\. Thus, teacher uncertainty and student mismatch help prioritize where to acquire additional evidence, while verified downstream outcomes determine how that evidence should modify the supervision target\.

These observations motivateSparse Probing and Outcome\-calibrated Targets for on\-policy distillation\(SPOT\), which reframes uncertainty\-aware distillation around two coupled decisions:*where to probe*and*what to distill*\.SPOTaddresses them through a three\-stage acquisition–exploration–exploitation procedure\. During acquisition, a lightweight position\-level scorests\_\{t\}prioritizes positions that satisfy three conditions: the teacher assigns meaningful probability to multiple next tokens, most of the teacher’s probability mass lies within a small top\-kkcandidate set, and the student either underweights or differently ranks those candidates\. Because these factors are multiplied, a low value on any one condition lowers the position’s overall probing priority\. The score therefore estimates*where additional evidence may be useful*; it does not identify which candidates yield successful student continuations, which is assessed only after the candidates are rolled out and verified\. During exploration, at each selected position,SPOTappends each candidate from the teacher’s top\-kkset in turn, rolls out a continuation under the student policy, and evaluates the completed continuation with a verifier\. During exploitation, the verified continuation values produce a closed\-form, KL\-regularized target: candidates with better downstream outcomes receive more probability, while the target remains anchored to the teacher distribution\.SPOTapplies this additional local loss only at positions where at least one tested candidate receives positive verifier reward\. This allows downstream evidence to modify the local token\-level target rather than serving only as a trajectory\-level score\.

In summary, our contributions are threefold:

- •A two\-decision formulation\.We formulate selective supervision in OPD as determining*where*additional outcome evidence is worth acquiring and*how*that evidence should be converted into a supervision target, separating position selection from target construction\.
- •Sparse probing and outcome\-calibrated targets\.We proposeSPOT, which probes positions via teacher uncertainty, top\-kkmass, and student mismatch, then derives a closed\-form target by reward\-tilting the teacher distribution with verifier\-scored student continuations\.
- •Empirical validation\.Across three evaluated Qwen student scales and six mathematical reasoning benchmarks,SPOTachieves the highest macro Pass@8 in all three settings and the highest or second\-highest macro Avg@8 among the compared methods\. These results indicate stronger multi\-sample solution coverage while maintaining competitive average accuracy\.

## 2Preliminaries

On\-Policy Distillation\.Let𝒟\\mathcal\{D\}denote the prompt distribution, and letπθ\\pi\_\{\\theta\}andπT\\pi\_\{T\}be the student and teacher policies over a shared vocabulary𝒱\\mathcal\{V\}\. Givenq∼𝒟q\\sim\\mathcal\{D\}, OPD samples a student trajectoryx=\(x1,…,xT\)∼πθ\(⋅∣q\)x=\(x\_\{1\},\\ldots,x\_\{T\}\)\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q\)and queries the teacher on each student\-induced prefixct=\(q,x<t\)c\_\{t\}=\(q,x\_\{<t\}\)\. At each prefix, it uses the reverse\-KL lossℒtOPD:=DKL\(πθ\(⋅∣ct\)∥πT\(⋅∣ct\)\)\\mathcal\{L\}^\{\\mathrm\{OPD\}\}\_\{t\}:=D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{t\}\)\\\|\\pi\_\{T\}\(\\cdot\\mid c\_\{t\}\)\)\. Averaging over the student trajectory gives

ℒOPD​\(θ\)=𝔼q∼𝒟,x∼πθ\(⋅∣q\)​\[1T​∑t=1TℒtOPD\]\.\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{q\\sim\\mathcal\{D\},\\,x\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q\)\}\\\!\\left\[\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathcal\{L\}^\{\\mathrm\{OPD\}\}\_\{t\}\\right\]\.\(1\)In practice, this objective is estimated from tokens sampled by a frozen behavior policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}and optimized with PPO\-style clipping\. Querying the teacher on student\-visited prefixes reduces state\-distribution mismatch and provides dense token\-level feedback\. However, the mode\-seeking reverse KL can under\-cover plausible alternatives when the teacher is uncertain\.

Entropy\-Aware On\-Policy Distillation\.EOPD\(Jinet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib1)\)augments OPD with mode\-covering supervision at positions of high teacher entropy\. The raw entropy and its vocabulary\-normalized counterpart are

HT\(ct\):=−∑v∈𝒱πT\(v∣ct\)logπT\(v∣ct\),H¯T\(ct\):=HT​\(ct\)log⁡\|𝒱\|∈\[0,1\]\.H\_\{T\}\(c\_\{t\}\):=\-\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{T\}\(v\\mid c\_\{t\}\)\\log\\pi\_\{T\}\(v\\mid c\_\{t\}\),\\qquad\\bar\{H\}\_\{T\}\(c\_\{t\}\):=\\frac\{H\_\{T\}\(c\_\{t\}\)\}\{\\log\|\\mathcal\{V\}\|\}\\in\[0,1\]\.\(2\)EOPD gates supervision usingHTH\_\{T\}, whereas our acquisition score later uses the bounded, vocabulary\-normalized entropyH¯T\\bar\{H\}\_\{T\}\. For efficiency, letStk:=TopKk\(πT\(⋅∣ct\)\)S\_\{t\}^\{k\}:=\\operatorname\{TopK\}\_\{k\}\(\\pi\_\{T\}\(\\cdot\\mid c\_\{t\}\)\)denote the teacher’s top\-kkcandidate set\. Forj∈\{T,θ\}j\\in\\\{T,\\theta\\\}, writeπ¯jk​\(v∣ct\):=πj​\(v∣ct\)/∑u∈Stkπj​\(u∣ct\)\\bar\{\\pi\}\_\{j\}^\{k\}\(v\\mid c\_\{t\}\):=\\pi\_\{j\}\(v\\mid c\_\{t\}\)/\\sum\_\{u\\in S\_\{t\}^\{k\}\}\\pi\_\{j\}\(u\\mid c\_\{t\}\)for the restriction ofπj\\pi\_\{j\}renormalized on this candidate set\. EOPD approximates forward KL as

ℒtFKL:=∑v∈Stkπ¯Tk​\(v∣ct\)​log⁡π¯Tk​\(v∣ct\)πθ​\(v∣ct\)\.\\mathcal\{L\}^\{\\mathrm\{FKL\}\}\_\{t\}:=\\sum\_\{v\\in S\_\{t\}^\{k\}\}\\bar\{\\pi\}\_\{T\}^\{k\}\(v\\mid c\_\{t\}\)\\log\\frac\{\\bar\{\\pi\}\_\{T\}^\{k\}\(v\\mid c\_\{t\}\)\}\{\\pi\_\{\\theta\}\(v\\mid c\_\{t\}\)\}\.\(3\)Only the teacher target is truncated and renormalized; the student retains its full\-vocabulary probabilities\. With entropy thresholdτ\\tauand forward\-KL weightα\\alpha, the per\-token objective is

ℒtEOPD:=ℒtOPD\+α​𝕀​\[HT​\(ct\)\>τ\]​ℒtFKL\.\\mathcal\{L\}^\{\\mathrm\{EOPD\}\}\_\{t\}:=\\mathcal\{L\}^\{\\mathrm\{OPD\}\}\_\{t\}\+\\alpha\\,\\mathbb\{I\}\\\!\\left\[H\_\{T\}\(c\_\{t\}\)\>\\tau\\right\]\\mathcal\{L\}^\{\\mathrm\{FKL\}\}\_\{t\}\.\(4\)Here,πj\\pi\_\{j\}always denotes a full\-vocabulary policy, whileπ¯jk\\bar\{\\pi\}\_\{j\}^\{k\}denotes its top\-kk\-renormalized shape—a distinction that separates probability mass from relative shape in our method\. EOPD promotes coverage, but still uses a scalar entropy criterion to decide*where*to intervene and the uncalibrated teacher prior to determine*what*to distill;SPOTrevisits both decisions\.

## 3Methodology

### 3\.1Overview

![Refer to caption](https://arxiv.org/html/2608.04419v1/x1.png)Figure 1:An overview of ourSPOTframework for on\-policy distillation\.SPOTfollows a three\-stage acquisition–exploration–exploitation procedure that addresses two coupled decisions:*where to probe*and*what to distill*\. As illustrated in[Figure1](https://arxiv.org/html/2608.04419#S3.F1), acquisition uses a lightweight position\-level score to prioritize where to acquire additional evidence; exploration estimates the downstream values of teacher\-proposed next\-token candidates through verifier\-scored student continuations; and exploitation combines these values with the teacher probabilities to construct outcome\-calibrated targets\. In this way, acquisition allocates the probing budget, while exploration and exploitation determine how the acquired evidence modifies local supervision\.

Algorithm 1SPOTTraining1:Student

πθ\\pi\_\{\\theta\}, teacher

πT\\pi\_\{T\}, verifier

RR, prompt set

𝒟\\mathcal\{D\}, learning rate

η\\eta
2:Position budget

MM, top\-

kksizes

ksk\_\{s\}\(scoring\) and

kpk\_\{p\}\(probing\), inverse temperature

γ\\gamma, branch\-loss weight

β\\beta
3:foreach training iterationdo

4:

πθold←πθ\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\\leftarrow\\pi\_\{\\theta\}
5:Rollout buffer

ℛ←∅\\mathcal\{R\}\\leftarrow\\emptyset
6:foreach prompt

qqin a batch sampled from

𝒟\\mathcal\{D\}do

7:Roll out

x∼πθold\(⋅∣q\)x\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid q\)
8:Query the teacher

πT\(⋅∣ct\)\\pi\_\{T\}\(\\cdot\\mid c\_\{t\}\)at every prefix

ct=\(q,x<t\)c\_\{t\}=\(q,x\_\{<t\}\)
9:Score every valid position by its position\-level acquisition score

st=H¯T​\(ct\)⋅Ctks⋅Gtkss\_\{t\}=\\bar\{H\}\_\{T\}\(c\_\{t\}\)\\cdot C\_\{t\}^\{k\_\{s\}\}\\cdot G\_\{t\}^\{k\_\{s\}\}
10:using Eqs\. \([5](https://arxiv.org/html/2608.04419#S3.E5)\)–\([7](https://arxiv.org/html/2608.04419#S3.E7)\)

11:Select sparse probing positions

ℬ=Top​\-⁡M​\(\{st\}t=1\|x\|\)\\mathcal\{B\}=\\operatorname\{Top\\text\{\-\}\}M\\bigl\(\\\{s\_\{t\}\\\}\_\{t=1\}^\{\|x\|\}\\bigr\)
12:foreach position

t∈ℬt\\in\\mathcal\{B\}and candidate

v∈Stkpv\\in S\_\{t\}^\{k\_\{p\}\}do

13:Sample

y∼πθold\(⋅∣ct,v\)y\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid c\_\{t\},v\)and estimate

V^t​\(v\)\\hat\{V\}\_\{t\}\(v\)using

R​\(q,x<t,v,y\)R\(q,x\_\{<t\},v,y\)
14:Retain positions with at least one positive\-reward candidate as

ℬ\+⊆ℬ\\mathcal\{B\}^\{\+\}\\subseteq\\mathcal\{B\}
15:Form reward\-tilted targets

π~T\(⋅∣ct\)\\tilde\{\\pi\}\_\{T\}\(\\cdot\\mid c\_\{t\}\)for

t∈ℬ\+t\\in\\mathcal\{B\}^\{\+\}using Eq\. \([10](https://arxiv.org/html/2608.04419#S3.E10)\)

16:Add the rollout and its supervision to

ℛ\\mathcal\{R\}
17:foreach gradient step on a mini\-batch sampled from

ℛ\\mathcal\{R\}do

18:Update

θ←θ−η​∇θℒ\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}\\mathcal\{L\}by minimizing

ℒ=1T​∑t=1TℒtOPD\+βmax⁡\{1,\|ℬ\+\|\}​∑t∈ℬ\+ℒtBranch\\mathcal\{L\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathcal\{L\}^\{\\mathrm\{OPD\}\}\_\{t\}\+\\frac\{\\beta\}\{\\max\\\{1,\|\\mathcal\{B\}^\{\+\}\|\\\}\}\\sum\_\{t\\in\\mathcal\{B\}^\{\+\}\}\\mathcal\{L\}^\{\\mathrm\{Branch\}\}\_\{t\}
19:using Eq\. \([12](https://arxiv.org/html/2608.04419#S3.E12)\)

### 3\.2SPOT: Sparse Probing and Outcome\-Calibrated Targets

Position Acquisition: Where to Probe\.Because probing next\-token candidates requires extra rollouts, acquisition prioritizes positions where several teacher candidates are plausible, a small candidate set captures most teacher mass, and the student does not already represent those candidates well\. Entropy alone cannot distinguish this pattern from uncertainty spread over a long tail\. Recent analyses also suggest that OPD gains depend on student–teacher compatibility and genuinely new teacher information\(Liet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib4)\)\. We therefore use student–teacher mismatch to signal potential correction needed, while verifier\-scored rollouts test the utility of the selected candidates\.

Starting from the normalized teacher entropyH¯T​\(ct\)\\bar\{H\}\_\{T\}\(c\_\{t\}\), letksk\_\{s\}denote the number of teacher candidates used to compute the position\-level acquisition score\. We first measure the teacher mass captured by its top\-ksk\_\{s\}candidate set:

Ctks=∑v∈StksπT​\(v∣ct\)\.C\_\{t\}^\{k\_\{s\}\}=\\sum\_\{v\\in S\_\{t\}^\{k\_\{s\}\}\}\\pi\_\{T\}\(v\\mid c\_\{t\}\)\.\(5\)The quantityCtksC\_\{t\}^\{k\_\{s\}\}is the teacher probability mass captured by its top\-ksk\_\{s\}candidates\. A large value means that a small candidate set represents most of the teacher distribution\. Combined with high entropy, this favors*positions with multiple plausible candidates*in that set over positions whose uncertainty is dispersed across a long probability tail\.

Teacher\-side structure, however, does not imply that the resulting supervision is useful to a student that may already represent the same alternatives\. We thus measure the student–teacher gap on the same candidate set:

Gtks=λmass​\(1−Atks\)\+λshape​DJS​\(π¯Tks∥π¯θoldks\),G\_\{t\}^\{k\_\{s\}\}=\\lambda\_\{\\mathrm\{mass\}\}\(1\-A\_\{t\}^\{k\_\{s\}\}\)\+\\lambda\_\{\\mathrm\{shape\}\}D\_\{\\mathrm\{JS\}\}\(\\bar\{\\pi\}\_\{T\}^\{k\_\{s\}\}\\,\\\|\\,\\bar\{\\pi\}\_\{\\theta\_\{\\mathrm\{old\}\}\}^\{k\_\{s\}\}\),\(6\)whereAtks=∑v∈Stksπθold​\(v∣ct\)A\_\{t\}^\{k\_\{s\}\}=\\sum\_\{v\\in S\_\{t\}^\{k\_\{s\}\}\}\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(v\\mid c\_\{t\}\)andλmass\+λshape=1\\lambda\_\{\\mathrm\{mass\}\}\+\\lambda\_\{\\mathrm\{shape\}\}=1\. HereDJSD\_\{\\mathrm\{JS\}\}denotes the Jensen–Shannon divergence normalized to\[0,1\]\[0,1\]; the two terms capture mass under\-coverage and relative\-shape mismatch, respectively\.

The final acquisition scorests\_\{t\}acts as a soft conjunction: a position receives a high score only when the teacher presents multiple plausible candidates, the top\-ksk\_\{s\}set captures most teacher probability mass, and the student either underweights or differently ranks those candidates:

st=H¯T​\(ct\)⋅Ctks⋅Gtks\.s\_\{t\}=\\bar\{H\}\_\{T\}\(c\_\{t\}\)\\cdot C\_\{t\}^\{k\_\{s\}\}\\cdot G\_\{t\}^\{k\_\{s\}\}\.\(7\)After masking special, padding, whitespace\-only, and punctuation\-only tokens, we allocate probing toℬ=Top​\-⁡M​\(\{st\}t=1\|x\|\)\\mathcal\{B\}=\\operatorname\{Top\\text\{\-\}\}M\(\\\{s\_\{t\}\\\}\_\{t=1\}^\{\|x\|\}\)\. Consequently, the rollout budget is reserved for compact teacher alternatives that the current student neither adequately covers nor matches\.

##### Sparse Probing and Outcome Calibration: What to Distill\.

Acquisition prioritizes where to probe, but a teacher’s local next\-token probability does not by itself predict downstream success\. Prior work suggests that it can become a weaker guide on deeper student\-generated prefixes\(Liet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib4)\); moreover, a teacher\-preferred token may lead to an incorrect solution, while a less\-preferred candidate may still enable a correct one\(Houet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib28)\)\. We therefore treat the teacher distribution as a proposal prior \(an initial preference over candidates\) and calibrate it using downstream outcomes\.

To obtain outcome\-grounded evidence, we performsparse probingat each selected positiont∈ℬt\\in\\mathcal\{B\}\. Letkpk\_\{p\}denote the number of teacher candidates probed at each selected position andNpN\_\{p\}the number of continuations sampled per candidate\. For eachv∈Stkpv\\in S\_\{t\}^\{k\_\{p\}\}, we appendvvto the prefixctc\_\{t\}and independently sampleNpN\_\{p\}continuationsy\(n\)∼πθold\(⋅∣ct,v\)y^\{\(n\)\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid c\_\{t\},v\),n=1,…,Npn=1,\\ldots,N\_\{p\}, from the frozen behavior policy\. These continuations probe the candidate\-conditioned branch induced byvv, whose student\-executable value is

V^t​\(v\)=𝔼y∼πθold\(⋅∣ct,v\)​\[R​\(q,x<t,v,y\)\],\\hat\{V\}\_\{t\}\(v\)=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid c\_\{t\},v\)\}\\\!\\left\[R\(q,x\_\{<t\},v,y\)\\right\],\(8\)which measures whether the current student can turn a teacher\-proposed local alternative into a high\-reward trajectory\. To avoid imposing additional supervision where exploration finds no viable alternative, we retain only positions for which at least one candidate has a positive estimated continuation value; the resulting set is denoted byℬ\+⊆ℬ\\mathcal\{B\}^\{\+\}\\subseteq\\mathcal\{B\}\. Because probing evaluates at mostM×kp×NpM\\times k\_\{p\}\\times N\_\{p\}candidate continuations per trajectory, its overhead is explicitly controlled by the position, candidate, and per\-candidate sampling budgets\.

To convert the probed outcomes into actionable supervision, we constructoutcome\-calibrated targetsat each retained positiont∈ℬ\+t\\in\\mathcal\{B\}^\{\+\}\. LetΔ​\(Stkp\)=\{ρt:ρt​\(v\)≥0,∑v∈Stkpρt​\(v\)=1\}\\Delta\(S\_\{t\}^\{k\_\{p\}\}\)=\\\{\\rho\_\{t\}:\\rho\_\{t\}\(v\)\\geq 0,\\,\\sum\_\{v\\in S\_\{t\}^\{k\_\{p\}\}\}\\rho\_\{t\}\(v\)=1\\\}denote the probability simplex over the teacher candidate set\. Rather than collapsing onto the empirically best candidate, we maximize expected downstream value within a KL trust region around the renormalized teacher prior:

maxρt∈Δ​\(Stkp\)∑v∈Stkpρt\(v\)V^t\(v\)s\.t\.DKL\(ρt∥π¯Tkp\(⋅∣ct\)\)≤ϵ\.\\max\_\{\\rho\_\{t\}\\in\\Delta\(S\_\{t\}^\{k\_\{p\}\}\)\}\\;\\sum\_\{v\\in S\_\{t\}^\{k\_\{p\}\}\}\\rho\_\{t\}\(v\)\\hat\{V\}\_\{t\}\(v\)\\quad\\text\{s\.t\.\}\\quad D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\rho\_\{t\}\\,\\\|\\,\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(\\cdot\\mid c\_\{t\}\)\\right\)\\leq\\epsilon\.\(9\)The derivation is deferred to Appendix[A](https://arxiv.org/html/2608.04419#A1)\. For the corresponding dual parameterγ\>0\\gamma\>0, which acts as an inverse temperature, the problem admits the closed\-form optimizer

π~T​\(v∣ct\)=π¯Tkp​\(v∣ct\)​exp⁡\(γ​V^t​\(v\)\)∑u∈Stkpπ¯Tkp​\(u∣ct\)​exp⁡\(γ​V^t​\(u\)\)\.\\tilde\{\\pi\}\_\{T\}\(v\\mid c\_\{t\}\)=\\frac\{\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(v\\mid c\_\{t\}\)\\exp\\\!\\left\(\\gamma\\hat\{V\}\_\{t\}\(v\)\\right\)\}\{\\sum\_\{u\\in S\_\{t\}^\{k\_\{p\}\}\}\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(u\\mid c\_\{t\}\)\\exp\\\!\\left\(\\gamma\\hat\{V\}\_\{t\}\(u\)\\right\)\}\.\(10\)
In log space, equation[10](https://arxiv.org/html/2608.04419#S3.E10)readslog⁡π~T​\(v∣ct\)=log⁡π¯Tkp​\(v∣ct\)\+γ​V^t​\(v\)−log⁡Zt\\log\\tilde\{\\pi\}\_\{T\}\(v\\mid c\_\{t\}\)=\\log\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(v\\mid c\_\{t\}\)\+\\gamma\\hat\{V\}\_\{t\}\(v\)\-\\log Z\_\{t\}, exposing each target log\-probability as*teacher log\-probability plus an outcome\-grounded continuation\-value bonus*, up to the normalizerZtZ\_\{t\}\. Subtracting this expression for anyu,v∈Stkpu,v\\in S\_\{t\}^\{k\_\{p\}\}yields the pairwise log\-odds decomposition

log⁡π~T​\(v∣ct\)π~T​\(u∣ct\)=log⁡π¯Tkp​\(v∣ct\)π¯Tkp​\(u∣ct\)\+γ​\(V^t​\(v\)−V^t​\(u\)\)\.\\log\\frac\{\\tilde\{\\pi\}\_\{T\}\(v\\mid c\_\{t\}\)\}\{\\tilde\{\\pi\}\_\{T\}\(u\\mid c\_\{t\}\)\}=\\log\\frac\{\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(v\\mid c\_\{t\}\)\}\{\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(u\\mid c\_\{t\}\)\}\+\\gamma\\\!\\left\(\\hat\{V\}\_\{t\}\(v\)\-\\hat\{V\}\_\{t\}\(u\)\\right\)\.\(11\)This identity makes the division of roles explicit: the teacher supplies the prior odds, while the verifier adds a correction proportional to relative downstream value\. Conditional ont∈ℬ\+t\\in\\mathcal\{B\}^\{\+\}, only value differences affect the target, making it invariant to uniform reward shifts\. Together with theℬ\+\\mathcal\{B\}^\{\+\}gate, outcome feedback first tests absolute viability and then corrects relative preference\. Asγ→0\\gamma\\to 0, the target recovers the teacher prior; largerγ\\gammaincreasingly favors candidates with higher estimated continuation values\.

Finally, we augment OPD with outcome\-calibrated local supervision at verifier\-supported positions:

ℒ=1T​∑t=1TℒtOPD\+βmax⁡\{1,\|ℬ\+\|\}​∑t∈ℬ\+ℒtBranch,\\mathcal\{L\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathcal\{L\}\_\{t\}^\{\\mathrm\{OPD\}\}\+\\frac\{\\beta\}\{\\max\\\{1,\|\\mathcal\{B\}^\{\+\}\|\\\}\}\\sum\_\{t\\in\\mathcal\{B\}^\{\+\}\}\\mathcal\{L\}\_\{t\}^\{\\mathrm\{Branch\}\},\(12\)whereℒtBranch=−∑v∈Stkpπ~T​\(v∣ct\)​log⁡πθ​\(v∣ct\)\\mathcal\{L\}\_\{t\}^\{\\mathrm\{Branch\}\}=\-\\sum\_\{v\\in S\_\{t\}^\{k\_\{p\}\}\}\\tilde\{\\pi\}\_\{T\}\(v\\mid c\_\{t\}\)\\log\\pi\_\{\\theta\}\(v\\mid c\_\{t\}\)\. The denominator averages this auxiliary loss over retained positions, andβ\\betacontrols its strength relative to trajectory\-wide OPD\. Whenℬ\+=∅\\mathcal\{B\}^\{\+\}=\\varnothing, the branch term vanishes and the objective reduces to standard OPD\.

## 4EXPERIMENTS

For a rigorous, controlled comparison ofSPOT, we follow the setup of recent OPD work\(Jinet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib1)\)\. Further details appear in Appendix[B](https://arxiv.org/html/2608.04419#A2)\.

### 4\.1Experimental Settings

Models and Training Data\.We use Qwen3\-8B, with thinking mode disabled, as the common teacher for all distillation methods, and Qwen3\-0\.6B\-Base, Qwen3\-1\.7B\-Base, and Qwen3\-4B\-Base as students\(Yanget al\.,[2025](https://arxiv.org/html/2608.04419#bib.bib7)\)\. The 0\.6B and 1\.7B students are trained on MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.04419#bib.bib8)\); the 4B student uses the more challenging DAPO dataset\([3](https://arxiv.org/html/2608.04419#bib.bib9)\)\.

Baselines\.We compareSPOTwith four baselines:

- •KD\(Hintonet al\.,[2015](https://arxiv.org/html/2608.04419#bib.bib2); Kim and Rush,[2016](https://arxiv.org/html/2608.04419#bib.bib3)\): off\-policy distillation using forward KL and cross\-entropy on teacher\-generated data\.
- •OPD\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib5); Liet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib4)\): on\-policy training on student rollouts with per\-token reverse\-KL supervision\.
- •GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib6)\): outcome\-based RL using group\-relative advantages from verifiable rewards, without teacher supervision\.
- •EOPD\(Jinet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib1)\): OPD augmented with a top\-kkforward\-KL term at positions selected by a fixed teacher\-entropy threshold\.

Evaluation\.We evaluate all models zero\-shot with the same prompt template and answer verifier on MATH\-500\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.04419#bib.bib8); Lightmanet al\.,[2023](https://arxiv.org/html/2608.04419#bib.bib10)\), AIME 2024\(Zhang and Math\-AI,[2024](https://arxiv.org/html/2608.04419#bib.bib13)\), AIME 2025\(Zhang and Math\-AI,[2025](https://arxiv.org/html/2608.04419#bib.bib14)\), AMC 2023\(Yanget al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib47)\), Minerva Math\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2608.04419#bib.bib11)\), and HMMT 2025\(balunović2026matharenaevaluatingllmsuncontaminated\)\. We sample eight responses per problem with temperature 1\.0, top\-ppsampling withp=0\.8p=0\.8, and a maximum response length of 8192 tokens\.Avg@8is the mean response accuracy, andPass@8is the fraction of problems solved at least once\. We report per\-benchmark scores and their unweighted macro\-average across the six benchmarks\.

### 4\.2Main Results

Consistent gains across student scales and training sets\.As shown in Table[1](https://arxiv.org/html/2608.04419#S4.T1),SPOTachieves the best macro Pass@8 at all three evaluated student scales and the best or second\-best macro Avg@8\. Relative to standard OPD,SPOTimproves macro Avg@8 by 0\.47–1\.48 points and macro Pass@8 by 4\.55–5\.28 points\. Relative to EOPD, the closest uncertainty\-aware OPD baseline,SPOTimproves macro Avg@8 by 0\.29–0\.68 points and macro Pass@8 by 2\.49–3\.19 points\. These gains persist from 0\.6B to 4B students and across MATH and DAPO\-Math\-14k training, demonstrating that the benefit is not tied to a particular capacity or training regime\.

Broader multi\-sample coverage without sacrificing average accuracy\.Avg@8 estimates per\-sample correctness, whereas Pass@8 measures whether eight samples reach at least one correct solution\. The markedly larger Pass@8 gains, together with preserved or improved Avg@8, reveal a favorable coverage–quality profile:SPOTplaces meaningful probability on viable alternatives without diluting an individual attempt\. This directly matchesSPOT’s goal of improving coverage without sacrificing average accuracy\. The comparison with EOPD is consistent with our motivating distinction: teacher entropy alone cannot determine whether uncertainty is concentrated in a compact candidate set, whether the student already represents those candidates, or which candidates lead to successful student continuations\.SPOTaddresses these gaps by using normalized teacher entropy, top\-ksk\_\{s\}probability mass, and student–teacher mismatch to allocate the probing budget, then using verifier\-scored continuation values to construct outcome\-calibrated local targets\.SPOTtherefore selectively expands coverage toward useful alternative reasoning modes rather than stylistic or erroneous variation\.

Table 1:Main results on six mathematical reasoning benchmarks\.Avg\.is the unweighted mean over the six benchmarks, computed before rounding\.Boldandunderlinedentries denote the best and second\-best results within each student setting, respectively\.MethodAvg@8Pass@8MATH500AMC23MinervaHMMTAIME24AIME25Avg\.MATH500AMC23MinervaHMMTAIME24AIME25Avg\.Student Model:Qwen3\-0\.6B\-Base\(Training Data: MATH\)KD47\.8023\.4314\.340\.212\.080\.8314\.7869\.6052\.5030\.881\.676\.676\.6728\.00GRPO53\.3328\.1316\.411\.044\.580\.8317\.3974\.4055\.0032\.723\.3310\.0010\.0030\.91OPD50\.1024\.6916\.040\.422\.501\.2515\.8373\.2057\.5031\.251\.6710\.006\.6730\.05EOPD50\.5027\.8115\.951\.464\.171\.2516\.8675\.0055\.0033\.465\.0013\.336\.6731\.41SPOT \(Ours\)50\.5028\.4415\.991\.885\.002\.0817\.3176\.4060\.0034\.566\.6713\.3316\.6734\.60Student Model:Qwen3\-1\.7B\-Base\(Training Data: MATH\)KD62\.8537\.8127\.071\.2510\.423\.3323\.7984\.2070\.0044\.123\.3320\.0016\.6739\.72GRPO67\.0038\.1328\.171\.889\.175\.4224\.9684\.0072\.5048\.166\.6720\.0016\.6741\.33OPD67\.0339\.0627\.621\.468\.336\.2524\.9684\.8070\.0047\.063\.3320\.0016\.6740\.31EOPD67\.6840\.3127\.901\.6710\.837\.5025\.9883\.6075\.0044\.496\.6726\.6720\.0042\.74SPOT \(Ours\)67\.4339\.6928\.452\.0812\.507\.5026\.2787\.8075\.0047\.436\.6733\.3323\.3345\.59Student Model:Qwen3\-4B\-Base\(Training Data: DAPO\-Math\-14k\)KD74\.7348\.1334\.653\.1312\.5012\.0830\.8792\.2080\.0051\.4713\.3326\.6723\.3347\.83GRPO79\.1853\.1339\.842\.7114\.1713\.3333\.7390\.0080\.0052\.2111\.6726\.6726\.6747\.87OPD80\.3855\.9437\.645\.8317\.9216\.2535\.6693\.0080\.0051\.1013\.3330\.0030\.0049\.57EOPD79\.7355\.6336\.087\.0817\.5016\.6735\.4593\.2082\.5051\.8416\.6733\.3333\.3351\.81SPOT \(Ours\)79\.8856\.2537\.737\.5017\.5017\.9236\.1393\.4085\.0054\.0423\.3333\.3336\.6754\.30

### 4\.3Ablation Study

#### 4\.3\.1Effect of the Branch\-Acquisition Score

Because verifier\-based candidate probing is costly,SPOTrestricts it to the top\-MMpositions\. Here,HHdenotes normalized teacher entropy,CCdenotes the teacher mass captured by its top\-ksk\_\{s\}candidates, andGGdenotes the student–teacher gap combining mass undercoverage and JS shape mismatch\. We compare five variants:Huses entropy alone;HCaddsCC;HGaddsGGbut omitsCC;HC\-MassusesHH,CC, and only the mass\-undercoverage term; andFulluses all components\. All runs share the teacher, data, schedule, probing budget, and evaluation protocol\.

Table 2:Branch\-acquisition score ablation\.Avg@8 / Pass@8 \(%\) for Qwen3\-0\.6B\-Base trained on MATH\.Macrodenotes the unweighted mean; best results are bold andFullis shaded\.Avg@8 / Pass@8Acquisition scoreMATH500AMC23AIME24AIME25MacroH:H¯T\\bar\{H\}\_\{T\}50\.50/76\.2025\.00 /60\.001\.67 / 3\.330\.42 / 3\.3319\.40 / 35\.72HC:H¯T​Ctks\\bar\{H\}\_\{T\}C\_\{t\}^\{k\_\{s\}\}50\.30 / 73\.2027\.50 / 57\.501\.67 / 3\.330\.00 / 0\.0019\.87 / 33\.51HG:H¯T​Gtks\\bar\{H\}\_\{T\}G\_\{t\}^\{k\_\{s\}\}50\.00 / 74\.2028\.13/ 55\.002\.08/10\.001\.25/10\.0020\.37 / 37\.30HC\-Mass:H¯T​Ctks​\(1−Atks\)\\bar\{H\}\_\{T\}C\_\{t\}^\{k\_\{s\}\}\(1\-A\_\{t\}^\{k\_\{s\}\}\)50\.75/ 75\.8027\.81 /62\.502\.08/ 6\.671\.25/ 6\.6720\.47/37\.91Full:H¯T​Ctks​Gtks\\bar\{H\}\_\{T\}C\_\{t\}^\{k\_\{s\}\}G\_\{t\}^\{k\_\{s\}\}50\.50/76\.4028\.44/60\.005\.00/13\.332\.08/16\.6721\.51/41\.60Table[2](https://arxiv.org/html/2608.04419#S4.T2)shows thatFullachieves the best macro Avg@8/Pass@8 \(21\.51/41\.60\), with its larger advantage in Pass@8 suggesting improved solution coverage\.HGunderperformsFullon both metrics across all benchmarks, supportingCtksC\_\{t\}^\{k\_\{s\}\}as a reliable filter against diffuse teacher uncertainty\.HC, which omits the student–teacher gap, also consistently underperformsFull, supporting the inclusion ofGtksG\_\{t\}^\{k\_\{s\}\}\.HC\-Massis the strongest partial variant but remains belowFull, suggesting that mass undercoverage and JS shape mismatch capture complementary student deficits\. Overall, the results support the multiplicative score as a soft conjunction of teacher ambiguity, top\-ksk\_\{s\}mass\.

#### 4\.3\.2Effect of Verifier\-Guided Calibration

Verifier\-guided calibration converts the teacher prior into an outcome\-aware branch target\. To isolate this component, we remove reward tilting and positive\-reward gating while retainingFullacquisition and the branch\-distillation objective\. This ablation therefore uses the uncalibrated teacher proposal prior at all probed positions\. All other training and evaluation settings match the Qwen3\-1\.7B\-Base main experiment\. Table[3](https://arxiv.org/html/2608.04419#S4.T3)shows that the full configuration improves macro Avg@8/Pass@8 by 3\.21/7\.38 points\. Pass@8 increases on all four benchmarks, with the largest gain on AIME24 \(\+13\.33\+13\.33\)\. On MATH500, Avg@8 changes by only\+0\.13\+0\.13while Pass@8 rises by\+2\.00\+2\.00, suggesting that verifier\-guided calibration primarily improves solution coverage in this setting\.

Table 3:Verifier\-guidance ablation\.Avg@8 / Pass@8 \(%\) for Qwen3\-1\.7B\-Base trained on MATH\.Macrodenotes the unweighted mean; best results are bold and the full model is shaded\.Avg@8 / Pass@8VariantMATH500AMC23AIME24AIME25Macrow/o verifier guidance67\.30 / 85\.8034\.06 / 67\.508\.33 / 20\.004\.58 / 16\.6728\.57 / 47\.49SPOT\(Full\)67\.43/87\.8039\.69/75\.0012\.50/33\.337\.50/23\.3331\.78/54\.87
#### 4\.3\.3Scaling with the Evaluation\-Time Sampling Budget

To test whetherSPOTimproves coverage beyond a fixed Pass@8 protocol, we varyk∈\{4,8,16,32,64\}k\\in\\\{4,8,16,32,64\\\}on AIME24, AIME25, and AMC23, holding decoding fixed\. Here,kkdenotes responses per problem\.

![Refer to caption](https://arxiv.org/html/2608.04419v1/x2.png)Figure 2:Pass@kkand its gain over OPD on AIME 2024, AIME 2025, and AMC 2023\. Each point is read from the corresponding summary of a separately generatedkk\-sample run\.Figure[2](https://arxiv.org/html/2608.04419#S4.F2)reveals a budget\-dependent separation\. The methods are close atk=4k=4, with EOPD leading on AIME25, whereasSPOTranks first or ties fromk=8k=8onward\. Atk=64k=64,SPOTretains 12\.50–16\.67\-point gains over OPD, showing that its coverage advantage extends beyond Pass@8 and remains visible as the sampling budget grows\.

#### 4\.3\.4Effect of Branch\-Distillation Weight

The coefficientβ\\betatrades off trajectory\-wide OPD against outcome\-calibrated branch supervision\.

![Refer to caption](https://arxiv.org/html/2608.04419v1/x3.png)Figure 3:Sensitivity to the branch\-distillation weight\.Pass@8 of Qwen3\-1\.7B\-Base forβ∈\{0\.05,0\.1,0\.5,1\.0\}\\beta\\in\\\{0\.05,0\.1,0\.5,1\.0\\\}\. The main configurationβ=0\.1\\beta=0\.1is marked as Ours\.Among the tested values in Figure[3](https://arxiv.org/html/2608.04419#S4.F3),β=0\.1\\beta=0\.1leads the three competition benchmarks and tiesβ=0\.5\\beta=0\.5on MATH500\. The competition benchmarks show greater observed sensitivity, with their mean Pass@8 lowest atβ=1\.0\\beta=1\.0\. This pattern is consistent with branch feedback serving as a calibrated correction to trajectory\-wide supervision: weaker weighting may underuse local outcome signals, whereas stronger weighting may overemphasize sparse branch targets\. We therefore useβ=0\.1\\beta=0\.1as a practical default\.

#### 4\.3\.5Out\-of\-Domain Generalization

Although trained exclusively on MATH,SPOTgeneralizes beyond the training distribution, with its clearest gains on deliberative reasoning while remaining competitive on broader knowledge and instruction following \(Appendix[C\.1](https://arxiv.org/html/2608.04419#A3.SS1), Table[9](https://arxiv.org/html/2608.04419#A3.T9)\)\. This pattern suggests that outcome\-calibrated branch supervision transfers reusable decision behavior rather than merely fitting math\-specific solution templates\. The non\-uniform gains further indicate that branch calibration complements, rather than replaces, token\-level uncertainty transfer and domain\-aligned supervision\.

#### 4\.3\.6Consistency Across Model Families

The Llama results \(Appendix[C\.2](https://arxiv.org/html/2608.04419#A3.SS2), Table[10](https://arxiv.org/html/2608.04419#A3.T10)\) preserve the main Qwen pattern:SPOTleads both macro metrics, with a clearer advantage in Pass@8 than Avg@8\. Together with the Qwen results across model scales and training sets, this supports the portability of outcome\-calibrated branch supervision beyond a single architecture and suggests that its primary benefit is consistently broader coverage of viable reasoning paths\.

## 5Related Work

Knowledge Distillation\.Knowledge distillation transfers capabilities from a high\-capacity teacher to a compact student by matching teacher predictive distributions or imitating teacher\-generated sequences\(Hintonet al\.,[2015](https://arxiv.org/html/2608.04419#bib.bib2); Kim and Rush,[2016](https://arxiv.org/html/2608.04419#bib.bib3)\)\. Recent work improves this paradigm through alternative divergence objectives, contrastive learning, and selective supervision based on sample difficulty or token importance\(Koet al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib18);[2025](https://arxiv.org/html/2608.04419#bib.bib19); Heet al\.,[2025](https://arxiv.org/html/2608.04419#bib.bib20); Guoet al\.,[2025](https://arxiv.org/html/2608.04419#bib.bib21)\)\. Despite their effectiveness, these methods train on contexts that are not sampled from the current student policy, creating exposure bias and compounding errors when the student conditions on its own predictions at inference time\(Bengioet al\.,[2015](https://arxiv.org/html/2608.04419#bib.bib22); Ranzatoet al\.,[2016](https://arxiv.org/html/2608.04419#bib.bib23)\)\.

On\-Policy Distillation for Reasoning Models\.On\-policy distillation trains students on self\-generated rollouts with dense token\-level teacher feedback, mitigating exposure bias and providing finer credit assignment than outcome\-only RL\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib5); Guet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib24); Lu and Lab,[2025](https://arxiv.org/html/2608.04419#bib.bib25)\)\. Recent work has broadened OPD along four directions: teacher access and transfer scope, through reward extrapolation, black\-box teachers, and multi\-teacher distillation\(Yanget al\.,[2026b](https://arxiv.org/html/2608.04419#bib.bib26); Yeet al\.,[2026a](https://arxiv.org/html/2608.04419#bib.bib27); Houet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib28)\); adaptive supervision, through uncertainty\- or disagreement\-aware token selection, teachability modeling, and trajectory\-aware guidance\(Jinet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib1); Xuet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib29); Wanget al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib30); Jiang and Ferraro,[2026](https://arxiv.org/html/2608.04419#bib.bib31)\); representation and training efficiency, through hidden\-state alignment and offline teacher scoring\(Yanget al\.,[2026a](https://arxiv.org/html/2608.04419#bib.bib32); Wuet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib33); Zihenget al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib37)\); and self\-distillation, where privileged or auxiliary context induces internal teachers for knowledge internalization\(Zhaoet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib35); Yeet al\.,[2026b](https://arxiv.org/html/2608.04419#bib.bib36)\)\. Together, these advances have established OPD as a general post\-training paradigm, with adoption in large\-scale systems such as Qwen3 and DeepSeek\-V4\(Yanget al\.,[2025](https://arxiv.org/html/2608.04419#bib.bib7); DeepSeek\-AIet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib34)\)\.

## 6Conclusion

Standard reverse\-KL OPD can under\-cover plausible alternatives, while teacher entropy alone cannot determine where limited probing is most useful or which candidates lead to successful student continuations\. We introducedSPOT, which uses normalized teacher entropy, top\-ksk\_\{s\}probability mass, and student–teacher mismatch to allocate a sparse probing budget, then uses verifier\-scored continuation values to construct KL\-regularized, outcome\-calibrated targets anchored to the teacher proposal prior\. Across the evaluated student scales, training sets, reasoning benchmarks, and model families,SPOTachieves the highest macro Pass@8 in every setting and the highest or second\-highest macro Avg@8, with controlled ablations supporting both selective acquisition and verifier guidance\. Overall, our results support separating where to probe from what to distill: uncertainty and student mismatch guide probing, while downstream outcomes calibrate supervision\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. Bachem \(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.External Links:2306\.13649,[Link](https://arxiv.org/abs/2306.13649)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p1.1),[2nd item](https://arxiv.org/html/2608.04419#S4.I1.i2.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- S\. Bengio, O\. Vinyals, N\. Jaitly, and N\. Shazeer \(2015\)Scheduled sampling for sequence prediction with recurrent neural networks\.External Links:1506\.03099,[Link](https://arxiv.org/abs/1506.03099)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- \[3\]\(2025\)DAPO: An Open\-Source LLM Reinforcement Learning System at Scale, author=Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and Yu Yue and Weinan Dai and Tiantian Fan and Gaohong Liu and Lingjun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Zhang and Wang Zhang and Hang Zhu and Jinhua Zhu and Jiaze Chen and Jiangjie Chen and Chengyi Wang and Hongli Yu and Yuxuan Song and Xiangpeng Wei and Hao Zhou and Jingjing Liu and Wei\-Ying Ma and Ya\-Qin Zhang and Lin Yan and Mu Qiao and Yonghui Wu and Mingxuan Wang\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p1.1)\.
- DeepSeek\-AI, A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling, C\. Lu, C\. Zhao, C\. Deng, C\. Hou, C\. Xu, C\. Shao, C\. Ruan, C\. Sun, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Li, D\. Ji, E\. Li, F\. Wei, F\. Lin, F\. Yuan, F\. Xia, F\. Dai, G\. Hao, G\. Chen, G\. Cao, G\. Meng, G\. Li, H\. Yu, H\. Zhang, H\. Xu, H\. Li, H\. Liang, H\. Zhang, H\. Luo, H\. Wei, H\. Yuan, H\. Zhang, H\. Luo, H\. Chen, H\. Ji, H\. Zhang, H\. Ding, H\. Tang, H\. Cao, H\. Gao, H\. Qu, H\. Zeng, J\. Yang, J\. Zhu, J\. Luo, J\. Song, J\. Yu, J\. Huang, J\. Cai, J\. Liang, J\. Zhou, J\. Ye, J\. Li, J\. Xu, J\. Hu, J\. Yang, J\. Chen, J\. Yan, J\. Chen, J\. Zhou, J\. Xiang, J\. Yuan, J\. Cheng, J\. Zhou, J\. Zhu, J\. Yu, J\. Sun, J\. Ran, J\. Jiang, J\. Qiu, J\. Li, J\. Zheng, J\. Song, K\. Dong, K\. Gao, K\. Guan, K\. Zhou, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Wang, L\. Xia, L\. Zhang, L\. Zhao, L\. Guo, L\. Luo, L\. Ma, L\. Zhu, L\. Wang, L\. Cai, L\. Zhang, L\. Chen, M\. Di, M\. Xu, M\. Mei, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, M\. Zhou, M\. Han, N\. Wang, P\. Huang, P\. Wang, P\. Cong, P\. Wang, P\. Zhang, Q\. Wang, Q\. Zhu, Q\. Li, Q\. Chen, Q\. Du, Q\. Jiang, R\. Tian, R\. Xu, R\. Lu, R\. Xu, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Chen, R\. Yin, R\. Xu, R\. Shen, R\. Zhang, R\. Chen, S\. Liu, S\. Lu, S\. Sun, S\. Zhou, S\. Chen, S\. Cai, S\. Nie, S\. Wu, S\. Chen, S\. Hu, S\. Liu, S\. Hu, S\. Ma, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. Yu, S\. Zhou, T\. Ni, T\. Yun, T\. Jin, T\. Pei, T\. Ye, T\. Lin, T\. Ji, T\. Cui, T\. Yue, T\. Yu, T\. Wang, W\. Zhang, W\. Xiao, W\. Zeng, W\. An, W\. Zhao, W\. Liu, W\. Liang, W\. Pang, W\. Luo, W\. Yao, W\. Gao, W\. Yang, W\. Huang, W\. Hou, W\. Zhang, W\. Ma, X\. Gao, X\. He, X\. Wang, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Chen, X\. Zhang, X\. Nie, X\. Sun, X\. Wang, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Liu, X\. Yu, X\. Li, X\. Yang, X\. Zhang, X\. Chen, X\. Wang, X\. Su, X\. Chen, X\. Lin, X\. Fu, Y\. Yan, Y\. Wang, Y\. Ma, Y\. Luo, Y\. Zhang, Y\. Xu, Y\. Ma, Y\. Huang, Y\. Li, Y\. Li, Y\. Xu, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Qian, Y\. Shao, Y\. Yu, Y\. Zhang, Y\. Ding, Y\. Shi, Y\. Wu, Y\. Xiong, Y\. Ma, Y\. He, Y\. Tang, Y\. Zhou, Y\. Luo, Y\. Zhong, Y\. Piao, Y\. Wang, Y\. Zhang, Y\. Chen, Y\. Tan, Y\. Wei, Y\. Ma, Y\. Liu, Y\. Yang, Y\. Guo, Y\. Wu, Y\. Wu, Y\. Li, Y\. Cheng, Y\. Ou, Y\. Xu, Y\. Li, Y\. Wang, Y\. Yang, Y\. Xu, Y\. Wu, Y\. Meng, Y\. Zou, Y\. Zha, Y\. Xiong, Y\. Chen, Y\. Lin, Y\. Cao, Y\. Wang, Y\. Zhang, Y\. Yan, Y\. Lin, Y\. Gu, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. Zhou, Y\. Huang, Z\. Wu, Z\. Wang, Z\. Zhao, Z\. Ren, Z\. Zhang, Z\. Sha, Z\. Fu, Z\. Ju, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Gao, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Huang, Z\. Chen, Z\. Wu, Z\. Ren, Z\. Wu, Z\. Li, Z\. Zhang, Z\. Xu, Z\. Wang, Z\. Qu, Z\. Gu, Z\. Zhu, Z\. Li, Z\. Zhang, Z\. Xie, Z\. Gao, Z\. Wan, Z\. Pan, and Z\. Yao \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- C\. Goddard and L\. Atkins \(2024\)DistillKit: flexible knowledge distillation for large language modelsExternal Links:[Link](https://github.com/arcee-ai/distillkit)Cited by:[Appendix B](https://arxiv.org/html/2608.04419#A2.SS0.SSS0.Px1.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2026\)MiniLLM: on\-policy distillation of large language models\.External Links:2306\.08543,[Link](https://arxiv.org/abs/2306.08543)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- Y\. Guo, W\. Yang, Z\. Sun, N\. Ding, Z\. Liu, and Y\. Lin \(2025\)Learning to focus: causal attention distillation via gradient\-guided token pruning\.External Links:2506\.07851,[Link](https://arxiv.org/abs/2506.07851)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- C\. He, Y\. Ding, J\. Guo, R\. Gong, H\. Qin, and X\. Liu \(2025\)DA\-KD: Difficulty\-aware knowledge distillation for efficient large language models\.InForty\-second International Conference on Machine Learning,Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.External Links:2103\.03874,[Link](https://arxiv.org/abs/2103.03874)Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p2.2)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.External Links:1503\.02531,[Link](https://arxiv.org/abs/1503.02531)Cited by:[1st item](https://arxiv.org/html/2608.04419#S4.I1.i1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- W\. Hou, S\. Peng, W\. Wang, Z\. Ruan, Y\. Zhang, Z\. Zhou, M\. Gao, Y\. Chen, K\. Wang, H\. Yang, C\. Zhang, Z\. Tian, H\. Hu, Y\. Yang, F\. Wu, and H\. Fan \(2026\)Uni\-opd: unifying on\-policy distillation with a dual\-perspective recipe\.External Links:2605\.03677,[Link](https://arxiv.org/abs/2605.03677)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p3.1),[§3\.2](https://arxiv.org/html/2608.04419#S3.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- Y\. Jiang and F\. Ferraro \(2026\)Bridging reasoning trajectories in on\-policy distillation via near\-future guidance\.External Links:2606\.00305,[Link](https://arxiv.org/abs/2606.00305)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. Lee \(2026\)Entropy\-aware on\-policy distillation of language models\.External Links:2603\.07079,[Link](https://arxiv.org/abs/2603.07079)Cited by:[Appendix B](https://arxiv.org/html/2608.04419#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.04419#S1.p2.1),[§2](https://arxiv.org/html/2608.04419#S2.p2.13),[4th item](https://arxiv.org/html/2608.04419#S4.I1.i4.p1.1),[§4](https://arxiv.org/html/2608.04419#S4.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- Y\. Kim and A\. M\. Rush \(2016\)Sequence\-level knowledge distillation\.External Links:1606\.07947,[Link](https://arxiv.org/abs/1606.07947)Cited by:[1st item](https://arxiv.org/html/2608.04419#S4.I1.i1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- J\. Ko, T\. Chen, S\. Kim, T\. Ding, L\. Liang, I\. Zharkov, and S\. Yun \(2025\)DistiLLM\-2: a contrastive approach boosts the distillation of llms\.External Links:2503\.07067,[Link](https://arxiv.org/abs/2503.07067)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: towards streamlined distillation for large language models\.External Links:2402\.03898,[Link](https://arxiv.org/abs/2402.03898)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- \[17\]Math\-Verify: Math Verification LibraryExternal Links:[Link](https://github.com/huggingface/math-verify)Cited by:[Appendix B](https://arxiv.org/html/2608.04419#A2.SS0.SSS0.Px6.p1.1)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. Misra \(2022\)Solving quantitative reasoning problems with language models\.External Links:2206\.14858,[Link](https://arxiv.org/abs/2206.14858)Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p2.2)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.External Links:2604\.13016,[Link](https://arxiv.org/abs/2604.13016)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p3.1),[§3\.2](https://arxiv.org/html/2608.04419#S3.SS2.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2608.04419#S3.SS2.p1.1),[2nd item](https://arxiv.org/html/2608.04419#S4.I1.i2.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p2.2)\.
- K\. Lu and T\. M\. Lab \(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/on\-policy\-distillationExternal Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- T\. P\. Minka \(2005\)Divergence measures and message passing\.External Links:[Link](https://api.semanticscholar.org/CorpusID:7585417)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p2.1)\.
- M\. Ranzato, S\. Chopra, M\. Auli, and W\. Zaremba \(2016\)Sequence level training with recurrent neural networks\.External Links:1511\.06732,[Link](https://arxiv.org/abs/1511.06732)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2608.04419#S1.p1.1),[3rd item](https://arxiv.org/html/2608.04419#S4.I1.i3.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)HybridFlow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,EuroSys ’25,pp\. 1279–1297\.External Links:[Link](http://dx.doi.org/10.1145/3689031.3696075),[Document](https://dx.doi.org/10.1145/3689031.3696075)Cited by:[Appendix B](https://arxiv.org/html/2608.04419#A2.SS0.SSS0.Px2.p1.1)\.
- Y\. Wang, S\. Lu, Y\. Gu, P\. Wang, Y\. Yang, Z\. Yan, C\. Xie, J\. Wu, and H\. Yang \(2026\)Not all disagreement is learnable: token teachability in on\-policy distillation\.External Links:2605\.26844,[Link](https://arxiv.org/abs/2605.26844)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- Y\. Wu, S\. Han, and H\. Cai \(2026\)Lightning opd: efficient post\-training for large reasoning models with offline on\-policy distillation\.External Links:2604\.13010,[Link](https://arxiv.org/abs/2604.13010)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. Geramifard \(2026\)TIP: token importance in on\-policy distillation\.External Links:2604\.14084,[Link](https://arxiv.org/abs/2604.14084)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p1.1),[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang,et al\.\(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p2.2)\.
- S\. Yang, G\. Zhu, B\. Song, H\. Wang, M\. Xia, X\. Zheng, Y\. Ma, Z\. Chen, W\. Wang, J\. Zhao, and G\. Chen \(2026a\)OPRD: on\-policy representation distillation\.External Links:2606\.06021,[Link](https://arxiv.org/abs/2606.06021)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. Lin \(2026b\)Learning beyond teacher: generalized on\-policy distillation with reward extrapolation\.External Links:2602\.12125,[Link](https://arxiv.org/abs/2602.12125)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- T\. Ye, L\. Dong, Z\. Chi, X\. Wu, S\. Huang, and F\. Wei \(2026a\)Black\-box on\-policy distillation of large language models\.External Links:2511\.10643,[Link](https://arxiv.org/abs/2511.10643)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei \(2026b\)On\-policy context distillation for language models\.External Links:2602\.12275,[Link](https://arxiv.org/abs/2602.12275)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- Y\. Zhang and T\. Math\-AI \(2024\)American invitational mathematics examination \(aime\) 2024\.Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p2.2)\.
- Y\. Zhang and T\. Math\-AI \(2025\)American invitational mathematics examination \(aime\) 2025\.Cited by:[§4\.1](https://arxiv.org/html/2608.04419#S4.SS1.p2.2)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.External Links:2601\.18734,[Link](https://arxiv.org/abs/2601.18734)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng \(2024\)SGLang: efficient execution of structured language model programs\.External Links:2312\.07104,[Link](https://arxiv.org/abs/2312.07104)Cited by:[Appendix B](https://arxiv.org/html/2608.04419#A2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ziheng, J\. Li, H\. Tang, Y\. N\. Wu, and D\. Terzopoulos \(2026\)Less is more: early stopping rollout for on\-policy distillation\.External Links:2605\.27028,[Link](https://arxiv.org/abs/2605.27028)Cited by:[§5](https://arxiv.org/html/2608.04419#S5.p2.1)\.

## Appendix ADerivation and Properties of Outcome\-Calibrated Targets

This appendix derives the outcome\-calibrated target and establishes several properties of its value–prior tradeoff\. Fix a retained positiont∈ℬ\+t\\in\\mathcal\{B\}^\{\+\}, and abbreviate

𝒮=Stkp,p​\(v\)=π¯Tkp​\(v∣ct\),V​\(v\)=V^t​\(v\)\.\\mathcal\{S\}=S\_\{t\}^\{k\_\{p\}\},\\qquad p\(v\)=\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(v\\mid c\_\{t\}\),\\qquad V\(v\)=\\hat\{V\}\_\{t\}\(v\)\.Here,VVis treated as the branch\-value estimate obtained after probing\. The candidate set𝒮\\mathcal\{S\}is finite, andp​\(v\)\>0p\(v\)\>0for everyv∈𝒮v\\in\\mathcal\{S\}\. We writeΔ​\(𝒮\)\\Delta\(\\mathcal\{S\}\)for the probability simplex over𝒮\\mathcal\{S\}\.

### A\.1Closed\-Form Derivation and Trust\-Region Equivalence

For any inverse\-temperature parameterγ\>0\\gamma\>0, consider the KL\-regularized problem

maxρ∈Δ​\(𝒮\)⁡\{𝔼v∼ρ​\[V​\(v\)\]−1γ​DKL​\(ρ∥p\)\}\.\\max\_\{\\rho\\in\\Delta\(\\mathcal\{S\}\)\}\\left\\\{\\mathbb\{E\}\_\{v\\sim\\rho\}\[V\(v\)\]\-\\frac\{1\}\{\\gamma\}D\_\{\\mathrm\{KL\}\}\(\\rho\\\|p\)\\right\\\}\.\(13\)The first term favors branches with high estimated downstream value, while the second anchors the target to the teacher prior\.

###### Proposition A\.1\(Closed\-form optimizer\)\.

The objective in Eq\. \([13](https://arxiv.org/html/2608.04419#A1.E13)\) has the unique optimizer

ργ​\(v\)=p​\(v\)​exp⁡\(γ​V​\(v\)\)∑u∈𝒮p​\(u\)​exp⁡\(γ​V​\(u\)\)\.\\rho\_\{\\gamma\}\(v\)=\\frac\{p\(v\)\\exp\\left\(\\gamma V\(v\)\\right\)\}\{\\sum\_\{u\\in\\mathcal\{S\}\}p\(u\)\\exp\\left\(\\gamma V\(u\)\\right\)\}\.\(14\)

###### Proof\.

LetZ​\(γ\)=∑u∈𝒮p​\(u\)​exp⁡\(γ​V​\(u\)\)Z\(\\gamma\)=\\sum\_\{u\\in\\mathcal\{S\}\}p\(u\)\\exp\(\\gamma V\(u\)\)\. For anyρ∈Δ​\(𝒮\)\\rho\\in\\Delta\(\\mathcal\{S\}\),

DKL​\(ρ∥ργ\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(\\rho\\\|\\rho\_\{\\gamma\}\)=∑v∈𝒮ρ​\(v\)​log⁡ρ​\(v\)​Z​\(γ\)p​\(v\)​exp⁡\(γ​V​\(v\)\)\\displaystyle=\\sum\_\{v\\in\\mathcal\{S\}\}\\rho\(v\)\\log\\frac\{\\rho\(v\)Z\(\\gamma\)\}\{p\(v\)\\exp\(\\gamma V\(v\)\)\}=DKL​\(ρ∥p\)−γ​𝔼ρ​\[V\]\+log⁡Z​\(γ\)\.\\displaystyle=D\_\{\\mathrm\{KL\}\}\(\\rho\\\|p\)\-\\gamma\\mathbb\{E\}\_\{\\rho\}\[V\]\+\\log Z\(\\gamma\)\.\(15\)Rearranging gives the Gibbs variational identity

𝔼ρ​\[V\]−1γ​DKL​\(ρ∥p\)=log⁡Z​\(γ\)γ−1γ​DKL​\(ρ∥ργ\)\.\\mathbb\{E\}\_\{\\rho\}\[V\]\-\\frac\{1\}\{\\gamma\}D\_\{\\mathrm\{KL\}\}\(\\rho\\\|p\)=\\frac\{\\log Z\(\\gamma\)\}\{\\gamma\}\-\\frac\{1\}\{\\gamma\}D\_\{\\mathrm\{KL\}\}\(\\rho\\\|\\rho\_\{\\gamma\}\)\.\(16\)The right\-hand side is maximized if and only ifDKL​\(ρ∥ργ\)=0D\_\{\\mathrm\{KL\}\}\(\\rho\\\|\\rho\_\{\\gamma\}\)=0, or equivalentlyρ=ργ\\rho=\\rho\_\{\\gamma\}\. ∎

The regularized and trust\-region views describe the same solution path\. Define the radius induced byγ\\gammaas

ϵ​\(γ\)=DKL​\(ργ∥p\)\.\\epsilon\(\\gamma\)=D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\)\.\(17\)
###### Proposition A\.2\(Trust\-region equivalence\)\.

For everyγ\>0\\gamma\>0,ργ\\rho\_\{\\gamma\}is the unique solution of

maxρ∈Δ​\(𝒮\)⁡𝔼ρ​\[V\]s\.t\.DKL​\(ρ∥p\)≤ϵ​\(γ\)\.\\max\_\{\\rho\\in\\Delta\(\\mathcal\{S\}\)\}\\mathbb\{E\}\_\{\\rho\}\[V\]\\quad\\textnormal\{s\.t\.\}\\quad D\_\{\\mathrm\{KL\}\}\(\\rho\\\|p\)\\leq\\epsilon\(\\gamma\)\.\(18\)Conversely, if the KL constraint is active and its KKT multiplier isη\>0\\eta\>0, the solution has the form in Eq\. \([14](https://arxiv.org/html/2608.04419#A1.E14)\) withγ=1/η\\gamma=1/\\eta\.

###### Proof\.

Suppose a feasibleρ\\rhoattained a larger expected value thanργ\\rho\_\{\\gamma\}\. SinceDKL​\(ρ∥p\)≤DKL​\(ργ∥p\)D\_\{\\mathrm\{KL\}\}\(\\rho\\\|p\)\\leq D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\), it would also attain a strictly larger objective in Eq\. \([13](https://arxiv.org/html/2608.04419#A1.E13)\), contradicting Proposition[A\.1](https://arxiv.org/html/2608.04419#A1.Thmproposition1)\. The same argument, together with the uniqueness of the regularized optimizer, excludes any distinct feasible optimizer with equal expected value\.

For the converse direction, introduce multipliersη≥0\\eta\\geq 0andλ∈ℝ\\lambda\\in\\mathbb\{R\}\. The Lagrangian for Eq\. \([18](https://arxiv.org/html/2608.04419#A1.E18)\) is

𝒥​\(ρ,η,λ\)=𝔼ρ​\[V\]−η​\(DKL​\(ρ∥p\)−ϵ\)\+λ​\(∑v∈𝒮ρ​\(v\)−1\)\.\\mathcal\{J\}\(\\rho,\\eta,\\lambda\)=\\mathbb\{E\}\_\{\\rho\}\[V\]\-\\eta\\left\(D\_\{\\mathrm\{KL\}\}\(\\rho\\\|p\)\-\\epsilon\\right\)\+\\lambda\\left\(\\sum\_\{v\\in\\mathcal\{S\}\}\\rho\(v\)\-1\\right\)\.When the constraint is active, stationarity with respect toρ​\(v\)\\rho\(v\)yields

V​\(v\)−η​\(log⁡ρ​\(v\)p​\(v\)\+1\)\+λ=0\.V\(v\)\-\\eta\\left\(\\log\\frac\{\\rho\(v\)\}\{p\(v\)\}\+1\\right\)\+\\lambda=0\.Normalizing the resulting probabilities givesρ​\(v\)∝p​\(v\)​exp⁡\(V​\(v\)/η\)\\rho\(v\)\\propto p\(v\)\\exp\(V\(v\)/\\eta\), which is Eq\. \([14](https://arxiv.org/html/2608.04419#A1.E14)\) withγ=1/η\\gamma=1/\\eta\. ∎

Thus, a sharedγ\\gammaparameterizes the regularized solution directly and induces a position\-dependent trust\-region radiusϵt\(γ\)=DKL\(ργ,t∥π¯Tkp\(⋅∣ct\)\)\\epsilon\_\{t\}\(\\gamma\)=D\_\{\\mathrm\{KL\}\}\\left\(\\rho\_\{\\gamma,t\}\\mathbin\{\\\|\}\\bar\{\\pi\}\_\{T\}^\{k\_\{p\}\}\(\\cdot\\mid c\_\{t\}\)\\right\)\. For a prescribed active radius, the corresponding inverse temperature is the reciprocal of the KL multiplier rather than the multiplier itself\.

### A\.2Geometry of the Calibration Path

The log\-partition function provides a compact characterization of howγ\\gammatrades off estimated branch value against deviation from the teacher\. Let

F​\(γ\)=log⁡Z​\(γ\)=log​∑v∈𝒮p​\(v\)​exp⁡\(γ​V​\(v\)\)\.F\(\\gamma\)=\\log Z\(\\gamma\)=\\log\\sum\_\{v\\in\\mathcal\{S\}\}p\(v\)\\exp\(\\gamma V\(v\)\)\.
###### Proposition A\.3\(Monotone value–prior tradeoff\)\.

Along the calibration path\{ργ\}γ≥0\\\{\\rho\_\{\\gamma\}\\\}\_\{\\gamma\\geq 0\}, withρ0=p\\rho\_\{0\}=pdefined by continuity,

dd​γ​𝔼ργ​\[V\]\\displaystyle\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\gamma\}\\mathbb\{E\}\_\{\\rho\_\{\\gamma\}\}\[V\]=Varργ⁡\(V\)≥0,\\displaystyle=\\operatorname\{Var\}\_\{\\rho\_\{\\gamma\}\}\(V\)\\geq 0,\(19\)dd​γ​DKL​\(ργ∥p\)\\displaystyle\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\gamma\}D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\)=γ​Varργ⁡\(V\)≥0\.\\displaystyle=\\gamma\\operatorname\{Var\}\_\{\\rho\_\{\\gamma\}\}\(V\)\\geq 0\.\(20\)IfVVis nonconstant on𝒮\\mathcal\{S\}, both quantities are strictly increasing forγ\>0\\gamma\>0\.

###### Proof\.

DifferentiatingFFgives

F′​\(γ\)=𝔼ργ​\[V\],F′′​\(γ\)=Varργ⁡\(V\)\.F^\{\\prime\}\(\\gamma\)=\\mathbb\{E\}\_\{\\rho\_\{\\gamma\}\}\[V\],\\qquad F^\{\\prime\\prime\}\(\\gamma\)=\\operatorname\{Var\}\_\{\\rho\_\{\\gamma\}\}\(V\)\.Moreover, Eq\. \([14](https://arxiv.org/html/2608.04419#A1.E14)\) implies

DKL​\(ργ∥p\)=γ​F′​\(γ\)−F​\(γ\)\.D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\)=\\gamma F^\{\\prime\}\(\\gamma\)\-F\(\\gamma\)\.Differentiating this identity proves Eqs\. \([19](https://arxiv.org/html/2608.04419#A1.E19)\)–\([20](https://arxiv.org/html/2608.04419#A1.E20)\)\. Becauseppand henceργ\\rho\_\{\\gamma\}have full support on𝒮\\mathcal\{S\}, the variance is positive wheneverVVis nonconstant\. ∎

This monotonicity makes the role ofγ\\gammaprecise: increasing it yields a target with higher estimated downstream value while moving it farther from the teacher prior\. It also yields a one\-to\-one correspondence betweenγ\\gammaand the induced radius over the nondegenerate portion of the path\. Specifically, let𝒜⋆=arg⁡maxv∈𝒮⁡V​\(v\)\\mathcal\{A\}^\{\\star\}=\\arg\\max\_\{v\\in\\mathcal\{S\}\}V\(v\)andP⋆=∑v∈𝒜⋆p​\(v\)P^\{\\star\}=\\sum\_\{v\\in\\mathcal\{A\}^\{\\star\}\}p\(v\)\. Then

limγ→0ργ=p,limγ→∞ργ​\(v\)=\{p​\(v\)/P⋆,v∈𝒜⋆,0,v∉𝒜⋆,\\lim\_\{\\gamma\\to 0\}\\rho\_\{\\gamma\}=p,\\qquad\\lim\_\{\\gamma\\to\\infty\}\\rho\_\{\\gamma\}\(v\)=\\begin\{cases\}p\(v\)/P^\{\\star\},&v\\in\\mathcal\{A\}^\{\\star\},\\\\ 0,&v\\notin\\mathcal\{A\}^\{\\star\},\\end\{cases\}\(21\)and the induced radius increases from0to−log⁡P⋆\-\\log P^\{\\star\}\. Notably, when several candidates share the maximum value, the large\-γ\\gammalimit preserves their relative teacher probabilities rather than selecting an arbitrary one\.

###### Corollary A\.4\(Diminishing returns of KL budget\)\.

Assume thatVVis nonconstant, and let

G​\(ϵ\)=maxρ:DKL​\(ρ∥p\)≤ϵ⁡𝔼ρ​\[V\]G\(\\epsilon\)=\\max\_\{\\rho:\\,D\_\{\\mathrm\{KL\}\}\(\\rho\\\|p\)\\leq\\epsilon\}\\mathbb\{E\}\_\{\\rho\}\[V\]for0<ϵ<−log⁡P⋆0<\\epsilon<\-\\log P^\{\\star\}\. Ifγ​\(ϵ\)\\gamma\(\\epsilon\)denotes the unique inverse temperature inducing radiusϵ\\epsilon, then

G′​\(ϵ\)=1γ​\(ϵ\),G′′​\(ϵ\)=−1γ​\(ϵ\)3​Varργ​\(ϵ\)⁡\(V\)<0\.G^\{\\prime\}\(\\epsilon\)=\\frac\{1\}\{\\gamma\(\\epsilon\)\},\\qquad G^\{\\prime\\prime\}\(\\epsilon\)=\-\\frac\{1\}\{\\gamma\(\\epsilon\)^\{3\}\\operatorname\{Var\}\_\{\\rho\_\{\\gamma\(\\epsilon\)\}\}\(V\)\}<0\.\(22\)Thus, relaxing the teacher\-centered KL budget improves the optimal estimated branch value, but with strictly diminishing marginal returns\.

###### Proof\.

Along the calibration path,G​\(ϵ​\(γ\)\)=𝔼ργ​\[V\]G\(\\epsilon\(\\gamma\)\)=\\mathbb\{E\}\_\{\\rho\_\{\\gamma\}\}\[V\]\. Proposition[A\.3](https://arxiv.org/html/2608.04419#A1.Thmproposition3)givesd​G/d​γ=Varργ⁡\(V\)\\mathrm\{d\}G/\\mathrm\{d\}\\gamma=\\operatorname\{Var\}\_\{\\rho\_\{\\gamma\}\}\(V\)andd​ϵ/d​γ=γ​Varργ⁡\(V\)\\mathrm\{d\}\\epsilon/\\mathrm\{d\}\\gamma=\\gamma\\operatorname\{Var\}\_\{\\rho\_\{\\gamma\}\}\(V\)\. Applying the chain rule once givesG′​\(ϵ\)=1/γG^\{\\prime\}\(\\epsilon\)=1/\\gamma; differentiating once more yields Eq\. \([22](https://arxiv.org/html/2608.04419#A1.E22)\)\. ∎

###### Corollary A\.5\(Finite\-temperature teacher anchoring\)\.

LetΔV=maxv⁡V​\(v\)−minv⁡V​\(v\)\\Delta\_\{V\}=\\max\_\{v\}V\(v\)\-\\min\_\{v\}V\(v\)\. For every finiteγ\>0\\gamma\>0andv∈𝒮v\\in\\mathcal\{S\},

exp⁡\(−γ​ΔV\)≤ργ​\(v\)p​\(v\)≤exp⁡\(γ​ΔV\)\.\\exp\(\-\\gamma\\Delta\_\{V\}\)\\leq\\frac\{\\rho\_\{\\gamma\}\(v\)\}\{p\(v\)\}\\leq\\exp\(\\gamma\\Delta\_\{V\}\)\.\(23\)Consequently, calibration preserves the teacher top\-kpk\_\{p\}support at every finite temperature\.

###### Proof\.

Sinceexp⁡\(γ​minv⁡V​\(v\)\)≤Z​\(γ\)≤exp⁡\(γ​maxv⁡V​\(v\)\)\\exp\(\\gamma\\min\_\{v\}V\(v\)\)\\leq Z\(\\gamma\)\\leq\\exp\(\\gamma\\max\_\{v\}V\(v\)\), the result follows directly fromργ​\(v\)/p​\(v\)=exp⁡\(γ​V​\(v\)\)/Z​\(γ\)\\rho\_\{\\gamma\}\(v\)/p\(v\)=\\exp\(\\gamma V\(v\)\)/Z\(\\gamma\)\. ∎

### A\.3Estimated\-Value Improvement under Teacher Anchoring

###### Proposition A\.6\(Improvement–deviation bounds\)\.

The outcome\-calibrated target satisfies

𝔼ργ​\[V\]−𝔼p​\[V\]≥1γ​DKL​\(ργ∥p\)≥0\.\\mathbb\{E\}\_\{\\rho\_\{\\gamma\}\}\[V\]\-\\mathbb\{E\}\_\{p\}\[V\]\\geq\\frac\{1\}\{\\gamma\}D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\)\\geq 0\.\(24\)Moreover, withϵ​\(γ\)=DKL​\(ργ∥p\)\\epsilon\(\\gamma\)=D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\),

0≤𝔼ργ​\[V\]−𝔼p​\[V\]≤ΔV​ϵ​\(γ\)2\.0\\leq\\mathbb\{E\}\_\{\\rho\_\{\\gamma\}\}\[V\]\-\\mathbb\{E\}\_\{p\}\[V\]\\leq\\Delta\_\{V\}\\sqrt\{\\frac\{\\epsilon\(\\gamma\)\}\{2\}\}\.\(25\)

###### Proof\.

Optimality ofργ\\rho\_\{\\gamma\}in Eq\. \([13](https://arxiv.org/html/2608.04419#A1.E13)\), using the teacher priorppas a feasible comparator, gives

𝔼ργ​\[V\]−1γ​DKL​\(ργ∥p\)≥𝔼p​\[V\],\\mathbb\{E\}\_\{\\rho\_\{\\gamma\}\}\[V\]\-\\frac\{1\}\{\\gamma\}D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\)\\geq\\mathbb\{E\}\_\{p\}\[V\],which proves Eq\. \([24](https://arxiv.org/html/2608.04419#A1.E24)\)\. For the upper bound, the expectation difference is at mostΔV​TV​\(ργ,p\)\\Delta\_\{V\}\\,\\mathrm\{TV\}\(\\rho\_\{\\gamma\},p\)\. Pinsker inequality givesTV​\(ργ,p\)≤DKL​\(ργ∥p\)/2\\mathrm\{TV\}\(\\rho\_\{\\gamma\},p\)\\leq\\sqrt\{D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\)/2\}, yielding Eq\. \([25](https://arxiv.org/html/2608.04419#A1.E25)\)\. ∎

Equation \([24](https://arxiv.org/html/2608.04419#A1.E24)\) formalizes the benefit of the exponential tilt, whereas Eq\. \([25](https://arxiv.org/html/2608.04419#A1.E25)\) formalizes its conservatism\. The guarantee concerns the probed estimateV=V^tV=\\hat\{V\}\_\{t\}; it does not by itself assert improvement in the unknown population branch value\.

### A\.4Preference Correction and the Binary\-Verifier Case

The target changes a teacher preference only when the outcome advantage is large enough to overcome the prior log\-odds\. In particular, for candidatesu,v∈𝒮u,v\\in\\mathcal\{S\},

ργ​\(v\)\>ργ​\(u\)⟺γ​\(V​\(v\)−V​\(u\)\)\>log⁡p​\(u\)p​\(v\)\.\\rho\_\{\\gamma\}\(v\)\>\\rho\_\{\\gamma\}\(u\)\\quad\\Longleftrightarrow\\quad\\gamma\\bigl\(V\(v\)\-V\(u\)\\bigr\)\>\\log\\frac\{p\(u\)\}\{p\(v\)\}\.\(26\)Thus,γ\\gammasets an explicit evidence threshold for reversing a teacher ranking\. Candidates with equal values preserve their teacher\-relative odds, and adding the same constant to all branch values leaves the target unchanged\.

The current training verifier is binary and uses one probe rollout per candidate, so the realized branch\-value estimates lie in\{0,1\}\\\{0,1\\\}\. This case admits a particularly direct interpretation\.

###### Proposition A\.7\(Binary\-verifier odds update\)\.

Let𝒮\+=\{v∈𝒮:V​\(v\)=1\}\\mathcal\{S\}^\{\+\}=\\\{v\\in\\mathcal\{S\}:V\(v\)=1\\\}, and define the prior and calibrated probability masses on successful branches as

P\+=∑v∈𝒮\+p​\(v\),Q\+=∑v∈𝒮\+ργ​\(v\)\.P\_\{\+\}=\\sum\_\{v\\in\\mathcal\{S\}^\{\+\}\}p\(v\),\\qquad Q\_\{\+\}=\\sum\_\{v\\in\\mathcal\{S\}^\{\+\}\}\\rho\_\{\\gamma\}\(v\)\.Then

Q\+=eγ​P\+1−P\+\+eγ​P\+,Q\+1−Q\+=eγ​P\+1−P\+,Q\_\{\+\}=\\frac\{e^\{\\gamma\}P\_\{\+\}\}\{1\-P\_\{\+\}\+e^\{\\gamma\}P\_\{\+\}\},\\qquad\\frac\{Q\_\{\+\}\}\{1\-Q\_\{\+\}\}=e^\{\\gamma\}\\frac\{P\_\{\+\}\}\{1\-P\_\{\+\}\},\(27\)Moreover,

DKL​\(ργ∥p\)=DKL​\(Bern⁡\(Q\+\)∥Bern⁡\(P\+\)\)\.D\_\{\\mathrm\{KL\}\}\(\\rho\_\{\\gamma\}\\\|p\)=D\_\{\\mathrm\{KL\}\}\\left\(\\operatorname\{Bern\}\(Q\_\{\+\}\)\\,\\\|\\,\\operatorname\{Bern\}\(P\_\{\+\}\)\\right\)\.\(28\)These identities hold whenever0<P\+<10<P\_\{\+\}<1\. Within either outcome group, the target preserves the teacher\-relative probabilities\.

###### Proof\.

For successful branches the exponential factor in Eq\. \([14](https://arxiv.org/html/2608.04419#A1.E14)\) iseγe^\{\\gamma\}, whereas for unsuccessful branches it is11\. Hence

Z​\(γ\)=eγ​P\+\+\(1−P\+\),Z\(\\gamma\)=e^\{\\gamma\}P\_\{\+\}\+\(1\-P\_\{\+\}\),and summing Eq\. \([14](https://arxiv.org/html/2608.04419#A1.E14)\) over𝒮\+\\mathcal\{S\}^\{\+\}gives Eq\. \([27](https://arxiv.org/html/2608.04419#A1.E27)\)\. If two candidates have the same binary value, their exponential factors cancel in their probability ratio, leaving the corresponding teacher odds unchanged\. Hence the conditional distributions within𝒮\+\\mathcal\{S\}^\{\+\}and its complement are unchanged\. Applying the chain rule for KL divergence over this binary partition yields Eq\. \([28](https://arxiv.org/html/2608.04419#A1.E28)\)\. ∎

Equation \([28](https://arxiv.org/html/2608.04419#A1.E28)\) shows that calibration spends its entire KL deviation on reallocating probability mass between the two outcome groups, introducing no within\-group distortion\. This proposition also clarifies the interaction between probing, gating, and calibration\. IfP\+=0P\_\{\+\}=0, the position is removed by theℬ\+\\mathcal\{B\}^\{\+\}gate\. IfP\+=1P\_\{\+\}=1, all candidates have equal value andργ=p\\rho\_\{\\gamma\}=p\. In the informative mixed case, outcome calibration multiplies the aggregate prior odds of successful versus unsuccessful branches by exactlyeγe^\{\\gamma\}, while retaining the finer teacher preferences within each group\.

## Appendix BImplementation Details

##### Off\-policy Training\.

Following\(Jinet al\.,[2026](https://arxiv.org/html/2608.04419#bib.bib1)\), KD is implemented with DistillKit\(Goddard and Atkins,[2024](https://arxiv.org/html/2608.04419#bib.bib42)\), with one teacher response sampled per training problem\. Table[4](https://arxiv.org/html/2608.04419#A2.T4)summarizes the off\-policy configuration\.

Table 4:Hyperparameters used for off\-policy distillation\.HyperparameterValueLearning rate1×10−51\\times 10^\{\-5\}LR scheduler typecosineOptimizerAdamWCE loss weight0\.5Forward\-KL loss weight0\.5Training batch size128Training epoch3Cutoff length4,096Top\-kk\(for FKL\)16
##### On\-policy Training\.

We implement OPD, EOPD, GRPO, andSPOTwithverl\(Shenget al\.,[2025](https://arxiv.org/html/2608.04419#bib.bib43)\)and generate rollouts asynchronously withSGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2608.04419#bib.bib45)\)\. All main runs use 4×\\timesA800 GPUs, FSDP2, and bfloat16\. Table[5](https://arxiv.org/html/2608.04419#A2.T5)summarizes the principal training hyperparameters\.

Table 5:Hyperparameters used for on\-policy distillation and GRPO\.HyperparameterOPD, EOPD, SPOTGRPOLearning rate3×10−63\\times 10^\{\-6\}3×10−63\\times 10^\{\-6\}LR scheduler typecosinecosineOptimizerAdamWAdamWTraining batch size128128Mini\-batch size3232Samples per prompt18Temperature1\.01\.0Top\-pp1\.0 \(Qwen\), 0\.8 \(Llama\)1\.0 \(Qwen\), 0\.8 \(Llama\)Max prompt length1,0241,024Max response length4,0964,096Clipping ratio0\.20\.2All methods use AdamW, weight decay0\.010\.01, gradient clipping at1\.01\.0, and four mini\-batch updates per rollout iteration; the distillation methods use no learning\-rate warmup\. EOPD applies a unit\-weight top\-16 forward\-KL loss at teacher\-entropy values of at least0\.80\.8, while GRPO uses KL and entropy coefficients of10−310^\{\-3\}and0, respectively\.

##### Training Schedule\.

Table[6](https://arxiv.org/html/2608.04419#A2.T6)records the scale\-specific data and effective schedules of the main on\-policy runs\.

Table 6:Training data and schedules\. Iterations denote rollout iterations\.StudentTraining setExamplesEpochsIterationsQwen3\-0\.6B\-BaseMATH7,5003174Qwen3\-1\.7B\-BaseMATH7,5003174Qwen3\-4B\-BaseDAPO\-Math14,1162220Training examples are shuffled, no validation set is used, and prompts exceeding the listed length limit are filtered\.

##### Chat Template\.

Following EOPD, the Qwen runs of OPD, EOPD, andSPOTuse the Qwen3\-8B teacher’s non\-thinking chat format\. With thinking disabled, the Qwen3\-Base tokenizer renders the same prompt:<\|im\_start\|\>user\\n\{query\}<\|im\_end\|\>\\n<\|im\_start\|\>assistant\\n<think\>\\n\\n</think\>\\n\\n\. GRPO instead uses the default Qwen3\-Base generation prompt, which omits the empty thinking block:<\|im\_start\|\>user\\n\{query\}<\|im\_end\|\>\\n<\|im\_start\|\>assistant\\n\.

##### SPOT\-specific Training\.

Table[7](https://arxiv.org/html/2608.04419#A2.T7)summarizes the hyperparameters specific toSPOT; the acquisition, probing, gating, and loss are defined in Algorithm[1](https://arxiv.org/html/2608.04419#alg1)and Section[3\.2](https://arxiv.org/html/2608.04419#S3.SS2)\.

Table 7:Hyperparameters used for SPOT\.HyperparameterValueSelected positionsMM2Scoring top\-kkksk\_\{s\}16Probing top\-kkkpk\_\{p\}4λmass,λshape\\lambda\_\{\\mathrm\{mass\}\},\\lambda\_\{\\mathrm\{shape\}\}0\.5,0\.50\.5,0\.5Reward\-tilt coefficientγ\\gamma1\.0Branch\-loss weightβ\\beta0\.1Probe rollouts per candidate1Training verifierbinary rule\-based boxed\-answer match
##### Evaluation Details\.

We use vLLM with bfloat16 precision and random seed 42, render one user message with the model’s native chat template and thinking disabled, and append the instructionPlease reason step by step, and put your final answer within \\boxed\{\}\.We extract the final balanced\\boxed\{\}expression and use Math\-Verify\([Kydlíček,](https://arxiv.org/html/2608.04419#bib.bib46)\)for symbolic equivalence; missing or unparsable answers are counted as incorrect\.

Table 8:Sizes of the evaluation benchmarks\.BenchmarkProblemsMATH\-500500AMC 202340Minerva Math272HMMT 2025 \(Feb\.\+Nov\.\)60AIME 202430AIME 202530

## Appendix CMore Ablation Experiment Results

### C\.1Out\-of\-Domain Generalization

Table[9](https://arxiv.org/html/2608.04419#A3.T9)reports the complete out\-of\-domain results analyzed in Section[4\.3\.5](https://arxiv.org/html/2608.04419#S4.SS3.SSS5)\. Together, these benchmarks separate deliberative reasoning, broad knowledge, and instruction following, providing a more diagnostic view of transfer than any single aggregate score\.

Table 9:Out\-of\-domain results for Qwen3\-1\.7B\-Base trained on MATH \(all values in %\)\. GPQA\-Diamond uses eight samples per question; MMLU\-Pro uses category\-matched five\-shot CoT demonstrations and reports Pass@1; and AlpacaEval 2\.0 reports win rate \(WR\) and length\-controlled win rate \(LC\-WR\)\.Boldandunderlinedentries denote the best and second\-best results, respectively\. TheSPOTcolumn is shaded\. AlpacaEval scores marked†use 803 \(EOPD\) and 800 \(SPOT\) valid judgments out of 805, respectively\.BenchmarkMetricKDGRPOOPDEOPDSPOT\(Ours\)GPQA\-DiamondAvg@821\.1019\.2625\.0027\.2129\.42Pass@862\.1855\.0567\.1769\.1980\.81MMLU\-ProPass@138\.1341\.8641\.2342\.9042\.26AlpacaEval 2\.0LC\-WR24\.1021\.8627\.0828\.13†27\.59†WR27\.5825\.7131\.1833\.23†34\.63†The comparison reveals a structured transfer profile rather than uniform dominance\. Relative to the strongest baseline on GPQA\-Diamond,SPOTimproves Avg@8 by 2\.21 points and Pass@8 by 11\.62 points, and it leads AlpacaEval WR by 1\.40 points\. On MMLU\-Pro,SPOTranks second at 42\.26: it trails EOPD by only 0\.64 points, while exceeding GRPO and OPD by 0\.40 and 1\.03 points, respectively\. This near\-best result indicates that branch calibration preserves broad subject\-matter reasoning despite training only on MATH, while EOPD’s edge suggests that token\-level uncertainty matching remains especially useful for heterogeneous knowledge questions\. On AlpacaEval, length control reverses the raw\-WR ordering, indicating that part of the preference gain is associated with response length\. Overall, outcome calibration transfers most clearly to deliberative reasoning, with method\-dependent tradeoffs on broader forms of generalization\.

### C\.2Consistency Across Model Families

To evaluate the generality of our method beyond Qwen3, we repeat the comparison with a Llama\-3\.2\-3B\-Instruct student trained on MATH for three epochs and a Llama\-3\.1\-8B\-Instruct teacher for the distillation methods\. All methods retain their native optimization protocols and are evaluated with eight samples under identical decoding\. Because this setting differs from the Qwen experiments in architecture, checkpoint type, and student–teacher scale, Table[10](https://arxiv.org/html/2608.04419#A3.T10)compares methods only within Llama; the experiment tests transfer of the relative method pattern, not absolute performance across families\.

Table 10:Results on the Llama model family\.Macro avg\.is the unweighted mean over the four benchmarks, computed before rounding\.Boldandunderlinedentries denote the best and second\-best results, respectively\. SPOT cells are shaded in blue for Avg@8 and green for Pass@8\.BenchmarkMetricGRPOOPDEOPDSPOTMATH500Avg@843\.6535\.9037\.7543\.78Pass@861\.6063\.6065\.2066\.20AMC23Avg@824\.0613\.1319\.0623\.44Pass@847\.5045\.0047\.5050\.00AIME24Avg@82\.080\.831\.672\.50Pass@816\.676\.6713\.3316\.67AIME25Avg@80\.000\.420\.000\.42Pass@80\.003\.330\.003\.33Macro avg\.Avg@817\.4512\.5714\.6217\.53Pass@831\.4429\.6531\.5134\.05##### Within\-family comparison\.

Within the Llama setting,SPOTraises macro Avg@8 from 12\.57 for OPD to 17\.53 and macro Pass@8 from 29\.65 to 34\.05\. Relative to the strongest baseline for each metric, however, the distinction is asymmetric: Avg@8 is essentially tied with GRPO \(17\.53 vs\. 17\.45\), whereas Pass@8 exceeds EOPD by 2\.54 points\. The benchmark\-level results show the same pattern\.SPOTis best or tied\-best in Pass@8 on all four benchmarks, while GRPO remains stronger in AMC23 Avg@8 and OPD tiesSPOTon AIME25\. Thus, the evidence supports a more consistent improvement in multi\-sample solution coverage than in average per\-sample accuracy, rather than uniform dominance on every benchmark\.

##### Cross\-family interpretation\.

This asymmetry mirrors the Qwen results: across the 0\.6B and 1\.7B students trained on MATH and the 4B student trained on DAPO,SPOTleads macro Pass@8 in every setting, while macro Avg@8 is best or nearly tied with the strongest baseline\. Its recurrence across Base and Instruct checkpoints, two model families, multiple capacities, and two training\-data regimes makes a Qwen\-specific explanation less plausible\. The aggregates are not directly comparable because the Llama panel contains four benchmarks whereas the Qwen panel contains six; moreover, architecture, capacity, initialization, teacher pairing, and training data are not independently controlled\. We therefore view these results as evidence of portability across the evaluated configurations, not architecture invariance or a causal scaling law\.