When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning

# When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
Source: [https://arxiv.org/html/2605.21606](https://arxiv.org/html/2605.21606)
Xiaogeng Liu1Xinyan Wang2Yingzi Ma2Yechao Zhang3Chaowei Xiao1 1Johns Hopkins University2University of Wisconsin–Madison 3Nanyang Technological University

###### Abstract

On\-policy self\-distillation \(OPSD\) trains a student on its own rollouts using a privileged teacher, but its standard objective weights all generated tokens equally, implicitly treating the privileged teacher target as equally reliable at every student\-visited prefix\. Existing entropy\-based OPD methods relax this uniformity by modulating token\-level supervision with teacher entropy, but high teacher entropy in reasoning has an ambiguous reliability meaning: it can reflect either non\-viable uncertainty or benign solution diversity\. To identify this phenomenon, we introduce a branch\-viability diagnostic\. Specifically, we record next\-token alternatives from the privileged\-answer teacher prompt, force each alternative after the student prompt plus its on\-policy spine prefix, and test whether the resulting student\-template continuation recovers the correct answer\. On Qwen3\-4B, we find that an oriented within\-sequence position score is the strongest tested predictor of teacher\-token reliability, reaching an area\-under\-ROC\-curve \(AUROC\) of0\.830\.83with a95%95\\%cluster\-bootstrap interval of\[0\.66,0\.95\]\[0\.66,0\.95\]; local uncertainty scores are at most0\.570\.57\. Motivated by this trajectory\-level structure, we propose Position\-Weighted On\-Policy Self\-Distillation \(PW\-OPSD\), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward\-KL target asOPSD\. In our comprehensive evaluations with different random seeds, the diagnostic\-derivedPW\-OPSDimproves AIME 2024 and AIME 2025 Avg@12 by\+1\.0\+1\.0and\+1\.1\+1\.1points, and a generalization evaluation on two larger\-scale models from different families, DeepSeek\-R1\-Distill\-Llama\-8B and Olmo\-3\-7B\-Think, also demonstrates consistent aggregate Avg@12 improvements\. These results show that teacher\-token reliability in reasoning distillation is trajectory\-structured and can be utilized without additional teacher computation\. The code is available at[https://github\.com/SaFo\-Lab/PW\-OPSD](https://github.com/SaFo-Lab/PW-OPSD)

## 1Introduction

On\-policy self\-distillation \(OPSD\)\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)is a practical recipe for distilling mathematical reasoning because it supervises the student on prefixes the student actually visits\. The student samples a rollout from the ordinary problem prompt, and a privileged copy of the model, conditioned on reference information, provides token\-level targets along that same rollout\. This construction avoids the trajectory mismatch of teacher\-generated demonstrations while still injecting privileged information at training time\.

The standardOPSDobjective nevertheless makes an implicit reliability assumption: every generated token receives the same forward\-KL supervision\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)\. This uniform objective treats the privileged teacher target as equally useful at every student\-visited prefix\. Recent adaptive, relaxed, or gated distillation objectives have challenged this uniformity assumption from complementary angles:EOPDaugments reverse\-KL OPD with forward KL on high\-teacher\-entropy tokens\(Jinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib3)\),ToDimixes forward and reverse KL per token using a teacher\-student probability log\-ratio\(Junget al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib4)\), REOPOLD treats the teacher\-student log\-likelihood ratio as a token reward with reward clipping and entropy\-based sampling\(Koet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib9)\), and GATES gates privileged self\-distillation by tutor consensus\(Steinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib5)\)\. Together, these works motivate nonuniform token\-level supervision through local entropy, teacher\-student mismatch, clipped token rewards, or consensus signals\.

![Refer to caption](https://arxiv.org/html/2605.21606v1/x1.png)Figure 1:Branch viability reveals a positional reliability structure in Qwen3\-4B reasoning traces\. Alternatives are selected by the same model under the privileged\-answer teacher prompt and rolled out under the ordinary student prompt; for high\-ambiguity candidate positions, these forced student\-context continuations fail more often at early positions and rarely fail late in the trace\.\(A\)Sorting candidates by normalized position therefore separates teacher\-unreliable branch points from viable diversity, whereas local uncertainty \(entropy\) scores yield near\-flat failure curves\.\(B\)The resulting empirical reliability curve motivates thePW\-OPSDweight schedule, which discounts early high\-ambiguity positions while preserving full\-strength supervision later\. See[section˜3\.2](https://arxiv.org/html/2605.21606#S3.SS2)for the details of the diagnostic experiments\.However, all of these adaptive criteria share a common limitation: they measure how ambiguous the teacher’s local distribution is, through entropy, teacher\-student mismatch, or tutor consensus, but token\-local ambiguity is*not*the same as low teacher reliability, and conflating the two reweights tokens for the wrong reason\. Our point of departure is the reliability meaning of these local signals\. Entropy\-based criteria are a natural way to detect uncertainty, but high teacher entropy is not the same as low teacher reliability\. A high\-entropy teacher distribution can reflect non\-viable uncertainty: several locally plausible next\-token alternatives receive probability mass, but some of them fail to recover the correct answer under continued student\-context generation\. However, it can also reflect benign diversity: the teacher assigns mass to multiple viable solution routes or surface realizations that still preserve correctness\. Thus a token\-local uncertainty score can identify ambiguity, but it does not distinguish non\-viable uncertainty from viable solution diversity\.

The practical cost is that any reweighting following local ambiguity alone will either amplify supervision at branch points where the teacher derails, or downweight supervision at benign branch points where the teacher stays correct\. Thus, the right object to estimate for adaptive token weighting is not teacher entropy but teacher*reliability*: a per\-context probability that the teacher’s local target is useful supervision for the student\. We formalize this*reliability*as a latent indicatorItI\_\{t\}for whether matching the teacher target at distillation contextctc\_\{t\}helps preserve or recover the correct solution\. The reliability\-weighted surrogate then weights the forward\-KL term byρ∗​\(ct\)=Pr⁡\(It=1∣ct\)\\rho^\{\*\}\(c\_\{t\}\)=\\Pr\(I\_\{t\}=1\\mid c\_\{t\}\)\.

To connect this latent quantity to observable behavior, we introduce a branch\-viability diagnostic\. Throughout, “teacher” and “student” refer to two prompt templates applied to the same model: the teacher template includes the privileged ground\-truth answer; the student template is the ordinary problem prompt\. Starting from a student\-template rollout that reaches the correct answer, we use the teacher template to propose high\-ambiguity next\-token alternatives, force each alternative after the student\-template prompt plus the spine prefix, and continue under the student template \(no privileged information\)\. We use student\-template continuation because a teacher\-template continuation can re\-use the privileged answer and recover from almost any forced branch, collapsing the labels\. If the forced alternatives usually fail to recover the correct answer, the candidate is labeled*real\-uncertain*: at that prefix, the teacher’s local alternatives are low\-reliability supervision for the student distribution\. If the alternatives remain viable, the ambiguity is treated as benign diversity rather than as evidence of low teacher reliability\.

On Qwen3\-4B\(Yanget al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib19)\), this diagnostic shows a clear positional pattern, as demonstrated in Fig\.[1](https://arxiv.org/html/2605.21606#S1.F1)\. In the correct\-spine subset, real\-uncertain candidates concentrate early in the normalized trace, while diversity candidates are more common later\. After within\-problem residualization, an oriented position score separates real\-uncertain candidates from diversity candidates with AUROC0\.830\.83and a95%95\\%cluster\-bootstrap interval of\[0\.66,0\.95\]\[0\.66,0\.95\]; the tested local uncertainty diagnostics reach residual AUROC at most0\.570\.57\. Thus the diagnostic supplies the direction of the reliability prior: early high\-ambiguity branch points are where the privileged teacher’s local alternatives most often become unreliable, while later positions are more reliable under forced continuation\.

Inspired by this finding, we propose Position\-Weighted On\-Policy Self\-Distillation \(PW\-OPSD\), which implements this reliability trend as a plug\-in approximation toρ∗​\(ct\)\\rho^\{\*\}\(c\_\{t\}\)\. It keeps the same student rollout, privileged teacher pass, and per\-vocabulary clipped forward\-KL surrogate asOPSD, but aggregates token losses with an increasing sigmoid function of normalized within\-sequence position\. The floor keeps early tokens partially supervised, while the increasing schedule assigns stronger weight to later positions where the diagnostic indicates higher teacher reliability\. The method therefore changes only the outer reliability\-weighted aggregation, adding no extra teacher pass or auxiliary verifier\.

We evaluatePW\-OPSDon Qwen3\-4B math reasoning benchmarks with the same maximum generation length of38,91238\{,\}912tokens as theOPSDreference evaluation\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)\. The diagnostic\-derived schedule matchesOPSDon MATH\-500 Avg@12 \(95\.3495\.34vs\.95\.3395\.33\) and improves Avg@12 by\+1\.0\+1\.0point on AIME 2024 and\+1\.1\+1\.1points on AIME 2025, with the same teacher cost\. The position\-schedule sweep further shows how reliability\-curve strength affects harder regimes: the aggressive schedule improves HMMT 2025 Avg@12 by\+1\.48\+1\.48points overOPSDand improves Maj@12 by\+1\.11\+1\.11points\. These results support position as a structural reliability prior and show that reasoning distillation benefits from modeling teacher reliability along the trajectory\.

## 2Related Work

#### On\-policy distillation\.

Knowledge distillation trains a student to match soft teacher probabilities rather than only hard labels\(Hintonet al\.,[2015](https://arxiv.org/html/2605.21606#bib.bib42)\)\. Sequence\-level distillation extends this idea from token labels to generated completions\(Kim and Rush,[2016](https://arxiv.org/html/2605.21606#bib.bib13)\)\. Modern LLM post\-training popularized instruction and feedback\-based tuning\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.21606#bib.bib26)\); in distillation, recent LLM methods often move supervision on policy, so the student is trained on prefixes it actually visits\(Agarwalet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib6); Guet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib7); Koet al\.,[2024](https://arxiv.org/html/2605.21606#bib.bib8); Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)\. This follows the interactive imitation\-learning intuition of querying expert feedback on learner\-visited states\(Rosset al\.,[2010](https://arxiv.org/html/2605.21606#bib.bib27)\); empirical work on LLM imitation also reports degradation under off\-policy imitation in multi\-step generation settings\(Gudibandeet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib15)\)\. GKD formalizes generalized on\-policy knowledge distillation\(Agarwalet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib6)\), MiniLLM optimizes sequence\-level reverse\-KL distillation on student/teacher\-mixed on\-policy samples through a policy\-gradient formulation\(Guet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib7)\), and DistiLLM studies skewed KL objectives for white\-box OPD on student\-generated outputs\(Koet al\.,[2024](https://arxiv.org/html/2605.21606#bib.bib8)\)\. SeeSong and Zheng \([2026](https://arxiv.org/html/2605.21606#bib.bib2)\)for a broader survey of OPD\.OPSDis the main baseline in this paper: it uses a privileged teacher prompt containing reference information to provide token\-level targets along student rollouts\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)\.PW\-OPSDkeeps the privileged on\-policy construction ofOPSDand makes the outer reliability weight trajectory\-structured rather than uniform\.

#### Divergence choice and adaptive token weighting\.

Several distillation methods change the divergence or token\-level controller between teacher and student\. MiniLLM replaces standard forward\-KL KD with sequence\-level reverse KL and optimizes the resulting on\-policy objective via policy\-gradient returns\(Guet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib7)\); DistiLLM uses skewed KL objectives to stabilize white\-box OPD on student\-generated outputs\(Koet al\.,[2024](https://arxiv.org/html/2605.21606#bib.bib8)\)\. Broaderff\-divergence work studies how divergence choice changes sequence\-level KD behavior\(Wenet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib25)\)\. AKL revisits the usual mode\-seeking/mode\-covering framing for LLM distillation, arguing that forward and reverse KL share the same asymptotic objective but that FKL emphasizes head probabilities while RKL emphasizes tail probabilities in early epochs, then proposing an adaptive KL mixture\(Wuet al\.,[2024](https://arxiv.org/html/2605.21606#bib.bib10)\)\. Recent adaptive or privileged distillation methods also depart from uniform token\-level supervision, but for different reasons and with different control signals\.EOPDapplies forward KL on high\-teacher\-entropy tokens while retaining reverse KL elsewhere in OPD\(Jinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib3)\); REOPOLD\(Koet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib9)\)interprets OPD as policy optimization with a teacher\-student log\-likelihood\-ratio token reward and relaxes strict imitation through reward clipping, entropy\-based dynamic sampling, and exploration\-to\-refinement training\. Outside this student\-rollout OPD divergence\-switching line,ToDiadaptively combines forward and reverse KL per token using a teacher\-student probability log\-ratio\(Junget al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib4)\); GATES instead gates privileged self\-distillation using consensus among tutor traces\(Steinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib5)\)\. These methods show that distillation supervision can be modulated per token through entropy, teacher\-student ratios, clipped rewards, or consensus signals\. However, these local control signals do not directly distinguish non\-viable uncertainty from benign solution diversity\. To explore this challenge, we propose a branch\-viability experiment in which the same model, under the privileged\-answer teacher prompt, selects forced alternatives and then rolls them out under the ordinary student prompt to test whether the correct answer is still recovered\. We compare branch viability against standard uncertainty diagnostics, including predictive entropy, MC\-dropout mutual information, and Dirichlet or evidential uncertainty\(Kendall and Gal,[2017](https://arxiv.org/html/2605.21606#bib.bib22); Gal and Ghahramani,[2015](https://arxiv.org/html/2605.21606#bib.bib11); Malinin and Gales,[2018](https://arxiv.org/html/2605.21606#bib.bib23); Sensoyet al\.,[2018](https://arxiv.org/html/2605.21606#bib.bib24)\)\. These baselines quantify local ambiguity, but they do not test whether high\-probability teacher alternatives preserve correctness under continued generation\.

## 3Method

PW\-OPSDis a reliability\-weighted version ofOPSDthat preserves the student rollout, privileged teacher pass, and per\-vocabulary clipped forward\-KL surrogate\. It changes the outer token aggregation through a position\-dependent reliability weight and a per\-sequence mean, motivated by a branch\-viability diagnostic showing that early high\-ambiguity branch points are often teacher\-unreliable, while local uncertainty scores are weak predictors of this reliability event\.

### 3\.1Background: On\-Policy Self\-Distillation

Letxstux\_\{\\mathrm\{stu\}\}be the ordinary student prompt andxtchx\_\{\\mathrm\{tch\}\}be the privileged teacher prompt that includes the reference solution\. On\-policy self\-distillation \(OPSD\)\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)first samples a completion from the current student,

y1:T∼πθ\(⋅∣xstu\),y\_\{1:T\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{\\mathrm\{stu\}\}\),and then queries a teacher copy of the same model on the privileged context and the student prefix\. At each valid generated positiontt, the teacher target is

𝐩topsd=ptch\(⋅∣xtch,y<t\),pt=πθ\(⋅∣xstu,y<t\)\.\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{t\}=p\_\{\\mathrm\{tch\}\}\(\\cdot\\mid x\_\{\\mathrm\{tch\}\},y\_\{<t\}\),\\qquad p\_\{t\}=\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{\\mathrm\{stu\}\},y\_\{<t\}\)\.Both distributions are temperature\-scaled before the softmax, matching the trainer path used for all forward\-KL losses\. With valid\-token setℳ\\mathcal\{M\}, the implementedOPSDobjective is

ℒOPSD=1\|ℳ\|​∑t∈ℳ∑j∈𝒱min⁡\(𝐩topsd​\(j\)​log⁡𝐩topsd​\(j\)pt​\(j\),τclip\)\.\\mathcal\{L\}\_\{\\textsc\{OPSD\}\{\}\}=\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\sum\_\{t\\in\\mathcal\{M\}\}\\sum\_\{j\\in\\mathcal\{V\}\}\\min\\\!\\left\(\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{t\}\(j\)\\log\\frac\{\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{t\}\(j\)\}\{p\_\{t\}\(j\)\},\\tau\_\{\\mathrm\{clip\}\}\\right\)\.\(1\)The clipping in[eq\.˜1](https://arxiv.org/html/2605.21606#S3.E1)is element\-wise over vocabulary terms: the quantity𝐩topsd​\(j\)​log⁡\(𝐩topsd​\(j\)/pt​\(j\)\)\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{t\}\(j\)\\log\(\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{t\}\(j\)/p\_\{t\}\(j\)\)is clipped for eachj∈𝒱j\\in\\mathcal\{V\}, the clipped terms are summed over the vocabulary, and the result is averaged over valid tokens\. This objective is the uniform\-weight baseline whose outer token aggregationPW\-OPSDwill modify\.

### 3\.2Motivation: Token\-Position Predicts Teacher Reliability

#### Branch\-viability protocol\.

The branch\-viability diagnostic asks whether a locally ambiguous teacher preference is reliable supervision for the student distribution\. Throughout the diagnostic, “teacher” and “student” refer to two prompt templates applied to the same Qwen3\-4B model: the*teacher*template includes the privileged ground\-truth answer, while the*student*template is the ordinary problem prompt without that information\.

For each problem, the model under the student template first generates an on\-policy rollout; this fixed token sequence is the*spine*\. The model is then re\-evaluated along the spine under the teacher template, which adds the privileged answer\. Candidate positions are pre\-selected from high top\-1616valid\-token truncated entropy on this teacher pass; Appendix[A](https://arxiv.org/html/2605.21606#A1)gives the filtering details\. The diagnostic is deliberately targeted: it studies positions where the teacher already signals local ambiguity, rather than arbitrary positions on the rollout\. The reliability\-weighted view in[section˜3\.3](https://arxiv.org/html/2605.21606#S3.SS3)identifies the latent reliability posterior as the formal object that this diagnostic probes behaviorally\.

At each candidate position, the teacher proposes preferred next\-token alternatives\. We then test whether each alternative leads back to the correct answer*from the student’s perspective*: we form a prefix consisting of the student\-template prompt, the spine truncated to the candidate position, and the forced alternative token, and complete the sequence under the student template \(no privileged information\)\. A candidate is labeled*real\-uncertain*when these student\-context continuations fail to recover the correct final answer; in this case the teacher’s local alternatives are unreliable targets for that student prefix\. A candidate is labeled*diversity*when the alternatives remain viable routes to the same answer\. We continue under the student template rather than the teacher template because student rollouts and deployment use the ordinary prompt without privileged information, and a teacher\-template continuation can recover from almost any forced branch by re\-using the ground\-truth answer in its prompt; that recovery affordance would mask the very reliability question the diagnostic is asking\. Appendix[A](https://arxiv.org/html/2605.21606#A1)gives the thresholded implementation details for the repeated forced\-continuation rollouts\.

#### Empirical finding\.

[Table˜1](https://arxiv.org/html/2605.21606#S3.T1)reports88real\-uncertain and271271diversity candidates from6161problems, restricted to the correct\-spine subset; the underlying failure\-rate\-by\-position curves are visualized in Fig\.[1](https://arxiv.org/html/2605.21606#S1.F1)\. Each feature is residualized within problem before scoring, so the AUROC reflects within\-problem token\-level separation rather than problem difficulty\. Confidence intervals are cluster bootstraps over problems\.

Position is the strongest predictor among the diagnostics we test\. We define normalized position asr~=\(spine\_pos\+0\.5\)/L\\widetilde\{r\}=\(\\text\{spine\\\_pos\}\+0\.5\)/L, whereLLis the length of the student spine\. Since the real\-uncertain candidates concentrate early,[table˜1](https://arxiv.org/html/2605.21606#S3.T1)reports the oriented early\-position score1−r~1\-\\widetilde\{r\}for the real\-uncertain label\. This score reaches AUROC0\.830\.83with95%95\\%CI\[0\.66,0\.95\]\[0\.66,0\.95\], whereas predictive entropy, MC\-dropout mutual information, Dirichlet precision, and top\-1616truncated entropy remain weak predictors\. Equivalently, raw positionr~\\widetilde\{r\}is negatively associated with teacher unreliability, which is whyPW\-OPSDuses an increasing reliability weight\.

Table 1:Branch\-viability classification on Qwen3\-4B\. Entries are within\-problem residualized AUROC for separating88*real\-uncertain*candidates \(whose forced student\-context continuations fail to recover the correct answer\) from271271*diversity*candidates \(whose continuations remain viable\) across6161problems, restricted to the correct\-spine subset and to high\-truncated\-entropy candidate positions\. For position, the scored feature is1−r~1\-\\widetilde\{r\}, so larger values correspond to earlier tokens\. Cluster\-bootstrap95%95\\%CIs are computed over problems\.The local diagnostics in[table˜1](https://arxiv.org/html/2605.21606#S3.T1)include standard teacher\-side uncertainty scores: predictive entropy\(Kendall and Gal,[2017](https://arxiv.org/html/2605.21606#bib.bib22)\), MC\-dropout mutual information\(Gal and Ghahramani,[2015](https://arxiv.org/html/2605.21606#bib.bib11)\), and Dirichlet precision and evidential\-style categorical uncertainty\(Malinin and Gales,[2018](https://arxiv.org/html/2605.21606#bib.bib23); Sensoyet al\.,[2018](https://arxiv.org/html/2605.21606#bib.bib24)\), plus our top\-1616truncated entropy\. They are included as diagnostic baselines for reliability prediction; implementation details for the MC\-dropout scores are in[Appendix˜B](https://arxiv.org/html/2605.21606#A2)\.

#### Implication for adaptive distillation signals\.

The weak scores in[table˜1](https://arxiv.org/html/2605.21606#S3.T1)call into question a common assumption behind recent adaptive, relaxed, or privileged distillation methods: that local uncertainty or teacher–student discrepancy can serve as a reliable control signal for distillation\. Existing methods instantiate this idea through teacher token entropy inEOPD\(Jinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib3)\), token\-wise teacher–student probability log\-ratios inToDi\(Junget al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib4)\), teacher–student log\-likelihood\-ratio rewards and entropy\-guided token\-level sampling in REOPOLD\(Koet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib9)\), and tutor\-answer consensus in GATES\(Steinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib5)\)\. These signals are useful for adapting objectives, rewards, or trajectory selection, but our diagnostic shows that they are not reliable proxies for branch viability: the same large or permissive signal can arise when the teacher is unreliable and when multiple continuations are genuinely viable\. Position provides a complementary structural signal, capturing where local ambiguity becomes unreliable for the student\.

### 3\.3Theoretical Interpretation: Reliability\-Weighted Distillation

Letct=\(xstu,xtch,y<t\)c\_\{t\}=\(x\_\{\\mathrm\{stu\}\},x\_\{\\mathrm\{tch\}\},y\_\{<t\}\)denote the distillation context, letqtq\_\{t\}be the privileged teacher target at that context, and letpt=πθ\(⋅∣xstu,y<t\)p\_\{t\}=\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{\\mathrm\{stu\}\},y\_\{<t\}\)be the student distribution\. For the interpretation, define the unclipped token divergenceDt​\(θ\)=KL⁡\(qt∥pt\)D\_\{t\}\(\\theta\)=\\operatorname\{KL\}\(q\_\{t\}\\,\\\|\\,p\_\{t\}\)\. Introduce a latent indicatorIt∈\{0,1\}I\_\{t\}\\in\\\{0,1\\\}, whereIt=1I\_\{t\}=1means that the teacher’s local target is reliable for the student prefix: matching it helps preserve or recover the correct solution rather than following an off\-path alternative\. The ideal reliability\-filtered risk is

R​\(θ\)=𝔼​\[It⋅Dt​\(θ\)\]\.R\(\\theta\)=\\mathbb\{E\}\\\!\\left\[I\_\{t\}\\cdot D\_\{t\}\(\\theta\)\\right\]\.\(2\)For fixedθ\\theta,Dt​\(θ\)D\_\{t\}\(\\theta\)is determined by the distillation context, so the tower property gives

R​\(θ\)=𝔼​\[ρ∗​\(ct\)⋅Dt​\(θ\)\],ρ∗​\(ct\)=Pr⁡\(It=1∣ct\)\.R\(\\theta\)=\\mathbb\{E\}\\\!\\left\[\\rho^\{\*\}\(c\_\{t\}\)\\cdot D\_\{t\}\(\\theta\)\\right\],\\qquad\\rho^\{\*\}\(c\_\{t\}\)=\\Pr\(I\_\{t\}=1\\mid c\_\{t\}\)\.\(3\)The Bayes\-optimal surrogate for this latent\-reliability risk weights each token by the posterior probability that the teacher target is reliable\. Appendix[C](https://arxiv.org/html/2605.21606#A3)gives the conditioning details\. On\-policy and adaptive distillation methods exercise related levers through sampling distributions, divergences, or weight functionals\(Agarwalet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib6); Guet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib7); Koet al\.,[2024](https://arxiv.org/html/2605.21606#bib.bib8); Wenet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib25); Wuet al\.,[2024](https://arxiv.org/html/2605.21606#bib.bib10); Junget al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib4); Jinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib3)\);PW\-OPSDuses normalized position as a low\-cost structural proxy for that posterior\.

The branch issue can also be summarized by a mixture identity\. Letht=\(xstu,y<t\)h\_\{t\}=\(x\_\{\\mathrm\{stu\}\},y\_\{<t\}\)denote the student\-side prefix\. Suppose the teacher target at that prefix is a mixture over latent successful branchesZZ,

qt=∑zα​\(z∣ht\)​qtz\.q\_\{t\}=\\sum\_\{z\}\\alpha\(z\\mid h\_\{t\}\)q\_\{t\}^\{z\}\.For any student distributionptp\_\{t\},

𝔼z∼α\(⋅∣ht\)​\[KL⁡\(qtz∥pt\)\]=KL⁡\(qt∥pt\)\+Iq​\(Yt;Z∣ht\),\\mathbb\{E\}\_\{z\\sim\\alpha\(\\cdot\\mid h\_\{t\}\)\}\\\!\\left\[\\operatorname\{KL\}\(q\_\{t\}^\{z\}\\,\\\|\\,p\_\{t\}\)\\right\]=\\operatorname\{KL\}\(q\_\{t\}\\,\\\|\\,p\_\{t\}\)\+I\_\{q\}\(Y\_\{t\};Z\\mid h\_\{t\}\),\(4\)whereIq\(⋅;⋅∣⋅\)I\_\{q\}\(\\cdot;\\cdot\\mid\\cdot\)is conditional mutual information under the teacher mixture\. The mutual\-information term quantifies branch\-specific variation hidden by the marginal teacher target; because it is independent ofptp\_\{t\}, the identity is not by itself a different gradient objective\. The corresponding sequence\-level identity accumulates this conditional mutual information across time; Appendix[D](https://arxiv.org/html/2605.21606#A4)gives the details and interpretation\.

The branch\-viability study supplies the task\-specific direction: early high\-ambiguity positions are most often labeled real\-uncertain, soPW\-OPSDdiscounts them with a lower outer weight\.

### 3\.4PW\-OPSD: Position\-Weighted On\-Policy Self\-Distillation

PW\-OPSDuses a deterministic structural proxy for the reliability posteriorρ∗​\(ct\)\\rho^\{\*\}\(c\_\{t\}\): a token’s relative position in its own student rollout\. For sequenceiiwith valid completion lengthLiL\_\{i\}, define the one\-based valid token indext∈\{1,…,Li\}t\\in\\\{1,\\ldots,L\_\{i\}\\\}and the per\-row position fraction

ri,t=t−0\.5Li\.r\_\{i,t\}=\\frac\{t\-0\.5\}\{L\_\{i\}\}\.The position weight is

wi,t=wmin\+\(1−wmin\)​σ​\(ri,t−τs\),w\_\{i,t\}=w\_\{\\min\}\+\(1\-w\_\{\\min\}\)\\sigma\\\!\\left\(\\frac\{r\_\{i,t\}\-\\tau\}\{s\}\\right\),\(5\)with defaults\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\)\. The floor keeps early tokens partially supervised, the threshold places the transition early in the rollout, and the scale avoids a hard discontinuity\. The schedule is increasing because raw position is positively associated with reliability in the branch\-viability diagnostic\.

The per\-row normalization inri,tr\_\{i,t\}and the per\-sequence reduction are a single design choice\. A position fraction is meaningful only relative to the sequence’s own valid length, not the batch\-padded maximum\. The corresponding loss therefore averages within each sequence before averaging across valid sequences; otherwise, longer rollouts would receive more gradient mass purely because they contain more tokens\.

WithBBvalid sequences in the batch, the implementedPW\-OPSDobjective is

ℒPW\-OPSD=1B​∑i=1B1Li​∑t=1Liwi,t​∑j∈𝒱min⁡\(𝐩i,topsd​\(j\)​log⁡𝐩i,topsd​\(j\)πθ​\(j∣xstui,y<ti\),τclip\)\.\\mathcal\{L\}\_\{\\textsc\{PW\-OPSD\}\{\}\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\frac\{1\}\{L\_\{i\}\}\\sum\_\{t=1\}^\{L\_\{i\}\}w\_\{i,t\}\\sum\_\{j\\in\\mathcal\{V\}\}\\min\\\!\\left\(\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{i,t\}\(j\)\\log\\frac\{\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{i,t\}\(j\)\}\{\\pi\_\{\\theta\}\(j\\mid x^\{i\}\_\{\\mathrm\{stu\}\},y^\{i\}\_\{<t\}\)\},\\tau\_\{\\mathrm\{clip\}\}\\right\)\.\(6\)The inner term is the same per\-vocabulary clipped forward KL as[eq\.˜1](https://arxiv.org/html/2605.21606#S3.E1)\.PW\-OPSDuses the ordinary single\-pass teacher target𝐩i,topsd\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{i,t\}; its additional computation overOPSDis only the scalar sigmoid in[eq\.˜5](https://arxiv.org/html/2605.21606#S3.E5)\.

## 4Experiments

[Section˜3\.2](https://arxiv.org/html/2605.21606#S3.SS2)suggests a simple reliability prior: early high\-ambiguity branch points are more likely to be teacher\-unreliable, while later positions are more reliable under the branch\-viability diagnostic\. This section tests whether the corresponding position\-weighted objective improves downstream reasoning behavior relative toOPSDunder the same rollout, teacher pass, and evaluation protocol\.

### 4\.1Setup

#### Models, baselines, and schedule\.

We train on the Qwen3\-4B checkpoint\(Yanget al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib19); Qwen Team,[2025](https://arxiv.org/html/2605.21606#bib.bib20)\)for the main comparison \([table˜2](https://arxiv.org/html/2605.21606#S4.T2)\), the position\-schedule sweep \([section˜4\.3](https://arxiv.org/html/2605.21606#S4.SS3)\), and the reduction\-positioning ablation \([table˜6](https://arxiv.org/html/2605.21606#A12.T6)\); the cross\-model evidence in[table˜4](https://arxiv.org/html/2605.21606#S4.T4)additionally uses DeepSeek\-R1\-Distill\-Llama\-8B \(DSR1\-L8B\)\(Guoet al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib21); DeepSeek\-AI,[2025](https://arxiv.org/html/2605.21606#bib.bib40)\)and Olmo\-3\-7B\-Think\(Allen Institute for AI,[2026](https://arxiv.org/html/2605.21606#bib.bib41)\)\.PW\-OPSDuses the Moderate schedule\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\), the diagnostic\-derived default\. We compare against three baselines:OPSD\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)as the uniform\-weight reference,EOPD\(Jinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib3)\)as a representative entropy\-conditioned adaptive\-KL baseline, and REOPOLD\(Koet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib9)\)as a cross\-family policy\-gradient adaptive on\-policy distillation baseline that controls for whether any adaptive per\-token signal recovers the downstream pattern attributed to position\. All methods share the OPSD privileged on\-policy chassis with LoRA\(Huet al\.,[2021](https://arxiv.org/html/2605.21606#bib.bib12)\)\(rank6464,α=128\\alpha=128\), are evaluated at the100100\-step checkpoint followingOPSD, and otherwise use their published defaults\. Full training hyperparameters, the cross\-family rationale, and thePW\-OPSDpseudocode are in[Appendices˜I](https://arxiv.org/html/2605.21606#A9)and[G](https://arxiv.org/html/2605.21606#A7)\.

#### Evaluation\.

We follow theOPSDevaluation setting\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\)with maximum generation length38,91238\{,\}912tokens:max\_new\_tokens=38912=38912,val\_n=12=12samples per problem, temperatureT=1\.0T=1\.0, top\-p=0\.95p=0\.95, top\-kkdisabled, andenable\_thinking=True\. Benchmarks are MATH\-500\(Hendryckset al\.,[2021](https://arxiv.org/html/2605.21606#bib.bib28); Lightmanet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib18); HuggingFaceH4,[2024](https://arxiv.org/html/2605.21606#bib.bib36)\), AIME 2024\(HuggingFaceH4,[2025](https://arxiv.org/html/2605.21606#bib.bib37)\), AIME 2025\(yentinglin,[2025](https://arxiv.org/html/2605.21606#bib.bib38)\), and HMMT 2025\(Dekonincket al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib35)\)\. We reportPass​@​12\\mathrm\{Pass\}@12,Avg​@​12\\mathrm\{Avg\}@12, andMaj​@​12\\mathrm\{Maj\}@12;[Appendix˜J](https://arxiv.org/html/2605.21606#A10)gives the formulas\. For each method–benchmark pair we run three random evaluation seeds \(main,11,22\) and report mean±\\pmacross\-seed sample standard deviation; the cross\-model assessment \([table˜4](https://arxiv.org/html/2605.21606#S4.T4)\) applies the same four\-benchmark three\-seed protocol to two additional checkpoints from different model families\.

### 4\.2Main results on Qwen3\-4B

Table 2:Qwen3\-4B results with maximum generation length38,91238\{,\}912tokens\. Entries report mean±\\pmstandard deviation across three evaluation seeds\. Bold marks the best value for each benchmark–metric column and any value within0\.10\.1pp of the leader\. The Aggressive row is included to show schedule sensitivity rather than as a separate method claim\.[Table˜2](https://arxiv.org/html/2605.21606#S4.T2)reports the Qwen3\-4B comparison with maximum generation length38,91238\{,\}912tokens\. We report the diagnostic\-derived*Moderate*schedule\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\)as the defaultPW\-OPSDconfiguration\. We also include*Aggressive*\(0\.05,0\.50,0\.05\)\(0\.05,0\.50,0\.05\)from the schedule sweep as a sensitivity variant that applies a stronger early discount\. The main pattern is that position weighting improves per\-sample reasoning accuracy on the AIME benchmarks while preserving near\-saturated MATH\-500 performance\. On MATH\-500,OPSD,EOPD, andPW\-OPSDModerate are tied within0\.010\.01pp Avg@12, whilePW\-OPSDAggressive gives the highest MATH\-500 Avg@12 and Maj@12 in the table\. On AIME 2024 and AIME 2025,PW\-OPSDModerate improves Avg@12 overOPSDby\+1\.0\+1\.0pp and\+1\.1\+1\.1pp, respectively\. The local or adaptive alternatives do not reproduce this Avg@12 pattern:EOPDtrailsOPSDby1\.61\.6pp on AIME 2024 and1\.01\.0pp on AIME 2025, while REOPOLD trails by1\.21\.2pp and4\.54\.5pp\. HMMT 2025 highlights the role of reliability\-curve strength\. The Moderate schedule improves Maj@12 overOPSDby\+1\.11\+1\.11pp but trails by0\.60\.6pp Avg@12 and4\.44\.4pp Pass@12\. Under the stronger early discount of the Aggressive schedule, HMMT 2025 Avg@12 rises to45\.3745\.37pp, a\+1\.48\+1\.48pp gain overOPSD, and Maj@12 remains\+1\.11\+1\.11pp aboveOPSD\. Among the four evaluated methods we compare,PW\-OPSDis the only one to improve overOPSD: Moderate gains\+1\.0\+1\.0pp on AIME 2024 and\+1\.1\+1\.1pp on AIME 2025, and Aggressive gains\+1\.5\+1\.5pp on HMMT 2025\.EOPDand REOPOLD instead regress on three of four benchmarks, losing1\.11\.1and2\.12\.1pp Avg@12 on average\.

### 4\.3Position\-schedule ablation

The main comparison fixes thePW\-OPSDschedule to\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\)\. Since the method introduces three scalar parameters, we also run a full schedule sweep over four settings to test whether the effect is tied to a single parameter tuple\.[Section˜4\.3](https://arxiv.org/html/2605.21606#S4.SS3)shows the schedule shapes;[Section˜4\.3](https://arxiv.org/html/2605.21606#S4.SS3)reports the completed Qwen3\-4B ablation evaluations across all four configurations under the same maximum generation length as the main table\.

The four settings vary how strongly early tokens are discounted\. Mild\(0\.50,0\.20,0\.20\)\(0\.50,0\.20,0\.20\)keeps a high early floor and uses the softest transition; Moderate\(0\.25,0\.30,0\.10\)\(0\.25,0\.30,0\.10\)is the chosen configuration from[Table˜2](https://arxiv.org/html/2605.21606#S4.T2); Sharp\(0\.10,0\.40,0\.05\)\(0\.10,0\.40,0\.05\)lowers the floor and makes the transition steeper; Aggressive\(0\.05,0\.50,0\.05\)\(0\.05,0\.50,0\.05\)applies the strongest early down\-weighting and delays the transition furthest\. Thus lowerwminw\_\{\\min\}makes the early\-token discount more aggressive, largerτ\\taumoves the transition later, and smallerssmakes it sharper\.

![[Uncaptioned image]](https://arxiv.org/html/2605.21606v1/x2.png)

ConfigBenchmarkPass@12Avg@12Maj@12MildMATH\-50098\.33±0\.1298\.33\\pm 0\.1295\.44±0\.0595\.44\\pm 0\.0596\.87±0\.1296\.87\\pm 0\.12AIME 202487\.78±3\.8587\.78\\pm 3\.8573\.98±0\.8573\.98\\pm 0\.8580\.00±0\.00\\mathbf\{80\.00\\pm 0\.00\}AIME 202584\.44±1\.92\\mathbf\{84\.44\\pm 1\.92\}67\.59±1\.1667\.59\\pm 1\.1673\.33±0\.0073\.33\\pm 0\.00HMMT 202562\.22±1\.9262\.22\\pm 1\.9245\.28±0\.7345\.28\\pm 0\.7347\.78±1\.9247\.78\\pm 1\.92ModerateMATH\-50098\.40±0\.00\\mathbf\{98\.40\\pm 0\.00\}95\.34±0\.1095\.34\\pm 0\.1096\.67±0\.1296\.67\\pm 0\.12AIME 202485\.56±1\.9285\.56\\pm 1\.9276\.20±0\.58\\mathbf\{76\.20\\pm 0\.58\}80\.00±0\.00\\mathbf\{80\.00\\pm 0\.00\}AIME 202583\.33±0\.0083\.33\\pm 0\.0067\.78±1\.2767\.78\\pm 1\.2774\.44±1\.9274\.44\\pm 1\.92HMMT 202560\.00±3\.3360\.00\\pm 3\.3343\.33±1\.2143\.33\\pm 1\.2152\.22±1\.92\\mathbf\{52\.22\\pm 1\.92\}SharpMATH\-50098\.13±0\.3198\.13\\pm 0\.3195\.36±0\.1595\.36\\pm 0\.1596\.60±0\.2096\.60\\pm 0\.20AIME 202488\.89±7\.70\\mathbf\{88\.89\\pm 7\.70\}75\.19±1\.5375\.19\\pm 1\.5380\.00±0\.00\\mathbf\{80\.00\\pm 0\.00\}AIME 202583\.33±0\.0083\.33\\pm 0\.0068\.80±0\.89\\mathbf\{68\.80\\pm 0\.89\}75\.56±3\.85\\mathbf\{75\.56\\pm 3\.85\}HMMT 202564\.44±1\.92\\mathbf\{64\.44\\pm 1\.92\}44\.26±1\.1244\.26\\pm 1\.1246\.67±3\.3346\.67\\pm 3\.33AggressiveMATH\-50098\.40±0\.20\\mathbf\{98\.40\\pm 0\.20\}95\.53±0\.04\\mathbf\{95\.53\\pm 0\.04\}97\.07±0\.12\\mathbf\{97\.07\\pm 0\.12\}AIME 202487\.78±1\.9287\.78\\pm 1\.9275\.19±0\.8575\.19\\pm 0\.8580\.00±0\.00\\mathbf\{80\.00\\pm 0\.00\}AIME 202583\.33±0\.0083\.33\\pm 0\.0067\.59±0\.8567\.59\\pm 0\.8575\.56±1\.92\\mathbf\{75\.56\\pm 1\.92\}HMMT 202560\.00±0\.0060\.00\\pm 0\.0045\.37±0\.58\\mathbf\{45\.37\\pm 0\.58\}52\.22±1\.92\\mathbf\{52\.22\\pm 1\.92\}
Figure 2:Per\-token weight schedules for the four configurations; top panels label the role of each\(wmin,τ,s\)\(w\_\{\\min\},\\tau,s\)knob\.Table 3:Position\-schedule sweep on Qwen3\-4B \(maximum generation length38,91238\{,\}912tokens\)\. Each cell is mean±\\pmstandard deviation across three evaluation seeds\. Bold marks the best value for each benchmark–metric column within the sweep\. Configurations: Mild\(0\.50,0\.20,0\.20\)\(0\.50,0\.20,0\.20\), Moderate\(0\.25,0\.30,0\.10\)\(0\.25,0\.30,0\.10\), Sharp\(0\.10,0\.40,0\.05\)\(0\.10,0\.40,0\.05\), Aggressive\(0\.05,0\.50,0\.05\)\(0\.05,0\.50,0\.05\)\.

On MATH\-500, every configuration sits within0\.20\.2pp Avg@12, well inside across\-seed noise\. On AIME 2024 Moderate holds the Avg@12 lead but the other three trail by at most2\.22\.2pp and all four tie on Maj@12; AIME 2025 Avg@12 is similarly clustered \(67\.5967\.59–68\.8068\.80\)\. HMMT 2025 separates the configurations more clearly: Aggressive and Mild gain\+1\.9\+1\.9–2\.02\.0pp Avg@12 over Moderate\. No configuration dominates uniformly\. We retain Moderate in the main table because it is the schedule chosen*a priori*from the diagnostic curves, not because it is the best across this sweep\. Averaged across the four benchmarks, all four schedules improve Avg@12 overOPSD\(70\.2770\.27\) by\+0\.30\+0\.30to\+0\.65\+0\.65pp and span only0\.350\.35pp among themselves, confirmingPW\-OPSDis robust to the choice of\(wmin,τ,s\)\(w\_\{\\min\},\\tau,s\)\.

#### Reduction vs\. positioning\.

PW\-OPSDdiffers fromOPSDalong two axes: a position\-dependent token weightwtw\_\{t\}\(positioning\) and a per\-rollout average of the token loss before the batch mean \(per\-sequence reduction\)\. The full2×22\{\\times\}2factorial in[Table˜6](https://arxiv.org/html/2605.21606#A12.T6)\(Appendix[L](https://arxiv.org/html/2605.21606#A12)\) shows the joint configuration is the only one that matches the AIME 2024 lead of[Table˜2](https://arxiv.org/html/2605.21606#S4.T2)\(\+1\.0\+1\.0pp Avg@12 overOPSD\)\.

### 4\.4Cross\-model transfer of the position schedule

To test whether the diagnostic\-derived Moderate schedule transfers beyond Qwen3\-4B, we evaluatePW\-OPSDon two larger models from different families: DeepSeek\-R1\-Distill\-Llama\-8B \(8B, Llama family\)\(Guoet al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib21); DeepSeek\-AI,[2025](https://arxiv.org/html/2605.21606#bib.bib40)\)and Olmo\-3\-7B\-Think \(7B, OLMo family\)\(Allen Institute for AI,[2026](https://arxiv.org/html/2605.21606#bib.bib41)\)\. For each model we comparePW\-OPSD\(Moderate,\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\)\) against theOPSDbaseline under the same training and evaluation protocols as the Qwen3\-4B main experiments: the full four\-benchmark suite \(MATH\-500, AIME 2024, AIME 2025, HMMT 2025\), three independent evaluation seeds per cell, and the maximum generation length of38,91238\{,\}912tokens described in[section˜4\.1](https://arxiv.org/html/2605.21606#S4.SS1)\. The Moderate schedule is held fixed across all three models with no per\-model retuning\. For DSR1\-L8B and Olmo\-3\-7B\-Think, both trained as reasoning models, we additionally apply a tokenizer\-level⟨think⟩\\langle\\text\{think\}\\rangle\-closure that replicatesenable\_thinking=Falseon chat templates that ignore the flag, matching the training\-time student prompt format used on Qwen3\-4B \(evaluation continues to useenable\_thinking=Trueas described in[section˜4\.1](https://arxiv.org/html/2605.21606#S4.SS1)\)\.

Table 4:Cross\-model Avg@12 evidence forPW\-OPSD\(Moderate, no per\-model retuning\) versus theOPSDbaseline ofZhaoet al\.\([2026](https://arxiv.org/html/2605.21606#bib.bib1)\)across three model families with maximum generation length38,91238\{,\}912tokens\. Entries report mean±\\pmacross\-seed sample standard deviation over three evaluation seeds; the Avg@12 column is the equal\-weight mean across the four benchmarks, computed from unrounded seed\-level values\. Bold marks the within\-model winner in each numeric column, including values within0\.10\.1pp of the leader\.[Table˜4](https://arxiv.org/html/2605.21606#S4.T4)shows that the same Moderate schedule gives positive Avg@12 gains on every tested model:\+0\.39\+0\.39pp on Qwen3\-4B \(4B parameters, Qwen family\),\+0\.35\+0\.35pp on DeepSeek\-R1\-Distill\-Llama\-8B \(8B, Llama family\), and\+0\.50\+0\.50pp on Olmo\-3\-7B\-Think \(7B, OLMo family\)\. The pattern spans Qwen, Llama, and OLMo checkpoints across a 4B–8B parameter range, with no per\-model schedule retuning, supporting the model\-family portability of the position\-rank reliability signal identified in[section˜3\.2](https://arxiv.org/html/2605.21606#S3.SS2)\.

## 5Conclusion, Limitations, and Future Work

In this paper, we study teacher\-token reliability in on\-policy self\-distillation\. To address whether high teacher entropy reflects low reliability or benign solution diversity, we design a branch\-viability diagnostic, showing empirically that reliability is positionally structured \(AUROC0\.830\.83vs\.≤0\.57\\leq 0\.57for local uncertainty\)\. Inspired by this finding, we introducePW\-OPSD, an increasing\-sigmoid position weight on theOPSDchassis\. Our experiments show thatPW\-OPSDimproves Avg@12 overOPSDon AIME 2024 \(\+1\.0\+1\.0pp\), AIME 2025 \(\+1\.1\+1\.1pp\), and on DeepSeek\-R1\-Distill\-Llama\-8B and Olmo\-3\-7B\-Think\. One limitation is thatPW\-OPSDis plug\-and\-play yet its gains overOPSDremain modest, leaving room to design more sophisticated methods built on the positional reliability finding\. To improve the proposed method, we aim to design more dedicated objectives that go beyond pure position weighting, such as position\-conditioned mixing of forward and reverse KL\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. Bachem \(2023\)On\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes\.arXiv preprint arXiv:2306\.13649\.Note:Accepted at ICLR 2024\. First two authors contributed equallyExternal Links:2306\.13649,[Link](https://arxiv.org/abs/2306.13649)Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2605.21606#S3.SS3.p1.9)\.
- Allen Institute for AI \(2026\)Olmo\-3\-7B\-Think\.Note:[https://huggingface\.co/allenai/Olmo\-3\-7B\-Think](https://huggingface.co/allenai/Olmo-3-7B-Think)Hugging Face model cardCited by:[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4),[§4\.4](https://arxiv.org/html/2605.21606#S4.SS4.p1.3)\.
- B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. R’e, and A\. Mirhoseini \(2024\)Large language monkeys: scaling inference compute with repeated sampling\.ArXivabs/2407\.21787\.Cited by:[Appendix J](https://arxiv.org/html/2605.21606#A10.SS0.SSS0.Px1.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. Pondé, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, I\. Babuschkin, S\. Balaji, S\. Jain, A\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.ArXivabs/2107\.03374\.Cited by:[Appendix J](https://arxiv.org/html/2605.21606#A10.SS0.SSS0.Px1.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1\-Distill\-Llama\-8B\.Note:[https://huggingface\.co/deepseek\-ai/DeepSeek\-R1\-Distill\-Llama\-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B)Hugging Face model cardCited by:[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4),[§4\.4](https://arxiv.org/html/2605.21606#S4.SS4.p1.3)\.
- J\. Dekoninck, N\. Jovanović, T\. Gehrunger, K\. Rögnvalddson, I\. Petrov, C\. Sun, and M\. Vechev \(2026\)Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs\.arXiv preprint arXiv:2605\.00674\.External Links:2605\.00674,[Link](https://arxiv.org/abs/2605.00674)Cited by:[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px6.p1.3),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px2.p1.12)\.
- Y\. Gal and Z\. Ghahramani \(2015\)Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning\.arXiv preprint arXiv:1506\.02142\.Note:12 pages, 6 figures; fixed a mistake with standard error and added a new table with updated results \(marked "Update \[October 2016\]"\); Published in ICML 2016External Links:1506\.02142,[Link](https://arxiv.org/abs/1506.02142)Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px2.p3.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2023\)MiniLLM: On\-Policy Distillation of Large Language Models\.arXiv preprint arXiv:2306\.08543\.Note:Published as a conference paper in ICLR 2024External Links:2306\.08543,[Link](https://arxiv.org/abs/2306.08543)Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2605.21606#S3.SS3.p1.9)\.
- A\. Gudibande, E\. Wallace, C\. B\. Snell, X\. Geng, H\. Liu, P\. Abbeel, S\. Levine, and D\. Song \(2023\)The false promise of imitating proprietary llms\.ArXivabs/2305\.15717\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Guha, R\. Marten, S\. S\. Keh, N\. Raoof, G\. Smyrnis, H\. Bansal, M\. Nezhurina, J\. Mercat, T\. Vu, Z\. Sprague, A\. Suvarna, B\. Feuer, L\. Chen, Z\. Khan, E\. Frankel, S\. Grover, C\. Choi, N\. Muennighoff, S\. Su, W\. Zhao, J\. Yang, S\. Pimpalgaonkar, K\. Sharma, C\. C\. Ji, Y\. Deng, S\. Pratt, V\. Ramanujan, J\. Saad\-Falcon, J\. Li, A\. Dave, A\. Albalak, K\. Arora, B\. Wulfe, C\. Hegde, G\. Durrett, S\. Oh, M\. Bansal, S\. Gabriel, A\. Grover, K\. Chang, V\. Shankar, A\. Gokaslan, M\. A\. Merrill, T\. Hashimoto, Y\. Choi, J\. Jitsev, R\. Heckel, M\. Sathiamoorthy, A\. G\. Dimakis, and L\. Schmidt \(2025\)OpenThoughts: data recipes for reasoning models\.ArXivabs/2506\.04178\.Cited by:[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang \(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4),[§4\.4](https://arxiv.org/html/2605.21606#S4.SS4.p1.3)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.ArXivabs/2103\.03874\.Cited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2),[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px6.p1.3),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px2.p1.12)\.
- G\. E\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.ArXivabs/1503\.02531\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px2.p1.14),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4)\.
- HuggingFaceH4 \(2024\)MATH\-500\.Note:[https://huggingface\.co/datasets/HuggingFaceH4/MATH\-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)Hugging Face dataset; MATH\-500 subset from PRM800K/MATH provenanceCited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2),[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px6.p1.3),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px2.p1.12)\.
- HuggingFaceH4 \(2025\)AIME 2024\.Note:[https://huggingface\.co/datasets/HuggingFaceH4/aime\_2024](https://huggingface.co/datasets/HuggingFaceH4/aime_2024)Hugging Face dataset for the 2024 competition; derived from AI\-MO/aimo\-validation\-aimeCited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2),[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px6.p1.3),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px2.p1.12)\.
- W\. Jin, T\. Min, Y\. Yang, S\. R\. Kadhe, Y\. Zhou, D\. Wei, N\. Baracaldo, and K\. Lee \(2026\)Entropy\-Aware On\-Policy Distillation of Language Models\.arXiv preprint arXiv:2603\.07079\.Note:16 pages, 11 figures, preprintExternal Links:2603\.07079,[Link](https://arxiv.org/abs/2603.07079)Cited by:[§F\.1](https://arxiv.org/html/2605.21606#A6.SS1.p2.1),[Table 5](https://arxiv.org/html/2605.21606#A6.T5.4.6.1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.21606#S1.p2.1),[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px3.p1.1),[§3\.3](https://arxiv.org/html/2605.21606#S3.SS3.p1.9),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4),[Table 2](https://arxiv.org/html/2605.21606#S4.T2.21.15.4.1)\.
- S\. Jung, S\. Yoon, D\. Kim, and H\. Lee \(2025\)ToDi: Token\-wise Distillation via Fine\-Grained Divergence Control\.arXiv preprint arXiv:2505\.16297\.Note:EMNLP 2025 \(Oral\)External Links:2505\.16297,[Link](https://arxiv.org/abs/2505.16297)Cited by:[§1](https://arxiv.org/html/2605.21606#S1.p2.1),[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px3.p1.1),[§3\.3](https://arxiv.org/html/2605.21606#S3.SS3.p1.9)\.
- A\. Kendall and Y\. Gal \(2017\)What uncertainties do we need in bayesian deep learning for computer vision?\.ArXivabs/1703\.04977\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px2.p3.1)\.
- Y\. Kim and A\. M\. Rush \(2016\)Sequence\-level knowledge distillation\.ArXivabs/1606\.07947\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Ko, S\. Abdali, Y\. J\. Kim, T\. Chen, and P\. Cameron \(2026\)Scaling Reasoning Efficiently via Relaxed On\-Policy Distillation\.arXiv preprint arXiv:2603\.11137\.External Links:2603\.11137,[Link](https://arxiv.org/abs/2603.11137)Cited by:[§F\.1](https://arxiv.org/html/2605.21606#A6.SS1.p2.1),[Appendix F](https://arxiv.org/html/2605.21606#A6.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.21606#S1.p2.1),[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4),[Table 2](https://arxiv.org/html/2605.21606#S4.T2.33.27.4.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: towards streamlined distillation for large language models\.ArXivabs/2402\.03898\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2605.21606#S3.SS3.p1.9)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient Memory Management for Large Language Model Serving with PagedAttention\.arXiv preprint arXiv:2309\.06180\.Note:SOSP 2023External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2)\.
- H\. Kydlíček \(2025\)Math\-Verify: math verification library\.Note:[https://github\.com/huggingface/math\-verify](https://github.com/huggingface/math-verify)Version 0\.6\.1Cited by:[Appendix J](https://arxiv.org/html/2605.21606#A10.SS0.SSS0.Px2.p1.8),[2nd item](https://arxiv.org/html/2605.21606#A11.I1.i2.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px5.p1.3)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2),[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px6.p1.3),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px2.p1.12)\.
- A\. Malinin and M\. Gales \(2018\)Predictive Uncertainty Estimation via Prior Networks\.arXiv preprint arXiv:1802\.10501\.External Links:1802\.10501,[Link](https://arxiv.org/abs/1802.10501)Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px2.p3.1)\.
- A\. Meurer, C\. P\. Smith, M\. Paprocki, O\. Čertík, S\. B\. Kirpichev, M\. Rocklin, A\. Kumar, S\. Ivanov, J\. K\. Moore, S\. Singh,et al\.\(2017\)SymPy: symbolic computing in python\.PeerJ Computer Science3,pp\. e103\.Cited by:[Appendix J](https://arxiv.org/html/2605.21606#A10.SS0.SSS0.Px2.p1.8),[2nd item](https://arxiv.org/html/2605.21606#A11.I1.i2.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px5.p1.3)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. E\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. J\. Lowe \(2022\)Training language models to follow instructions with human feedback\.ArXivabs/2203\.02155\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. Chintala \(2019\)PyTorch: an imperative style, high\-performance deep learning library\.ArXivabs/1912\.01703\.Cited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2)\.
- Qwen Team \(2025\)Qwen3\-4B\.Note:[https://huggingface\.co/Qwen/Qwen3\-4B](https://huggingface.co/Qwen/Qwen3-4B)Hugging Face model cardCited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4)\.
- S\. Ross, G\. J\. Gordon, and J\. A\. Bagnell \(2010\)A Reduction of Imitation Learning and Structured Prediction to No\-Regret Online Learning\.arXiv preprint arXiv:1011\.0686\.Note:Appearing in the 14th International Conference on Artificial Intelligence and Statistics \(AISTATS 2011\)External Links:1011\.0686,[Link](https://arxiv.org/abs/1011.0686)Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Sensoy, M\. Kandemir, and L\. M\. Kaplan \(2018\)Evidential deep learning to quantify classification uncertainty\.ArXivabs/1806\.01768\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px2.p3.1)\.
- siyanzhao \(2026\)OpenThoughts\-Math\-30k OPSD\.Note:[https://huggingface\.co/datasets/siyanzhao/Openthoughts\_math\_30k\_opsd](https://huggingface.co/datasets/siyanzhao/Openthoughts_math_30k_opsd)Hugging Face dataset; train split with 29,434 examples; loaded by the OPSD training codeCited by:[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[Appendix H](https://arxiv.org/html/2605.21606#A8.SS0.SSS0.Px1.p1.2)\.
- M\. Song and M\. Zheng \(2026\)A Survey of On\-Policy Distillation for Large Language Models\.arXiv preprint arXiv:2604\.00626\.External Links:2604\.00626,[Link](https://arxiv.org/abs/2604.00626)Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Stein, F\. Huang, and T\. Goldstein \(2026\)GATES: self\-distillation under privileged context with consensus gating\.ArXivabs/2602\.20574\.Cited by:[§1](https://arxiv.org/html/2605.21606#S1.p2.1),[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2605.21606#S3.SS2.SSS0.Px3.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. H\. Chi, and D\. Zhou \(2022\)Self\-consistency improves chain of thought reasoning in language models\.ArXivabs/2203\.11171\.Cited by:[Appendix J](https://arxiv.org/html/2605.21606#A10.SS0.SSS0.Px1.p1.1)\.
- Y\. Wen, Z\. Li, W\. Du, and L\. Mou \(2023\)F\-divergence minimization for sequence\-level knowledge distillation\.ArXivabs/2307\.15190\.Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2605.21606#S3.SS3.p1.9)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. Rush \(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Q\. Liu and D\. Schlangen \(Eds\.\),Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.emnlp-demos.6/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2)\.
- T\. Wu, C\. Tao, J\. Wang, R\. Yang, Z\. Zhao, and N\. Wong \(2024\)Rethinking Kullback\-Leibler Divergence in Knowledge Distillation for Large Language Models\.arXiv preprint arXiv:2404\.02657\.Note:COLING 2025External Links:2404\.02657,[Link](https://arxiv.org/abs/2404.02657)Cited by:[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2605.21606#S3.SS3.p1.9)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.21606#S1.p6.4),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4)\.
- yentinglin \(2025\)AIME 2025\.Note:[https://huggingface\.co/datasets/yentinglin/aime\_2025](https://huggingface.co/datasets/yentinglin/aime_2025)Hugging Face datasetCited by:[Appendix A](https://arxiv.org/html/2605.21606#A1.SS0.SSS0.Px1.p1.2),[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px6.p1.3),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px2.p1.12)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.ArXivabs/2601\.18734\.Cited by:[1st item](https://arxiv.org/html/2605.21606#A11.I1.i1.p1.1),[§F\.1](https://arxiv.org/html/2605.21606#A6.SS1.p2.1),[Table 5](https://arxiv.org/html/2605.21606#A6.T5.2.2.3),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px2.p1.14),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px3.p1.1),[Appendix I](https://arxiv.org/html/2605.21606#A9.SS0.SSS0.Px4.p1.6),[§1](https://arxiv.org/html/2605.21606#S1.p1.1),[§1](https://arxiv.org/html/2605.21606#S1.p2.1),[§1](https://arxiv.org/html/2605.21606#S1.p8.7),[§2](https://arxiv.org/html/2605.21606#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2605.21606#S3.SS1.p1.2),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px1.p1.4),[§4\.1](https://arxiv.org/html/2605.21606#S4.SS1.SSS0.Px2.p1.12),[Table 2](https://arxiv.org/html/2605.21606#S4.T2.9.3.4.1),[Table 4](https://arxiv.org/html/2605.21606#S4.T4)\.

## Appendix ABranch\-viability protocol details

#### Models, prompts, and software\.

The diagnostic uses oneQwen/Qwen3\-4Bcheckpoint\[Qwen Team,[2025](https://arxiv.org/html/2605.21606#bib.bib20)\]\(HuggingFace snapshot1cfa9a72…3b3df60c, dtypebfloat16\) with no adapters loaded\. “Teacher” and “student” denote two prompt templates applied to this single checkpoint: the teacher template includes the privileged ground\-truth answer, and the student template is the ordinary problem prompt\. vLLM and HuggingFace load this checkpoint as separate software backends, not as different trained weights\. Problems are drawn from three sources, sampled without replacement \(see attrition below\):MATH\-500test split\[Hendryckset al\.,[2021](https://arxiv.org/html/2605.21606#bib.bib28), Lightmanet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib18), HuggingFaceH4,[2024](https://arxiv.org/html/2605.21606#bib.bib36)\],AIME 2024\[HuggingFaceH4,[2025](https://arxiv.org/html/2605.21606#bib.bib37)\], andAIME 2025\[yentinglin,[2025](https://arxiv.org/html/2605.21606#bib.bib38)\]\. We usevllm 0\.11\.0withtensor\_parallel\_size=4\(TP=4=4\) for generation\[Kwonet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib29)\],transformers 4\.57\.1for the HuggingFace \(HF\) teacher forward passes\[Wolfet al\.,[2020](https://arxiv.org/html/2605.21606#bib.bib30)\], andtorch 2\.8\.0\+cu128on4×4\{\\times\}H100 80GB GPUs\[Paszkeet al\.,[2019](https://arxiv.org/html/2605.21606#bib.bib31)\]\.

#### Problem attrition\.

Phase A samples2424MATH\-500,3030AIME 2024, and3030AIME 2025 problems \(8484total\)\. Phase B produces23\+21\+18=6223\+21\+18=62correct\-spine problems\. After Phase F labeling, one AIME 2024 correct\-spine problem has only gray candidates, so the binary\-labeled pool used in[table˜1](https://arxiv.org/html/2605.21606#S3.T1)contains23\+20\+18=6123\+20\+18=61usable problems\. Phase C then proposes up to55high\-truncated\-entropy candidates per problem \(with the spacing / plausibility filters below\); Phase E rolls out66student continuations per \(candidate, alternative\) pair\. The[table˜1](https://arxiv.org/html/2605.21606#S3.T1)report restricts to high\-truncated\-entropy candidate positions and to the88candidates labeled*real\-uncertain*plus271271*diversity*candidates across these6161problems\.[Figure˜1](https://arxiv.org/html/2605.21606#S1.F1)uses the broader continuous\-viability subset before binary thresholding\.

#### Random seeds\.

Phase A fallback sampling, Phase E forced continuations, and ther~\\widetilde\{r\}residualization use vLLM seed0; Phase D MC\-dropout masks use a per\-position random state derived from the model’s default generator\. The cluster bootstrap in Phase G uses NumPy seed0\.

#### Score residualization and AUROC\.

Within each problempp, every raw uncertainty scoreuu\(or its sign\-oriented form for position\) is mean\-centered:u′=u−u¯pu^\{\\prime\}=u\-\\bar\{u\}\_\{p\}\. We compute AUROC on the residualized scores against the binary label defined in Phase F, using the standardMann​\-​Whitney​U\\mathrm\{Mann\\text\{\-\}Whitney\}\\,Uformula with mid\-rank tie handling\. The cluster bootstrap resamples problems \(not candidates\) with replacement and recomputes the within\-problem residualization on each resample, giving the95%95\\%intervals reported in[table˜1](https://arxiv.org/html/2605.21606#S3.T1)\.

- •Phase A\.vLLM \(TP=44\) student greedy decode of8484problems in total \(2424from MATH\-500,3030from AIME 2024, and3030from AIME 2025; the protocol is run independently per dataset and the labeled candidates are pooled in Phase G\), with fallback toT=0\.7T\{=\}0\.7for problems where greedy does not reach\\boxed\\backslash\\mathrm\{boxed\}within1616K tokens\.
- •Phase B\.HF forward of the same checkpoint under the teacher template \(privileged\-answer prompt\+\+spine\), giving per\-position teacher distributions and top\-M=16M\{=\}16valid\-token truncated entropy\.
- •Phase C\.Apply junk / EOS / special filters, plausibility filterp2≥0\.02∧p2/p1≥0\.10p\_\{2\}\\geq 0\.02\\wedge p\_\{2\}/p\_\{1\}\\geq 0\.10,6464\-token spacing constraint, and skip post\-\\boxed\\backslash\\mathrm\{boxed\}positions\. Keep up to55high\-truncated\-entropy candidate positions per problem; for each, record the teacher pass’s top\-33valid child tokens as forced alternatives\.
- •Phase D\.HF teacher MC dropout \(M=5M\{=\}5,p=0\.1p\{=\}0\.1, last44layers\) at each candidate; recordHfullH\_\{\\mathrm\{full\}\}, MI, andlog⁡κ^\\log\\hat\{\\kappa\}\.
- •Phase E\.vLLM \(TP=44\)*student*\-context forced continuation: each forced child token from Phase C is appended to the student\-template prompt\+\+spine truncated to the candidate position, and rollouts continue under the student template \(no privileged information\)\. We use student\-template continuation rather than teacher\-template continuation because a teacher\-template continuation can re\-use the privileged answer in its prompt and recover from almost any forced child, collapsing the labels toward diversity\. Across the three datasets, up to\(24\+30\+30\)×5×3×6=7560\(24\+30\+30\)\\times 5\\times 3\\times 6=7560attempted rollouts atT=1\.0T\{=\}1\.0,top\_p=0\.95=0\.95, withmax\_tokensdynamically clipped per request to fitmax\_model\_len=32=32K\.
- •Phase F\.For each forced child, define viability as the fraction of its66student\-context continuations whose extracted boxed answer matches the ground truth\. Label a candidate*diversity*if at least two of its children have viability≥Vhigh=0\.75\\geq V\_\{\\mathrm\{high\}\}\{=\}0\.75; label it*real\-uncertain*if every child has viability<Vlow=0\.40<V\_\{\\mathrm\{low\}\}\{=\}0\.40and the mean child viability is<Vlow<V\_\{\\mathrm\{low\}\}; otherwise label “gray” and exclude from the AUROC\.
- •Phase G\.AUROC \+ area\-under\-precision\-recall\-curve \(AUPRC\) \+ cluster\-bootstrap by problem \(20002000resamples, multiplicity\-preserved\); scatter and histograms\.

#### Privileged\-info teacher prompt\.

We use the following template \(wrapped in the Qwen3 chat template withenable\_thinking=True\):

> You are a privileged teacher\. Solve the problem and end your solution with the correct boxed answer\. Problem:\{problem\} Privileged ground\-truth final answer:\{answer\} Use the ground\-truth answer above\. Produce a complete step\-by\-step solution that ends with\\boxed\{\{\\backslash\\mathrm\{boxed\}\\\{\\\{answer\}\}\\\}\\\}\.

This is a simplified privileged\-info injection in the spirit ofOPSD’s training\-time teacher prompt; it is not pixel\-identical to theOPSDofficial template but conditions the teacher’s autoregressive distribution on the ground\-truth answer for the duration of generation\.

## Appendix BMC\-dropout diagnostic implementation

MC dropout is diagnostic\-only in this paper\. It is used to computeHfullH\_\{\\mathrm\{full\}\}, MI, andlog⁡κ^\\log\\hat\{\\kappa\}scores for[table˜1](https://arxiv.org/html/2605.21606#S3.T1); it is not used to construct a training target or a token weight\. All training methods use the ordinary single\-pass teacher target𝐩topsd\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{t\}\.

#### MC dropout via forward hooks\.

Modern Qwen / Llama transformer blocks do not expose annn\.Dropoutsubmodule\. We inject MC dropout at inference time via forward hooks on the lastL=4L=4transformer layers; the hook appliesF\.dropout​\(h,p,training=True\)\\mathrm\{F\.dropout\}\(h,p,\\mathrm\{training\}\{=\}\\mathrm\{True\}\)to the layer output\. We usep=0\.1p=0\.1throughout andM=5M=5MC samples\. The unperturbed privileged\-teacher forward provides𝐩topsd\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{t\}; the perturbed forwards are used only for the diagnostic uncertainty scores\. Hooks are removed in atry/finallyblock to prevent leaking dropout into subsequent forward passes\.

#### Numerical stability ofκ^t\\hat\{\\kappa\}\_\{t\}\.

The moment\-matching estimator forκ^t\\hat\{\\kappa\}\_\{t\}can produce negative values when the sample variance trace exceeds the categorical variance trace1−‖𝐩¯t‖221\-\\\|\\bar\{\\mathbf\{p\}\}\_\{t\}\\\|\_\{2\}^\{2\}\(an artifact of smallMM\)\. For[table˜1](https://arxiv.org/html/2605.21606#S3.T1), we do not apply any training\-style floor or log\-normalization\. We store the rawκ^t\\hat\{\\kappa\}\_\{t\}estimate and computelog⁡max⁡\(κ^t,ϵ\)\\log\\max\(\\hat\{\\kappa\}\_\{t\},\\epsilon\)only at score time, withϵ=10−6\\epsilon=10^\{\-6\}for the log transform\.

#### Tail\-bucket aggregation\.

For very large vocabularies \(Qwen3 vocab≈152\\approx 152K\), computing the full softmax per MC sample at every position is memory\-bound\. We compute moment statistics token\-by\-token and aggregate∑m‖𝐩t\(m\)‖22\\sum\_\{m\}\\\|\\mathbf\{p\}^\{\(m\)\}\_\{t\}\\\|\_\{2\}^\{2\}incrementally to avoid materializing the full per\-sample distribution beyond one forward at a time\. In Phase B of the branch\-viability experiment we still materialize the per\-position softmax once for top\-MMselection; this is a one\-time pass per problem\.

## Appendix CReliability\-weighted surrogate details

[Section˜3\.3](https://arxiv.org/html/2605.21606#S3.SS3)conditions the reliability posterior on the distillation contextct=\(xstu,xtch,y<t\)c\_\{t\}=\(x\_\{\\mathrm\{stu\}\},x\_\{\\mathrm\{tch\}\},y\_\{<t\}\)\. This conditioning makes the tower\-property step explicit\. For fixedθ\\thetaand a fixed teacher, the token divergenceDt​\(θ\)=KL⁡\(qt∥pt\)D\_\{t\}\(\\theta\)=\\operatorname\{KL\}\(q\_\{t\}\\,\\\|\\,p\_\{t\}\)is determined byctc\_\{t\}, so

𝔼​\[It​Dt​\(θ\)\]=𝔼​\[𝔼​\[It​Dt​\(θ\)∣ct\]\]=𝔼​\[Dt​\(θ\)​Pr⁡\(It=1∣ct\)\]\.\\mathbb\{E\}\[I\_\{t\}D\_\{t\}\(\\theta\)\]=\\mathbb\{E\}\\\!\\left\[\\mathbb\{E\}\[I\_\{t\}D\_\{t\}\(\\theta\)\\mid c\_\{t\}\]\\right\]=\\mathbb\{E\}\\\!\\left\[D\_\{t\}\(\\theta\)\\Pr\(I\_\{t\}=1\\mid c\_\{t\}\)\\right\]\.Conditioning only on the student prefixht=\(xstu,y<t\)h\_\{t\}=\(x\_\{\\mathrm\{stu\}\},y\_\{<t\}\)would require an additional assumption, because the privileged teacher target can vary with the teacher prompt and reference information even when the student prefix is fixed\.

The risk in[eq\.˜2](https://arxiv.org/html/2605.21606#S3.E2)is also a distillation surrogate on sampled rollouts\. As in standardOPSDimplementations, the rollout is sampled from the current student and then treated as a fixed training example for the KL update; gradients pass through the student probabilities on visited prefixes, not through the sampling operation that produced those prefixes\. The main\-text interpretation uses the unclipped divergenceDt​\(θ\)D\_\{t\}\(\\theta\)for notational clarity, while the implemented objectives use the element\-wise clipped forward\-KL surrogate described in[eq\.˜1](https://arxiv.org/html/2605.21606#S3.E1)\.

## Appendix DBranch\-mixture identity and interpretation

[Equation˜4](https://arxiv.org/html/2605.21606#S3.E4)gives the token\-level identity used in the main text\. With joint distributionq​\(z,yt∣ht\)=α​\(z∣ht\)​qtz​\(yt\)q\(z,y\_\{t\}\\mid h\_\{t\}\)=\\alpha\(z\\mid h\_\{t\}\)q\_\{t\}^\{z\}\(y\_\{t\}\)and marginalqt​\(yt\)=∑zα​\(z∣ht\)​qtz​\(yt\)q\_\{t\}\(y\_\{t\}\)=\\sum\_\{z\}\\alpha\(z\\mid h\_\{t\}\)q\_\{t\}^\{z\}\(y\_\{t\}\),

𝔼z∼α​KL⁡\(qtz∥pt\)\\displaystyle\\mathbb\{E\}\_\{z\\sim\\alpha\}\\operatorname\{KL\}\(q\_\{t\}^\{z\}\\,\\\|\\,p\_\{t\}\)=∑z,ytq​\(z,yt∣ht\)​log⁡qtz​\(yt\)pt​\(yt\)\\displaystyle=\\sum\_\{z,y\_\{t\}\}q\(z,y\_\{t\}\\mid h\_\{t\}\)\\log\\frac\{q\_\{t\}^\{z\}\(y\_\{t\}\)\}\{p\_\{t\}\(y\_\{t\}\)\}=KL⁡\(qt∥pt\)\+Iq​\(Yt;Z∣ht\)\.\\displaystyle=\\operatorname\{KL\}\(q\_\{t\}\\,\\\|\\,p\_\{t\}\)\+I\_\{q\}\(Y\_\{t\};Z\\mid h\_\{t\}\)\.The mutual\-information term is independent ofptp\_\{t\}\. The identity therefore decomposes branch\-specific variation hidden by the marginal teacher target; it is not, by itself, a proof that marginal forward KL has a different gradient objective\. The empirical branch\-viability diagnostic supplies the task\-specific direction used byPW\-OPSD: early high\-ambiguity positions more often correspond to unreliable supervision\.

The same branch\-mixture view yields a sequence\-level form\. Let the teacher’s branch\-conditioned next\-token distribution beqtz​\(yt∣y<t\)q\_\{t\}^\{z\}\(y\_\{t\}\\mid y\_\{<t\}\), and let the corresponding sequence distribution factorise autoregressively as

qz​\(y1:T\)=∏t=1Tqtz​\(yt∣y<t\)\.q^\{z\}\(y\_\{1:T\}\)=\\prod\_\{t=1\}^\{T\}q\_\{t\}^\{z\}\(y\_\{t\}\\mid y\_\{<t\}\)\.Its marginal under the latent branch priorα\\alphaisq​\(y1:T\)=∑zα​\(z\)​qz​\(y1:T\)q\(y\_\{1:T\}\)=\\sum\_\{z\}\\alpha\(z\)\\,q^\{z\}\(y\_\{1:T\}\)\. For a student sequence distributionp​\(y1:T\)=∏t=1Tpt​\(yt∣y<t\)p\(y\_\{1:T\}\)=\\prod\_\{t=1\}^\{T\}p\_\{t\}\(y\_\{t\}\\mid y\_\{<t\}\),

𝔼z∼α​\[KL⁡\(qz∥p\)\]=KL⁡\(q∥p\)\+∑tIq​\(Yt;Z∣Y<t\)\.\\mathbb\{E\}\_\{z\\sim\\alpha\}\\\!\\left\[\\operatorname\{KL\}\(q^\{z\}\\,\\\|\\,p\)\\right\]=\\operatorname\{KL\}\(q\\,\\\|\\,p\)\+\\sum\_\{t\}I\_\{q\}\(Y\_\{t\};Z\\mid Y\_\{<t\}\)\.\(7\)Thus the value of branch\-conditioned sequence forward KL differs from marginal sequence forward KL by the sum of conditional mutual\-information terms between the next token and the latent branch\. As in the token\-level identity, this sum quantifies branch ambiguity but is independent of the student distribution\.

## Appendix EPosition\-schedule hyperparameters

The Moderate schedule used in the main table sets\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\), placing the transition near the empirical early\-to\-late change in the branch\-viability diagnostic while keeping a nonzero floor on early\-token supervision\. The full position sweep in[section˜4\.3](https://arxiv.org/html/2605.21606#S4.SS3)covers four schedules:Mild\(0\.50,0\.20,0\.20\)\(0\.50,0\.20,0\.20\),Moderate\(0\.25,0\.30,0\.10\)\(0\.25,0\.30,0\.10\)\(the a\-priori choice from the diagnostic curve in[fig\.˜1](https://arxiv.org/html/2605.21606#S1.F1)\),Sharp\(0\.10,0\.40,0\.05\)\(0\.10,0\.40,0\.05\), andAggressive\(0\.05,0\.50,0\.05\)\(0\.05,0\.50,0\.05\)\. The main table reports both Moderate \(the a\-priori headline\) and Aggressive \(the sweep configuration that recovers the HMMT 2025 cell on Avg@12\)\. We did not tune\(wmin,τ,s\)\(w\_\{\\min\},\\tau,s\)on any evaluation benchmark; the four configurations were chosen before observing any of the Avg@12/Pass@12/Maj@12 values\.

## Appendix FAdaptive\-loss template and method comparison

[Table˜5](https://arxiv.org/html/2605.21606#A6.T5)compares the three per\-token distribution\-matching objectives evaluated in[table˜2](https://arxiv.org/html/2605.21606#S4.T2)along the dimensions that the template makes explicit: target distribution, adaptive signal or reliability proxy, and reduction over tokens\. REOPOLD\[Koet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib9)\]is also evaluated in[table˜2](https://arxiv.org/html/2605.21606#S4.T2); it does not fit this template because its gradient flows through the rolled\-out token’s log\-prob rather than through a distributional forward KL\. Its role in the comparison is summarized in[section˜F\.1](https://arxiv.org/html/2605.21606#A6.SS1)\.

### F\.1Comparison to Related Adaptive Losses

[Algorithm˜1](https://arxiv.org/html/2605.21606#alg1)\([Appendix˜G](https://arxiv.org/html/2605.21606#A7)\) gives the explicit one\-step pseudocode\. For reference, the scalar\-weighted clipped\-FKL template is

ℒwFKL=1\|ℳ\|​∑t∈ℳwt​∑j∈𝒱min⁡\(𝐩t​\(j\)​log⁡𝐩t​\(j\)πθ​\(j∣xstu,y<t\),τclip\)\.\\mathcal\{L\}\_\{\\mathrm\{wFKL\}\}=\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\sum\_\{t\\in\\mathcal\{M\}\}w\_\{t\}\\sum\_\{j\\in\\mathcal\{V\}\}\\min\\\!\\left\(\\mathbf\{p\}\_\{t\}\(j\)\\log\\frac\{\\mathbf\{p\}\_\{t\}\(j\)\}\{\\pi\_\{\\theta\}\(j\\mid x\_\{\\mathrm\{stu\}\},y\_\{<t\}\)\},\\tau\_\{\\mathrm\{clip\}\}\\right\)\.\(8\)Different adaptive losses instantiate or modify this template through the choice of target, reliability proxy, inner divergence, and reduction\.

OPSDuses a uniform weight\[Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\]\.EOPD\[Jinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib3)\]gates a forward\-KL augmentation to reverse KL using teacher entropy\.PW\-OPSDkeeps the OPSD forward\-KL inner loss and uses the position schedulewi,tw\_\{i,t\}from[eq\.˜5](https://arxiv.org/html/2605.21606#S3.E5)in its outer token aggregation\. REOPOLD is also included in the experimental comparison but is a policy\-gradient distillation variant rather than a per\-token forward\-KL weighting rule\[Koet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib9)\], so it does not instantiate the template above;[section˜4\.1](https://arxiv.org/html/2605.21606#S4.SS1)states the cross\-family rationale\.[Table˜5](https://arxiv.org/html/2605.21606#A6.T5)\(Appendix[F](https://arxiv.org/html/2605.21606#A6)\) lays out the three per\-token distribution\-matching objectives in a side\-by\-side table\.

Table 5:Per\-token distribution\-matching distillation objectives compared in[table˜2](https://arxiv.org/html/2605.21606#S4.T2), parametrized by target distribution, adaptive signal or reliability proxy, and reduction over tokens\. REOPOLD is also evaluated in[table˜2](https://arxiv.org/html/2605.21606#S4.T2)but follows a policy\-gradient route rather than a per\-token forward\-KL weighting, so it is not instantiated in this template\.

## Appendix GPW\-OPSD training pseudocode

[Algorithm˜1](https://arxiv.org/html/2605.21606#alg1)gives one training step ofPW\-OPSD\. The procedure computes student and teacher log\-probabilities at the distillation temperature, forms the forward\-KL tensor with no reduction, clamps each vocabulary element, sums over the vocabulary, applies the position weight, averages over valid tokens within each sequence, and then averages over valid sequences\. The sampled rollout is fixed for this update, as discussed in[Appendix˜C](https://arxiv.org/html/2605.21606#A3)\.

Algorithm 1One training step ofPW\-OPSD\.1:Parameters

θ\\theta, prompts

\{\(xstui,xtchi\)\}i=1B\\\{\(x^\{i\}\_\{\\mathrm\{stu\}\},x^\{i\}\_\{\\mathrm\{tch\}\}\)\\\}\_\{i=1\}^\{B\}, schedule

\(wmin,τ,s\)\(w\_\{\\min\},\\tau,s\), distillation temperature

TdistillT\_\{\\mathrm\{distill\}\}, clip

τclip\\tau\_\{\\mathrm\{clip\}\}\.

2:Sample student rollouts

yi∼πθ\(⋅∣xstui\)y^\{i\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x^\{i\}\_\{\\mathrm\{stu\}\}\)\.

3:Let

LiL\_\{i\}be the number of valid generated tokens before EOS or truncation\.

4:Score visited prefixes at

TdistillT\_\{\\mathrm\{distill\}\}under the privileged teacher context and ordinary student context to obtain

𝐩i,topsd\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{i,t\}and

pi,tp\_\{i,t\}for each valid

tt\.

5:foreach sequence

iiand valid token

t∈\{1,…,Li\}t\\in\\\{1,\\ldots,L\_\{i\}\\\}do

6:

ri,t←\(t−0\.5\)/Lir\_\{i,t\}\\leftarrow\(t\-0\.5\)/L\_\{i\}
7:

wi,t←wmin\+\(1−wmin\)​σ​\(\(ri,t−τ\)/s\)w\_\{i,t\}\\leftarrow w\_\{\\min\}\+\(1\-w\_\{\\min\}\)\\sigma\(\(r\_\{i,t\}\-\\tau\)/s\)
8:

ℓi,t←∑j∈𝒱min⁡\(𝐩i,topsd​\(j\)​log⁡𝐩i,topsd​\(j\)pi,t​\(j\),τclip\)\\ell\_\{i,t\}\\leftarrow\\sum\_\{j\\in\\mathcal\{V\}\}\\min\\\!\\left\(\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{i,t\}\(j\)\\log\\frac\{\\mathbf\{p\}^\{\\mathrm\{opsd\}\}\_\{i,t\}\(j\)\}\{p\_\{i,t\}\(j\)\},\\,\\tau\_\{\\mathrm\{clip\}\}\\right\)
9:endfor

10:

ℒPW\-OPSD←B−1​∑i=1BLi−1​∑t=1Liwi,t​ℓi,t\\mathcal\{L\}\_\{\\textsc\{PW\-OPSD\}\{\}\}\\leftarrow B^\{\-1\}\\sum\_\{i=1\}^\{B\}L\_\{i\}^\{\-1\}\\sum\_\{t=1\}^\{L\_\{i\}\}w\_\{i,t\}\\ell\_\{i,t\}
11:Update

θ\\thetausing

∇θℒPW\-OPSD\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\textsc\{PW\-OPSD\}\{\}\}\.

## Appendix HImplementation conventions inherited from OPSD

The following implementation conventions are held fixed across the methods reported in this paper\.

#### Right\-padded prompt collator \+ batch\-max loss slicing\.

The upstreamOPSDdata\_collatorright\-pads prompts and the trainer slices loss only on the firstbatch\_max\_prompt\_lentokens of each completion; this is held constant across all methods in[table˜2](https://arxiv.org/html/2605.21606#S4.T2)\. OpenThoughts\-Math\-30k\[siyanzhao,[2026](https://arxiv.org/html/2605.21606#bib.bib34)\]prompts vary in length \(median9393tokens, max826826\), so right\-padding produces a per\-batch prompt\-PAD gap\. Methods are compared on the same gap\.

#### Train/eval prompt template gap\.

Training prompts use theOPSDreference student templateProblem: \{problem\}\\n\\n Please reason\.\.\.withenable\_thinking=False; evaluation prompts use the simpler form\{problem\}\\n\\n Please reason\.\.\.withenable\_thinking=True\. This training/evaluation prompt\-template difference is shared across all methods reported in this paper\.

#### Per\-vocabulary clip semantics\.

The clipped forward KL in[eq\.˜1](https://arxiv.org/html/2605.21606#S3.E1)clamps each token/vocabulary entry ofqt​\(j\)​log⁡\(qt​\(j\)/pt​\(j\)\)q\_\{t\}\(j\)\\log\(q\_\{t\}\(j\)/p\_\{t\}\(j\)\)element\-wise viaF\.kl\_div\(reduction=’none’\)\.clamp\(max=tau\_clip\)before the inner sum over the vocabulary\. This matches theOPSDreference implementation\.

#### Gradient\-accumulation token\-mean\.

With per\-microbatch token\-mean reduction and gradient accumulation=2=2, theOPSDloss computes the mean of two per\-microbatch token\-means rather than the exact token\-weighted mean over the effective batch\. This is the standard HuggingFace Trainer behavior and is held fixed for theOPSDbaseline\.PW\-OPSD’s per\-sequence reduction does not have this issue because each microbatch contains the same number of sequences\.

#### vLLM rollout seed\.

The vLLM colocate\-mode rollout sampler is seeded asaccelerator\.process\_index // tp\_size, independent of the trainer\-\-seedflag\. Reruns of the same\-\-seed Ntherefore produce slightly different rollouts\. All evaluation results use seeded vLLMSamplingParams\.seed, which is reported per evaluation run\.

## Appendix IEvaluation setup

#### Models\.

The Qwen3\-4B checkpoint\[Yanget al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib19), Qwen Team,[2025](https://arxiv.org/html/2605.21606#bib.bib20)\]is used in the main comparison \([table˜2](https://arxiv.org/html/2605.21606#S4.T2)\), the position\-schedule sweep \([section˜4\.3](https://arxiv.org/html/2605.21606#S4.SS3)\), and the reduction\-positioning ablation \([table˜6](https://arxiv.org/html/2605.21606#A12.T6)\); the cross\-model evidence in[table˜4](https://arxiv.org/html/2605.21606#S4.T4)additionally uses DeepSeek\-R1\-Distill\-Llama\-8B\[Guoet al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib21), DeepSeek\-AI,[2025](https://arxiv.org/html/2605.21606#bib.bib40)\]and Olmo\-3\-7B\-Think\[Allen Institute for AI,[2026](https://arxiv.org/html/2605.21606#bib.bib41)\], two larger models from different families\. In every comparison we train LoRA adapters on a fixed checkpoint and merge the selected adapter before evaluation, so within each model block, differences between method rows come only from the distillation objective and not from the checkpoint weights\.

#### Training\.

We use theOPSDprivileged on\-policy setup\[Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\]\. Across methods, the local implementation holds fixed preprocessing and prompt templates, full\-vocabulary clipped forward\-KL conventions, LoRA\[Huet al\.,[2021](https://arxiv.org/html/2605.21606#bib.bib12)\]rank6464,α=128\\alpha=128, dropout0\.050\.05, learning rate5×10−65\{\\times\}10^\{\-6\},max\_completion\_length=1024=1024, distillation temperatureTdistill=1\.1T\_\{\\mathrm\{distill\}\}=1\.1, KL clipτclip=0\.05\\tau\_\{\\mathrm\{clip\}\}=0\.05, a fixed teacher with LoRA disabled in the privileged forward pass, and seed4242\. The local launcher uses4×4\{\\times\}H100 GPUs with effective batch size3232\(per\-device batch44with gradient accumulation22\)\. All methods are evaluated at the100100\-step checkpoint, following the OPSD evaluation horizon ofZhaoet al\.\[[2026](https://arxiv.org/html/2605.21606#bib.bib1)\]\.PW\-OPSDuses\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\)for the diagnostic\-derived default schedule\.

#### Baselines\.

We comparePW\-OPSDagainst three baselines run under our common training and evaluation protocol:OPSD\[Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\]as the uniform\-weight reference,EOPD\[Jinet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib3)\]as a representative entropy\-conditioned adaptive\-KL alternative, and REOPOLD\[Koet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib9)\]as a representative policy\-gradient adaptive on\-policy distillation method\. REOPOLD differs from the per\-token forward\-KL family in gradient form, because its gradient flows through the rolled\-out token’s log\-prob rather than through a distributional forward KL\. It nevertheless answers the same operational question of how to derive a per\-token training signal from teacher\-student log\-likelihood\-ratio rewards on sampled rollout tokens\. Including REOPOLD therefore provides a cross\-family reference point that controls for the possibility that any adaptive token weighting recovers the downstream pattern attributed to position\. All baselines use their published default hyperparameters under the common training and evaluation protocol described above\.

#### Evaluation setting\.

We use a maximum generation length of38,91238\{,\}912tokens:max\_new\_tokens=38912=38912,N=12N=12samples per problem, temperatureT=1\.0T=1\.0, top\-p=0\.95p=0\.95, top\-kkdisabled, andenable\_thinking=True, matching theOPSDevaluation setup\[Zhaoet al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib1)\]\.

#### Metrics\.

We reportAvg​@​12\\mathrm\{Avg\}@12,Pass​@​12\\mathrm\{Pass\}@12, andMaj​@​12\\mathrm\{Maj\}@12, which together expose per\-sample accuracy, search\-style success under repeated attempts, and stability under aggregation\. Majority vote uses math\-equivalence clustering withsympy\[Meureret al\.,[2017](https://arxiv.org/html/2605.21606#bib.bib32)\]andmath\_verify\[Kydlíček,[2025](https://arxiv.org/html/2605.21606#bib.bib39)\]; unformatted predictions are placed in a singleINVALIDcluster that participates in the plurality count and is scored incorrect when selected\. Formal per\-problem definitions are deferred to[Appendix˜J](https://arxiv.org/html/2605.21606#A10)\.

#### Benchmarks\.

We evaluate on MATH\-500\[Hendryckset al\.,[2021](https://arxiv.org/html/2605.21606#bib.bib28), Lightmanet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib18), HuggingFaceH4,[2024](https://arxiv.org/html/2605.21606#bib.bib36)\], AIME 2024\[HuggingFaceH4,[2025](https://arxiv.org/html/2605.21606#bib.bib37)\], AIME 2025\[yentinglin,[2025](https://arxiv.org/html/2605.21606#bib.bib38)\], and HMMT February 2025\. The HMMT set is a locally cleaned parquet derived from MathArena’shmmt\_feb\_2025release\[Dekonincket al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib35)\], with SHA\-256 recorded in the appendix\. For each method–benchmark pair we run three random evaluation seeds \(main,11,22\) and report mean±\\pmacross\-seed sample standard deviation\. The cross\-model assessment \([table˜4](https://arxiv.org/html/2605.21606#S4.T4)\) extends this protocol to DeepSeek\-R1\-Distill\-Llama\-8B and Olmo\-3\-7B\-Think under the same four benchmarks\.

## Appendix JEvaluation metric definitions

#### Multi\-sample evaluation for reasoning\.

Reasoning models are commonly evaluated with repeated sampling because a single completion can understate the chance that the model finds a correct solution\. Pass@N measures whether any ofNNsamples succeeds\[Chenet al\.,[2021](https://arxiv.org/html/2605.21606#bib.bib16), Brownet al\.,[2024](https://arxiv.org/html/2605.21606#bib.bib17)\], while self\-consistency and majority vote aggregate multiple reasoning paths\[Wanget al\.,[2022](https://arxiv.org/html/2605.21606#bib.bib14)\]\. We report Avg@12, Pass@12, and Maj@12 throughout\. These metrics expose different behaviors of a distillation objective: per\-sample accuracy, search\-style success under repeated attempts, and stability under aggregation\.

#### Formal definitions\.

For a single problem withN=12N\{=\}12generated solutions\{y\(1\),…,y\(N\)\}\\\{y^\{\(1\)\},\\ldots,y^\{\(N\)\}\\\}, the predicted answer is extracted as the content of the last\\boxed\{\.\.\.\}iny\(i\)y^\{\(i\)\}; samples without a parseable boxed answer are assigned the cluster keyINVALID\. Letci∈\{0,1\}c\_\{i\}\\in\\\{0,1\\\}indicate whethery\(i\)y^\{\(i\)\}is graded correct against the gold answer \(usingmath\_verify\[Kydlíček,[2025](https://arxiv.org/html/2605.21606#bib.bib39)\]with a normalized string\-equality fallback for parsing failures\), and letaia\_\{i\}be the math\-equivalence cluster key obtained by grouping the extracted answers under thesympy\[Meureret al\.,[2017](https://arxiv.org/html/2605.21606#bib.bib32)\]andmath\_verifynormalization pipeline\. The per\-problem metrics are

Avg​@​N\\displaystyle\\mathrm\{Avg\}@N=1N​∑i=1Nci,\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}c\_\{i\},\(9\)Pass​@​N\\displaystyle\\mathrm\{Pass\}@N=𝟏​\[∑i=1Nci≥1\],\\displaystyle=\\mathbf\{1\}\\\!\\left\[\\textstyle\\sum\_\{i=1\}^\{N\}c\_\{i\}\\geq 1\\right\],\(10\)Maj​@​N\\displaystyle\\mathrm\{Maj\}@N=\{0,a⋆=INVALID,ci⋆,otherwise,\\displaystyle=\\begin\{cases\}0,&a^\{\\star\}=\\texttt\{INVALID\},\\\\ c\_\{i^\{\\star\}\},&\\text\{otherwise\},\\end\{cases\}\(11\)where the plurality cluster key isa⋆∈arg⁡maxa∈\{a1,…,aN\}⁡\#​\{j:aj=a\}a^\{\\star\}\\in\\arg\\max\_\{a\\in\\\{a\_\{1\},\\ldots,a\_\{N\}\\\}\}\\\#\\\{j:a\_\{j\}=a\\\}, with ties broken by smallest first occurrencea⋆=amin⁡\{i:ai∈arg⁡max\}a^\{\\star\}=a\_\{\\min\\\{i:a\_\{i\}\\in\\arg\\max\\\}\}, and the representative indexi⋆=min⁡\{i:ai=a⋆\}i^\{\\star\}=\\min\\\{i:a\_\{i\}=a^\{\\star\}\\\}\. TheINVALIDcluster*participates in the plurality count*but is scored incorrect when selected, so a problem on which the model produces no parseable answer in the majority of samples scoresMaj​@​N=0\\mathrm\{Maj\}@N=0even if some minority of samples were correct\. Reported numbers are means of these per\-problem metrics across the benchmark, optionally averaged across evaluation seeds with a sample \(Bessel\-corrected\) standard deviation\.

## Appendix KReproducibility notes

The training and evaluation code accompanying this paper is available at[https://github\.com/SaFo\-Lab/PW\-OPSD](https://github.com/SaFo-Lab/PW-OPSD)\. The main dataset sources and evaluation conventions are as follows\.

- •Datasets\.MATH\-500 is HuggingFaceHuggingFaceH4/MATH\-500\[Hendryckset al\.,[2021](https://arxiv.org/html/2605.21606#bib.bib28), Lightmanet al\.,[2023](https://arxiv.org/html/2605.21606#bib.bib18), HuggingFaceH4,[2024](https://arxiv.org/html/2605.21606#bib.bib36)\]; AIME 2024 is HuggingFaceHuggingFaceH4/aime\_2024\[HuggingFaceH4,[2025](https://arxiv.org/html/2605.21606#bib.bib37)\]; AIME 2025 is HuggingFaceyentinglin/aime\_2025\[yentinglin,[2025](https://arxiv.org/html/2605.21606#bib.bib38)\]; HMMT February 2025 is a locally cleaned parquet derived from the MathArenahmmt\_feb\_2025release\[Dekonincket al\.,[2026](https://arxiv.org/html/2605.21606#bib.bib35)\], SHA\-25687bfb23d2c887fab12b42fcc2b2dd8cb5a9d1070e591490dac8755bc366ea25e; the training corpus is OpenThoughts\-Math\-30k\[Guhaet al\.,[2025](https://arxiv.org/html/2605.21606#bib.bib33), siyanzhao,[2026](https://arxiv.org/html/2605.21606#bib.bib34)\], the same corpus used byZhaoet al\.\[[2026](https://arxiv.org/html/2605.21606#bib.bib1)\]\.
- •Maj@N evaluation\.Maj@N clusters allNNpredictions by math\-equivalence usingsympy\[Meureret al\.,[2017](https://arxiv.org/html/2605.21606#bib.bib32)\]andmath\_verify\[Kydlíček,[2025](https://arxiv.org/html/2605.21606#bib.bib39)\], with unformatted predictions placed in a singleINVALIDcluster that participates in the plurality count and is scored incorrect when selected \(see[Appendix˜J](https://arxiv.org/html/2605.21606#A10)for the formal definition\)\.

## Appendix LReduction×\\timespositioning ablation

PW\-OPSDdiffers fromOPSDalong two axes that are easy to conflate\. The first is*positioning*:PW\-OPSDmultiplies the per\-token loss by a position\-dependent scalarwt∈\[wmin,1\]w\_\{t\}\\in\[w\_\{\\min\},1\], whileOPSDuseswt≡1w\_\{t\}\\equiv 1\. The second is*reduction*:PW\-OPSDaverages the per\-token loss within each rollout before averaging across the batch \(a per\-sequence reduction\), whileOPSD’s training script accumulates the loss in a global pool that effectively pools tokens across the batch first\. Both changes alter how single\-rollout positions contribute to the gradient, and a naive comparison conflates them\. We run the full2×22\{\\times\}2factorial under identical training and evaluation regimes;[Table˜6](https://arxiv.org/html/2605.21606#A12.T6)reports Avg@12 on AIME 2024\.

Table 6:2×22\{\\times\}2ablation on Qwen3\-4B AIME 2024: reduction \(uniform vs\. per\-sequence\) by positioning \(none vs\. position\-weighted\)\. Avg@12 mean±\\pmsample standard deviation across three evaluation seeds\. Bold marks the column maximum\. The diagonal \(OPSDandPW\-OPSDModerate\) reports the same evaluation runs as the corresponding rows of[Table˜2](https://arxiv.org/html/2605.21606#S4.T2); small differences in the standard\-deviation digits reflect the use of sample \(Bessel\-corrected\) standard deviation here\. The position\-weighted rows fix the schedule to Moderate\(wmin,τ,s\)=\(0\.25,0\.30,0\.10\)\(w\_\{\\min\},\\tau,s\)=\(0\.25,0\.30,0\.10\)\.Only the joint configuration \(row 4\) matches the AIME 2024 lead of[Table˜2](https://arxiv.org/html/2605.21606#S4.T2); switching either axis alone underperforms by∼1\.5\\sim 1\.5pp\. The two axes are complementary on AIME 2024 rather than independently sufficient\.