针对On-Policy蒸馏的教师-学生差异校准

# Calibrating Teacher–Student Discrepancy for On-Policy Distillation
Source: [https://arxiv.org/html/2609.21619](https://arxiv.org/html/2609.21619)
Qiangqiang He1Jin Li2MingCai Chen3 1State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China2College of Software Engineering, Southeast University, Nanjing, China3Nanjing University of Posts and Telecommunications, Nanjing, Chinaqqh@smail\.nju\.edu\.cnjin\_li@seu\.edu\.cnchenmc@njupt\.edu\.cn

###### Abstract

On\-policy distillation \(OPD\) improves reasoning models by learning the token\-level discrepancy between a stronger teacher and an on\-policy student\. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher–student discrepancy and indiscriminately learned by standard OPD during training\. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher\-side likelihood shifts, thereby encouraging the student to learn more of the teacher’s own deviation\. We introduceCalibrated On\-Policy Distillation \(Cal\-OPD\), which estimates the teacher’s self\-deviation region through positive and negative privileged interventions and calibrates the original teacher–student discrepancy by retaining only the component that lies beyond this region\. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52–65% of the original teacher–student discrepancy as the optimization signal, Cal\-OPD consistently outperforms standard OPD and its variants across model scales\.

## 1Introduction

Knowledge distillation \(KD\) transfers capabilities from a stronger teacher to a weaker student\([Hinton et al\., 2015](https://arxiv.org/html/2609.21619#bib.bib1)\), but conventional off\-policy distillation suffers from distribution mismatch between fixed distillation data and the student’s evolving policy distribution\([Agarwal et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib2);[Gu et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib3)\)\. On\-policy distillation \(OPD\) mitigates this mismatch by training directly on student\-generated trajectories and using teacher–student token\-level likelihood discrepancies as dense supervision\([Agarwal et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib2);[Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4);[Jin et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib5)\)\. Compared with reinforcement learning with verifiable rewards \(RLVR\), which relies on sparse outcome\-level rewards\([Wen et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib6);[Guo et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib7);[Yu et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib8)\), OPD provides token\-level guidance throughout the trajectory, enabling more direct and efficient reasoning post\-training\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4);[Jin et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib5)\)\.

Standard OPD implicitly treats the teacher likelihood assigned to each student token as an equally reliable reference\. Yet its stability varies substantially across tokens\. Recent studies on reasoning models suggest that reasoning tokens tend to remain relatively stable, whereas stylistic or surface\-form tokens, such as discourse markers and formatting choices, are substantially more sensitive to contextual interventions even when both the question and student rollout are held fixed\([Pan et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib9);[He et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib10)\)\. Related work further shows that the reliability of teacher supervision can vary across tokens and reasoning positions\([Liu et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib11)\)\. We refer to this token\-specific variability in teacher likelihood asTeacher Self\-Deviation \(TSD\)\. Consequently, the observed teacher–student discrepancy reflects not only the underlying capability gap but also deviations arising from the teacher itself, which standard OPD indiscriminately incorporates into the learning signal\.

Privileged OPD extends standard OPD by conditioning a stronger teacher on additional training\-time information, such as reference solutions, final answers, or hints\([Ye et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib12);[Yu et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib13);[Kaur et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib14)\)\. While such privileged context is intended to improve teacher supervision, it also induces further shifts in the teacher distribution\. Existing studies show that these shifts can introduce information\-asymmetry effects\([Yu et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib13)\), shortcut behavior\([Tian et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib15)\), and even degrade performance in thinking models\([Kaur et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib14)\)\. From the perspective of teacher self\-deviation, privileged OPD therefore amplifies the teacher\-side deviations already embedded in the teacher–student discrepancy, causing the student to learn more context\-induced teacher variation, as illustrated in Figure[1](https://arxiv.org/html/2609.21619#S1.F1)\.

![Refer to caption](https://arxiv.org/html/2609.21619v1/Cal_OPD_Intro.png)Figure 1:Conceptual comparison of standard OPD, privileged OPD, and Cal\-OPD\. Cal\-OPD removes the TSD\-explained discrepancy and retains only the residual beyond the TSD region\.To address this issue, we proposeCalibrated On\-Policy Distillation \(Cal\-OPD\), which first estimates the teacher’s self\-deviation region and then learns only the teacher–student discrepancy that lies beyond it\. Specifically, Cal\-OPD applies positive and negative privileged interventions, whose contrasting semantics provide complementary probes of the teacher’s contextual variability and enable an approximation of its token\-level self\-deviation region\. Unlike privileged OPD, these interventions are not directly distilled into the student, but are instead used to measure and calibrate the teacher reference\. Cal\-OPD then removes the portion of the original discrepancy covered by the estimated self\-deviation region, retaining only the residual as the learning signal and setting it to zero when the student likelihood falls within this estimated region\.

Our contributions are summarized as follows:

- •We identify and empirically characterizeTeacher Self\-Deviation \(TSD\), showing that teacher likelihood is not an equally stable reference across tokens and contexts, and that learning these teacher\-side deviations, which are further amplified by privileged context, can substantially degrade OPD for long\-CoT reasoning\.
- •We introduceCalibrated On\-Policy Distillation \(Cal\-OPD\), which uses positive and negative privileged interventions to estimate the teacher’s self\-deviation region and retains only the teacher–student discrepancy beyond it\. Privileged information is thus used to calibrate the teacher reference rather than directly supervise the student\.
- •Extensive experiments on mathematical reasoning benchmarks show that Cal\-OPD retains only about 52–65% of the original teacher–student discrepancy for optimization, yet consistently outperforms standard OPD and its variants across model scales, while alleviating the response\-length expansion observed in standard and privileged OPD\.

## 2Understanding Teacher Self\-Deviation

### 2\.1Preliminaries

#### On\-Policy Distillation\.

LetπS=πθ\\pi\_\{S\}=\\pi\_\{\\theta\}denote the student policy parameterized byθ\\theta, and letπT\\pi\_\{T\}denote a fixed teacher policy\. Given a problemx∼𝒟x\\sim\\mathcal\{D\}, the student generates an on\-policy rollout

y=\(y1,…,yT\)∼πS\(⋅∣x\),y=\(y\_\{1\},\\ldots,y\_\{T\}\)\\sim\\pi\_\{S\}\(\\cdot\\mid x\),\(1\)wherey<t=\(y1,…,yt−1\)y\_\{<t\}=\(y\_\{1\},\\ldots,y\_\{t\-1\}\)denotes the prefix of the student response preceding tokenyty\_\{t\}\. For each student\-generated token, we evaluate the teacher and student under the same problemxxand student response prefixy<ty\_\{<t\}, and define their token\-level log\-likelihoods and discrepancy as

ℓtT=log⁡πT​\(yt∣x,y<t\),ℓtS=log⁡πS​\(yt∣x,y<t\),AtOPD=ℓtT−ℓtS\.\\ell\_\{t\}^\{T\}=\\log\\pi\_\{T\}\(y\_\{t\}\\mid x,y\_\{<t\}\),\\qquad\\ell\_\{t\}^\{S\}=\\log\\pi\_\{S\}\(y\_\{t\}\\mid x,y\_\{<t\}\),\\qquad A\_\{t\}^\{\\mathrm\{OPD\}\}=\\ell\_\{t\}^\{T\}\-\\ell\_\{t\}^\{S\}\.\(2\)OPD usesAtOPDA\_\{t\}^\{\\mathrm\{OPD\}\}as the token\-level advantage for optimizing the student:

ℒOPD\(θ\)=−𝔼x∼𝒟,y∼πS\(⋅∣x\)\[1T∑t=1Tsg\(AtOPD\)logπS\(yt∣x,y<t\)\],\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,y\\sim\\pi\_\{S\}\(\\cdot\\mid x\)\}\\left\[\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\operatorname\{sg\}\\\!\\left\(A\_\{t\}^\{\\mathrm\{OPD\}\}\\right\)\\log\\pi\_\{S\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\right\],\(3\)wheresg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)denotes the stop\-gradient operator\. Accordingly,AtOPD\>0A\_\{t\}^\{\\mathrm\{OPD\}\}\>0increases the student likelihood ofyty\_\{t\}, whereasAtOPD<0A\_\{t\}^\{\\mathrm\{OPD\}\}<0decreases it\. This formulation treats the teacher likelihoodℓtT\\ell\_\{t\}^\{T\}as the token\-level reference against which the student is optimized\.

#### Teacher Self\-Deviation\.

For a fixed problemxx, student rolloutyy, and token positiontt, we introduce a teacher\-side contextual interventionccwhile keeping the evaluated student trajectory unchanged\. The intervention is provided only to the teacher, together with the problemxx, and precedes the student rollout prefixy<ty\_\{<t\}in the teacher context\. We usec0=∅c\_\{0\}=\\varnothingto denote the original setting without additional context\. The teacher likelihoods under interventionccand the original settingc0c\_\{0\}, together with the induced likelihood variation, are defined as

ℓtT\(c\)=logπT\(yt∣x,c,y<t\),ℓtT\(c0\)=logπT\(yt∣x,y<t\)=ℓtT,ΔtT​\(c\)=ℓtT​\(c\)−ℓtT​\(c0\)\.\\begin\{gathered\}\\ell\_\{t\}^\{T\}\(c\)=\\log\\pi\_\{T\}\(y\_\{t\}\\mid x,c,y\_\{<t\}\),\\qquad\\ell\_\{t\}^\{T\}\(c\_\{0\}\)=\\log\\pi\_\{T\}\(y\_\{t\}\\mid x,y\_\{<t\}\)=\\ell\_\{t\}^\{T\},\\\\ \\Delta\_\{t\}^\{T\}\(c\)=\\ell\_\{t\}^\{T\}\(c\)\-\\ell\_\{t\}^\{T\}\(c\_\{0\}\)\.\\end\{gathered\}\(4\)Sincexx,y<ty\_\{<t\}, andyty\_\{t\}remain fixed throughout the comparison,ΔtT​\(c\)\\Delta\_\{t\}^\{T\}\(c\)captures the change in teacher likelihood induced by the additional contextcc, rather than any change in the evaluated trajectory\. We refer to this context\-induced variation in teacher likelihood asteacher self\-deviation \(TSD\)\.

Given an intervention set𝒞\\mathcal\{C\}, we use the induced likelihood variations to estimate the downward and upward magnitudes of TSD:

d^t↓=max⁡\(0,−minc∈𝒞⁡ΔtT​\(c\)\),d^t↑=max⁡\(0,maxc∈𝒞⁡ΔtT​\(c\)\)\.\\hat\{d\}\_\{t\}^\{\\downarrow\}=\\max\\left\(0,\\,\-\\min\_\{c\\in\\mathcal\{C\}\}\\Delta\_\{t\}^\{T\}\(c\)\\right\),\\qquad\\hat\{d\}\_\{t\}^\{\\uparrow\}=\\max\\left\(0,\\,\\max\_\{c\\in\\mathcal\{C\}\}\\Delta\_\{t\}^\{T\}\(c\)\\right\)\.\(5\)Bothd^t↓\\hat\{d\}\_\{t\}^\{\\downarrow\}andd^t↑\\hat\{d\}\_\{t\}^\{\\uparrow\}are non\-negative and measure the maximum downward and upward deviations in teacher log\-likelihood, respectively\. These quantities yield an empirical estimate of theTSD region:

ℛ^tT​\(𝒞\)=\[ℓtT−d^t↓,ℓtT\+d^t↑\]\.\\hat\{\\mathcal\{R\}\}\_\{t\}^\{T\}\(\\mathcal\{C\}\)=\\left\[\\ell\_\{t\}^\{T\}\-\\hat\{d\}\_\{t\}^\{\\downarrow\},\\;\\ell\_\{t\}^\{T\}\+\\hat\{d\}\_\{t\}^\{\\uparrow\}\\right\]\.\(6\)Accordingly,ℛ^tT​\(𝒞\)\\hat\{\\mathcal\{R\}\}\_\{t\}^\{T\}\(\\mathcal\{C\}\)is a finite\-intervention approximation to the underlying teacher self\-deviation regionℛtT\\mathcal\{R\}\_\{t\}^\{T\}, rather than an exhaustive characterization of all possible contextual variation\. TSD characterizes variability in the teacher likelihood, while student\-generated rollouts provide the trajectories on which this variability is measured\.

### 2\.2Teacher Self\-Deviation Is Not Reliable Knowledge

TSD captures contextual changes in the teacher likelihood assigned to a fixed token\. If TSDfaithfully reflected task\-relevant knowledge, its variation should be systematically tied to the information introduced by the context\. In particular, TSD shoulddepend on the presence of task\-specific informationandrespond consistently to the correctness of that information\. Otherwise, the observed likelihood shift cannot be reliably interpreted as a knowledge signal\. Table[1](https://arxiv.org/html/2609.21619#S2.T1)summarizes the contextual interventions used in our analysis, spanning task\-agnostic instructions, evaluative feedback, and answer\- and solution\-level privileged information\. We conduct this analysis with Qwen3\-1\.7B as the student and Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4)\)as the teacher, both in thinking mode, over 6,528 questions sampled from DAPO\-17k\([Yu et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib8)\), with one response per question and approximately 60 million response tokens evaluated across all intervention conditions\. Details of data collection and preparation for TSD analysis are provided in Appendix[A\.2](https://arxiv.org/html/2609.21619#A1.SS2)\.

Table 1:Contextual interventions used to probe TSD\. The shared problem contextxxis omitted for brevity\. For answer\- and solution\-level interventions, the positive and negative variants share the same template and differ only in the inserted privileged content\.#### TSD Emerges Without Task Knowledge\.

We first find thatsubstantial TSD emerges even in the absence of external task\-specific knowledge, and that the affected token positions are largely preserved when richer privileged information is subsequently introduced\. This indicates that part of the teacher\-side variability amplified by privileged context is already present before answer\- or solution\-level knowledge is provided\.

To quantify this effect, we group the positive and negative variants of each intervention group as

𝒞g=\{cgpos,cgneg\},g∈\{inst,eval,ans,sol\}\.\\mathcal\{C\}\_\{g\}=\\left\\\{c\_\{g\}^\{\\mathrm\{pos\}\},c\_\{g\}^\{\\mathrm\{neg\}\}\\right\\\},\\qquad g\\in\\\{\\mathrm\{inst\},\\mathrm\{eval\},\\mathrm\{ans\},\\mathrm\{sol\}\\\}\.\(7\)Given a thresholdτ\\tau, we define the set of tokens exhibitingsignificant TSDunder group𝒞g\\mathcal\{C\}\_\{g\}as

𝒮g​\(τ\)=\{t:maxc∈𝒞g⁡\|ΔtT​\(c\)\|\>τ\}\.\\mathcal\{S\}\_\{g\}\(\\tau\)=\\left\\\{t:\\max\_\{c\\in\\mathcal\{C\}\_\{g\}\}\\left\|\\Delta\_\{t\}^\{T\}\(c\)\\right\|\>\\tau\\right\\\}\.\(8\)Figure[2](https://arxiv.org/html/2609.21619#S2.F2)\(a\) reports the prevalence of significant TSD under each intervention, together with the union prevalence defined by𝒮g​\(τ\)\\mathcal\{S\}\_\{g\}\(\\tau\)\. Atτ=0\.01\\tau=0\.01, task\-agnostic instructions induce significant TSD on20\.2%20\.2\\%and25\.8%25\.8\\%of tokens under the positive and negative variants, respectively, with their union covering29\.8%29\.8\\%of all tokens\. This prevalence is comparable to the29\.1%29\.1\\%observed under evaluative feedback and the29\.4%29\.4\\%under answer\-level privilege, despite task\-agnostic interventions providing neither an answer nor a solution\. Solution\-level privilege further expands the affected set to39\.6%39\.6\\%, showing that richer privileged context broadens TSD rather than creating it from scratch\.

We measure whether TSD under less informative interventions persists under richer privileged contexts\. For groups𝒞i\\mathcal\{C\}\_\{i\}and𝒞j\\mathcal\{C\}\_\{j\}, we define the retention of significant TSD as

Reti→j⁡\(τ\)=\|𝒮i​\(τ\)∩𝒮j​\(τ\)\|\|𝒮i​\(τ\)\|\.\\operatorname\{Ret\}\_\{i\\rightarrow j\}\(\\tau\)=\\frac\{\\left\|\\mathcal\{S\}\_\{i\}\(\\tau\)\\cap\\mathcal\{S\}\_\{j\}\(\\tau\)\\right\|\}\{\\left\|\\mathcal\{S\}\_\{i\}\(\\tau\)\\right\|\}\.\(9\)Figure[2](https://arxiv.org/html/2609.21619#S2.F2)\(b\) reveals a clear asymmetry\. Under solution\-level privilege,98\.3%98\.3\\%,98\.6%98\.6\\%, and98\.2%98\.2\\%of tokens exhibiting significant TSD under task\-agnostic instructions, evaluative feedback, and answer\-level privilege remain significant, respectively, whereas only74\.2%74\.2\\%,72\.6%72\.6\\%, and72\.9%72\.9\\%of solution\-level significant\-TSD tokens remain significant under these less informative interventions\. Thus, richer privileged context largely preserves previously affected positions while extending TSD to additional positions\. Together, these results show thattask knowledge is not necessary for TSD to emerge, while richer privileged contexts mainly broaden the affected set rather than introducing an entirely new deviation pattern\.

\(a\) Prevalence of Significant TSD\(b\) Retention of Significant TSDFigure 2:TSD across contextual interventions\. Significant TSD emerges even under task\-agnostic interventions, and the affected token positions are largely retained under richer privileged contexts\.Figure 3:Consistency of TSD under contrasting interventions\. TSD occurs at overlapping token positions, remains directionally aligned, and is dominated by a shared common\-mode component\.
#### TSD Is Largely Insensitive to Intervention Semantics\.

We find that TSD is largelyinsensitive to both semantic polarity and correctness\. To quantify this consistency, we compare the positive and negative variants of evaluative, answer\-level, and solution\-level interventions using three metrics\. For intervention groupgg, letΔt\+=ΔtT​\(cgpos\)\\Delta\_\{t\}^\{\+\}=\\Delta\_\{t\}^\{T\}\(c\_\{g\}^\{\\mathrm\{pos\}\}\)andΔt−=ΔtT​\(cgneg\)\\Delta\_\{t\}^\{\-\}=\\Delta\_\{t\}^\{T\}\(c\_\{g\}^\{\\mathrm\{neg\}\}\), with corresponding significant\-TSD setsSg\+​\(τ\)=\{t:\|Δt\+\|\>τ\}S\_\{g\}^\{\+\}\(\\tau\)=\\\{t:\|\\Delta\_\{t\}^\{\+\}\|\>\\tau\\\}andSg−​\(τ\)=\{t:\|Δt−\|\>τ\}S\_\{g\}^\{\-\}\(\\tau\)=\\\{t:\|\\Delta\_\{t\}^\{\-\}\|\>\\tau\\\}\.Overlapmeasures whether significant TSD occurs at the same token positions, whileAgreementmeasures whether the two interventions shift teacher likelihood in the same direction at jointly significant positions:

Ovlg⁡\(τ\)=\|Sg\+​\(τ\)∩Sg−​\(τ\)\|\|Sg\+​\(τ\)∪Sg−​\(τ\)\|,Agrg⁡\(τ\)=∑t∈Sg\+​\(τ\)∩Sg−​\(τ\)\[Δt\+Δt−\>0\]\|Sg\+​\(τ\)∩Sg−​\(τ\)\|\.\\operatorname\{Ovl\}\_\{g\}\(\\tau\)=\\frac\{\|S\_\{g\}^\{\+\}\(\\tau\)\\cap S\_\{g\}^\{\-\}\(\\tau\)\|\}\{\|S\_\{g\}^\{\+\}\(\\tau\)\\cup S\_\{g\}^\{\-\}\(\\tau\)\|\},\\qquad\\operatorname\{Agr\}\_\{g\}\(\\tau\)=\\frac\{\\sum\_\{t\\in S\_\{g\}^\{\+\}\(\\tau\)\\cap S\_\{g\}^\{\-\}\(\\tau\)\}\\mathbf\{1\}\\\!\\left\[\\Delta\_\{t\}^\{\+\}\\Delta\_\{t\}^\{\-\}\>0\\right\]\}\{\|S\_\{g\}^\{\+\}\(\\tau\)\\cap S\_\{g\}^\{\-\}\(\\tau\)\|\}\.\(10\)We further measure how much paired variation is shared between the two interventions\. LetIg​\(τ\)=Sg\+​\(τ\)∩Sg−​\(τ\)I\_\{g\}\(\\tau\)=S\_\{g\}^\{\+\}\(\\tau\)\\cap S\_\{g\}^\{\-\}\(\\tau\)denote the set of jointly significant tokens\. We decompose the paired deviations into shared and contrastive components and define theShared Deviation Ratio \(SDR\)as

Mt=Δt\+\+Δt−2,Dt=Δt\+−Δt−2,SDRg⁡\(τ\)=∑t∈Ig​\(τ\)\|Mt\|∑t∈Ig​\(τ\)\(\|Mt\|\+\|Dt\|\)\.M\_\{t\}=\\frac\{\\Delta\_\{t\}^\{\+\}\+\\Delta\_\{t\}^\{\-\}\}\{2\},\\qquad D\_\{t\}=\\frac\{\\Delta\_\{t\}^\{\+\}\-\\Delta\_\{t\}^\{\-\}\}\{2\},\\qquad\\operatorname\{SDR\}\_\{g\}\(\\tau\)=\\frac\{\\sum\_\{t\\in I\_\{g\}\(\\tau\)\}\|M\_\{t\}\|\}\{\\sum\_\{t\\in I\_\{g\}\(\\tau\)\}\\left\(\|M\_\{t\}\|\+\|D\_\{t\}\|\\right\)\}\.\(11\)A higher shared deviation ratio indicates that, among jointly significant tokens, paired TSD is dominated by variation shared across interventions, with less attributable to their semantic difference\.

Figure[3](https://arxiv.org/html/2609.21619#S2.F3)reveals that TSD remains highly structured under contrasting intervention semantics\. Reversing answer correctness still yields88\.7%88\.7\\%directional agreement and a74\.7%74\.7\\%shared deviation ratio, showing that paired likelihood shifts are dominated by a shared response rather than the correctness contrast itself\. Solution\-level interventions exhibit the highest positional overlap at80\.3%80\.3\\%, yet the lowest shared deviation ratio at66\.1%66\.1\\%, suggesting that richer context alters how TSD varies more than where it emerges\. Across all groups, substantial overlap, high directional agreement, and predominantly shared variation persist\. These results show thatTSD is largely insensitive to intervention semantics and correctness, further indicating that such teacher likelihood shifts cannot be reliably interpreted as task knowledge\. Additional analyses across multiple thresholds and model scales in Appendix[A\.3](https://arxiv.org/html/2609.21619#A1.SS3)reproduce these findings\.

Table 2:Token forms with the highest and lowest significant\-TSD ratesρτ​\(v\)\\rho\_\{\\tau\}\(v\)undercsolposc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}atτ=0\.01\\tau=0\.01, restricted to forms occurring more than 20,000 times\.\|Δ\|¯\\overline\{\|\\Delta\|\}denotes the mean TSD magnitude over significant occurrences\. Values are rounded for display; rankings are based on the unroundedρτ​\(v\)\\rho\_\{\\tau\}\(v\)\. For compactness,altern\.abbreviatesalternatively\.

### 2\.3TSD Concentrates on Surface\-Form Tokens

TSD isstrongly concentrated on surface\-form tokens rather than tokens carrying mathematical content\. High\-TSD forms are natural\-language markers that organize, qualify, or redirect the reasoning text without directly encoding problem\-specific mathematical information, whereas low\-TSD forms are dominated by numbers, mathematical symbols, and notation\. Undercsolposc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}, we merge tokenizer variants corresponding to the same surface form and restrict the analysis to forms occurring more than 20,000 times\. For each token formvv, we define the significant\-TSD rate asρτ​\(v\)=Pr⁡\(\|ΔtT​\(csolpos\)\|\>τ∣yt=v\)\\rho\_\{\\tau\}\(v\)=\\Pr\\\!\\left\(\|\\Delta\_\{t\}^\{T\}\(c\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}\)\|\>\\tau\\mid y\_\{t\}=v\\right\)\. Unlike occurrence counts,ρτ​\(v\)\\rho\_\{\\tau\}\(v\)measures how often a token exhibits significant TSD when it appears, thereby controlling for differences in token frequency\.

Table[2](https://arxiv.org/html/2609.21619#S2.T2)reveals a clear separation between token forms with the highest and lowest significant\-TSD rates atτ=0\.01\\tau=0\.01\. The 18 highest\-ranked forms are dominated by natural\-language surface expressions such asmaybe,however,therefore,consider, andalternatively, withρτ\\rho\_\{\\tau\}exceeding89%89\\%throughout, meaning significant TSD appears in about nine of ten occurrences\. In contrast, the 18 lowest\-ranked forms consist predominantly of digits, mathematical symbols, and notation such as0,√\\surd,θ\\theta, andfrac, all withρτ\\rho\_\{\\tau\}below9\.3%9\.3\\%\. This nearly order\-of\-magnitude separation shows that significant TSD occurs far more frequently on surface\-form tokens than on tokens directly expressing mathematical content\. The large\|Δ\|¯\\overline\{\|\\Delta\|\}values in Table[2](https://arxiv.org/html/2609.21619#S2.T2)mainly reflect the stronger teacher\-side shifts induced by solution\-level privilege\. Appendix[A\.4](https://arxiv.org/html/2609.21619#A1.SS4)extends the analysis to the top\-24 and bottom\-24 token forms across different contextual interventions, showing that weaker interventions substantially reduce deviation magnitudes while preserving the same separation between surface\-form tokens and mathematical or symbolic forms\. Appendix[A\.5](https://arxiv.org/html/2609.21619#A1.SS5)provides a trajectory\-level case study illustrating how TSD is distributed throughout a complete reasoning trace\.

## 3Calibrated On\-Policy Distillation

The preceding analysis shows that teacher likelihood is not an equally reliable pointwise reference: it exhibits substantial TSD that cannot be reliably interpreted as a knowledge signal\. We therefore proposeCalibrated On\-Policy Distillation \(Cal\-OPD\), which decomposes the observed teacher–student discrepancy into a TSD\-explained component and a calibrated residual, and distills only the discrepancy that remains beyond the teacher’s estimated self\-deviation region\.

AtOPD⏟Teacher–Student Discrepancy=AtCal⏟Calibrated Teacher–Student Discrepancy\+AtTSD⏟TSD\-Explained Discrepancy\.\\underbrace\{A\_\{t\}^\{\\mathrm\{OPD\}\}\}\_\{\\text\{Teacher\-\-Student Discrepancy\}\}=\\underbrace\{A\_\{t\}^\{\\mathrm\{Cal\}\}\}\_\{\\text\{Calibrated Teacher\-\-Student Discrepancy\}\}\+\\underbrace\{A\_\{t\}^\{\\mathrm\{TSD\}\}\}\_\{\\text\{TSD\-Explained Discrepancy\}\}\.\(12\)
For each tokenyty\_\{t\}, Cal\-OPD probes the teacher with two contrasting interventionscposc^\{\\mathrm\{pos\}\}andcnegc^\{\\mathrm\{neg\}\}, producingΔtpos=ΔtT​\(cpos\)\\Delta\_\{t\}^\{\\mathrm\{pos\}\}=\\Delta\_\{t\}^\{T\}\(c^\{\\mathrm\{pos\}\}\)andΔtneg=ΔtT​\(cneg\)\\Delta\_\{t\}^\{\\mathrm\{neg\}\}=\\Delta\_\{t\}^\{T\}\(c^\{\\mathrm\{neg\}\}\)\. We define their maximum downward and upward deviations asd^t↓=max⁡\(0,−Δtpos,−Δtneg\)\\hat\{d\}\_\{t\}^\{\\downarrow\}=\\max\(0,\-\\Delta\_\{t\}^\{\\mathrm\{pos\}\},\-\\Delta\_\{t\}^\{\\mathrm\{neg\}\}\)andd^t↑=max⁡\(0,Δtpos,Δtneg\)\\hat\{d\}\_\{t\}^\{\\uparrow\}=\\max\(0,\\Delta\_\{t\}^\{\\mathrm\{pos\}\},\\Delta\_\{t\}^\{\\mathrm\{neg\}\}\)\. Since two interventions provide only a finite probe of the underlying TSD, we introduce a relaxation factorλ≥1\\lambda\\geq 1and estimate the TSD region as

ℛ^tT=\[ℓtT−λ​d^t↓,ℓtT\+λ​d^t↑\]=\[LtT,UtT\]\.\\hat\{\\mathcal\{R\}\}\_\{t\}^\{T\}=\\left\[\\ell\_\{t\}^\{T\}\-\\lambda\\hat\{d\}\_\{t\}^\{\\downarrow\},\\;\\ell\_\{t\}^\{T\}\+\\lambda\\hat\{d\}\_\{t\}^\{\\uparrow\}\\right\]=\[L\_\{t\}^\{T\},U\_\{t\}^\{T\}\]\.\(13\)
Cal\-OPD then removes the portion of the teacher–student discrepancy covered by the estimated TSD region and retains only the residual beyond its boundary:

AtCal=\[LtT−ℓtS\]\+−\[ℓtS−UtT\]\+,\[z\]\+=max⁡\(z,0\)\.A\_\{t\}^\{\\mathrm\{Cal\}\}=\\left\[L\_\{t\}^\{T\}\-\\ell\_\{t\}^\{S\}\\right\]\_\{\+\}\-\\left\[\\ell\_\{t\}^\{S\}\-U\_\{t\}^\{T\}\\right\]\_\{\+\},\\qquad\[z\]\_\{\+\}=\\max\(z,0\)\.\(14\)WhenℓtS∈ℛ^tT\\ell\_\{t\}^\{S\}\\in\\hat\{\\mathcal\{R\}\}\_\{t\}^\{T\}, the observed teacher–student discrepancy is fully covered by the estimated TSD region andAtCal=0A\_\{t\}^\{\\mathrm\{Cal\}\}=0; otherwise,AtCalA\_\{t\}^\{\\mathrm\{Cal\}\}retains only the discrepancy beyond the nearest boundary\. Accordingly,AtTSD=AtOPD−AtCalA\_\{t\}^\{\\mathrm\{TSD\}\}=A\_\{t\}^\{\\mathrm\{OPD\}\}\-A\_\{t\}^\{\\mathrm\{Cal\}\}\. Cal\-OPD follows the standard OPD objective in Eq\.[3](https://arxiv.org/html/2609.21619#S2.E3), replacingAtOPDA\_\{t\}^\{\\mathrm\{OPD\}\}withAtCalA\_\{t\}^\{\\mathrm\{Cal\}\}as the token\-level advantage\.

Table 3:Main results on mathematical reasoning benchmarks\. The best result among distillation methods for each teacher–student configuration is shown inbold\. Subscripts on Cal\-OPD indicate gains over the corresponding student\. For compactness,4B\-2507and30B\-2507denote Qwen3\-4B\-Thinking\-2507 and Qwen3\-30B\-A3B\-Thinking\-2507, respectively\.Model / MethodAMC23AIME24AIME25AIME26HMMT26MATH500Avg\.4B\-2507→\\rightarrowQwen3\-1\.7BStudent80\.840\.632\.529\.221\.890\.249\.2Teacher94\.856\.553\.154\.225\.295\.463\.2OPD82\.039\.635\.034\.622\.990\.650\.8ExOPD81\.743\.135\.434\.225\.690\.651\.8EOPD81\.644\.036\.738\.819\.790\.651\.9Uni\-OPD81\.442\.134\.432\.722\.589\.850\.5Privileged\-OPD83\.339\.830\.431\.021\.889\.449\.3Cal\-OPD84\.5\+3\.7\\mathbf\{84\.5\}\_\{\+3\.7\}45\.0\+4\.4\\mathbf\{45\.0\}\_\{\+4\.4\}37\.1\+4\.6\\mathbf\{37\.1\}\_\{\+4\.6\}36\.3\+7\.136\.3\_\{\+7\.1\}24\.8\+3\.024\.8\_\{\+3\.0\}90\.8\+0\.6\\mathbf\{90\.8\}\_\{\+0\.6\}53\.1\+3\.9\\mathbf\{53\.1\}\_\{\+3\.9\}30B\-2507→\\rightarrowQwen3\-4BStudent95\.264\.856\.357\.930\.195\.466\.6Teacher97\.874\.865\.466\.033\.996\.972\.5OPD94\.262\.157\.555\.231\.195\.065\.9ExOPD92\.066\.959\.456\.333\.795\.667\.3EOPD95\.362\.754\.660\.033\.596\.767\.1Uni\-OPD93\.467\.959\.857\.931\.495\.467\.6Privileged\-OPD93\.458\.852\.154\.031\.394\.464\.0Cal\-OPD96\.3\+1\.1\\mathbf\{96\.3\}\_\{\+1\.1\}67\.1\+2\.367\.1\_\{\+2\.3\}61\.3\+5\.0\\mathbf\{61\.3\}\_\{\+5\.0\}60\.2\+2\.3\\mathbf\{60\.2\}\_\{\+2\.3\}33\.3\+3\.233\.3\_\{\+3\.2\}95\.8\+0\.495\.8\_\{\+0\.4\}69\.0\+2\.4\\mathbf\{69\.0\}\_\{\+2\.4\}
## 4Experiments

### 4\.1Experimental Setup

#### Datasets\.

We use DAPO\-17K\([Yu et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib8)\)filtered by Qwen3\-235B\-A22B\-Instruct\-2507\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4)\)as the training dataset\. The filtering and solution\-annotation procedure is described in Appendix[A\.2](https://arxiv.org/html/2609.21619#A1.SS2)\. For evaluation, we consider six reasoning benchmarks: AMC23, AIME24, AIME25, AIME26, HMMT26, and MATH500\([Lightman et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib16)\)\. These benchmarks span a range of difficulty, from standard problem solving to high\-difficulty competition mathematics\.

#### Models and Baselines\.

We evaluate Cal\-OPD on two teacher–student configurations from the Qwen3 family\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4)\): Qwen3\-4B\-Thinking\-2507→\\rightarrowQwen3\-1\.7B and Qwen3\-30B\-A3B\-Thinking\-2507→\\rightarrowQwen3\-4B, covering two distinct scale regimes\. We compare against five representative OPD baselines: standard OPD\([Agarwal et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib2)\), which directly optimizes teacher–student discrepancies; ExOPD\([Yang et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib17)\), augmented with reward extrapolation; EOPD\([Jin et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib5)\), with entropy\-aware forward\-KL supervision; Uni\-OPD\([Hou et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib18)\), with exploration and outcome\-guided calibration; and Privileged\-OPD\([Ye et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib12);[Kaur et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib14)\), where the teacher is additionally conditioned on the ground\-truth reference solution\. This evaluates if Cal\-OPD remains effective across varying capacity gaps and diverse modifications of the standard OPD objective\. We report Avg@16 accuracy, computed by sampling 16 independent responses per problem and averaging their binary correctness scores across the full evaluation set\.

#### Implementation Details\.

We implement all methods withverl\([Sheng et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib19)\)and train on 8 NVIDIA H20 GPUs, with 4 GPUs hosting the student and 4 hosting the teacher\. All methods are trained for 100 steps, with 256 trajectories per step, one rollout per question unless otherwise specified, and a learning rate of1×10−61\\times 10^\{\-6\}\. During training, we set response length to 16,384, temperature to1\.01\.0, and top\-ppto1\.01\.0; during evaluation, we use response length of 20,480, temperature0\.60\.6, and top\-ppto0\.950\.95\. For Cal\-OPD, we usecevalposc\_\{\\mathrm\{eval\}\}^\{\\mathrm\{pos\}\}andcevalnegc\_\{\\mathrm\{eval\}\}^\{\\mathrm\{neg\}\}as contrasting interventions and set the relaxation factor toλ=5\\lambda=5\. Method\-specific hyperparameters are provided in Appendix[A\.6](https://arxiv.org/html/2609.21619#A1.SS6)\.

Table 4:Effect of contextual interventions used for TSD estimation on Cal\-OPD performance for Qwen3\-4B\-Thinking\-2507→\\rightarrowQwen3\-1\.7B\.

### 4\.2Main Results

Table[3](https://arxiv.org/html/2609.21619#S3.T3)shows that Cal\-OPD achieves the highest average performance in both teacher–student configurations:53\.153\.1for Qwen3\-4B\-Thinking\-2507→\\rightarrowQwen3\-1\.7B and69\.069\.0for Qwen3\-30B\-A3B\-Thinking\-2507→\\rightarrowQwen3\-4B\. This corresponds to gains of\+3\.9\+3\.9and\+2\.4\+2\.4over the student baselines, and\+2\.3\+2\.3and\+3\.1\+3\.1over standard OPD\. Cal\-OPD also achieves the best result on 7 of 12 benchmark–configuration pairs, demonstrating consistent gains across model scales and reasoning benchmarks\. In contrast, standard OPD improves the 4B→\\rightarrow1\.7B student only modestly, from49\.249\.2to50\.850\.8, and even degrades the 30B→\\rightarrow4B student from66\.666\.6to65\.965\.9, despite both teachers being substantially stronger than their students\. This shows that raw teacher–student discrepancy isnot uniformly beneficial supervision: optimizing all discrepancies also learns teacher\-side deviations, whereas Cal\-OPD removes the TSD\-explained component and retains a more effective signal\.

Privileged\-OPD exhibits the strongest degradation, obtaining the lowest average performance among distillation methods in both configurations, at49\.349\.3and64\.064\.0, respectively\. In the 4B→\\rightarrow1\.7B setting, it nearly eliminates the gain from standard OPD, while in the 30B→\\rightarrow4B setting it falls2\.62\.6points below the student\. Together with our earlier finding that privileged context substantially amplifies TSD, these results suggest that directly distilling the privileged teacher can transfer teacher\-side deviation alongside task\-relevant information, offsetting the benefit of stronger supervision\. Cal\-OPD mitigates this effect by filtering out the TSD\-explained component before distillation\.

### 4\.3Analysis and Ablations

#### Effect of Contextual Interventions on Cal\-OPD Performance\.

Table[4](https://arxiv.org/html/2609.21619#S4.T4)compares different intervention sets withλ=5\\lambda=5throughout, while Figure[4](https://arxiv.org/html/2609.21619#S4.F4)shows their calibration dynamics\.𝒞eval\\mathcal\{C\}\_\{\\mathrm\{eval\}\}achieves the best performance, reaching an average of53\.153\.1\. Although evaluative feedback introduces external judgment, it provides no task\-specific solution knowledge and yields stronger calibration than instruction interventions, which retain nearly70%70\\%of the teacher–student discrepancy\. In contrast,𝒞sol\\mathcal\{C\}\_\{\\mathrm\{sol\}\}causes the largest degradation, with the average falling to49\.049\.0while retaining only about20%20\\%of the discrepancy\. This suggests that solution\-induced TSD contains a larger task\-relevant component, such that using it for calibration over\-filters useful teacher supervision\. Overall,𝒞eval\\mathcal\{C\}\_\{\\mathrm\{eval\}\}provides a stronger probe of TSD without the excessive filtering induced by solution\-level privilege\. Further training dynamics, including entropy and other measures, are provided in Appendix[A\.7](https://arxiv.org/html/2609.21619#A1.SS7)\.

\(a\) Retained OPD Discrepancy\(b\) Zero\-Advantage Token Ratio\(c\) Average Response LengthFigure 4:Training dynamics of Cal\-OPD under different TSD\-estimation interventions for Qwen3\-4B\-Thinking\-2507→\\rightarrowQwen3\-1\.7B\.
#### Effects of Calibration on OPD Training Dynamics\.

Figure[4](https://arxiv.org/html/2609.21619#S4.F4)shows the training dynamics of the main𝒞eval\\mathcal\{C\}\_\{\\mathrm\{eval\}\}configuration\. The retained\-discrepancy ratio decreases from65%65\\%to52%52\\%, while the zero\-advantage token ratio increases from27%27\\%to34%34\\%, indicating stronger filtering of the OPD signal\. Calibration also alters response\-length dynamics\. Standard OPD expands the average response length from about9\.89\.8K to11\.811\.8K tokens, whereas Cal\-OPD ends at only9\.39\.3K after an initial decrease to8\.58\.5K\. Despite the additional teacher computation for estimating TSD, the shorter trajectories make Cal\-OPD approximately1\.26×1\.26\\timesfaster to train\. Together with the main results, these dynamics show that filtering the TSD\-explained component yields a more effective and efficient supervision signal\. A complete efficiency analysis is provided in Appendix[A\.8](https://arxiv.org/html/2609.21619#A1.SS8)\.

\(a\) Impact ofλ\\boldsymbol\{\\lambda\}on Performance\(b\) Impact ofλ\\boldsymbol\{\\lambda\}on Retained OPD DiscrepancyFigure 5:Effect of the relaxation factorλ\\lambdaon Cal\-OPD performance and retained discrepancy ratio\.
#### Effect of the Relaxation Factor𝝀\\boldsymbol\{\\lambda\}\.

Figure[5](https://arxiv.org/html/2609.21619#S4.F5)shows that Cal\-OPD achieves optimal performance atλ=5\\lambda=5, retaining approximately 52% of the teacher–student discrepancy\. Increasingλ\\lambdabeyond 5 sharply degrades downstream performance, eventually dropping 1\.7 points below theλ=1\\lambda=1baseline atλ=80\\lambda=80\. Asλ\\lambdaincreases, the estimated TSD region expands too broadly, reducing the retained discrepancy to roughly 20% and over\-filtering task\-relevant supervision\. We further compare against a TSD\-threshold filtering baseline matched in retained teacher–student discrepancy in Appendix[A\.9](https://arxiv.org/html/2609.21619#A1.SS9)\. Cal\-OPD remains stronger, indicating that its gains are not due to signal attenuation alone\.

## 5Conclusion

We show that the teacher–student discrepancy used by standard OPD contains substantial teacher self\-deviation \(TSD\), which emerges even without task\-specific knowledge, remains largely insensitive to intervention semantics and correctness, and concentrates on surface\-form tokens\. Based on these findings, we introduce Calibrated On\-Policy Distillation \(Cal\-OPD\), which estimates the teacher’s self\-deviation region and removes the TSD\-explained portion of the OPD signal before optimization\. Across different teacher–student scales and mathematical reasoning benchmarks, Cal\-OPD consistently outperforms standard OPD and its variants while retaining only 52–65% of the original discrepancy and mitigating response\-length expansion during training\. These results suggest that effective on\-policy distillation depends not on indiscriminately learning more teacher–student discrepancy, but on identifying which discrepancy is truly worth learning\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21246–21263\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2609.21619#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)H\. Chen, S\. Wu, X\. Quan, R\. Wang, M\. Yan, and J\. ZhangMcc\-kd: multi\-cot consistent knowledge distillation\.InFindings of the association for computational linguistics: EMNLP 2023,pp\. 6805–6820\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Daiet al\.\(2024\)C\. Dai, K\. Li, W\. Zhou, and S\. HuImprove student’s reasoning generalizability through cascading decomposed cots distillation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15623–15643\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Ding and Zhang \(2026\)Y\. Ding and R\. ZhangDoes on\-policy distillation really distill? from noisy teacher to self\-improvement\.arXiv preprint arXiv:2608\.31046\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1)\.
- Fenget al\.\(2024\)T\. Feng, Y\. Li, L\. Chenglin, H\. Chen, F\. Yu, and Y\. ZhangTeaching small language models reasoning through counterfactual distillation\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 5831–5842\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMinillm: knowledge distillation of large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 32694–32717\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p1.1)\.
- Heet al\.\(2026\)Q\. He, Z\. Wu, and Z\. WangNot all tokens deserve equal credit: counterfactual sensitivity credit reallocation for long\-cot reasoning\.arXiv preprint arXiv:2607\.27888\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p2.1)\.
- Heoet al\.\(2026\)B\. Heo, J\. Hwang, S\. Yun, and D\. HanOn\-policy delta distillation for multilingual math reasoning\.arXiv preprint arXiv:2608\.05802\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p1.1)\.
- Hoet al\.\(2023\)N\. Ho, L\. Schmid, and S\. YunLarge language models are reasoning teachers\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 14852–14882\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Houet al\.\(2026a\)W\. Hou, S\. Peng, W\. Wang, Z\. Ruan, Y\. Zhang, Z\. Zhou, M\. Gao, Y\. Chen, K\. Wang, H\. Yang,et al\.Uni\-opd: unifying on\-policy distillation with a dual\-perspective recipe\.arXiv preprint arXiv:2605\.03677\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1),[§A\.6](https://arxiv.org/html/2609.21619#A1.SS6.SSS0.Px6.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px2.p1.1)\.
- Houet al\.\(2026b\)Z\. Hou, X\. Tang, H\. An, J\. Zhang, W\. Wang, Y\. Han, G\. Li, X\. Hao, H\. Guo, W\. Hu,et al\.DASH: divergence\-adaptive supervision horizons for on\-policy self\-distillation of reasoning models\.arXiv preprint arXiv:2608\.06243\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1)\.
- Hsiehet al\.\(2023\)C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. PfisterDistilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the association for computational linguistics: ACL 2023,pp\. 8003–8017\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Hübotteret al\.\(2026\)J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Ichiharaet al\.\(2026\)Y\. Ichihara, N\. Iwase, M\. A\. Quamar, and J\. KomiyamaPrivileged solutions or context\-induced teacher behavior? dissecting on\-policy self\-distillation\.arXiv preprint arXiv:2608\.09228\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Jiaoet al\.\(2020\)X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. LiuTinybert: distilling bert for natural language understanding\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 4163–4174\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Jinet al\.\(2026\)W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. LeeEntropy\-aware on\-policy distillation of language models\.arXiv preprint arXiv:2603\.07079\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1),[§A\.6](https://arxiv.org/html/2609.21619#A1.SS6.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px2.p1.1)\.
- Kauret al\.\(2026\)S\. Kaur, N\. Ri, Y\. He, L\. Fowl, and S\. AroraRethinking on\-policy self\-distillation for thinking models\.arXiv preprint arXiv:2607\.05184\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1),[§A\.6](https://arxiv.org/html/2609.21619#A1.SS6.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px2.p1.1)\.
- Keet al\.\(2026\)J\. Ke, Z\. Wen, W\. Li, C\. He, and L\. ZhangRespecting self\-uncertainty in on\-policy self\-distillation for efficient llm reasoning\.arXiv preprint arXiv:2605\.13255\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1)\.
- Kim and Rush \(2016\)Y\. Kim and A\. M\. RushSequence\-level knowledge distillation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 1317–1327\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Koet al\.\(2024\)J\. Ko, S\. Kim, T\. Chen, and S\. YunDistillm: towards streamlined distillation for large language models\.arXiv preprint arXiv:2402\.03898\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)L\. H\. Li, J\. Hessel, Y\. Yu, X\. Ren, K\. Chang, and Y\. ChoiSymbolic chain\-of\-thought distillation: small models can also “think” step\-by\-step\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2665–2679\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2026a\)Y\. Li, B\. Wang, Y\. Liang, Y\. Tian, D\. Fu, and N\. VasconcelosOn\-policy self\-distillation without any supervision\.arXiv preprint arXiv:2608\.06296\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1)\.
- Liet al\.\(2026b\)Y\. Li, M\. Zhang, D\. Shen, and Y\. SunPHF: privileged hidden flow for on\-policy self\-distillation\.arXiv preprint arXiv:2606\.29340\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026\)X\. Liu, X\. Wang, Y\. Ma, Y\. Zhang, and C\. XiaoWhen are teacher tokens reliable? position\-weighted on\-policy self\-distillation for reasoning\.arXiv preprint arXiv:2605\.21606\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p2.1)\.
- Mitraet al\.\(2023\)A\. Mitra, L\. Del Corro, S\. Mahajan, A\. Codas, C\. Simoes, S\. Agarwal, X\. Chen, A\. Razdaibiedina, E\. Jones, K\. Aggarwal,et al\.Orca 2: teaching small language models how to reason\.arXiv preprint arXiv:2311\.11045\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Panet al\.\(2026\)L\. Pan, S\. Tao, Y\. Zhai, L\. Zhang, Z\. Liu, B\. Ding, A\. Liu, and L\. WenRLCSD: reinforcement learning with contrastive on\-policy self\-distillation\.arXiv preprint arXiv:2606\.11709\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p2.1)\.
- Penalozaet al\.\(2026\)E\. Penaloza, D\. Vattikonda, N\. Gontier, A\. Lacoste, L\. Charlin, and M\. CacciaPrivileged information distillation for language models\.arXiv preprint arXiv:2602\.04942\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Sanget al\.\(2026\)H\. Sang, Y\. Xu, Z\. Zhou, R\. He, Z\. Wang, and J\. SunOn\-policy self\-distillation for reasoning compression\.arXiv e\-prints,pp\. arXiv–2603\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Sanhet al\.\(2019\)V\. Sanh, L\. Debut, J\. Chaumond, and T\. WolfDistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Shenet al\.\(2026a\)G\. Shen, X\. Cheng, C\. Zhao, L\. Huang, J\. Li, D\. Zhao, and X\. YuAnti\-self\-distillation for reasoning rl via pointwise mutual information\.arXiv preprint arXiv:2605\.11609\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Shenet al\.\(2026b\)Z\. Shen, J\. Tong, S\. Yan, C\. Shen, H\. Chen, W\. Ye, X\. Hu, R\. Miao, H\. Wang, J\. Zhao,et al\.Purified opsd: on\-policy self\-distillation without losing how to think\.arXiv preprint arXiv:2607\.02234\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[§A\.6](https://arxiv.org/html/2609.21619#A1.SS6.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px3.p1.1)\.
- Shrestha and Tessier \(2026\)S\. Shrestha and A\. TessierRethinking privileged information in on\-policy self\-distillation\.arXiv preprint arXiv:2608\.18271\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Steinet al\.\(2026\)A\. Stein, F\. Huang, and T\. GoldsteinGATES: self\-distillation under privileged context with consensus gating\.arXiv preprint arXiv:2602\.20574\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Tan and Hong \(2026\)Z\. Tan and Y\. HongSelf\-supervised on\-policy distillation for reasoning language models\.arXiv preprint arXiv:2605\.17497\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1)\.
- Tianet al\.\(2026\)K\. Tian, S\. Liu, Z\. Yan, S\. Xia, S\. Dong, and Y\. WangVicur: visual cues as recoverable privilege for multimodal on\-policy distillation\.arXiv preprint arXiv:2606\.05718\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p3.1)\.
- Wenet al\.\(2026\)X\. Wen, Z\. Liu, S\. Zheng, S\. Ye, Z\. Wu, Y\. Wang, Z\. Xu, X\. Liang, J\. Li, Z\. Miao,et al\.Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 49450–49483\.Cited by:[§1](https://arxiv.org/html/2609.21619#S1.p1.1)\.
- Wuet al\.\(2026\)J\. Wu, Y\. Wang, E\. Su, C\. Tang, and S\. TangDAPD: dual\-anchored policy distillation\.arXiv preprint arXiv:2608\.01735\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§A\.2](https://arxiv.org/html/2609.21619#A1.SS2.SSS0.Px1.p1.1),[§A\.2](https://arxiv.org/html/2609.21619#A1.SS2.SSS0.Px2.p1.1),[§A\.2](https://arxiv.org/html/2609.21619#A1.SS2.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.21619#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2026a\)W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. LinLearning beyond teacher: generalized on\-policy distillation with reward extrapolation\.arXiv preprint arXiv:2602\.12125\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§A\.6](https://arxiv.org/html/2609.21619#A1.SS6.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2026b\)Y\. Yang, X\. Wang, and W\. ZhangOGLS\-sd: on\-policy self\-distillation with outcome\-guided logit steering for llm reasoning\.arXiv preprint arXiv:2605\.12400\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1)\.
- Yeet al\.\(2026\)T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. WeiOn\-policy context distillation for language models\.arXiv preprint arXiv:2602\.12275\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1),[§A\.6](https://arxiv.org/html/2609.21619#A1.SS6.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px2.p1.1)\.
- Yuet al\.\(2026a\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[§A\.2](https://arxiv.org/html/2609.21619#A1.SS2.p1.1),[§A\.6](https://arxiv.org/html/2609.21619#A1.SS6.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.21619#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.21619#S4.SS1.SSS0.Px1.p1.1)\.
- Yuet al\.\(2026b\)X\. Yu, G\. Li, Q\. Si, G\. Zhang, Y\. Xu, C\. Wang, S\. Dong, K\. Tuo, X\. Zeng, K\. Feng,et al\.DOPD: dual on\-policy distillation\.arXiv preprint arXiv:2606\.30626\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.21619#S1.p3.1)\.
- Zhanget al\.\(2026\)K\. Zhang, Y\. Tian, D\. Zhao, Y\. Li, Y\. Liu, V\. M\. Patel, and D\. FuOn\-policy distillation with best\-of\-n teacher rollout selection\.arXiv preprint arXiv:2605\.09725\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2024\)S\. Zhang, X\. Zhang, Z\. Sun, Y\. Chen, and J\. XuDual\-space knowledge distillation for large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 18164–18181\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p1.1)\.
- Zhang \(2026\)W\. ZhangBeyond absolute imitation: anchored residual guidance for privileged on\-policy distillation\.arXiv preprint arXiv:2606\.10385\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026a\)S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. GroverSelf\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px1.p2.1),[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026b\)X\. Zhao, L\. Zhang, Z\. Xu, Z\. Wang, X\. Caiyue, S\. Zhao, and Q\. LiIs more privileged information better? from solution traces to problem\-solving structure in self\-distilled reasoning\.arXiv preprint arXiv:2608\.01589\.Cited by:[§A\.1](https://arxiv.org/html/2609.21619#A1.SS1.SSS0.Px2.p1.1)\.

## Appendix AAppendix

### A\.1Related Work

#### Knowledge Distillation and On\-Policy Distillation\.

Knowledge distillation \(KD\) transfers capabilities from a stronger teacher to a weaker student and has become a standard framework for model compression and capability transfer\([Hinton et al\., 2015](https://arxiv.org/html/2609.21619#bib.bib1)\)\. Sequence\-level and pretrained language\-model distillation extended this paradigm to generation and compression\([Kim and Rush, 2016](https://arxiv.org/html/2609.21619#bib.bib20);[Sanh et al\., 2019](https://arxiv.org/html/2609.21619#bib.bib21);[Jiao et al\., 2020](https://arxiv.org/html/2609.21619#bib.bib22)\), while LLM\-specific methods introduced reverse or student\-aware divergence objectives and output\-space alignment\([Gu et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib3);[Ko et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib23);[Zhang et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib24)\)\. For reasoning, teacher\-generated rationales and reasoning strategies have been transferred through Fine\-tune\-CoT\([Ho et al\., 2023](https://arxiv.org/html/2609.21619#bib.bib25)\), Distilling Step\-by\-Step\([Hsieh et al\., 2023](https://arxiv.org/html/2609.21619#bib.bib26)\), SCoTD\([Li et al\., 2023](https://arxiv.org/html/2609.21619#bib.bib27)\), MCC\-KD\([Chen et al\., 2023](https://arxiv.org/html/2609.21619#bib.bib28)\), Orca 2\([Mitra et al\., 2023](https://arxiv.org/html/2609.21619#bib.bib29)\), counterfactual distillation\([Feng et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib30)\), and decomposed CoT distillation\([Dai et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib31)\); DeepSeek\-R1 further demonstrates that CoT reasoning behaviors can be distilled into smaller models\([Guo et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib7)\)\.

Many sequence\- and rationale\-level distillation methods remain off\-policy, training on teacher\-generated or otherwise fixed trajectories\. On\-policy distillation instead provides teacher supervision on trajectories sampled from the student itself, reducing the training–inference distribution mismatch\([Agarwal et al\., 2024](https://arxiv.org/html/2609.21619#bib.bib2)\), and has been adopted in large\-scale reasoning post\-training such as Qwen3\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4)\)\. Recent extensions modify the supervision signal through reward extrapolation\([Yang et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib17)\), entropy\- or uncertainty\-aware objectives\([Jin et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib5);[Ke et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib34)\), outcome\-guided calibration\([Hou et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib18)\), divergence\-adaptive supervision horizons\([Hou et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib40)\), and teacher\-versus\-base delta signals\([Heo et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib41)\)\. On\-policy self\-distillation further removes the need for a separate stronger teacher through privileged or behavior\-conditioned self\-teachers\([Zhao et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib32);[Sang et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib33)\), self\-generated supervision\([Tan and Hong, 2026](https://arxiv.org/html/2609.21619#bib.bib36)\), and internal consistency\([Li et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib39)\), while on\-policy context distillation internalizes teacher\-only context such as prior experience or optimized system prompts\([Ye et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib12)\)\.

#### Privileged and Context\-Augmented Distillation\.

Recent LLM distillation strengthens the teacher with information unavailable to the student at inference time\. On\-Policy Self\-Distillation \(OPSD\) conditions a self\-teacher on verified reference solutions\([Zhao et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib32)\), SDPO conditions the self\-teacher on environment feedback\([Hübotter et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib43)\), and privileged\-information distillation studies forms of training\-only information\([Penaloza et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib44)\)\. Related methods distill teacher\-only experience or system prompts\([Ye et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib12)\), document\-grounded evidence\([Stein et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib45)\), or behavioral instructions for reasoning compression\([Sang et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib33)\)\. For mathematical reasoning, Anti\-Self\-Distillation reverses harmful privileged\-teacher guidance\([Shen et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib35)\), DOPD dynamically routes token\-level supervision to mitigate privilege illusion\([Yu et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib13)\), PHF transfers privileged hidden\-state dynamics\([Li et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib46)\), AR\-OPD anchors privileged guidance to a locally compatible view\([Zhang, 2026](https://arxiv.org/html/2609.21619#bib.bib37)\), Purified OPSD removes reference\-induced non\-transferable components\([Shen et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib38)\), and DAPD introduces dual anchoring against information asymmetry\([Wu et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib42)\)\. Recent analyses further show that privileged context can degrade thinking models\([Kaur et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib14)\), that references from other problems can retain comparable gains\([Ichihara et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib47)\), that correct references do not provide consistent benefits\([Shrestha and Tessier, 2026](https://arxiv.org/html/2609.21619#bib.bib48)\), and that structured privileged information can outperform raw solution traces\([Zhao et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib49)\)\. Together, these findings suggest that privileged context changes not only the information available to the teacher, but also the teacher behavior induced by that context, complicating the interpretation of privileged teacher likelihoods as transferable knowledge\.

#### Reliability of Teacher Supervision\.

Recent work has also questioned the assumption that teacher supervision is uniformly reliable across tokens and trajectories\. Entropy\-Aware OPD adapts distillation to teacher uncertainty\([Jin et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib5)\), Uni\-OPD calibrates teacher guidance using outcome\-level order consistency\([Hou et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib18)\), and Position\-Weighted OPSD shows that teacher\-token reliability is strongly structured by reasoning position\([Liu et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib11)\)\. OGLS\-SD calibrates privileged teacher logits using outcome contrast\([Yang et al\., 2026b](https://arxiv.org/html/2609.21619#bib.bib50)\), while BRTS selects teacher rollouts according to correctness and student alignment\([Zhang et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib52)\)\. Related studies further show that reasoning tokens remain relatively stable while stylistic or surface\-form tokens are more sensitive to contextual interventions\([Pan et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib9);[He et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib10)\), and that privileged context can induce shortcut behavior\([Tian et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib15)\)\. More recently,[Ding and Zhang \(2026\)](https://arxiv.org/html/2609.21619#bib.bib51)reveal substantial, scale\-dependent noise in OPD teacher supervision and show that removing noisy supervision can leave student performance largely unchanged\. These studies establish that teacher supervision is heterogeneous, but characterize reliability primarily through uncertainty, outcomes, positions, trajectory quality, or specific privileged\-reference effects\. In contrast, the variation of teacher likelihoods under controlled contextual interventions has not been explicitly modeled as a finite\-intervention self\-deviation region for calibrating the teacher–student discrepancy in OPD\.

### A\.2Data Collection and Preparation for TSD Analysis

Our TSD analysis is conducted on a collection of student\-generated reasoning trajectories derived from DAPO\-17K\([Yu et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib8)\)\. Since DAPO\-17K provides problems and ground\-truth answers but does not include reference solutions, we first construct a solution\-augmented subset, then generate on\-policy student rollouts, and finally rescore the fixed trajectories under different teacher\-side contextual interventions\. Table[5](https://arxiv.org/html/2609.21619#A1.T5)summarizes the configurations used in this process\.

#### Reference\-Solution Preparation\.

We use Qwen3\-235B\-A22B\-Instruct\-2507\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4)\)to generate one reference solution for each problem in DAPO\-17K\. The model is used in its no\-thinking mode with a maximum response length of 12,288 tokens, temperature1\.01\.0, and top\-pp1\.01\.0\. All generations use the following shared system prompt:

System PromptYou are a helpful math assistant\. Please solve the math problem\.You must enclose your final answer exactly within \\boxed\{\}\.

We extract the final answer enclosed in\\boxed\{\}and compare it with the ground\-truth answer provided by DAPO\-17K\. Solutions with incorrect final answers are discarded\. After filtering, we retain 15,560 question–solution pairs, covering90\.18%90\.18\\%of the 17,255 problems, with 1,695 problems unsolved\. The retained reference solutions contain an average of 7,636 characters, a median of 5,679 characters, a 95th percentile of 19,493 characters, and a range of 445–39,695 characters\.

#### Student Rollout Collection\.

Starting from the 15,560 retained questions, we generate one on\-policy response per question using Qwen3\-1\.7B\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4)\)in thinking mode\. We set the maximum response length to 16,384 tokens, temperature to1\.01\.0, and top\-ppto1\.01\.0, while using the same system prompt as above\. Rollout generation is stopped once the cumulative number of student\-generated response tokens reaches approximately 60 million\. This results in 6,528 question–response pairs used for subsequent TSD analysis\. The student responses contain an average of 9,270 tokens and a median of 7,758 tokens\.

#### Training Data Usage\.

All distillation methods in our main experiments are trained on the same 15,560 question–solution pairs\.

Table 5:Configurations used for preparing the data for TSD analysis\. Solution generation and student rollout collection use one generation per question\. Teacher rescoring does not involve sampling, since the student trajectory is kept fixed\.\(a\)Qwen3\-8B

\(b\)Qwen3\-4B\-Thinking\-2507

Figure 6:Prevalence of significant TSD across thresholds and teacher models\. Although the absolute prevalence decreases with stricter thresholds, substantial TSD continues to emerge under task\-agnostic interventions, while solution\-level privilege consistently broadens the affected set\.\(a\)Qwen3\-8B

\(b\)Qwen3\-4B\-Thinking\-2507

Figure 7:Retention of significant TSD across thresholds and teacher models\. Less informative interventions retain nearly all affected token positions under solution\-level privilege, whereas retention in the reverse direction is substantially lower\.
#### Construction of Privileged Contexts\.

For the answer\-level interventions in Table[1](https://arxiv.org/html/2609.21619#S2.T1), the positive variant uses the ground\-truth answer associated with the current problem\. To constructcansnegc\_\{\\mathrm\{ans\}\}^\{\\mathrm\{neg\}\}, we randomly generate an incorrect answer with the same number of digits as the corresponding correct answer, thereby controlling for differences in answer length\.

For the solution\-level interventions,csolposc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}uses the verified reference solution generated for the current problem\. To constructcsolnegc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{neg\}\}, we randomly select the reference solution of a different problem whose tokenized length differs from that of the current reference solution by less than10%10\\%\. This length\-matching constraint reduces the possibility that differences between positive and negative solution\-level interventions are driven primarily by contextual length rather than content\.

\(a\)Qwen3\-8B

\(b\)Qwen3\-4B\-Thinking\-2507

Figure 8:Semantic consistency of TSD across thresholds and teacher models\. Contrasting interventions consistently exhibit substantial positional overlap, high directional agreement, and predominantly shared deviation across thresholds and model scales\.
#### Teacher Likelihood Rescoring\.

We use Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib4)\)as the teacher and evaluate each fixed Qwen3\-1\.7B rollout under nine teacher contexts:

c0,cinstpos,cinstneg,cevalpos,cevalneg,canspos,cansneg,csolpos,csolneg\.\\begin\{gathered\}c\_\{0\},\\quad c\_\{\\mathrm\{inst\}\}^\{\\mathrm\{pos\}\},\\;c\_\{\\mathrm\{inst\}\}^\{\\mathrm\{neg\}\},\\ c\_\{\\mathrm\{eval\}\}^\{\\mathrm\{pos\}\},\\;c\_\{\\mathrm\{eval\}\}^\{\\mathrm\{neg\}\},\\ c\_\{\\mathrm\{ans\}\}^\{\\mathrm\{pos\}\},\\;c\_\{\\mathrm\{ans\}\}^\{\\mathrm\{neg\}\},\\ c\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\},\\;c\_\{\\mathrm\{sol\}\}^\{\\mathrm\{neg\}\}\.\\end\{gathered\}\(15\)The shared system prompt above is included in every condition and is therefore not itself treated as a contextual intervention\. For all nine contexts, the problemxx, student rolloutyy, prefixy<ty\_\{<t\}, and evaluated tokenyty\_\{t\}are kept identical; only the additional teacher\-side context is changed\. The teacher does not generate new trajectories\. Instead, we recompute the token\-level log\-likelihood assigned to every student\-generated token under each context, allowing the resulting likelihood differences to be attributed directly to the contextual intervention\.

Consequently, the TSD analysis is based on approximately 60 million student\-generated tokens, with each token rescored under the baseline context and eight contextual interventions\.

### A\.3TSD Semantic Consistency Across Thresholds and Model Scales

We further examine whether the TSD patterns reported in the main text persist across significance thresholds and teacher scales\. Using the same Qwen3\-1\.7B student trajectories described in Appendix[A\.2](https://arxiv.org/html/2609.21619#A1.SS2), we repeat the analysis with Qwen3\-8B and Qwen3\-4B\-Thinking\-2507 as teachers underτ∈\{0\.01,0\.05,0\.10\}\\tau\\in\\\{0\.01,0\.05,0\.10\\\}\. We consider the prevalence of significant TSD, retention across intervention groups, and semantic consistency across the six teacher–threshold configurations\.

#### Prevalence of Significant TSD\.

Figure[6](https://arxiv.org/html/2609.21619#A1.F6)shows that increasingτ\\taureduces the absolute prevalence of significant TSD, while preserving the relative pattern across intervention groups\. For Qwen3\-8B, the union prevalence under task\-agnostic instructions decreases from29\.8%29\.8\\%atτ=0\.01\\tau=0\.01to10\.1%10\.1\\%atτ=0\.10\\tau=0\.10, yet remains comparable to evaluative feedback at29\.1%29\.1\\%and9\.2%9\.2\\%, respectively\. The same pattern holds for Qwen3\-4B\-Thinking\-2507, where task\-agnostic instructions remain close to evaluative feedback across all thresholds\. Solution\-level privilege consistently produces the highest prevalence, supporting the conclusion that richer privileged context broadens the affected set rather than being necessary for TSD to emerge\.

#### Retention Across Intervention Groups\.

Figure[7](https://arxiv.org/html/2609.21619#A1.F7)exhibits a consistent asymmetric retention pattern across thresholds and teacher models\. Token positions affected under instruction, evaluation, and answer interventions are largely retained under solution\-level privilege\. Even atτ=0\.10\\tau=0\.10, their retention into the solution group remains at least93\.4%93\.4\\%for Qwen3\-8B and94\.3%94\.3\\%for Qwen3\-4B\-Thinking\-2507\. The reverse retention is substantially lower and decreases as the threshold becomes stricter\. This asymmetry indicates that richer privileged context predominantly extends an existing set of TSD\-sensitive positions rather than replacing it with a distinct deviation pattern\.

#### Semantic Consistency\.

Figure[8](https://arxiv.org/html/2609.21619#A1.F8)shows that the semantic\-consistency pattern also persists across thresholds and teacher models, and across all three intervention groups\. Asτ\\tauincreases, positional overlap generally decreases because fewer tokens remain significant under both contrasting interventions, while directional agreement remains consistently high\. Atτ=0\.10\\tau=0\.10, answer\-level agreement reaches93\.6%93\.6\\%for Qwen3\-8B and94\.5%94\.5\\%for Qwen3\-4B\-Thinking\-2507, with corresponding SDR values of76\.9%76\.9\\%and79\.6%79\.6\\%\. Evaluative interventions show the same trend, whereas solution\-level interventions consistently exhibit higher positional overlap but lower SDR\. Thus, stricter thresholds change which tokens remain significant without altering the broader finding that contrasting semantics induce strongly aligned and predominantly shared TSD\.

Across all configurations, the qualitative conclusions of the main analysis remain unchanged\. TSD emerges substantially without task\-specific knowledge, richer privileged context largely preserves and expands previously affected token positions, and contrasting intervention semantics leave much of the resulting TSD directionally aligned and shared\.These results show that the observed TSD patterns are not specific to a particular significance threshold or teacher scale\.

Table 6:Top\-24 and bottom\-24 token forms undercinstposc\_\{\\mathrm\{inst\}\}^\{\\mathrm\{pos\}\}andcinstnegc\_\{\\mathrm\{inst\}\}^\{\\mathrm\{neg\}\}atτ=0\.01\\tau=0\.01\. The final row reports unweighted means ofρτ\\rho\_\{\\tau\}and\|Δ\|¯\\overline\{\|\\Delta\|\}over each 24\-token subset\.Table 7:Top\-24 and bottom\-24 token forms undercevalposc\_\{\\mathrm\{eval\}\}^\{\\mathrm\{pos\}\}andcevalnegc\_\{\\mathrm\{eval\}\}^\{\\mathrm\{neg\}\}atτ=0\.01\\tau=0\.01\. The final row reports unweighted means ofρτ\\rho\_\{\\tau\}and\|Δ\|¯\\overline\{\|\\Delta\|\}over each 24\-token subset\.Table 8:Top\-24 and bottom\-24 token forms undercansposc\_\{\\mathrm\{ans\}\}^\{\\mathrm\{pos\}\}andcansnegc\_\{\\mathrm\{ans\}\}^\{\\mathrm\{neg\}\}atτ=0\.01\\tau=0\.01\. The final row reports unweighted means ofρτ\\rho\_\{\\tau\}and\|Δ\|¯\\overline\{\|\\Delta\|\}over each 24\-token subset\.

### A\.4Token\-Level TSD Patterns Across Contextual Interventions

To examine whether the token\-level structure of TSD depends on a particular contextual intervention, we repeat the token\-form analysis under all eight contextual interventions using Qwen3\-8B atτ=0\.01\\tau=0\.01\. Following the main analysis, we merge tokenizer variants corresponding to the same normalized surface form and retain forms occurring more than 20,000 times\. Tables[6](https://arxiv.org/html/2609.21619#A1.T6)–[9](https://arxiv.org/html/2609.21619#A1.T9)report the top\-24 and bottom\-24 forms ranked byρτ​\(v\)\\rho\_\{\\tau\}\(v\)for each positive–negative intervention pair, together with their significant\-TSD rates and mean deviation magnitudes\.

#### Stability Across Intervention Polarity\.

Positive and negative variants within each intervention group produce substantially overlapping token rankings\. Their top\-24 sets share18/2418/24forms for instruction,22/2422/24for evaluation,24/2424/24for answer\-level privilege, and23/2423/24for solution\-level privilege\. The corresponding bottom\-24 overlaps are24/2424/24,23/2423/24,24/2424/24, and24/2424/24\. Thus, reversing intervention polarity or correctness generally leaves the token forms most prone to TSD largely unchanged\. This is especially true for forms that remain comparatively stable across conditions, while the instruction\-level top set shows somewhat more variation than the other pairs\.

#### Stability Across Intervention Types\.

The same structure persists across different types of contextual intervention\. Any two of the eight intervention conditions share at least18/2418/24top\-ranked forms and22/2422/24bottom\-ranked forms, while16/2416/24top forms and22/2422/24bottom forms appear in every ranking\. Across all conditions, high\-ρτ\\rho\_\{\\tau\}forms are consistently dominated by natural\-language surface expressions such asmaybe,however,therefore,consider, andthink, whereas low\-ρτ\\rho\_\{\\tau\}forms are dominated by digits, mathematical symbols, and notation\. This separation is also quantitatively strong: the mean top\-24ρτ\\rho\_\{\\tau\}ranges from61\.1%61\.1\\%to91\.5%91\.5\\%, compared with only3\.5%3\.5\\%to8\.2%8\.2\\%for the bottom\-24, corresponding to an approximately1111–18×18\\timesgap within individual conditions\.

#### Context Modulates Strength More Than Token Identity\.

While the ranking structure remains stable, the magnitude of TSD varies substantially across contextual interventions\. Under positive interventions, the mean\|Δ\|¯\\overline\{\|\\Delta\|\}of the top\-24 forms increases from0\.0800\.080under instruction and0\.0840\.084under evaluation to0\.1820\.182under answer\-level privilege and0\.4360\.436under solution\-level privilege\. The bottom\-24 exhibits the same overall trend, rising from0\.0830\.083and0\.0910\.091to0\.1110\.111and0\.2900\.290, respectively\. Negative interventions show a similar overall increase: the top\-24 means are0\.1210\.121,0\.1080\.108,0\.2440\.244, and0\.2650\.265for instruction, evaluation, answer\-level privilege, and solution\-level privilege, respectively, while the corresponding bottom\-24 means are0\.1020\.102,0\.1020\.102,0\.1080\.108, and0\.1990\.199\. Thus, richer privileged context can substantially amplify teacher\-side likelihood shifts, particularly at the solution level\. Crucially, however, this amplification does not substantially alter which token forms occupy the two ends of the ranking: largely the same surface\-form tokens remain TSD\-prone, while numerical and symbolic forms remain comparatively stable\. These results indicate that contextual interventions primarily modulate the strength of teacher self\-deviation, whereas its token\-level concentration remains stable across intervention semantics and privilege types\.

Table 9:Top\-24 and bottom\-24 token forms undercsolposc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}andcsolnegc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{neg\}\}atτ=0\.01\\tau=0\.01\. The final row reports unweighted means ofρτ\\rho\_\{\\tau\}and\|Δ\|¯\\overline\{\|\\Delta\|\}over each 24\-token subset\.

### A\.5Trajectory\-Level Case Study of TSD

We complement the aggregate analysis with two trajectory\-level examples illustrating how TSD is distributed within complete reasoning traces\. We use the same Qwen3\-1\.7B student and Qwen3\-8B teacher configuration and visualize TSD undercsolposc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}\. Red and blue indicate decreases and increases in teacher log\-likelihood, respectively, with darker shades denoting larger\|ΔtT\|\|\\Delta\_\{t\}^\{T\}\|; tokens with\|ΔtT\|≤0\.1\|\\Delta\_\{t\}^\{T\}\|\\leq 0\.1are left uncolored\. Since the student trajectory is fixed, all highlighted variation is induced solely by the additional reference solution\.

Case 1: Correct Student RolloutOutcome: Correct

Problem\.Calculate the least integer greater than5\(−6\)\(−5\)\(−4\)⋯\(2\)\(3\)\(4\)5^\{\(\-6\)\(\-5\)\(\-4\)\\cdots\(2\)\(3\)\(4\)\}\.

Answer\.The student rollout outputs2\\boxed\{2\}, which matches the gold answer\.

TSD Legend\.\[−0\.3,−0\.1\)\[\-0\.3,\-0\.1\)\[−0\.7,−0\.3\)\[\-0\.7,\-0\.3\)\[−1\.5,−0\.7\)\[\-1\.5,\-0\.7\)\[−3,−1\.5\)\[\-3,\-1\.5\)<−3<\-3

\(0\.1,0\.3\]\(0\.1,0\.3\]\(0\.3,0\.7\]\(0\.3,0\.7\]\(0\.7,1\.5\]\(0\.7,1\.5\]\(1\.5,3\]\(1\.5,3\]\>3\>3

Student trajectory excerpt undercsolposc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}\.

Okay,soI need to find the least integer greater than$5^\{\(\-6\)\(\-5\)\(\-4\)\.\.\.\(2\)\(3\)\(4\)\}$\.Hmm, let mestartby understandingtheexponenthere\. The exponent istheproduct ofallintegers from \-6 up to 4\.Wait,letme confirm:theexponentis\(\-6\)\(\-5\)\(\-4\)\.\.\.\(2\)\(3\)\(4\)\.Sothat’smultiplyingallintegersfrom\-6to4,butIneed tocheckifthat’scorrect\.Letmecounthowmanynumbersarethere\.

Starting from \-6upto 4\.Let’ssee:from\-6 to4inclusive\.Thenumberoftermswouldbe 4 \- \(\-6\) \+ 1 = 11 terms\.Butwait,letmecheck:

Numbers:\-6, \-5, \-4, \-3, \-2, \-1, 0, 1, 2, 3, 4\.Yes, that’s 11 numbers\.Butwait,theproblemsays"\(\-6\)\(\-5\)\(\-4\)\.\.\.\(2\)\(3\)\(4\)"\.Sotheproductisfrom \-6to4,including0?Wait,but0 isincludedhere\.However,whenyou multiplyby0, the entire product becomes 0\.Butthentheexponentwouldbe0, so5^0 is 1\.Thenthe least integer greater than 1 is 2\.Butthatseemstoostraightforward\.MaybeImadea mistake here\.

Wait,butholdon\. Let mecheckagain\. Theproblemsays"\(\-6\)\(\-5\)\(\-4\)\.\.\.\(2\)\(3\)\(4\)"\.Istheproductfrom\-6upto 4,including0?Butifthat’s the case,thentheproductis zero,sotheexponentis zero,and5^0is1\.Therefore, the least integer greater than 1 is 2\.Butthatseemstooeasy\.Maybethere’s a misunderstanding here\.

Alternatively, maybe theproductisfrom\-6to4,excluding0? But theproblemdidn’t specify\.Wait,theoriginalproblemsays"\(\-6\)\(\-5\)\(\-4\)\.\.\.\(2\)\(3\)\(4\)"\.Theellipsisisbetween \-4 and 2,somaybetheproductisfrom\-6upto 4, butskipping0?Wait,buthow?Ifyou have\-6, \-5, \-4,\.\.\.,2, 3, 4\.Soifyou start at \-6andgo up to 4,butnotincluding0? But0 is between \-6and 4\.Sounlessthere’s a typo,buttheproblemiswrittenas\(\-6\)\(\-5\)\(\-4\)\.\.\.\(2\)\(3\)\(4\)\. Somaybetheproductis from \-6 to 4,including0?Butthenthe productiszero\. Butmaybethere’s amistakehere\.

Wait,butletmecheckiftheproductisfrom\-6 to 4,butnotincluding0\.Ifthat’s the case, thentheproductis\(\-6\)\(\-5\)\(\-4\)\.\.\.\(\-1\)\(1\)\(2\)\(3\)\(4\)\.Let’scheckhowmanyterms that would be\. From \-6 to \-1is6numbers, then1,2, 3, 4\.So total of 6 \+ 4 = 10 terms\.Butthentheproductwouldbe\(\-6\)\(\-5\)\.\.\.\(\-1\)\*1\*2 \* 3 \* 4\.Letme compute that\.

First,theproductfrom\-6 to \-1is\(\-1\)^6\*6\! = 1 \* 720 = 720\. Thenmultiplyby 1\*2\*3\*4 = 24\. So total product is 720 \* 24=17,280\.Then5^17280isanextremelylarge number\.Thenthe least integer greater than thatwouldbe5^17280 \+ 1,butthatseemswaytoobig\.Buttheproblemsays"the least integer greater than5^\{\.\.\.\}"somaybetheanswerissomethingelse\.Buttheoriginalproblemiswrittenas\(\-6\)\(\-5\)\.\.\.\(2\)\(3\)\(4\)\. Soiftheproduct includes 0,thenit’szero,so5^0is1\.Thentheleastinteger greater than 1 is 2\. Butmaybetheproblemis writtenwithadifferentintention\.

Wait,maybetheexponentisfrom\-6 to 4, but not including0? But how?Ifyouhavefrom\-6 to 4,butnotincluding0, thentheproductis\(\-6\)\(\-5\)\.\.\.\(\-1\)\(1\)\(2\)\(3\)\(4\)\.WhichiswhatIthoughtearlier\. But then theexponentis17280,so 5^17280 is agiganticnumber,andthe least integer greater than thatwouldbe 5^17280 \+ 1\. But thatseemslikeanunreasonableanswer\.However,maybethere’s amistakeinmycalculation\.

Wait,letme check the product again\.Iftheproductis from \-6 to 4,excluding0,then:

First,theproductfrom\-6 to \-1is\(\-6\)\(\-5\)\(\-4\)\(\-3\)\(\-2\)\(\-1\)\. Letmecompute that:

\(\-6\)\* \(\-5\) = 30

30 \* \(\-4\) = \-120

\-120 \* \(\-3\) = 360

360 \* \(\-2\) = \-720

\-720 \* \(\-1\) = 720\.Sothat’s 720\.

\[Remaining trajectory omitted\.\]

Case 2: Incorrect Student RolloutOutcome: Incorrect

Problem\.Alex the Kat has written6161problems for a math contest, and there are a total of187187problems submitted\. How many more problems does he need to write and submit before he has written half of the total problems?

Answer and error\.The student outputs93\.593\.5, whereas the gold answer is6565\. It incorrectly keeps the total fixed at187187and treats187/2=93\.5187/2=93\.5as the required additional count\.

TSD Legend\.\[−0\.3,−0\.1\)\[\-0\.3,\-0\.1\)\[−0\.7,−0\.3\)\[\-0\.7,\-0\.3\)\[−1\.5,−0\.7\)\[\-1\.5,\-0\.7\)\[−3,−1\.5\)\[\-3,\-1\.5\)<−3<\-3

\(0\.1,0\.3\]\(0\.1,0\.3\]\(0\.3,0\.7\]\(0\.3,0\.7\]\(0\.7,1\.5\]\(0\.7,1\.5\]\(1\.5,3\]\(1\.5,3\]\>3\>3

Student trajectory excerpt undercsolposc\_\{\\mathrm\{sol\}\}^\{\\mathrm\{pos\}\}\.

Okay,let’stryto figure outthisproblemstepby step\. So, Alex the Kat has already written 61 problemsfora math contest\.Thetotal number of problemssubmittedis187\.Thequestion is asking how many more problemsheneeds to write and submitbeforehe has written half of the total problems\.

First,Ineed tounderstandwhatexactlyisbeing asked here\. Thetotalnumberof problemssubmittedis187,butIthinkthatmightnotbedirectlyrelevant\.Wait,actually,theproblem saysAlexhas written 61problems, and there areatotal of 187 problems submitted\.Hmm,maybeIneed to clarifythat\.

Wait,maybethetotalnumber of problems in the contestis187? Or is 187 thenumberof problemssubmittedbyothers?Thewordingisa bitconfusing\. Let mereaditagain:"Alex the Kat has written 61 problems for a math contest, and there are a total of 187 problems submitted\.How many more problems does he need to write and submit before he has written half of the total problems?"

So,thetotalnumberof problems submitted is 187\.ButAlexhas written 61\.Wait,maybe"submitted" refers to the totalnumberof problemsthathave been submittedbyeveryone,including Alex?Oris187 the total number of problemsinthe contest?

Thisisabit ambiguous\.Butlet’sthink\.IfAlexhaswritten 61problems, andthereare187 problems submitted in total,maybethetotalnumberof problemsinthe contest is 187?Butthen hehaswritten 61,so he needs to writehalfof187,which is 93\.5?Butthatdoesn’t make sensebecauseyou can’twritehalf a problem\.Wait,maybethetotalnumber of problemsisdifferent\.

Alternatively,maybethetotalnumber of problemsinthe contest is187,andAlex has written 61\.Thenheneeds to writehalfofthetotal,whichis187/2=93\.5\.Butsinceyoucan’thavehalf a problem,maybeit’s94? Buttheproblemsays"howmany more problems does he need to write and submit,"so maybeit’sexpectingafractionalanswer?Butthatseemsodd\.

Wait,maybethetotalnumber of problems is not187\.Letmere\-examine the problem\.

"Alex the Kat has written 61 problems for a math contest, and there are a total of 187 problems submitted\. How many more problems does he need to write and submit before he has written half of the total problems?"

Hmm\.Maybe"submitted"refersto the total number of problems that have been submitted by all participants,includingAlex?So,Alexhas written 61problems, andthereare 187 problemssubmittedin total\.So,thetotal number of problemsinthe contest is 187?ButthenAlex haswritten61,so he needs to write half of 187,which is 93\.5\. Butsinceyoucan’t write half a problem,maybetheansweris94? Buttheproblemisprobablyexpectinganexactanswer\.

Alternatively,maybethe total number of problemsinthe contest isnot187\.Maybe187 is thetotalnumber of problemsthathave been submitted,andAlexhaswritten 61\.Then, thetotalnumber of problemsinthe contestis187?Butthenheneeds to write half of that,whichis 93\.5\. But again,thisseemsodd\.

Wait,maybetheproblem isthatAlexhas written 61problems, andthereare 187 problemsintotal\.So,heneeds to write half of 187,which is 93\.5\.Butsincehecan’t write half a problem, maybeweneed toroundup?Buttheproblemmightbeexpectinganexact answer\.

Alternatively,maybetheproblemis thatthetotal number of problems submittedis187,andAlexhaswritten 61\. So, the total number of problemsthathavebeen submittedis187\.ButAlexneedsto writehalfof the totalproblems\.Wait,butifthetotalproblemssubmitted is 187,thenhalfof that is 93\.5\.So,heneedstowrite93\.5moreproblems?Butthat’s not possible\.

Wait,maybeIneed to think differently\.Maybethetotal number of problemsinthe contest is187\.Alexhas written 61,sohe needs to writehalfof187, which is 93\.5\.Therefore, he needs to write 93\.5moreproblems\. Butsinceyoucan’t write half a problem,maybetheansweris 94?Buttheproblemsays"how many more problems does he need to write and submit,"somaybeit’s expectingafractionalanswer?Butinreality,problemsarewholenumbers\.

\[Remaining trajectory omitted\.\]

#### TSD Throughout a Correct Trajectory\.

Case 1 shows a rollout that correctly concludes2\\boxed\{2\}after recognizing that the exponent contains zero\. Despite reaching the correct reasoning early, the student repeatedly revisits its interpretation\. TSD is distributed throughout this process rather than being confined to task\-critical mathematical steps\. Strong deviations frequently occur on discourse and reasoning\-management expressions such asHmm,Wait,But,However,Maybe, andAlternatively, whereas much of the numerical and symbolic content remains comparatively stable\. This qualitatively matches the token\-level pattern in Table[2](https://arxiv.org/html/2609.21619#S2.T2)\.

#### TSD Is Not Necessarily Localized Around Reasoning Errors\.

Case 2 shows an incorrect rollout in which the student repeatedly treats187187as a fixed total and reasons from187/2=93\.5187/2=93\.5\. The correct relation is

61\+x=187\+x2,61\+x=\\frac\{187\+x\}\{2\},\(16\)wherexxdenotes the number of additional problems Alex needs to write, which yieldsx=65x=65\. However, the TSD pattern does not concentrate around this identifiable conceptual error or the repeated occurrence of93\.593\.5\. Instead, substantial deviations again appear broadly on expressions such asFirst,Hmm,Wait,Alternatively, andTherefore\. Thus, even when the privileged solution directly resolves the student’s mistake, the induced likelihood variation does not behave as a localized reasoning\-error signal\.

#### Takeaway\.

Across both correct and incorrect trajectories, TSD is broadly distributed and particularly pronounced on tokens that organize or redirect reasoning, rather than selectively aligning with mathematical correctness or error locations\. These examples provide a trajectory\-level illustration of why context\-induced teacher variation should not be indiscriminately treated as transferable supervision, further motivating the calibration mechanism in Cal\-OPD\.

### A\.6Training Details and Hyperparameters

We implement all methods in theverlframework\([Sheng et al\., 2025](https://arxiv.org/html/2609.21619#bib.bib19)\)and train on 8 NVIDIA H20 GPUs, with 4 GPUs hosting the student and 4 hosting the teacher\. All methods share a common training recipe unless otherwise specified\. We first summarize the shared configuration, then describe method\-specific hyperparameters and the algorithmic role each plays\.

#### Shared Training Configuration\.

Table[10](https://arxiv.org/html/2609.21619#A1.T10)lists the optimization and training\-loop settings common to all baselines and Cal\-OPD\. All methods are trained for 100 steps, with a maximum response length of 16,384 tokens during training, temperature1\.01\.0, and top\-pp1\.01\.0\. We use AdamW with a learning rate of1×10−61\\times 10^\{\-6\}, no learning\-rate warmup, weight decay0\.010\.01, a cosine schedule, and gradient clipping at1\.01\.0\. Each training step processes 256 on\-policy trajectories, with one rollout per question, one PPO epoch, and a mini\-batch size of 256\. The entropy coefficient and KL regularization are set to zero because the OPD distillation objective replaces the standard RL KL penalty\. We use GRPO as the advantage estimator with standard deviation normalization within each group\([Yu et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib8)\)\.

CategoryHyperparameterValueOptimizerLearning rate1×10−61\\times 10^\{\-6\}LR warmup ratio0\.0Weight decay0\.01LR schedulercosineGradient clip1\.0Training loopTraining steps100Max prompt length2,048Max response length16,384Temperature1\.0Top\-pp1\.0PPO epochs1Train batch size256 trajectories/stepPPO mini\-batch size256PPO micro\-batch size1 \+ dynamic batchingEntropy coefficient0\.0KL regularizationdisabledDistillationStudent chunk size1024Teacher chunk size128Teacher max logprobs16Token selection ratio1\.0Policy loss modereinforceAdvantage and lossLoss max clamp10Advantage estimatorGRPOTable 10:Shared training configuration for all methods\.
#### Distillation\-Specific Configuration\.

The distillation module operates on student\-generated trajectories and queries the teacher for token\-level log\-probabilities\. We set the student chunk size to 1024 and the teacher chunk size to 128, which controls the number of tokens processed per forward pass and is chosen to balance GPU memory against throughput\. The teacher returns log\-probabilities over its top\-16 tokens by default, providing a dense yet bounded supervision signal\. All student\-generated tokens participate in the distillation loss without filtering, corresponding to a token selection ratio of 1\.0 with random selection\. The policy loss is computed inreinforcemode, and the per\-token loss is clamped at a maximum value of 10 to prevent destabilizing updates from outlier advantages\.

#### Method\-Specific Hyperparameters\.

Table[11](https://arxiv.org/html/2609.21619#A1.T11)summarizes the hyperparameters that differ across baselines\. Below, we describe the algorithmic role of each\.

Table 11:Method\-specific hyperparameters\. Entries marked “none” use the shared configuration unchanged\.
#### ExOPD\.

ExOPD extrapolates the teacher\-derived reward beyond the teacher’s own performance level with a scaling factorλexo\>1\\lambda\_\{\\mathrm\{exo\}\}\>1\([Yang et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib17)\); we setλexo=1\.25\\lambda\_\{\\mathrm\{exo\}\}=1\.25\. It maintains both the current policy and the pre\-RL base model\. To avoid storing duplicate weights, we parameterize the student with LoRA adapters \(rank 64, alpha 128, all linear layers\) and switch between base and policy by disabling or enabling the adapter\. We setloss\_max\_clamp=null andloss\_agg\_mode=token\-mean\.

#### EOPD\.

EOPD augments the reverse\-KL distillation objective with a forward\-KL term applied selectively to high\-entropy teacher tokens\([Jin et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib5)\)\. The entropy thresholdτent=0\.8\\tau\_\{\\mathrm\{ent\}\}=0\.8determines which tokens receive the forward\-KL treatment: tokens whose teacher entropy exceeds this threshold are supervised with forward KL to preserve generation diversity, while lower\-entropy tokens retain the standard reverse\-KL objective\. The mixing coefficientαent=1\.0\\alpha\_\{\\mathrm\{ent\}\}=1\.0controls the weight of the forward\-KL term relative to the reverse\-KL term\. To compute teacher entropy, EOPD requires one additional log\-probability slot beyond the top\-16 used by other methods; we therefore set teacher max logprobs to 17 for EOPD only\.

#### Uni\-OPD\.

Uni\-OPD introduces a dual\-perspective recipe that combines student\-side data balancing with teacher\-side outcome\-guided margin calibration\([Hou et al\., 2026a](https://arxiv.org/html/2609.21619#bib.bib18)\)\. On the student side, an online correctness\-aware filter reshapes each training batch to maintain a target correct\-to\-incorrect ratioρcorr=0\.5\\rho\_\{\\mathrm\{corr\}\}=0\.5; we implement this by training on 16 questions per step withn=16n\{=\}16rollouts per question, yielding 256 trajectories per step, and applying sample filtering to achieve the target ratio\. The mini\-batch size is correspondingly set to 16 questions\. On the teacher side, Uni\-OPD calibrates token\-level teacher margins against trajectory\-level outcome rewards\. The margin scope is set togroup, meaning calibration is performed within each group of rollouts for the same question\. The trajectory reduction ismean, the margin direction isspread\(which spreads apart the margins of correct and incorrect trajectories\), and the target margin isδmargin=0\.4\\delta\_\{\\mathrm\{margin\}\}=0\.4\. The loss aggregation mode is token\-mean, and the loss is not clamped\.

#### Privileged\-OPD\.

Privileged\-OPD conditions the teacher on additional training\-time information, specifically the verified reference solution for each problem\([Ye et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib12);[Kaur et al\., 2026](https://arxiv.org/html/2609.21619#bib.bib14)\)\. Because reference solutions are substantially longer than the problem statements, we increase the maximum prompt length from the shared default to 12,288 tokens to accommodate the concatenation of problem, reference solution, and student rollout prefix\. Solutions exceeding this length are truncated\. All other settings follow the standard OPD configuration\. In our implementation, the privileged context is prepended to the problem and student prefix in the teacher’s input, while the student receives only the problem and its own rollout, preserving the information asymmetry that Privileged\-OPD exploits\.

#### Cal\-OPD\.

Cal\-OPD follows the shared configuration above\. It additionally uses the evaluative feedback interventionscevalposc\_\{\\mathrm\{eval\}\}^\{\\mathrm\{pos\}\}andcevalnegc\_\{\\mathrm\{eval\}\}^\{\\mathrm\{neg\}\}to probe the teacher’s self\-deviation region, with relaxation factorλ=5\\lambda=5\. These interventions are used only to estimate the TSD region and are not directly distilled into the student\. The distillation loss then operates on the calibrated advantageAtCalA\_\{t\}^\{\\mathrm\{Cal\}\}in place of the rawAtOPDA\_\{t\}^\{\\mathrm\{OPD\}\}, as described in Section[3](https://arxiv.org/html/2609.21619#S3)\.

### A\.7Additional Training Dynamics

\(a\) Average Response Length

\(b\) Entropy

\(c\) Top\-16 Token Overlap

\(d\) Top\-16 Probability\-Mass Overlap

Figure 9:Training dynamics of OPD and Cal\-OPD on Qwen3\-1\.7B\.#### OPD vs\. Cal\-OPD with evaluative calibration\.

Figure[9](https://arxiv.org/html/2609.21619#A1.F9)reveals qualitatively different training dynamics between OPD and Cal\-OPD with𝒞eval\\mathcal\{C\}\_\{\\mathrm\{eval\}\}\. OPD progressively increases response length while reducing student entropy, whereas Cal\-OPD maintains substantially shorter responses and higher entropy throughout training\. This contrast is also reflected in how each method aligns with the teacher’s top\-16 distribution\. LetHHdenote the student entropy,Top16Tok\\mathrm\{Top16Tok\}the top\-16 token overlap, andTop16Mass\\mathrm\{Top16Mass\}the top\-16 probability\-mass overlap\. The dynamics can be summarized as

HCal​\-​OPD\\displaystyle H\_\{\\mathrm\{Cal\\text\{\-\}OPD\}\}\>HOPD,\\displaystyle\>H\_\{\\mathrm\{OPD\}\},\(17\)Top16TokCal​\-​OPD\\displaystyle\\mathrm\{Top16Tok\}\_\{\\mathrm\{Cal\\text\{\-\}OPD\}\}\>Top16TokOPD,\\displaystyle\>\\mathrm\{Top16Tok\}\_\{\\mathrm\{OPD\}\},Top16MassCal​\-​OPD\\displaystyle\\mathrm\{Top16Mass\}\_\{\\mathrm\{Cal\\text\{\-\}OPD\}\}<Top16MassOPD\.\\displaystyle<\\mathrm\{Top16Mass\}\_\{\\mathrm\{OPD\}\}\.In particular, the top\-16 token overlap under Cal\-OPD steadily increases to approximately0\.6930\.693, whereas OPD peaks earlier and then slightly declines\. Meanwhile, OPD achieves a higher top\-16 probability\-mass overlap, indicating stronger matching of the teacher’s probability allocation\.

Together, these dynamics suggest that OPD increasingly fits the teacher’s dominant probability pattern, whereas Cal\-OPD preserves a broader student distribution while covering more teacher\-supported token patterns, enabling the student to capture multiple plausible reasoning modes rather than overfitting to a particular one\.

#### Cal\-OPD with instruction\-level calibration\.

In contrast to evaluative calibration, instruction\-level calibration with𝒞inst\\mathcal\{C\}\_\{\\mathrm\{inst\}\}yields dynamics thatclosely resemble standard OPD, indicating that its calibration signal is too weak to meaningfully separate TSD from transferable knowledge\. As shown in Figure[9](https://arxiv.org/html/2609.21619#A1.F9), the student entropy under𝒞inst\\mathcal\{C\}\_\{\\mathrm\{inst\}\}follows the same declining trend as OPD and stabilizes around0\.200\.20, whereas the other Cal\-OPD variants maintain substantially higher entropy\. Its top\-16 probability\-mass overlap is also nearly identical to that of OPD, stabilizing around0\.8860\.886, which shows that it continues to match the teacher’s dominant probability allocation as strongly as the uncalibrated objective\. Similarly, its top\-16 token overlap remains low at approximately0\.6840\.684, again close to OPD and well below the levels reached by𝒞eval\\mathcal\{C\}\_\{\\mathrm\{eval\}\},𝒞ans\\mathcal\{C\}\_\{\\mathrm\{ans\}\}, and𝒞sol\\mathcal\{C\}\_\{\\mathrm\{sol\}\}\. These trends indicate that, without task\-specific information or strong semantic contrast, the estimated TSD region is too narrow to filter the teacher–student discrepancy effectively, and Cal\-OPD largely degenerates to standard OPD behavior\.

#### Cal\-OPD with solution\-level calibration\.

Solution\-level calibration with𝒞sol\\mathcal\{C\}\_\{\\mathrm\{sol\}\}lies at the opposite extreme: it over\-filters the teacher–student discrepancy and causes the supervision signal to collapse\. As shown in Figure[9](https://arxiv.org/html/2609.21619#A1.F9), the student entropy under𝒞sol\\mathcal\{C\}\_\{\\mathrm\{sol\}\}is the highest among all methods and continues to rise throughout training, stabilizing near0\.360\.36, while its average response length is the lowest and steadily decreases to roughly5\.55\.5K tokens\. More strikingly, its top\-16 probability\-mass overlap is not only the lowest among all variants but also falls below its initial value, stabilizing around0\.8620\.862\. Its top\-16 token overlap remains moderate at approximately0\.6880\.688, below𝒞eval\\mathcal\{C\}\_\{\\mathrm\{eval\}\}and𝒞ans\\mathcal\{C\}\_\{\\mathrm\{ans\}\}, but above OPD\. These trends indicate that the TSD region estimated from solution\-level privilege is excessively broad, so that a large fraction of the teacher–student discrepancy is classified as TSD and zeroed out\. As a result, the student receives too few effective supervision signals to follow the teacher distribution, and instead drifts toward a high\-entropy, short\-response regime\.Over\-calibration therefore degrades the signal more severely than no calibration at all, which is consistent with its lowest downstream performance among all variants\.

\(a\) Entropy

\(b\) Top\-16 Probability\-Mass Overlap

\(c\) Top\-16 Token Overlap

Figure 10:Training dynamics comparison between privileged OPD and Cal\-OPD with solution\-level calibration\.
#### Failure Modes of Privileged\-OPD and Cal\-OPD with Solution\-Level Calibration\.

Both methods shorten responses and degrade performance, but for different reasons\. All top\-16 metrics are computed against the unprivileged teacher\. Privileged\-OPD keeps student entropy low and stable \(around0\.250\.25\) while its top\-16 probability\-mass overlap rises to about0\.8800\.880, indicating that the student becomes overconfident and concentrates on a narrow token set\. In contrast, Cal\-OPD with𝒞sol\\mathcal\{C\}\_\{\\mathrm\{sol\}\}drives entropy up to about0\.360\.36, and its top\-16 probability\-mass overlap even drops to roughly0\.8590\.859, showing that it fails to match the unprivileged teacher\. We attribute this to two distinct failure modes: Privileged\-OPD suffers fromoverconfident shortcut collapse, where the student imitates the privileged teacher’s sharp distribution but loses diversity; Cal\-OPD with𝒞sol\\mathcal\{C\}\_\{\\mathrm\{sol\}\}suffers fromsignal collapse, where an excessively broad TSD region zeros out useful supervision, causing the student to drift into a high\-entropy, short\-response regime\.

### A\.8Efficiency Analysis

Table 12:Efficiency comparison between Cal\-OPD and OPD across different teacher–student configurations\.#### Efficiency of Cal\-OPD\.

Cal\-OPD requires two additional teacher forward passes per step to probe the TSD region, which introduces extra computation\. Table[12](https://arxiv.org/html/2609.21619#A1.T12)compares the wall\-clock time per step and the total time for 100 steps between Cal\-OPD and OPD across three teacher–student configurations\. For the Qwen3\-4B\-Thinking\-2507→\\rightarrowQwen3\-1\.7B setting, Cal\-OPD is actually faster than OPD \(14\.90 vs\. 18\.80 minutes per step\), because Cal\-OPD avoids the response\-length expansion that OPD exhibits, and the teacher is not large enough for the extra forward passes to dominate\. In the other two configurations, where the teacher \(30B\-A3B\) is substantially larger than the student \(8B or 4B\), the two additional teacher forward passes become the main computational overhead, making Cal\-OPD slower per step \(50\.05 vs\. 35\.19 and 48\.76 vs\. 31\.26 minutes\)\. Overall, Cal\-OPD remains practically efficient: its extra cost is bounded by two teacher forward passes, and it can even reduce total training time when the teacher–student size gap is moderate\.

### A\.9Retention\-Matched Control on Teacher–Student Discrepancy

\(a\) Qwen3\-4B\-Thinking\-2507→\\rightarrowQwen3\-1\.7B

\(b\) Qwen3\-30B\-A3B\-Thinking\-2507→\\rightarrowQwen3\-4B

Figure 11:Gradient norm dynamics of OPD training across different teacher–student pairs\.Figure[11](https://arxiv.org/html/2609.21619#A1.F11)reports the pre\-clipping gradient norm during OPD training\. In both teacher–student configurations, the norm is initially large, around120120and6565, and gradually decreases to approximately3535and1515\. All methods apply gradient clipping with thresholdC=1C=1, so the update is computed as

𝐠~=𝐠⋅min⁡\(1,C‖𝐠‖\)\.\\tilde\{\\mathbf\{g\}\}=\\mathbf\{g\}\\cdot\\min\\\!\\left\(1,\\frac\{C\}\{\\\|\\mathbf\{g\}\\\|\}\\right\)\.\(18\)Since‖𝐠‖\\\|\\mathbf\{g\}\\\|is far above11throughout training, the raw gradient is scaled by roughlyC/‖𝐠‖C/\\\|\\mathbf\{g\}\\\|, corresponding to an attenuation of one to two orders of magnitude\.Thus, substantial signal attenuation is already inherent to standard OPD, and the gains of Cal\-OPD cannot be attributed to a smaller overall gradient magnitude\. Cal\-OPD instead changes the*relative*contribution of individual token advantages by removing the TSD\-explained component, while leaving the global clipping behavior unchanged\.

To rule out the possibility that Cal\-OPD gains merely from attenuating the optimization signal, we construct three retention\-matched baselines that mimic its signal reduction without TSD calibration\.

- •Advantage\-Sync \(Advantage\-Level Synchronization\)\.At each training stepkk, we read Cal\-OPD’s advantage retention ratiorkr\_\{k\}, defined as the ratio between the sum of absolute calibrated advantages and the sum of absolute original teacher–student discrepancies\. We then scale standard OPD’s token\-level advantages by the same factor,A~tOPD=rk⋅AtOPD\\tilde\{A\}\_\{t\}^\{\\mathrm\{OPD\}\}=r\_\{k\}\\cdot A\_\{t\}^\{\\mathrm\{OPD\}\}, so that the global advantage magnitude matches Cal\-OPD step by step\.
- •Token\-Sync \(Token\-Level Synchronization\)\.At each training stepkk, we read Cal\-OPD’s zero\-advantage token ratiozkz\_\{k\}, i\.e\., the fraction of tokens whose calibrated advantage is zero\. We then randomly select the same fractionzkz\_\{k\}of tokens in standard OPD and set their advantages to zero, keeping the remaining advantages unchanged\. This matches Cal\-OPD’s sparsity pattern in terms of the number of discarded tokens, but not which tokens are discarded\.
- •TSD\-Filter \(TSD\-Threshold Filtering\)\.For each token, we compute the maximum absolute deviation induced by the positive and negative privileged interventions, and discard the token entirely if this deviation exceeds a thresholdτTSD\\tau\_\{\\mathrm\{TSD\}\}\. We evaluate three thresholds,τTSD∈\{0\.1,0\.05,0\.01\}\\tau\_\{\\mathrm\{TSD\}\}\\in\\\{0\.1,0\.05,0\.01\\\}, corresponding to increasingly aggressive filtering\. Unlike Cal\-OPD, which retains the residual discrepancy beyond the estimated TSD region, this baseline removes the token from the loss altogether\.

These controls match Cal\-OPD in global advantage scale, token sparsity, and TSD\-based token selection, respectively, but omit the calibrated residual\.

Table 13:Retention\-matched control on teacher–student discrepancy for Qwen3\-4B\-Thinking\-2507→\\rightarrowQwen3\-1\.7B\.Table[13](https://arxiv.org/html/2609.21619#A1.T13)compares Cal\-OPD against three retention\-matched controls\.Advantage\-Syncapplies the same global scaling to OPD’s advantages and achieves50\.850\.8, identical to OPD, showing that reducing the overall advantage magnitude alone brings no benefit\.Token\-Syncrandomly masks the same fraction of tokens and even drops to50\.150\.1, indicating that matching the sparsity level without selecting the right tokens is insufficient\.TSD\-FilterwithτTSD=0\.1\\tau\_\{\\mathrm\{TSD\}\}=0\.1reaches52\.752\.7, the strongest control, but still falls short of Cal\-OPD; more aggressive thresholds of0\.050\.05and0\.010\.01degrade to51\.151\.1and49\.449\.4, respectively\. These results show that neither global attenuation, nor random token removal, nor hard TSD\-based filtering can reproduce the gains of Cal\-OPD\.The advantage of Cal\-OPD therefore stems from its soft calibration mechanism, which retains the discrepancy beyond the estimated TSD region, rather than from signal attenuation or token sparsity alone\.