I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

# I-SDPO: Instance-Level Adaptive Self-DistillationPolicy Optimization
Source: [https://arxiv.org/html/2608.12957](https://arxiv.org/html/2608.12957)
## I\-SDPO: Instance\-Level Adaptive Self\-Distillation Policy Optimization

Xinhong MaZezhong TanZiqiang DongAffiliation:Qwen Large Model Application Team, Alibaba

###### Abstract

Group Relative Policy Optimization \(GRPO\) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect\. Privileged self\-distillation can fill this gap with dense token supervision, yet applying it throughout training creates a different failure mode: the teacher is a biased, low\-variance surrogate for the reward objective, so persistent imitation can oppose reward\-improving updates after the policy becomes capable of producing successful trajectories\. We introduce I\-SDPO \(Instance\-Level Adaptive Self\-Distillation Policy Optimization\), which treats teacher reliance as capability\-dependent\. I\-SDPO makes one routing decision per input instance and shares it across that instance’s rollout group: all\-incorrect groups use a privileged self\-distillation objective, whereas any\-success groups remain intact for GRPO\. This design uses imitation only where group\-relative rewards are uninformative\. A local analysis characterizes when teacher and reward directions align and shows that a non\-vanishing biased distillation weight induces an optimization bias floor\. The routing rule automatically reduces the expected distillation rate as success probability rises, withdrawing teacher influence without a hand\-designed schedule\. On SciKnowEval, I\-SDPO obtains the best result in all four scientific domains and improves average mean@16 accuracy from 56\.67% with GRPO to 70\.31%, with a maximum domain gain of 18\.24 points\.

## 1Introduction

Reinforcement learning \(RL\) post\-training has emerged as a critical paradigm for improving the reasoning capabilities of large language models \(LLMs\)\([Ouyang et al\. 2022](https://arxiv.org/html/2608.12957#bib.bib16);[Guo et al\. 2025](https://arxiv.org/html/2608.12957#bib.bib7)\)\. Among RL approaches, Group Relative Policy Optimization \(GRPO\)\([Shao et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib20)\)has gained widespread adoption due to its simplicity and effectiveness: it eliminates the need for a separate critic model by computing advantages relative to other samples within the same rollout group\.

However, GRPO suffers from a fundamental limitation that we term the*degenerate gradient problem*\. When allKKresponses sampled for a given prompt are incorrect, the within\-group advantages collapse to near\-zero values, as all rewards are similar\. Formally, if rewardsr1≈r2≈⋯≈rK≈0r\_\{1\}\\approx r\_\{2\}\\approx\\cdots\\approx r\_\{K\}\\approx 0, then the advantageAi=ri−r¯≈0A\_\{i\}=r\_\{i\}\-\\bar\{r\}\\approx 0for allii, producing negligible policy gradients\. This problem is particularly severe during early training when model capability is low, or on challenging problems where correct solutions are rare—precisely the situations where learning is most needed\.

Privileged self\-distillation supplies the missing dense signal: a teacher that also observes the ground\-truth solution can assign token\-level targets along student\-generated trajectories, an approach recently instantiated for LLM reasoning by on\-policy self\-distillation\([Zhao et al\. 2026](https://arxiv.org/html/2608.12957#bib.bib32)\)\. This signal has a favorable bias–variance profile early in training\. It is biased because matching the teacher distribution is not the same objective as maximizing sequence\-level reward, but it has much lower variance than waiting for rare successful samples\([Menon et al\. 2021](https://arxiv.org/html/2608.12957#bib.bib14)\)\. Early on, this bias can be preferable to a zero or highly variable RL update\.

The same bias becomes consequential later\. The privileged teacher is not a reward oracle: it scores tokens under a context containing both the correct solution and the student’s possibly erroneous prefix; it may prefer a particular reasoning style even when the reward accepts multiple valid solutions; and, as an exponential moving average \(EMA\) of the student, it shares the student’s systematic errors\. Thus, a fixed self\-distillation objective can continue pulling toward a teacher\-specific distribution after reward\-bearing trajectories become available\. This is not merely “noisy supervision\.” It is an objective\-mismatch problem whose relative cost grows as the RL estimator becomes more informative\. Prior analyses likewise connect distillation to a bias–variance trade\-off and show that repeated self\-distillation can eventually over\-regularize a learner\([Menon et al\. 2021](https://arxiv.org/html/2608.12957#bib.bib14);[Mobahi et al\. 2020](https://arxiv.org/html/2608.12957#bib.bib15)\)\.

The central question is therefore not whether the teacher is useful in general, but*when it should be trusted*\. We propose I\-SDPO \(Instance\-Level Adaptive Self\-Distillation Policy Optimization\), based on a capability\-dependent rule:make one routing decision per input instance and preserve its complete rollout group\. An all\-incorrect group has no relative reward signal and is routed to privileged self\-distillation\. If at least one response succeeds, the group remains under GRPO; in mixed groups, this preserves the contrast between successful and unsuccessful trajectories\. Unlike sample\-level routing \(SRPO\), I\-SDPO never overwrites the negative samples that make this comparison possible\.

This paper makes three contributions:

- •We formulate privileged self\-distillation as a biased, low\-variance surrogate for reward optimization\. A token\-space alignment criterion and a local quadratic analysis explain both its early utility and the bias floor caused by persistent teacher influence\.
- •We introduce instance\-level routing over rollout groups, which invokes self\-distillation exactly on all\-incorrect groups and otherwise preserves group\-relative RL\. Under conditionally independent sampling, its expected distillation rate is\(1−p\)K\(1\-p\)^\{K\}and therefore decreases automatically with policy success probabilitypp\.
- •On four SciKnowEval domains, I\-SDPO reaches 70\.31% average mean@16 accuracy, outperforming GRPO, pure SDPO, and sample\-level SRPO\. Training dynamics and KL\-direction ablations are consistent with capability\-dependent teacher trust\.

## 2Related Work

##### Reinforcement Learning for LLM Post\-Training\.

RL\-based post\-training traces back to RLHF\([Ouyang et al\. 2022](https://arxiv.org/html/2608.12957#bib.bib16)\), which uses Proximal Policy Optimization \(PPO\)\([Schulman et al\. 2017](https://arxiv.org/html/2608.12957#bib.bib19)\)to align LLMs with human preferences\. Direct Preference Optimization \(DPO\)\([Rafailov et al\. 2023](https://arxiv.org/html/2608.12957#bib.bib18)\)simplifies this by eliminating the reward model, directly optimizing preferences via a closed\-form objective\. More recently, GRPO\([Shao et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib20)\)removes the critic model entirely, computing group\-relative advantages from multiple sampled responses\. DeepSeek\-R1\([Guo et al\. 2025](https://arxiv.org/html/2608.12957#bib.bib7)\)demonstrates that GRPO\-style training can incentivize strong reasoning capabilities\. Dr\. GRPO\([Liu et al\. 2025](https://arxiv.org/html/2608.12957#bib.bib13)\)identifies and corrects length\-dependent biases in the original GRPO formulation\. Our work builds on GRPO as the RL backbone and addresses its degenerate gradient problem on all\-incorrect rollout groups\.

##### Classical and Online Self\-Distillation\.

Knowledge distillation transfers a teacher’s predictive distribution or representations to a student\([Hinton et al\. 2015](https://arxiv.org/html/2608.12957#bib.bib8)\)\. Self\-distillation removes the need for an independently stronger teacher and has taken several forms\. Born\-Again Networks train successive generations of the same architecture\([Furlanello et al\. 2018](https://arxiv.org/html/2608.12957#bib.bib5)\); other methods transfer knowledge between deep and shallow sections of one network\([Zhang et al\. 2019](https://arxiv.org/html/2608.12957#bib.bib28)\), from earlier optimization snapshots\([Yang et al\. 2019](https://arxiv.org/html/2608.12957#bib.bib25)\), across distorted views of an example\([Xu & Liu 2019](https://arxiv.org/html/2608.12957#bib.bib23)\), between same\-class samples\([Yun et al\. 2020](https://arxiv.org/html/2608.12957#bib.bib27)\), or from progressively refined past predictions\([Kim et al\. 2021](https://arxiv.org/html/2608.12957#bib.bib9)\)\. Online variants construct teachers jointly with the student, using peer networks, native ensembles, or weight\-averaged models\([Zhang et al\. 2018](https://arxiv.org/html/2608.12957#bib.bib29);[Lan et al\. 2018](https://arxiv.org/html/2608.12957#bib.bib12);[Tarvainen & Valpola 2017](https://arxiv.org/html/2608.12957#bib.bib21)\)\. These methods differ in where the target is obtained, but all reuse knowledge already present in the model, training process, or data rather than importing an independent source of task competence\.

##### Theoretical Views of Self\-Distillation\.

Several accounts interpret self\-distillation primarily as regularization\. Teacher predictions can act as instance\-specific label smoothing\([Zhang & Sabuncu 2020](https://arxiv.org/html/2608.12957#bib.bib31)\); statistically, approximate teacher probabilities trade lower target variance against teacher\-induced bias\([Menon et al\. 2021](https://arxiv.org/html/2608.12957#bib.bib14)\)\. In a Hilbert\-space setting, repeated self\-distillation progressively restricts the learned function: a few rounds can reduce overfitting, while further rounds can underfit\([Mobahi et al\. 2020](https://arxiv.org/html/2608.12957#bib.bib15)\)\. These results do not directly characterize autoregressive Transformers, but they establish an important qualification to the usual “dense supervision” intuition: self\-generated targets can help without being unbiased, and continued agreement is not necessarily continued learning\.

##### Distillation for Autoregressive Language Models\.

Sequence\-level distillation trains on teacher\-generated sequences\([Kim & Rush 2016](https://arxiv.org/html/2608.12957#bib.bib10)\), whereas recent white\-box methods optimize token distributions on student or mixed trajectories\. MiniLLM advocates reverse KL for generative distillation\([Gu et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib6)\); GKD generalizes both the divergence and the use of on\-policy student generations\([Agarwal et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib1)\); and DistiLLM combines a skew\-KL objective with adaptive off\-policy sampling\([Ko et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib11)\)\. Token\-level teaching behavior is heterogeneous, so applying a single teaching mode uniformly can reduce performance\([Zhong et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib33)\)\. Self\-distillation has also been adapted to multilingual transfer and low\-bit LLM training\([Zhang et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib30);[Du et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib3)\), illustrating its breadth beyond conventional model compression\.

##### Privileged On\-Policy Self\-Distillation and RL\.

Most closely related, Self\-Distilled Reasoner introduces on\-policy self\-distillation \(OPSD\), where the same LLM serves as a student conditioned on the question and as a teacher additionally conditioned on a verified reasoning trace; the two distributions are matched over student\-generated rollouts\([Zhao et al\. 2026](https://arxiv.org/html/2608.12957#bib.bib32)\)\. OPSD establishes privileged\-context self\-distillation as a token\-efficient standalone reasoning objective\. Our work asks a complementary question exposed by longer training: when should this teacher cease to control optimization? I\-SDPO couples the privileged objective to verifiable rewards, applying it only to all\-incorrect groups and preserving group\-relative RL once successful evidence exists\. This capability\-dependent routing differs from uniform loss blending, pure OPSD, and self\-reward or iterative preference schemes\([Yuan et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib26);[Pang et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib17)\)\.

## 3Method

### 3\.1Preliminaries

##### Group Relative Policy Optimization \(GRPO\)\.

Given a promptxxand a policyπθ\\pi\_\{\\theta\}, GRPO samplesKKresponses\{y1,…,yK\}\\\{y\_\{1\},\\ldots,y\_\{K\}\\\}and scores each with a reward functionr⁡\(x,yi\)r\(x,y\_\{i\}\)\. The advantage for sampleiiis computed as:

Ai=r⁡\(x,yi\)−1K​∑j=1Kr⁡\(x,yj\)A\_\{i\}=r\(x,y\_\{i\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}r\(x,y\_\{j\}\)\(1\)The policy is updated using a clipped surrogate objective following Dr\. GRPO\([Liu et al\. 2025](https://arxiv.org/html/2608.12957#bib.bib13)\):

ℒGRPO=−∑i=1K∑t=1Timin\(ρi,tAi,clip\(ρi,t,1−ϵ,1\+ϵ\)Ai\)\\mathcal\{L\}\_\{\\text\{GRPO\}\}=\-\\sum\_\{i=1\}^\{K\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\min\\\!\\Big\(\\rho\_\{i,t\}\\,A\_\{i\},\\;\\text\{clip\}\(\\rho\_\{i,t\},1\{\-\}\\epsilon,1\{\+\}\\epsilon\)\\,A\_\{i\}\\Big\)\(2\)whereρi,t=πθ​\(yi,t∣x,yi,<t\)/πθold​\(yi,t∣x,yi,<t\)\\rho\_\{i,t\}=\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)/\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)is the per\-token importance ratio\.

##### Self\-Distillation Policy Optimization \(SDPO\)\.

SDPO provides dense token\-level supervision by minimizing the divergence between the student policyπθ\\pi\_\{\\theta\}and a teacherπtea\\pi\_\{\\text\{tea\}\}that observes privileged information\. Following on\-policy distillation over student\-generated trajectories\([Agarwal et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib1);[Zhao et al\. 2026](https://arxiv.org/html/2608.12957#bib.bib32)\), we interpolate forward and reverse KL divergences:

ℒSDPO=\(1−α\)KL\(πtea∥πθ\)\+αKL\(πθ∥πtea\)\\mathcal\{L\}\_\{\\text\{SDPO\}\}=\(1\{\-\}\\alpha\)\\,\\text\{KL\}\(\\pi\_\{\\text\{tea\}\}\\\|\\pi\_\{\\theta\}\)\+\\alpha\\,\\text\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{tea\}\}\)\(3\)whereKL\(πtea∥πθ\)\\text\{KL\}\(\\pi\_\{\\text\{tea\}\}\\\|\\pi\_\{\\theta\}\)is the forward KL,KL\(πθ∥πtea\)\\text\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{tea\}\}\)is the reverse KL, andα∈\[0,1\]\\alpha\\in\[0,1\]controls their balance\. The choice matters for autoregressive generation: reverse KL is more mode\-seeking, while forward KL places greater weight on teacher coverage\([Gu et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib6);[Ko et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib11)\)\. Thus,α=0\\alpha\{=\}0uses forward KL only,α=1\\alpha\{=\}1uses reverse KL only, andα=0\.5\\alpha\{=\}0\.5weights both directions equally\. We useα=0\.5\\alpha\{=\}0\.5by default and analyze this choice in Appendix[C](https://arxiv.org/html/2608.12957#A3)\.

### 3\.2The Degenerate Gradient Problem

We formalize the motivation for I\-SDPO\. Consider a rollout group where allKKsamples receive identical \(or near\-identical\) rewardsri≈cr\_\{i\}\\approx cfor some constantcc\. The GRPO advantages become:

Ai=ri−r¯≈c−c=0,∀i∈\{1,…,K\}A\_\{i\}=r\_\{i\}\-\\bar\{r\}\\approx c\-c=0,\\quad\\forall i\\in\\\{1,\\ldots,K\\\}\(4\)Consequently,ℒGRPO≈0\\mathcal\{L\}\_\{\\text\{GRPO\}\}\\approx 0, and the policy gradient vanishes\. With binary rewards, this occurs whenever all responses are incorrect\. \(An all\-correct group is also constant\-reward, but it does not represent a failure to discover a successful trajectory\.\) For a fixed prompt, letptp\_\{t\}denote the per\-sample success probability at steptt\. Under conditionally independent sampling, the probability of an all\-incorrect group is:

f⁡\(t\)=\(1−pt\)Kf\(t\)=\(1\-p\_\{t\}\)^\{K\}\(5\)Forpt=0\.1p\_\{t\}=0\.1andK=16K=16,f⁡\(t\)≈0\.185f\(t\)\\approx 0\.185—nearly one in five groups has no relative reward signal\. Atpt=0\.2p\_\{t\}=0\.2and0\.30\.3, the same probability falls rapidly to0\.0280\.028and0\.00330\.0033, respectively\. Prompt heterogeneity changes the dataset average to𝔼x​\[\(1−pt​\(x\)\)K\]\\mathbb\{E\}\_\{x\}\[\(1\-p\_\{t\}\(x\)\)^\{K\}\]but preserves the capability dependence\.

### 3\.3Why Teacher Trust Should Depend on Capability

Dense supervision is not automatically reward\-aligned\. At token states=\(x,y<t\)s=\(x,y\_\{<t\}\), letpsp\_\{s\},qsq\_\{s\}, andusu\_\{s\}denote the student distribution, privileged\-teacher distribution, and a conceptual reward\-compatible target distribution\. For the forward\-KL component, the descent direction with respect to student logits isqs−psq\_\{s\}\-p\_\{s\}, whereas an ideal local direction would beus−psu\_\{s\}\-p\_\{s\}\. Their alignment is

Γs=\(qs−ps\)⊤​\(us−ps\)=12​\(∥qs−ps∥22\+∥us−ps∥22−∥qs−us∥22\)\.\\Gamma\_\{s\}=\(q\_\{s\}\-p\_\{s\}\)^\{\\top\}\(u\_\{s\}\-p\_\{s\}\)=\\tfrac\{1\}\{2\}\\\!\\left\(\\lVert q\_\{s\}\-p\_\{s\}\\rVert\_\{2\}^\{2\}\+\\lVert u\_\{s\}\-p\_\{s\}\\rVert\_\{2\}^\{2\}\-\\lVert q\_\{s\}\-u\_\{s\}\\rVert\_\{2\}^\{2\}\\right\)\.\(6\)Thus, teacher supervision is locally helpful only whenΓs\>0\\Gamma\_\{s\}\>0\. Privileged context can improveqsq\_\{s\}, but it does not guarantee this condition: the teacher is evaluated after the student’s possibly erroneous prefix, and sequence reward may admit many correct continuations that differ from the demonstrated solution\. Reverse KL changes the exact logit gradient, but the interpolated objective in Eq\.[3](https://arxiv.org/html/2608.12957#S3.E3)retains the same dependence on a potentially mismatchedqsq\_\{s\}\.

This yields a capability\-dependent bias–variance trade\-off\. Before the policy discovers successful trajectories, the GRPO direction is zero or estimated from rare events; a biased but dense teacher direction can therefore be substantially more useful\. Once successful and unsuccessful samples coexist, the group\-relative reward supplies an on\-policy comparison\. The variance\-reduction benefit of imitation then shrinks, while any mismatch betweenqsq\_\{s\}and reward\-compatible behavior remains\. Moreover, teacher and student share an architecture and are coupled by EMA, so agreement may increase either because the student learns or because the teacher inherits the student’s errors\. EMA smooths temporal fluctuations but cannot remove such shared bias\.

The following local model makes the persistent\-bias mechanism explicit\.

###### Proposition 1\(Bias floor under persistent distillation\)\.

Suppose that near a reward optimumθ⋆\\theta^\{\\star\}, the reward and distillation losses satisfyℒR​\(θ\)≈12​∥θ−θ⋆∥H2\\mathcal\{L\}\_\{R\}\(\\theta\)\\approx\\frac\{1\}\{2\}\\lVert\\theta\-\\theta^\{\\star\}\\rVert\_\{H\}^\{2\}andℒD​\(θ\)≈12​∥θ−\(θ⋆\+b\)∥H2\\mathcal\{L\}\_\{D\}\(\\theta\)\\approx\\frac\{1\}\{2\}\\lVert\\theta\-\(\\theta^\{\\star\}\+b\)\\rVert\_\{H\}^\{2\}forH≻0H\\succ 0\. MinimizingℒR\+λ​ℒD\\mathcal\{L\}\_\{R\}\+\\lambda\\mathcal\{L\}\_\{D\}with constantλ≥0\\lambda\\geq 0gives

θλ⋆=θ⋆\+λ1\+λ​b,ℒR​\(θλ⋆\)−ℒR​\(θ⋆\)≈12​\(λ1\+λ\)2​∥b∥H2\.\\theta\_\{\\lambda\}^\{\\star\}=\\theta^\{\\star\}\+\\frac\{\\lambda\}\{1\+\\lambda\}b,\\qquad\\mathcal\{L\}\_\{R\}\(\\theta\_\{\\lambda\}^\{\\star\}\)\-\\mathcal\{L\}\_\{R\}\(\\theta^\{\\star\}\)\\approx\\frac\{1\}\{2\}\\left\(\\frac\{\\lambda\}\{1\+\\lambda\}\\right\)^\{2\}\\lVert b\\rVert\_\{H\}^\{2\}\.\(7\)

The proposition is a local explanatory model, not a global convergence theorem for Transformers; its proof is in Appendix[A](https://arxiv.org/html/2608.12957#A1)\. It predicts a bias floor whenever teacher mismatchbband effective distillation weightλ\\lambdaboth persist\. It also explains why pure SDPO can improve rapidly at first yet plateau or deteriorate later: the supervision that replaces an absent reward gradient early can become the dominant, misaligned force once reward optimization is feasible\. Related theory connects distillation to imperfect\-teacher bias and repeated self\-distillation to progressively stronger regularization\([Menon et al\. 2021](https://arxiv.org/html/2608.12957#bib.bib14);[Mobahi et al\. 2020](https://arxiv.org/html/2608.12957#bib.bib15)\)\.

### 3\.4I\-SDPO: Instance\-Level Routing

I\-SDPO addresses the degenerate gradient problem through instance\-level routing over rollout groups\. Each input instance is a promptxix\_\{i\}, and the routing variable indexed byiiis shared by its complete rollout group\{yi1,…,yiK\}\\\{y\_\{i\}^\{1\},\\ldots,y\_\{i\}^\{K\}\\\}\. We define:

ci\\displaystyle c\_\{i\}=\[∃j:r\(xi,yij\)≥τ\]\\displaystyle=\\mathbf\{1\}\\\!\\left\[\\exists\\,j:r\(x\_\{i\},y\_\{i\}^\{j\}\)\\geq\\tau\\right\]\(correctness indicator\)\(8\)mi\\displaystyle m\_\{i\}=\[ground\-truth available for​xi\]\\displaystyle=\\mathbf\{1\}\\\!\\left\[\\text\{ground\-truth available for \}x\_\{i\}\\right\]\(teacher feasibility\)\(9\)whereτ\\tauis a reward threshold for correctness\. The routing masks are:

ziSDPO\\displaystyle z\_\{i\}^\{\\text\{SDPO\}\}=\(1−ci\)⋅mi\\displaystyle=\(1\-c\_\{i\}\)\\cdot m\_\{i\}\(10\)ziGRPO\\displaystyle z\_\{i\}^\{\\text\{GRPO\}\}=1−ziSDPO\\displaystyle=1\-z\_\{i\}^\{\\text\{SDPO\}\}\(11\)
The routing logic is:

- •If the instance has any correct response \(ci=1c\_\{i\}=1\): all samples go to GRPO \(ziGRPO=1z\_\{i\}^\{\\text\{GRPO\}\}=1\); a mixed group therefore retains its reward contrast\.
- •If all responses are wrong and ground\-truth exists \(ci=0,mi=1c\_\{i\}=0,m\_\{i\}=1\): all samples go to SDPO \(ziSDPO=1z\_\{i\}^\{\\text\{SDPO\}\}=1\)\.
- •If all responses are wrong but no ground\-truth \(ci=0,mi=0c\_\{i\}=0,m\_\{i\}=0\): defaults to GRPO\.

The combined loss is:

ℒI\-SDPO=∑i\(ziGRPO⋅ℒGRPO\(i\)\+ziSDPO⋅ℒSDPO\(i\)\)∑i\(ziGRPO\+ziSDPO\)\\mathcal\{L\}\_\{\\text\{I\-SDPO\}\}=\\frac\{\\sum\_\{i\}\\left\(z\_\{i\}^\{\\text\{GRPO\}\}\\cdot\\mathcal\{L\}\_\{\\text\{GRPO\}\}^\{\(i\)\}\+z\_\{i\}^\{\\text\{SDPO\}\}\\cdot\\mathcal\{L\}\_\{\\text\{SDPO\}\}^\{\(i\)\}\\right\)\}\{\\sum\_\{i\}\\left\(z\_\{i\}^\{\\text\{GRPO\}\}\+z\_\{i\}^\{\\text\{SDPO\}\}\\right\)\}\(12\)
##### Contrast with Sample\-Level Routing \(SRPO\)\.

In SRPO, each incorrect sample is routed to SDPO even when a correct peer exists\. Consider a group with 15 incorrect and one correct response\. SRPO replaces the learning objective on the 15 negative trajectories, weakening the within\-group comparison that identifies which behavior led to success\. I\-SDPO instead keeps the group intact for GRPO and invokes teacher supervision only when the group contains no successful trajectory\. The routing decision is therefore also a decision about*teacher trust*: observed policy success makes reward supervision preferable to imitation\.

### 3\.5Privileged Teacher for Self\-Distillation

For instances routed to SDPO, the teacher modelπtea\\pi\_\{\\text\{tea\}\}generates token\-level soft targets for every rollout in the corresponding group\. The teacher shares the student’s architecture but receives*privileged information*: the correct solution is prepended to the prompt context before computing log\-probabilities over the student’s response tokens\.

Specifically, for a promptxxwith ground\-truth solutiony∗y^\{\*\}, the teacher input is constructed as:

xtea=\[x∥‘‘Correct solution: ’’∥y∗∥‘‘Correctly solve the original question\.’’\]x\_\{\\text\{tea\}\}=\[x\\,\\\|\\,\\texttt\{\`\`Correct solution: ''\}\\,\\\|\\,y^\{\*\}\\,\\\|\\,\\texttt\{\`\`Correctly solve the original question\.''\}\]\(13\)The teacher then computesπtea\(⋅∣xtea,y<t\)\\pi\_\{\\text\{tea\}\}\(\\cdot\\mid x\_\{\\text\{tea\}\},y\_\{<t\}\)for the same response tokensyythat the student generated\. This instantiates privileged on\-policy self\-distillation\([Zhao et al\. 2026](https://arxiv.org/html/2608.12957#bib.bib32)\)and follows the Learning Using Privileged Information \(LUPI\) paradigm\([Vapnik & Vashist 2009](https://arxiv.org/html/2608.12957#bib.bib22)\): the teacher leverages information available during training but not at inference\.

The teacher model is maintained as an Exponential Moving Average \(EMA\) of the student\([Tarvainen & Valpola 2017](https://arxiv.org/html/2608.12957#bib.bib21)\):

θtea←\(1−τema\)​θtea\+τema​θstudent\\theta\_\{\\text\{tea\}\}\\leftarrow\(1\-\\tau\_\{\\text\{ema\}\}\)\\,\\theta\_\{\\text\{tea\}\}\+\\tau\_\{\\text\{ema\}\}\\,\\theta\_\{\\text\{student\}\}\(14\)whereτema\\tau\_\{\\text\{ema\}\}is the update rate\. Unrolling Eq\.[14](https://arxiv.org/html/2608.12957#S3.E14)shows that past student parameters receive geometrically decaying weights\. Withτema=0\.05\\tau\_\{\\text\{ema\}\}=0\.05, the characteristic averaging horizon is roughly1/τema=201/\\tau\_\{\\text\{ema\}\}=20updates\. This smooths short\-term target variation, but creates an unavoidable trade\-off: faster tracking reduces teacher independence, while slower tracking increases staleness\. Neither setting removes systematic teacher–reward mismatch; instance\-level routing limits how long that mismatch influences optimization\.

### 3\.6Entropy\-Aware Dynamic Weighting

Not all teacher targets are equally reliable\. We use low teacher entropy as a pragmatic confidence proxy and introduce entropy\-aware weighting:

wi,t=exp\(−β⋅Httea\)1\|𝒮\|∑\(j,s\)∈𝒮exp\(−β⋅Hstea\)w\_\{i,t\}=\\frac\{\\exp\(\-\\beta\\cdot H\_\{t\}^\{\\text\{tea\}\}\)\}\{\\frac\{1\}\{\|\\mathcal\{S\}\|\}\\sum\_\{\(j,s\)\\in\\mathcal\{S\}\}\\exp\(\-\\beta\\cdot H\_\{s\}^\{\\text\{tea\}\}\)\}\(15\)whereHttea=−∑vptea\(v\)logptea\(v\)H\_\{t\}^\{\\text\{tea\}\}=\-\\sum\_\{v\}p\_\{\\text\{tea\}\}\(v\)\\log p\_\{\\text\{tea\}\}\(v\)is the teacher entropy at positiontt,β\\betacontrols sensitivity, and𝒮\\mathcal\{S\}contains all SDPO\-routed token positions\. Low\-entropy positions receive larger weights and high\-entropy positions are down\-weighted\. Importantly, confidence is not correctness: this heuristic reduces exposure to uncertain targets but cannot correct a confidently biased teacher\. The weighted SDPO loss becomes:

ℒSDPO\(i\)=∑t=1Tiwi,t⋅ℓKL\-mix​\(i,t\)\\mathcal\{L\}\_\{\\text\{SDPO\}\}^\{\(i\)\}=\\sum\_\{t=1\}^\{T\_\{i\}\}w\_\{i,t\}\\cdot\\ell\_\{\\text\{KL\-mix\}\}\(i,t\)\(16\)whereℓKL\-mix​\(i,t\)\\ell\_\{\\text\{KL\-mix\}\}\(i,t\)is the per\-token forward–reverse KL interpolation from Eq\.[3](https://arxiv.org/html/2608.12957#S3.E3)\.

### 3\.7Self\-Annealing Property

Instance\-level routing converts policy competence into an adaptive effective distillation weight\.

###### Proposition 2\(Self\-Annealing\)\.

For a fixed prompt, suppose theKKrollouts are conditionally independent with per\-sample success probabilityptp\_\{t\}\. Ifptp\_\{t\}is non\-decreasing, then the expected SDPO routing probabilityf⁡\(t\)=\(1−pt\)Kf\(t\)=\(1\-p\_\{t\}\)^\{K\}is non\-increasing\.

The result follows because\(1−p\)K\(1\-p\)^\{K\}decreases inpp\. If the per\-routed\-group distillation scale is fixed, its expected contribution is nevertheless multiplied byf⁡\(t\)f\(t\)\. Hence, as reward\-bearing groups become common, I\-SDPO drives the effectiveλ\\lambdain Proposition[1](https://arxiv.org/html/2608.12957#Thmproposition1)toward zero and removes the corresponding bias floor\. This capability\-coupled withdrawal distinguishes I\-SDPO from a time\-based decay schedule: two prompts at the same training step can receive different supervision according to whether the current policy can solve them\.

The complete training procedure is summarized in Algorithm[1](https://arxiv.org/html/2608.12957#alg1)\.

Algorithm 1I\-SDPO: Instance\-Level Adaptive Self\-Distillation Policy Optimization0:Policy

πθ\\pi\_\{\\theta\}, teacher

πtea\\pi\_\{\\text\{tea\}\}, prompts

𝒟\\mathcal\{D\}, reward function

rr, threshold

τ\\tau, EMA rate

τema\\tau\_\{\\text\{ema\}\}
1:foreach training iterationdo

2:Sample batch of prompts

\{xi\}\\\{x\_\{i\}\\\}from

𝒟\\mathcal\{D\}
3:Generate

KKresponses per prompt:

\{yi1,…,yiK\}∼πθ\(⋅\|xi\)\\\{y\_\{i\}^\{1\},\\ldots,y\_\{i\}^\{K\}\\\}\\sim\\pi\_\{\\theta\}\(\\cdot\|x\_\{i\}\)
4:Compute rewards

r⁡\(xi,yij\)r\(x\_\{i\},y\_\{i\}^\{j\}\)for all samples

5:foreach prompt

xix\_\{i\}do

6:if

∃j:r⁡\(xi,yij\)≥τ\\exists\\,j:r\(x\_\{i\},y\_\{i\}^\{j\}\)\\geq\\tauthen

7:Route to GRPO: compute

ℒGRPO\(i\)\\mathcal\{L\}\_\{\\text\{GRPO\}\}^\{\(i\)\}via Eq\.[2](https://arxiv.org/html/2608.12957#S3.E2)

8:elseifground\-truth

yi∗y\_\{i\}^\{\*\}availablethen

9:Construct teacher input

xteax\_\{\\text\{tea\}\}with

yi∗y\_\{i\}^\{\*\}
10:Compute teacher log\-probs:

πtea\(⋅\|xtea,yi,<tj\)\\pi\_\{\\text\{tea\}\}\(\\cdot\|x\_\{\\text\{tea\}\},y\_\{i,<t\}^\{j\}\)
11:Route to SDPO: compute

ℒSDPO\(i\)\\mathcal\{L\}\_\{\\text\{SDPO\}\}^\{\(i\)\}via Eq\.[16](https://arxiv.org/html/2608.12957#S3.E16)

12:else

13:Route to GRPO \(fallback\)

14:endif

15:endfor

16:Update

θ\\thetausing

ℒI\-SDPO\\mathcal\{L\}\_\{\\text\{I\-SDPO\}\}\(Eq\.[12](https://arxiv.org/html/2608.12957#S3.E12)\)

17:Update teacher:

θtea←\(1−τema\)​θtea\+τema​θ\\theta\_\{\\text\{tea\}\}\\leftarrow\(1\-\\tau\_\{\\text\{ema\}\}\)\\theta\_\{\\text\{tea\}\}\+\\tau\_\{\\text\{ema\}\}\\theta\(Eq\.[14](https://arxiv.org/html/2608.12957#S3.E14)\)

18:endfor

## 4Experiments

### 4\.1Experimental Setup

##### Model and Data\.

We use Qwen3\-8B\([Yang et al\. 2025](https://arxiv.org/html/2608.12957#bib.bib24)\)as the base model\. Training and evaluation are conducted on the SciKnowEval benchmark\([Feng et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib4)\), which assesses scientific knowledge across four domains:biology,material science,chemistry, andphysics\. Each domain provides problems with verifiable answers, enabling rule\-based reward computation\.

##### Baselines\.

We compare I\-SDPO against three baselines:

- •GRPO: Standard Group Relative Policy Optimization\([Shao et al\. 2024](https://arxiv.org/html/2608.12957#bib.bib20)\)without any self\-distillation\.
- •SDPO: Pure self\-distillation applied to all samples without RL, serving as the distillation\-only baseline\.
- •SRPO: Sample\-level routing where each individual incorrect sample receives SDPO, even if correct peers exist in the same rollout group\.

##### Hyperparameters\.

All methods use a learning rate of5×10−65\\times 10^\{\-6\}, forward–reverse KL interpolationα=0\.5\\alpha=0\.5, teacher EMA update rateτema=0\.05\\tau\_\{\\text\{ema\}\}=0\.05, entropy weightingβ=1\.0\\beta=1\.0, and importance sampling clip ratio of 2\.0 for the SDPO branch\. The GRPO branch uses a clip ratioϵ=0\.2\\epsilon=0\.2following Dr\. GRPO\([Liu et al\. 2025](https://arxiv.org/html/2608.12957#bib.bib13)\)without standard deviation normalization\. Training runs for 2 epochs with 16 rollout samples per prompt\. All experiments are conducted using the VERL training framework\. For distillation\-based methods, we retain the teacher’s 100 highest\-probability vocabulary tokens when computing the distillation loss \(distillation\_topk=100=100\); Appendix[C](https://arxiv.org/html/2608.12957#A3)studies the sensitivity to this choice\.

##### Evaluation\.

We report mean@16 accuracy: for each test problem, 16 responses are sampled and the mean accuracy is computed\. This metric captures both the model’s problem\-solving ability and the consistency of its responses\.

### 4\.2Main Results

Table 1:Main results \(mean@16 accuracy, %\) on SciKnowEval across four scientific domains after 2 epochs of training\.Boldindicates the best result per domain\.Table[1](https://arxiv.org/html/2608.12957#S4.T1)reports final performance\. I\-SDPO is best in every domain and reaches 70\.31% average mean@16 accuracy\. It improves over GRPO by 13\.64 points on average and by 18\.24 points on chemistry, the largest domain\-level gain\. Pure SDPO also improves the average over GRPO, from 56\.67% to 65\.74%, showing that dense privileged supervision is valuable when reward\-only learning is sparse\. Its 4\.57\-point gap behind I\-SDPO is equally informative: teacher supervision is useful, but applying it indiscriminately is not the best final objective\.

Routing granularity accounts for a further difference\. I\-SDPO exceeds sample\-level SRPO in biology, material science, chemistry, and physics by 5\.98, 1\.59, 2\.47, and 7\.19 points, respectively, for a 4\.30\-point average gain\. These results are consistent with preserving mixed groups for reward learning: once a group contains both outcomes, its incorrect trajectories are valuable negative evidence rather than merely samples to be imitated under privileged context\.

### 4\.3Training Dynamics Analysis

We next examine whether the optimization dynamics match the capability\-dependent account in Section[3\.3](https://arxiv.org/html/2608.12957#S3.SS3)\.

##### Reward Curves\.

Figure[1](https://arxiv.org/html/2608.12957#S4.F1)shows mean training reward\. Across domains, methods containing self\-distillation rise faster than pure GRPO early in training\. This is the regime in which all\-incorrect groups are common and token\-level targets replace otherwise absent relative\-reward updates\. Final accuracy in Table[1](https://arxiv.org/html/2608.12957#S4.T1), however, favors I\-SDPO over pure SDPO\. Together, the observations support a phase\-dependent interpretation: dense imitation is most useful for bootstrapping, whereas reward optimization should dominate after successful trajectories become available\.

![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure1/biology.png)\(a\)Biology
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure1/material.png)\(b\)Material Science
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure1/chemistry.png)\(c\)Chemistry
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure1/physics.png)\(d\)Physics

Figure 1:Mean training reward across SciKnowEval domains\. Distillation\-based methods improve rapidly in the early phase, when reward\-bearing rollouts are scarce; I\-SDPO retains this bootstrapping effect while reducing teacher use as the policy improves\.
##### Self\-Annealing Behavior\.

Figure[2](https://arxiv.org/html/2608.12957#S4.F2)shows the fraction of samples routed to GRPO and the all\-wrong group fraction\. The GRPO fraction trends upward \(from approximately 0\.78 to 0\.96 on physics and 0\.65 to 0\.90 on biology\), while all\-wrong groups become less frequent\. This is the feedback predicted by Proposition[2](https://arxiv.org/html/2608.12957#Thmproposition2): improved sampling success directly reduces the effective weight of the biased surrogate rather than relying on elapsed training time\.

![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/grpo_sample_fraction_biology.png)\(a\)Biology
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/grpo_sample_fraction_material.png)\(b\)Material
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/grpo_sample_fraction_chemistry.png)\(c\)Chemistry
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/grpo_sample_fraction_physics.png)\(d\)Physics
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/all_wrong_rollout_fraction_biology.png)\(e\)Biology
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/all_wrong_rollout_fraction_material.png)\(f\)Material
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/all_wrong_rollout_fraction_chemistry.png)\(g\)Chemistry
![Refer to caption](https://arxiv.org/html/2608.12957v1/Figures/Figure2/all_wrong_rollout_fraction_physics.png)\(h\)Physics

Figure 2:Capability\-dependent routing in I\-SDPO\. \(Top\) The fraction assigned to GRPO generally increases during training\. \(Bottom\) The all\-wrong group fraction correspondingly decreases, so teacher influence is withdrawn as successful sampling becomes more likely\.
##### Lower Intrusion on Moderate\-Difficulty Data\.

A notable difference from SRPO is that I\-SDPO assigns a larger fraction to GRPO on material science and physics \(approximately 0\.90–0\.96 versus 0\.82–0\.90\)\. The stronger final results despite less distillation coverage argue against “more teacher supervision is always better\.” They instead favor selective intervention: keep the low\-variance teacher for groups with no reward contrast, and preserve on\-policy evidence everywhere else\.

##### Forward–Reverse KL Ablation\.

Figure[3](https://arxiv.org/html/2608.12957#S4.F3)compares mean@16 accuracy trajectories forα∈\{0,0\.5,1\}\\alpha\\in\\\{0,0\.5,1\\\}in Eq\.[3](https://arxiv.org/html/2608.12957#S3.E3)\. The balanced objective \(α=0\.5\\alpha=0\.5\) has the highest final accuracy in all four domains\. This is consistent with combining the mode\-covering tendency of forward KL and the sharper mode preference of reverse KL, although balancing divergence directions does not by itself remove teacher–reward mismatch\. Appendix[C](https://arxiv.org/html/2608.12957#A3)gives the quantitative results\.

\(a\)Biology\(b\)Material Science\(c\)Chemistry\(d\)Physics
Figure 3:Effect of the forward–reverse KL interpolation on I\-SDPO across four SciKnowEval domains\.α=0\\alpha=0uses forward KL only,α=1\\alpha=1uses reverse KL only, andα=0\.5\\alpha=0\.5weights both directions equally\. Curves show the raw mean@16 evaluation accuracy recorded during training\.

### 4\.4Capability\-Dependent Teacher Trust

The theory and results suggest a single organizing principle: the correct amount of teacher trust is determined by the policy’s current ability to produce endogenous reward supervision\. At low capability, GRPO often observes no outcome variation; the teacher then supplies a biased but low\-variance direction\. At higher capability, mixed groups expose which sampled behaviors succeed, while the privileged teacher remains tied to a particular solution and to the student’s own EMA history\. The signal\-to\-bias ratio of distillation therefore falls even if the teacher becomes numerically closer to the student\.

This view explains why a fixed SDPO objective can help most at the beginning yet become limiting later\. Student–teacher disagreement is not a stationary measure of teaching value: easy, aligned disagreements are absorbed first, leaving residual KL pressure concentrated on difficult prefixes, alternative valid modes, and confidently miscalibrated targets\. Continued minimization can then reduce KL without improving sequence reward\. This mechanism complements views of self\-distillation as instance\-specific smoothing and progressively stronger regularization\([Zhang & Sabuncu 2020](https://arxiv.org/html/2608.12957#bib.bib31);[Mobahi et al\. 2020](https://arxiv.org/html/2608.12957#bib.bib15)\)\. EMA delays and smooths this process, but cannot create independent knowledge or correct shared error\.

I\-SDPO operationalizes the principle without estimating gradient alignment directly\. The binary event “no successful response amongKKsamples” is an observable proxy for whether relative reward is informative\. Its probability\(1−p\)K\(1\-p\)^\{K\}falls steeply with capability, so the algorithm moves from teacher\-led bootstrapping to reward\-led refinement at an instance\-dependent rate\. Instance\-level routing matters because the decision is shared by all rollouts of the same input: whenever successful evidence exists, the complete group comparison is preserved, whereas sample\-level routing partially discards it\.

## 5Discussion

##### Connection to Curriculum Learning\.

I\-SDPO resembles a curriculum\([Bengio et al\. 2009](https://arxiv.org/html/2608.12957#bib.bib2)\), but the curriculum is over*supervision sources*, not a fixed ordering of examples\. The same prompt receives dense privileged supervision when the current policy fails to sample any success and group\-relative feedback once it does\. Difficulty is therefore model\- and time\-dependent\.

##### Connection to Learning Using Privileged Information\.

The teacher follows Learning Using Privileged Information \(LUPI\)\([Vapnik & Vashist 2009](https://arxiv.org/html/2608.12957#bib.bib22)\): ground\-truth solutions are available during training but not inference\. Our analysis adds an important qualification\. Privilege can make targets more informative without making them identical to the reward\-optimal policy, especially when the teacher evaluates an erroneous student prefix\. I\-SDPO therefore uses privilege as a temporary scaffold rather than a permanent optimization target\.

##### Scope of the theory\.

Equation[6](https://arxiv.org/html/2608.12957#S3.E6)is an exact local statement for the forward\-KL logit direction, while Proposition[1](https://arxiv.org/html/2608.12957#Thmproposition1)is a quadratic approximation around a reward optimum\. They identify concrete mechanisms—directional mismatch, persistent bias, and capability\-dependent variance reduction—but do not establish global convergence or predict every non\-convex training trajectory\. The empirical dynamics should consequently be read as consistent with the account, not as proof of its assumptions\.

## 6Limitations

We acknowledge two limitations:

Model scale\.Our experiments use only Qwen3\-8B\. Results on other model families and scales may differ, particularly because stronger base policies change the frequency and duration of all\-incorrect rollout groups\.

Benchmark diversity\.Evaluation is limited to SciKnowEval\. The relative value of privileged distillation and group\-relative rewards may change on general mathematical or open\-ended reasoning tasks with different reward structure and solution multiplicity\.

## 7Conclusion

We presented I\-SDPO, which treats privileged self\-distillation as a capability\-dependent scaffold rather than a uniformly reliable target\. The central distinction is a bias–variance one: distillation supplies a dense, low\-variance direction when all sampled rewards are uninformative, but its teacher\-specific bias can obstruct reward optimization after the policy begins producing successful trajectories\. Instance\-level routing uses the existence of such trajectories to choose one objective for the corresponding rollout group and automatically withdraws the teacher as capability improves\.

On SciKnowEval, I\-SDPO achieves the best result in all four domains and 70\.31% average mean@16 accuracy, improving over GRPO, pure SDPO, and sample\-level SRPO by 13\.64, 4\.57, and 4\.30 points, respectively\. The reward and routing dynamics are consistent with the proposed transition from teacher\-led bootstrapping to reward\-led refinement\. More broadly, our results indicate that the value of self\-distillation lies not only in the quality of its targets, but in restricting those targets to the phase in which they are preferable to the policy’s own reward evidence\.

#### Reproducibility Statement

We provide full hyperparameter specifications in Section[4\.1](https://arxiv.org/html/2608.12957#S4.SS1)\. The method description in Section[3](https://arxiv.org/html/2608.12957#S3)and Algorithm[1](https://arxiv.org/html/2608.12957#alg1)contain sufficient detail for re\-implementation\. Code will be released upon publication\.

## References

- Agarwal et al\. \(2024\)Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem\.On\-policy distillation of language models: Learning from self\-generated mistakes\.In*The Twelfth International Conference on Learning Representations \(ICLR\)*, 2024\.
- Bengio et al\. \(2009\)Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston\.Curriculum learning\.In*Proceedings of the 26th Annual International Conference on Machine Learning \(ICML\)*, pp\. 41–48, 2009\.
- Du et al\. \(2024\)DaYou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu\.BitDistiller: Unleashing the potential of sub\-4\-bit LLMs via self\-distillation\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 102–116, Bangkok, Thailand, 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.7\.
- Feng et al\. \(2024\)Kehua Feng, Kewen Zhao, Jingyuan Sun, Yujin Zhu, and Guoyin Wang\.SciKnowEval: Evaluating multi\-level scientific knowledge of large language models\.*arXiv preprint arXiv:2406\.09098*, 2024\.
- Furlanello et al\. \(2018\)Tommaso Furlanello, Zachary C\. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar\.Born again neural networks\.In*Proceedings of the 35th International Conference on Machine Learning \(ICML\)*, volume 80 of*Proceedings of Machine Learning Research*, pp\. 1607–1616\. PMLR, 2018\.
- Gu et al\. \(2024\)Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang\.MiniLLM: Knowledge distillation of large language models\.In*The Twelfth International Conference on Learning Representations*, 2024\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al\.DeepSeek\-R1: Incentivizing reasoning capability in LLMs via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- Hinton et al\. \(2015\)Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- Kim et al\. \(2021\)Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang\.Self\-knowledge distillation with progressive refinement of targets\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp\. 6567–6576, 2021\.
- Kim & Rush \(2016\)Yoon Kim and Alexander M\. Rush\.Sequence\-level knowledge distillation\.In*Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pp\. 1317–1327, 2016\.
- Ko et al\. \(2024\)Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se\-Young Yun\.DistiLLM: Towards streamlined distillation for large language models\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 24872–24895\. PMLR, 2024\.
- Lan et al\. \(2018\)Xu Lan, Xiatian Zhu, and Shaogang Gong\.Knowledge distillation by on\-the\-fly native ensemble\.In*Advances in Neural Information Processing Systems*, volume 31, 2018\.
- Liu et al\. \(2025\)Yuxuan Liu, Zhenyu Liu, Lu Yin, Yelong Shen, Dongkuan Xu, Jiawei Han, and Shiwei Liu\.Dr\. GRPO: Removing bias from group relative policy optimization\.*arXiv preprint arXiv:2503\.20783*, 2025\.
- Menon et al\. \(2021\)Aditya K\. Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar\.A statistical perspective on distillation\.In*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, pp\. 7632–7642\. PMLR, 2021\.
- Mobahi et al\. \(2020\)Hossein Mobahi, Mehrdad Farajtabar, and Peter L\. Bartlett\.Self\-distillation amplifies regularization in hilbert space\.In*Advances in Neural Information Processing Systems*, volume 33, 2020\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al\.Training language models to follow instructions with human feedback\.In*Advances in Neural Information Processing Systems*, volume 35, pp\. 27730–27744, 2022\.
- Pang et al\. \(2024\)Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston\.Iterative reasoning preference optimization\.*arXiv preprint arXiv:2404\.19733*, 2024\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Tarvainen & Valpola \(2017\)Antti Tarvainen and Harri Valpola\.Mean teachers are better role models: Weight\-averaged consistency targets improve semi\-supervised deep learning results\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Vapnik & Vashist \(2009\)Vladimir Vapnik and Akshay Vashist\.A new learning paradigm: Learning using privileged information\.*Neural Networks*, 22\(5\-6\):544–557, 2009\.
- Xu & Liu \(2019\)Ting\-Bing Xu and Cheng\-Lin Liu\.Data\-distortion guided self\-distillation for deep neural networks\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 33, pp\. 5565–5572, 2019\.doi:10\.1609/aaai\.v33i01\.33015565\.
- Yang et al\. \(2025\)An Yang, Baosong Yang, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yang et al\. \(2019\)Chenglin Yang, Lingxi Xie, Chi Su, and Alan L\. Yuille\.Snapshot distillation: Teacher\-student optimization in one generation\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 2859–2868, 2019\.
- Yuan et al\. \(2024\)Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston\.Self\-rewarding language models\.*arXiv preprint arXiv:2401\.10020*, 2024\.
- Yun et al\. \(2020\)Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin\.Regularizing class\-wise predictions via self\-knowledge distillation\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 13876–13885, 2020\.
- Zhang et al\. \(2019\)Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma\.Be your own teacher: Improve the performance of convolutional neural networks via self distillation\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp\. 3713–3722, 2019\.
- Zhang et al\. \(2018\)Ying Zhang, Tao Xiang, Timothy M\. Hospedales, and Huchuan Lu\.Deep mutual learning\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pp\. 4320–4328, 2018\.
- Zhang et al\. \(2024\)Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu\.Enhancing multilingual capabilities of large language models through self\-distillation from resource\-rich languages\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 11189–11204, Bangkok, Thailand, 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.603\.
- Zhang & Sabuncu \(2020\)Zhilu Zhang and Mert R\. Sabuncu\.Self\-distillation as instance\-specific label smoothing\.In*Advances in Neural Information Processing Systems*, volume 33, 2020\.
- Zhao et al\. \(2026\)Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover\.Self\-distilled reasoner: On\-policy self\-distillation for large language models\.*arXiv preprint arXiv:2601\.18734*, 2026\.
- Zhong et al\. \(2024\)Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao\.Revisiting knowledge distillation for autoregressive language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 10900–10913, Bangkok, Thailand, 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.587\.

## Appendix AAnalysis Details

##### Token\-direction identity\.

Letzsz\_\{s\}be the student logits andps=softmax⁡\(zs\)p\_\{s\}=\\operatorname\{softmax\}\(z\_\{s\}\)\. For the forward\-KL term,∇zsKL\(qs∥ps\)=ps−qs\\nabla\_\{z\_\{s\}\}\\operatorname\{KL\}\(q\_\{s\}\\\|p\_\{s\}\)=p\_\{s\}\-q\_\{s\}; hence its negative\-gradient direction isqs−psq\_\{s\}\-p\_\{s\}\. Replacingqsq\_\{s\}with a conceptual reward\-compatible targetusu\_\{s\}givesus−psu\_\{s\}\-p\_\{s\}\. Equation[6](https://arxiv.org/html/2608.12957#S3.E6)then follows from the polarization identity2​a⊤​b=∥a∥22\+∥b∥22−∥a−b∥222a^\{\\top\}b=\\lVert a\\rVert\_\{2\}^\{2\}\+\\lVert b\\rVert\_\{2\}^\{2\}\-\\lVert a\-b\\rVert\_\{2\}^\{2\}witha=qs−psa=q\_\{s\}\-p\_\{s\}andb=us−psb=u\_\{s\}\-p\_\{s\}\.

##### Proof of Proposition[1](https://arxiv.org/html/2608.12957#Thmproposition1)\.

Under the stated local approximations, the gradient of the combined objective is

∇θ\(ℒR\+λ​ℒD\)≈H⁡\(θ−θ⋆\)\+λ​H​\(θ−θ⋆−b\)\.\\nabla\_\{\\theta\}\(\\mathcal\{L\}\_\{R\}\+\\lambda\\mathcal\{L\}\_\{D\}\)\\approx H\(\\theta\-\\theta^\{\\star\}\)\+\\lambda H\(\\theta\-\\theta^\{\\star\}\-b\)\.\(17\)Setting this expression to zero and usingH≻0H\\succ 0yields\(1\+λ\)​\(θ−θ⋆\)=λ​b\(1\+\\lambda\)\(\\theta\-\\theta^\{\\star\}\)=\\lambda b, and thereforeθλ⋆=θ⋆\+λ1\+λ​b\\theta\_\{\\lambda\}^\{\\star\}=\\theta^\{\\star\}\+\\frac\{\\lambda\}\{1\+\\lambda\}b\. Substitution into the quadratic approximation ofℒR\\mathcal\{L\}\_\{R\}gives Eq\.[7](https://arxiv.org/html/2608.12957#S3.E7)\. In particular, the excess reward loss is positive wheneverb≠0b\\neq 0andλ\>0\\lambda\>0, and vanishes as routing drives the effectiveλ\\lambdato zero\.

##### EMA memory\.

Unrolling Eq\.[14](https://arxiv.org/html/2608.12957#S3.E14)at updatettgives

θtea,t=\(1−τema\)t​θtea,0\+τema​∑j=1t\(1−τema\)t−j​θstudent,j\.\\theta\_\{\\mathrm\{tea\},t\}=\(1\-\\tau\_\{\\mathrm\{ema\}\}\)^\{t\}\\theta\_\{\\mathrm\{tea\},0\}\+\\tau\_\{\\mathrm\{ema\}\}\\sum\_\{j=1\}^\{t\}\(1\-\\tau\_\{\\mathrm\{ema\}\}\)^\{t\-j\}\\theta\_\{\\mathrm\{student\},j\}\.\(18\)Ignoring the vanishing initialization term, these weights form a geometric age distribution with mean age\(1−τema\)/τema\(1\-\\tau\_\{\\mathrm\{ema\}\}\)/\\tau\_\{\\mathrm\{ema\}\}\. Forτema=0\.05\\tau\_\{\\mathrm\{ema\}\}=0\.05, the mean age is 19 updates and the half\-life islog⁡\(1/2\)/log⁡\(0\.95\)≈13\.5\\log\(1/2\)/\\log\(0\.95\)\\approx 13\.5updates\. This quantifies smoothing and lag; it does not imply that EMA corrects systematic errors\.

## Appendix BDetailed Experimental Results

Table[2](https://arxiv.org/html/2608.12957#A2.T2)provides a detailed view of the training dynamics metrics at different training checkpoints\.

Table 2:Training dynamics at selected checkpoints for I\-SDPO on biology\.
## Appendix CHyperparameter Sensitivity

##### Distillation top\-kk\.

This experiment tests whether the performance of instance\-level and sample\-level routing depends on the number of teacher vocabulary tokens retained by the distillation objective\. We varydistillation\_topkover\{20,40,60,80,100\}\\\{20,40,60,80,100\\\}while keeping all other training settings fixed\. Table[3](https://arxiv.org/html/2608.12957#A3.T3)reports the resulting mean@16 accuracy\.

Table 3:Sensitivity todistillation\_topkfor I\-SDPO and SRPO on SciKnowEval \(mean@16 accuracy, %\)\.Boldindicates the best top\-kkvalue within each row\. The average is computed across the four domains\.Neither routing method improves monotonically with top\-kk\. For I\-SDPO,k=40k=40performs best on biology and material science, whereask=100k=100performs best on chemistry, physics, and the cross\-domain average; SRPO shows a similarly mixed pattern and peaks on average atk=60k=60\. Smaller values remain competitive, so retaining more teacher tokens does not necessarily strengthen distillation\. We hypothesize that many instances are easy enough for the teacher to concentrate task\-relevant mass on a small token set; largerkkthen mainly adds a low\-probability tail\. Because we did not measure retained probability mass directly, this remains a hypothesis\. We usek=100k=100in the main experiments\.

##### Forward–reverse KL balance\.

We compareα∈\{0,0\.5,1\}\\alpha\\in\\\{0,0\.5,1\\\}in Eq\.[3](https://arxiv.org/html/2608.12957#S3.E3)with the remaining settings fixed to test whether distillation benefits from balancing the mode\-covering behavior of forward KL and the mode\-seeking behavior of reverse KL\. Figure[3](https://arxiv.org/html/2608.12957#S4.F3)reports the raw TensorBoard mean@16 measurements\.

The balanced objective \(α=0\.5\\alpha=0\.5\) achieves the highest final accuracy in every domain: 50\.25% on biology, 73\.80% on material science, 81\.16% on chemistry, and 67\.42% on physics\. Its 68\.16% cross\-domain average exceeds forward KL alone \(60\.39%\) and reverse KL alone \(61\.63%\)\. The two directions are complementary: forward KL promotes coverage, whereas reverse KL sharpens high\-probability modes but can become overly mode\-seeking\. Because each curve is a single run, we compare their recorded final values and treat trajectory details descriptively\.