SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision

# SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
Source: [https://arxiv.org/html/2606.28562](https://arxiv.org/html/2606.28562)
Chia\-Hsuan Lee, Zelei Cheng, Yu Wang, Renkun Ni Sambit Sahu, Shi\-Xiong Zhang, William Campbell Capital One

###### Abstract

On\-policy distillation \(OPD\) has a property absent in offline distillation and RL:*teacher supervision quality depends on student competence*\. Incoherent rollouts yield noisy gradients; already\-mastered tokens yield redundant ones\. This creates waste at three scales—tokens, training phases, and prompts—yet existing methods supervise uniformly\. We introduceSEAD, which uses entropy as a unified probe of this competence\-dependent degradation at three scales: \(1\) joint teacher–student entropy partitions tokens into zones receiving tailored divergences or zero gradient \(∼50%\{\\sim\}50\\%skipped\); \(2\) a cosine schedule anneals from forward to reverse KL as competence grows; \(3\) a competence\-gated curriculum introduces prompts easy\-to\-hard\. These components are symbiotically necessary: token selection requires coherent rollouts \(curriculum\), annealing requires monotonic improvement \(also curriculum\)\. On OLMo\-3 \(7B→\\to32B\), SEAD achieves \+4\.8 avg accuracy over vanilla OPD across six math benchmarks, with ablations confirming super\-additive interactions\.

## 1Introduction

Large reasoning models \(49B\+\) achieve strong performance but are prohibitively expensive for deployment\. Knowledge distillation compresses these capabilities into smaller students\. The dominant*off\-policy*paradigm—training on static teacher\-generated traces—suffers from exposure bias: prediction errors compound autoregressively at inference\(Agarwalet al\.,[2024](https://arxiv.org/html/2606.28562#bib.bib1); Song and Zheng,[2026](https://arxiv.org/html/2606.28562#bib.bib7)\)\.

On\-policy distillation \(OPD\) addresses this by training on student\-generated rollouts scored by the teacher\(Agarwalet al\.,[2024](https://arxiv.org/html/2606.28562#bib.bib1); Lu and Thinking Machines Lab,[2025](https://arxiv.org/html/2606.28562#bib.bib2)\), matching or exceeding RL methods like GRPO\(Koet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib3); Jinet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib4); Yanget al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib5)\)\. However, existing OPD methods apply supervision*uniformly*—the same divergence, on every token, at every phase, for every prompt\.

We identify a single structural problem underlying this uniformity:in OPD, supervision quality depends on student competence\. Unlike off\-policy KD \(teacher traces are always coherent\) or RL \(binary rewards are equally reliable\), OPD’s per\-token teacher corrections are only informative when the student’s rollouts are sufficiently coherent\. This competence\-dependent degradation manifests at three scales:

- •Token level:∼50%\{\\sim\}50\\%of tokens are deterministic for both models—supervising them wastes compute\. Among the remaining, some call for sharpening \(RKL\) while others require diversity preservation \(FKL\)\(Jinet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib4)\)\.
- •Temporal level:The optimal divergence evolves from mode\-covering \(early\) to mode\-seeking \(late\), yet prior solutions use manual two\-stage switches\(Koet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib3); Xuet al\.,[2026b](https://arxiv.org/html/2606.28562#bib.bib15)\)\.
- •Prompt level:Problems beyond the student’s capability produce incoherent rollouts on which teacher supervision is noise\(Liet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib18); Zhaoet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib6)\)\.

These are not three independent problems but three symptoms of*ignoring how supervision quality varies with competence*\. We observe that*entropy*provides a unified observable for this quantity across all three scales\.

We introduceSEAD, a framework operationalizing this principle:

1. 1\.Token\-level: Sparse Entropy\-Adaptive Divergence\.Joint teacher\-student entropy partitions tokens into Zone A \(skip\), Zone B \(RKL\), Zone C \(FKL\)—jointly determining selection*and*divergence type\.
2. 2\.Temporal: Competence\-Driven Annealing\.Continuous FKL→\\toRKL transition tracking the evolving active token composition\.
3. 3\.Prompt\-level: Competence\-Gated Curriculum\.First prompt\-level curriculum for OPD, addressing an open problem flagged byZhaoet al\.\([2026](https://arxiv.org/html/2606.28562#bib.bib6)\); Liet al\.\([2026](https://arxiv.org/html/2606.28562#bib.bib18)\)\.

Crucially, these are symbiotically necessary: token selection requires coherent rollouts \(curriculum\), and annealing requires monotonic competence growth \(also curriculum\)\. The ablation \(Section[4](https://arxiv.org/html/2606.28562#S4)\) confirms super\-additive interactions\.

We validate SEAD on OLMo\-3 \(7B→\\to32B\) and Nemotron \(8B→\\to49B\) across MATH\-500, Minerva\-Math, AIME 2024/2025, AMC 2023, and OlympiadBench\. The full framework achieves \+4\.8 average over vanilla OPD\.

![Refer to caption](https://arxiv.org/html/2606.28562v1/fig/sys_v3.png)Figure 1:Overview of SEAD\.First, the*competence\-gated curriculum*\(left, yellow\) selects prompts within the student’s current ability frontier \(di≤c​\(t\)d\_\{i\}\\leq c\(t\)\), ensuring rollouts are coherent enough for meaningful supervision\. The studentπθ\\pi\_\{\\theta\}generates a rollout, which the teacherπte\\pi\_\{\\text\{te\}\}scores with per\-token logits\. From these, we compute joint entropy \(HθH\_\{\\theta\},HteH\_\{\\text\{te\}\}\) and partition tokens into three zones:Zone A\(gray\)—both models confident, zero gradient \(∼50%\{\\sim\}50\\%of tokens\);Zone B\(blue\)—teacher confident but student uncertain, supervised via reverse KL to sharpen toward the teacher’s mode;Zone C\(red\)—teacher uncertain at reasoning forks, supervised via forward KL to preserve multi\-path diversity\. A*temporal annealing*schedule \(bottom\-right\) modulates Zone C asα⋅ℒFKL\+\(1−α\)⋅ℒRKL\\alpha\\cdot\\mathcal\{L\}\_\{\\text\{FKL\}\}\+\(1\-\\alpha\)\\cdot\\mathcal\{L\}\_\{\\text\{RKL\}\}, smoothly transitioning from exploration \(α=0\.8\\alpha\{=\}0\.8\) to refinement \(α=0\\alpha\{=\}0\) over training\. The combined loss updates the student\. See Algorithm[1](https://arxiv.org/html/2606.28562#alg1)for the full procedure\.
## 2Method

### 2\.1Preliminaries and Problem Setting

Letπθ\\pi\_\{\\theta\}andπte\\pi\_\{\\text\{te\}\}denote the student’s and teacher’s policies respectively\. For a given prompt𝐪\\mathbf\{q\}, let𝐜t=\(𝐪,x1,…,xt−1\)\\mathbf\{c\}\_\{t\}=\(\\mathbf\{q\},x\_\{1\},\\dots,x\_\{t\-1\}\)denote the context up to steptt, andxt∈𝒱x\_\{t\}\\in\\mathcal\{V\}the generated token\. Each OPD iteration: \(1\) samples rollouts𝐱∼πθold\(⋅\|𝐪\)\\mathbf\{x\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|\\mathbf\{q\}\); \(2\) queries teacher logits on the rollout; \(3\) updatesπθ\\pi\_\{\\theta\}via a divergence loss\. The two standard choices areforward KL\(mode\-covering\):

ℒFKL​\(t\)=∑v∈𝒱πte​\(v\|𝐜t\)​log⁡πte​\(v\|𝐜t\)πθ​\(v\|𝐜t\)\\mathcal\{L\}\_\{\\text\{FKL\}\}\(t\)=\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\text\{te\}\}\(v\|\\mathbf\{c\}\_\{t\}\)\\log\\frac\{\\pi\_\{\\text\{te\}\}\(v\|\\mathbf\{c\}\_\{t\}\)\}\{\\pi\_\{\\theta\}\(v\|\\mathbf\{c\}\_\{t\}\)\}\(1\)andreverse KL\(mode\-seeking\):

ℒRKL​\(t\)=∑v∈𝒱πθ​\(v\|𝐜t\)​log⁡πθ​\(v\|𝐜t\)πte​\(v\|𝐜t\)\\mathcal\{L\}\_\{\\text\{RKL\}\}\(t\)=\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\|\\mathbf\{c\}\_\{t\}\)\\log\\frac\{\\pi\_\{\\theta\}\(v\|\\mathbf\{c\}\_\{t\}\)\}\{\\pi\_\{\\text\{te\}\}\(v\|\\mathbf\{c\}\_\{t\}\)\}\(2\)Forward KL encourages the student to cover all modes of the teacher, preserving diversity at reasoning branch points\. Reverse KL drives the student to concentrate on the teacher’s high\-probability modes, yielding sharper outputs but risking premature entropy collapse\(Koet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib3)\)\.

###### Assumption 1\(OPD supervision quality — modulated PL condition\)\.

Letℓi​\(θ\)\\ell\_\{i\}\(\\theta\)denote the per\-prompt OPD loss andℓi⋆=infθℓi​\(θ\)\\ell\_\{i\}^\{\\star\}=\\inf\_\{\\theta\}\\ell\_\{i\}\(\\theta\)\. There exists a non\-decreasing functionϕ:\[0,1\]→\[0,1\]\\phi\\colon\[0,1\]\\to\[0,1\]withϕ​\(0\)=0\\phi\(0\)=0,ϕ​\(1\)=1\\phi\(1\)=1, and a constantμ\>0\\mu\>0such that for each promptii: :

‖∇ℓi​\(θ\)‖2≥ϕ​\(pi​\(θ\)\)​μ​\(ℓi​\(θ\)−ℓi⋆\)\.\\\|\\nabla\\ell\_\{i\}\(\\theta\)\\\|^\{2\}\\geq\\phi\(p\_\{i\}\(\\theta\)\)\\,\\mu\\,\(\\ell\_\{i\}\(\\theta\)\-\\ell\_\{i\}^\{\\star\}\)\.\(3\)

Intuitively, gradient signal scales with competence: when the student cannot solve a problem at all \(pi≈0p\_\{i\}\\approx 0\), its rollouts are incoherent andϕ​\(pi\)≈0\\phi\(p\_\{i\}\)\\approx 0, so the lower bound is almost a zero regardless of the teacher’s quality\. Entropy serves as the observable proxy for this principle: joint teacher–student entropy reveals whether a token position is informative, redundant, or noise\. Assumption[1](https://arxiv.org/html/2606.28562#Thmassumption1)motivates a unified principle:*allocate compute only where supervision quality is high*\. SEAD operationalizes this at three granularities: \(i\)*token\-level*—skip tokens where both teacher and student are already confident \(zero information gain\); \(ii\)*temporal*—shift from exploratory FKL to sharpening RKL as competence grows; \(iii\)*prompt\-level*—restrict training to prompts where rollouts are coherent enough for the teacher to provide meaningful corrections\. We present these in order of increasing scope, noting that the prompt\-level curriculum \(Sec\.[2\.5](https://arxiv.org/html/2606.28562#S2.SS5)\) is the foundational enabler: it ensures the entropy landscape is well\-behaved, which the token\-level selector requires, and it guarantees monotonic competence growth, which the temporal annealer assumes\.

### 2\.2The Unified SEAD Objective

SEAD integrates token\-level selection, temporal annealing, and prompt\-level curriculum into a single loss\. Letℬ\\mathcal\{B\}and𝒞\\mathcal\{C\}denote the sets of token indices assigned to RKL and FKL respectively \(formally defined in Sec\.[2\.3](https://arxiv.org/html/2606.28562#S2.SS3)\), and letλ\>0\\lambda\>0be a hyperparameter balancing the intrinsic scale differences between the two divergence terms:

ℒSEAD​\(θ,tstep\)=1\|ℬ∪𝒞\|​\(∑t∈ℬℒRKL​\(t\)\+α​\(tstep\)⋅λ​∑t∈𝒞ℒFKL​\(t\)\),𝐪∼Uniform​\(𝒟​\(tstep\)\)\\mathcal\{L\}\_\{\\text\{SEAD\}\}\(\\theta,t\_\{\\text\{step\}\}\)=\\frac\{1\}\{\|\\mathcal\{B\}\\cup\\mathcal\{C\}\|\}\\left\(\\sum\_\{t\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{\\text\{RKL\}\}\(t\)\+\\alpha\(t\_\{\\text\{step\}\}\)\\cdot\\lambda\\sum\_\{t\\in\\mathcal\{C\}\}\\mathcal\{L\}\_\{\\text\{FKL\}\}\(t\)\\right\),\\quad\\mathbf\{q\}\\sim\\text\{Uniform\}\(\\mathcal\{D\}\(t\_\{\\text\{step\}\}\)\)\(4\)Three control variables—all functions of training progress, all serving the same principle \(allocate compute only where supervision quality is high\):

- •Zone partition\{𝒜,ℬ,𝒞\}\\\{\\mathcal\{A\},\\mathcal\{B\},\\mathcal\{C\}\\\}: which tokens receive gradient and which divergence \(Sec\.[2\.3](https://arxiv.org/html/2606.28562#S2.SS3)\);
- •Annealing coefficientα​\(tstep\)\\alpha\(t\_\{\\text\{step\}\}\): FKL/RKL balance evolving with competence \(Sec\.[2\.4](https://arxiv.org/html/2606.28562#S2.SS4)\);
- •Eligible set𝒟​\(tstep\)=\{𝐪i:di≤c​\(tstep\)\}\\mathcal\{D\}\(t\_\{\\text\{step\}\}\)=\\\{\\mathbf\{q\}\_\{i\}:d\_\{i\}\\leq c\(t\_\{\\text\{step\}\}\)\\\}: prompts within the competence frontier \(Sec\.[2\.5](https://arxiv.org/html/2606.28562#S2.SS5)\)\.

Algorithm 1SEAD: Competence\-Aware On\-Policy Distillation0:Student

πθ\\pi\_\{\\theta\}, teacher

πte\\pi\_\{\\text\{te\}\}, prompts

𝒬\\mathcal\{Q\}with difficulty scores

\{di\}\\\{d\_\{i\}\\\}, zone percentiles

\(ρA,ρB,ρC\)\(\\rho\_\{A\},\\rho\_\{B\},\\rho\_\{C\}\), annealing schedule

α​\(⋅\)\\alpha\(\\cdot\), competence function

c​\(⋅\)c\(\\cdot\)
1:Precompute per\-prompt difficulty

di=1−pid\_\{i\}=1\-p\_\{i\}via student pass rate

2:for

tstep=1t\_\{\\text\{step\}\}=1to

TtotalT\_\{\\text\{total\}\}do

3:

𝒟←\{𝐪i:di≤c​\(tstep\)\}\\mathcal\{D\}\\leftarrow\\\{\\mathbf\{q\}\_\{i\}:d\_\{i\}\\leq c\(t\_\{\\text\{step\}\}\)\\\}\{Competence\-gated curriculum\}

4:Sample prompt batch

\{𝐪i\}⊂𝒟\\\{\\mathbf\{q\}\_\{i\}\\\}\\subset\\mathcal\{D\}; generate rollouts

𝐱i∼πθold\(⋅\|𝐪i\)\\mathbf\{x\}\_\{i\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|\\mathbf\{q\}\_\{i\}\)
5:Query teacher

πte\(⋅\|𝐜t\)\\pi\_\{\\text\{te\}\}\(\\cdot\|\\mathbf\{c\}\_\{t\}\); compute

Hte​\(t\)H\_\{\\text\{te\}\}\(t\),

Hθ​\(t\)H\_\{\\theta\}\(t\)per position

6:Assign tokens to Zone A, B, C via percentile thresholds

7:

α←α​\(tstep\)\\alpha\\leftarrow\\alpha\(t\_\{\\text\{step\}\}\)\{Competence\-driven annealing\}

8:

ℒ←1\|ℬ∪𝒞\|​\(∑t∈ℬℒRKL​\(t\)\+α​λ​∑t∈𝒞ℒFKL​\(t\)\)\\mathcal\{L\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{B\}\\cup\\mathcal\{C\}\|\}\\\!\\left\(\\sum\_\{t\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{\\text\{RKL\}\}\(t\)\+\\alpha\\lambda\\sum\_\{t\\in\\mathcal\{C\}\}\\mathcal\{L\}\_\{\\text\{FKL\}\}\(t\)\\right\)
9:Update

θ\\thetavia clipped gradient step on

ℒ\\mathcal\{L\}
10:endfor

### 2\.3Token\-Level: Sparse Entropy\-Adaptive Divergence

We define token\-level entropyHte​\(t\)=−∑vπte​\(v\|𝐜t\)​log⁡πte​\(v\|𝐜t\)H\_\{\\text\{te\}\}\(t\)=\-\\sum\_\{v\}\\pi\_\{\\text\{te\}\}\(v\|\\mathbf\{c\}\_\{t\}\)\\log\\pi\_\{\\text\{te\}\}\(v\|\\mathbf\{c\}\_\{t\}\)andHθ​\(t\)H\_\{\\theta\}\(t\)analogously\. To maintain computational tractability and avoid the prohibitive overhead of a full softmax over the entire vocabulary𝒱\\mathcal\{V\}during training,Hte​\(t\)H\_\{\\text\{te\}\}\(t\)is approximated using the top\-kkvocabulary subset\. SEAD partitions tokens into

- •Zone A\(Skip,∼ρA%\{\\sim\}\\rho\_\{A\}\\%\): BothHte​\(t\)H\_\{\\text\{te\}\}\(t\)andHθ​\(t\)H\_\{\\theta\}\(t\)low\.Zero gradient—the vast majority of tokens \(connectives, formatting, deterministic steps\) fall here\.
- •Zone B\(RKL,∼ρB%\{\\sim\}\\rho\_\{B\}\\%\):Hte​\(t\)H\_\{\\text\{te\}\}\(t\)low,Hθ​\(t\)H\_\{\\theta\}\(t\)high\. Student should*sharpen*toward the confident teacher\.
- •Zone C\(FKL,∼ρC%\{\\sim\}\\rho\_\{C\}\\%\):Hte​\(t\)H\_\{\\text\{te\}\}\(t\)high\. Genuine reasoning forks—student should*cover*teacher modes\.

The per\-step loss is:

ℒSEADtoken=1\|ℬ∪𝒞\|​\(∑t∈ℬℒRKL​\(t\)\+λ​∑t∈𝒞ℒFKL​\(t\)\)\\mathcal\{L\}\_\{\\text\{SEAD\}\}^\{\\text\{token\}\}=\\frac\{1\}\{\|\\mathcal\{B\}\\cup\\mathcal\{C\}\|\}\\left\(\\sum\_\{t\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{\\text\{RKL\}\}\(t\)\+\\lambda\\sum\_\{t\\in\\mathcal\{C\}\}\\mathcal\{L\}\_\{\\text\{FKL\}\}\(t\)\\right\)\(5\)with defaultsρA=50\\rho\_\{A\}=50,ρB=40\\rho\_\{B\}=40,ρC=10\\rho\_\{C\}=10\. Unlike EOPD\(Jinet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib4)\)\(teacher entropy only\) and TIP\(Xuet al\.,[2026a](https://arxiv.org/html/2606.28562#bib.bib10)\)\(weighting, single divergence\), SEAD jointly determines both*selection*and*divergence type*via*joint*teacher\-student entropy, with extreme sparsity \(∼20%\{\\sim\}20\\%active\)\.

#### Sparse selection is approximately lossless\.

The following theorem shows that Zone A tokens contribute negligible gradient, justifying their exclusion\. The key precondition is that Zone A tokens have genuinely low entropy for*both*teacher and student—a property that holds when the curriculum \(Sec\.[2\.5](https://arxiv.org/html/2606.28562#S2.SS5)\) ensures coherent rollouts, preventing spurious low\-entropy assignments from misaligned contexts\.

###### Theorem 1\.

Under mild regularity \(bounded score functions\) and the assumption that Zone A tokens satisfyHte​\(t\)≤τH\_\{\\mathrm\{te\}\}\(t\)\\leq\\tau,Hθ​\(t\)≤τH\_\{\\theta\}\(t\)\\leq\\tauwithTV​\(πte,πθ\)≤δ​\(τ\)\\mathrm\{TV\}\(\\pi\_\{\\mathrm\{te\}\},\\pi\_\{\\theta\}\)\\leq\\delta\(\\tau\), the full gradientggand sparse gradientg^\\hat\{g\}\(computed on Zones B∪\\cupC only\) satisfy:

‖g−\(1−s\)​g^‖≤s​G​\(2​τ\+2​δ​\(τ\)\+τ/2​log⁡\|𝒱\|\)=O​\(s​G​τ​log⁡\|𝒱\|\)\\\|g\-\(1\-s\)\\hat\{g\}\\\|\\leq s\\,G\\bigl\(\\sqrt\{2\\tau\}\+2\\,\\delta\(\\tau\)\+\\sqrt\{\\tau/2\}\\,\\log\|\\mathcal\{V\}\|\\bigr\)=O\(s\\,G\\sqrt\{\\tau\}\\,\\log\|\\mathcal\{V\}\|\)\(6\)wheres=\|𝒜\|/N<1s=\|\\mathcal\{A\}\|/N<1is the skip fraction \(NN= sequence length\),GGbounds score function norms, and\|𝒱\|\|\\mathcal\{V\}\|is the vocabulary size\.

The bound vanishes asτ→0\\tau\\to 0: Zone A tokens are selected precisely for near\-zero entropy, making their gradient contributions negligible\. Crucially, this guarantee requires the entropy landscape to be*well\-behaved*—which holds when curriculum ensures coherent rollouts\.

#### Connection to temporal scheduling\.

As the student improves, Zone A expands and the active set shifts toward Zone B\. The optimal FKL/RKL ratio among active tokens thus evolves, motivating a temporal schedule\.

### 2\.4Temporal: Competence\-Driven Divergence Annealing

As the student masters more tokens, the active token composition shifts: early in training Zone C \(both uncertain\) is large, calling for mode\-covering FKL; late in training Zone B \(teacher confident, student uncertain\) dominates, calling for mode\-seeking RKL\. The annealing schedule tracks this:

α​\(tstep\)=αend\+αstart−αend2​\(1\+cos⁡\(tstepTtotal⋅π\)\)\\alpha\(t\_\{\\text\{step\}\}\)=\\alpha\_\{\\text\{end\}\}\+\\frac\{\\alpha\_\{\\text\{start\}\}\-\\alpha\_\{\\text\{end\}\}\}\{2\}\\left\(1\+\\cos\\\!\\left\(\\frac\{t\_\{\\text\{step\}\}\}\{T\_\{\\text\{total\}\}\}\\cdot\\pi\\right\)\\right\)\(7\)decreasing fromαstart=0\.8\\alpha\_\{\\text\{start\}\}=0\.8toαend=0\.0\\alpha\_\{\\text\{end\}\}=0\.0\. Combined with SEAD zones,α\\alphamodulates Zone C’s FKL weight in Eq\.[4](https://arxiv.org/html/2606.28562#S2.E4), implementing a continuous exploration→\\rightarrowrefinement transition that generalizes the discrete two\-stage switches ofKoet al\.\([2026](https://arxiv.org/html/2606.28562#bib.bib3)\)andXuet al\.\([2026b](https://arxiv.org/html/2606.28562#bib.bib15)\)\.

Temporal annealing assumes monotonically improving competence—an assumption that can be violated under uniform prompt sampling\. The following curriculum ensures this precondition holds\.

### 2\.5Prompt\-Level: Competence\-Gated Curriculum

The deepest manifestation of the supervision quality principle \(Assumption[1](https://arxiv.org/html/2606.28562#Thmassumption1)\): on prompts far beyond the student’s capability,pi​\(θ\)≈0p\_\{i\}\(\\theta\)\\approx 0impliesϕ​\(pi​\(θ\)\)≈0\\phi\(p\_\{i\}\(\\theta\)\)\\approx 0, so the modulated PL condition yields vanishing gradient signal\. The rollouts are incoherent and teacher corrections amount to noise\. This degradation is absent in off\-policy KD \(teacher traces are always coherent,ϕ≡1\\phi\\equiv 1\) and RL \(binary rewards are equally reliable regardless of rollout quality\)\. The curriculum keepsϕ​\(pi​\(θ\)\)\\phi\(p\_\{i\}\(\\theta\)\)bounded away from zero on all eligible prompts—a precondition for both Theorem[1](https://arxiv.org/html/2606.28562#Thmtheorem1)\(the entropy landscape must be well\-behaved for Zone A tokens to genuinely have low entropy\) and temporal annealing \(competence must grow monotonically for the schedule to track the correct FKL/RKL ratio\)\.

#### Difficulty estimation\.

We sampleKKrollouts per prompt fromπθ0\\pi\_\{\\theta\_\{0\}\}and compute difficultydi=1−pid\_\{i\}=1\-p\_\{i\}wherepi=1K​∑k𝟏​\[correct​\(𝐱i\(k\)\)\]p\_\{i\}=\\frac\{1\}\{K\}\\sum\_\{k\}\\mathbf\{1\}\[\\mathrm\{correct\}\(\\mathbf\{x\}^\{\(k\)\}\_\{i\}\)\]\. Pass rate is a binary discretization of rollout entropy, maintaining the entropy\-as\-unified\-signal principle\.

#### Competence\-based progression\.

FollowingPlatanioset al\.\([2019](https://arxiv.org/html/2606.28562#bib.bib22)\), a competence functionc​\(t\)=min⁡\(1,c0​\(1\+t/T\)p\)c\(t\)=\\min\(1,c\_\{0\}\(1\+t/T\)^\{p\}\)grows monotonically, with eligible set𝒟​\(t\)=\{𝐪i:di≤c​\(t\)\}\\mathcal\{D\}\(t\)=\\\{\\mathbf\{q\}\_\{i\}:d\_\{i\}\\leq c\(t\)\\\}\. In OPD,c​\(t\)c\(t\)delineates prompts where rollouts are coherent enough for informative teacher supervision\.

#### Staleness of difficulty scores\.

Becausedid\_\{i\}is computed from the initial studentπθ0\\pi\_\{\\theta\_\{0\}\}, the relative ordering of prompts may drift as training progresses\. While monotonic improvement guarantees that initially “easy” prompts remain solvable, mid\-tier prompts may improve at different rates\. Consequently, we rely on the curriculum as a coarse filter rather than a strict total ordering, trading precise difficulty tracking for computational efficiency\. The primary risk is minor sample inefficiency, not gradient corruption, as the curriculum’s only theoretical requirement is maintaining a minimum competence floorϕ​\(pi​\(θ\)\)≥c\>0\\phi\(p\_\{i\}\(\\theta\)\)\\geq c\>0on all eligible prompts\.

Under Assumption[1](https://arxiv.org/html/2606.28562#Thmassumption1), and assumingβ\\beta\-smoothness of the loss and bounded gradient varianceσ2\\sigma^\{2\}\(see Appendix[A\.3](https://arxiv.org/html/2606.28562#A1.SS3)\), we have the following theorem:

###### Theorem 2\(Curriculum convergence\)\.

The curriculum achievesϵ\\epsilon\-accuracy inTC=𝒪​\(β​σ2/\(\(c​μ\)2​ϵ\)\)T^\{\\textup\{C\}\}=\\mathcal\{O\}\(\\beta\\sigma^\{2\}/\(\(c\\mu\)^\{2\}\\epsilon\)\)steps, versusTU≥Ω​\(β​σ2/\(\(ϕ¯U​μ\)2​ϵ\)\)T^\{\\textup\{U\}\}\\geq\\Omega\(\\beta\\sigma^\{2\}/\(\(\\bar\{\\phi\}^\{\\textup\{U\}\}\\mu\)^\{2\}\\epsilon\)\)for uniform sampling, whereϕ¯U=1n​∑iϕ​\(pi​\(θ\)\)\\bar\{\\phi\}^\{\\textup\{U\}\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\phi\(p\_\{i\}\(\\theta\)\)is the average modulation under uniform sampling\. This yields a theoretical speedup factor ofS2=\(c/ϕ¯U\)2S^\{2\}=\(c/\\bar\{\\phi\}^\{\\textup\{U\}\}\)^\{2\}in the worst case—a separation unique to OPDand absent in off\-policy KD \(ϕ≡1\\phi\\equiv 1\) or RL \(unbiased gradients regardless of competence\)\.

###### Corollary 3\(Mutual reinforcement\)\.

The three SEAD components form a virtuous cycle, not a circular dependency\. The resolution is*staged bootstrapping*:

1. 1\.Curriculum provides the base\.By restricting to prompts withϕ​\(pi\)≥c\>0\\phi\(p\_\{i\}\)\\geq c\>0, the curriculum ensures coherent rollouts, which in turn make the entropy landscape reliable \(low\-entropy tokens in Zone A genuinely agree between teacher and student\)\.
2. 2\.Reliable entropy enables sparse selection\.Given a well\-behaved entropy landscape, Theorem[1](https://arxiv.org/html/2606.28562#Thmtheorem1)guarantees that skipping Zone A introduces negligible error, bounded by𝒪​\(s​G​τmax\)\\mathcal\{O\}\(s\\,G\\sqrt\{\\tau\_\{\\max\}\}\)\.
3. 3\.Sparse selection \+ monotonic competence enable annealing\.With competence growing monotonically \(Theorem[2](https://arxiv.org/html/2606.28562#Thmtheorem2)\) and Zone composition shifting predictably, the cosine scheduleα​\(t\)\\alpha\(t\)correctly tracks the exploration→\\torefinement transition\.

As stages progress, the competence lower boundcctightens, Zone A entropy thresholdτ\\taudecreases, and the sparse gradient bound improves: all three components strengthen jointly\.

## 3Evaluation

### 3\.1Experimental Setup

#### Models\.

We evaluate our approach on two distinct model families to demonstrate generalizability\. The first isNemotron\(Nano\-8B student, Super\-49B teacher\)and the second isOLMo\(7B\-Instruct\-SFT student, 32B\-Instruct teacher\), which allows us to rigorously test our method in the challenging small teacher\-student capability gap regime\.

#### Training\.

We trained our models using 4 nodes\. Each node is equipped with 8 H100 80GB GPUs and 1TB memory\. We utilize a rigorously deduplicated version of the DAPO\-Math\-17K dataset\(Yuet al\.,[2025](https://arxiv.org/html/2606.28562#bib.bib25)\)for all training phases\. More training details could be found in Appendix[B\.2](https://arxiv.org/html/2606.28562#A2.SS2)\.

#### Evaluation\.

Our evaluation suite comprehensively assesses mathematical reasoning capabilities across varying difficulty levels\. We report greedy Pass@1 accuracy on standard reasoning benchmarks, including MATH\-500, Minerva\-Math, and OlympiadBench\. To evaluate performance on highly complex, competition\-level mathematics, we utilize the AMC 2023 and AIME 2024/2025 datasets, reporting the Pass@32 metric \(the average pass rate over 32 independently sampled trajectories per problem\)\.

#### Baselines\.

We benchmark our approach against several strong paradigms: Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28562#bib.bib37)\), Vanilla OPD utilizing full\-token Reverse Kullback\-Leibler \(RKL\) divergence\(Lu and Thinking Machines Lab,[2025](https://arxiv.org/html/2606.28562#bib.bib2)\), and On\-Policy Self\-Distillation \(OPSD\)\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib6)\)\.

### 3\.2Main Results

We present the results of OLMo model pair in Table[1](https://arxiv.org/html/2606.28562#S3.T1)and leave Nemotron results in Appendix[C\.1](https://arxiv.org/html/2606.28562#A3.SS1)\.

Table 1:Results on OLMo model pair \(OLMo\-7B student, OLMo\-32B teacher\)\.#### The Distillation Challenge\.

As shown in Table[1](https://arxiv.org/html/2606.28562#S3.T1), bridging the massive 17\.2\-point performance gap between the OLMo teacher and student is challenging\. While the teacher establishes a strong upper bound \(75\.4 average\), standard off\-policy distillation \(OPD\) baselines fail to meaningfully improve upon the base student\. OPSD and GRPO stagnate near the student’s 58\.2 average, while Vanilla OPD \(RKLk=1k=1\) yields a marginal\+1\.0\+1\.0improvement\.

#### Efficacy of Proposed Components\.

KL Annealing alone provides a modest gain \(\+0\.2 avg\), confirming that divergence scheduling without token\-level structure has limited impact\. Token Zones alone similarly yields marginal improvement \(\+0\.3\), as zone\-based sparsity without a matching temporal schedule leaves the FKL/RKL balance untuned\. The critical finding is their*synergy*: combining Token Zones with KL Annealing produces a \+4\.5 jump to 63\.7 average—a20×20\\timesamplification over either component in isolation\. This validates the symbiotic design: zone sparsity concentrates gradient on informative tokens, while annealing ensures the divergence applied to those tokens evolves appropriately\. Adding the competence\-gated curriculum \(SEAD full\) further improves to 64\.0, with the easy\-to\-hard ordering providing the coherent rollouts that stabilize entropy\-based zone assignment\.

#### Combined Method Performance\.

Combining Token Zones with KL Annealing yields robust performance, achieving a 63\.7 average \(a\+5\.5\+5\.5absolute increase over the base student\), with pronounced gains on the most rigorous datasets, including\+7\.7\+7\.7points on AIME24 and\+7\.5\+7\.5points on AIME25\. Ultimately, the full SEAD method further pushes the average to 64\.0\. These results indicate that stabilizing the token zone strategy with an annealed KL penalty effectively mitigates baseline stagnation, drastically elevating student capabilities and closing the teacher\-student gap\.

### 3\.3Ablation Study

We conduct a232^\{3\}factorial ablation on OLMo\-3\-7B/32B to isolate three components:T\(Token SEAD, 50/40/10 zone assignment\),A\(KL annealing from 0 to target\), andC\(curriculum\)\. Table[2](https://arxiv.org/html/2606.28562#S3.T2)reveals three findings: Curriculum \(C\) is the single strongest factor, yielding \+4\.20 avg\. accuracy alone, while annealing \(A\) in isolation provides negligible gain \(\+0\.22\)\. T\+A jumps to \+4\.52, a 20×\\timesamplification over A alone, demonstrating that entropy\-adaptive zone sparsity requires a matching schedule to be effective\. The three\-way combination T\+A\+C achieves the best overall result \(64\.00 avg\., \+5\.2 AIME25\), with each pairwise interaction contributing complementary gains\. T\+A matches curriculum\-only \(63\.70 vs\. 63\.38\), and A\+C \(63\.97\) narrows the gap to the full system\. This confirms that SEAD’s token\-level zone assignment, annealing, and curriculum address orthogonal bottlenecks: zone sparsity targets*which*tokens receive gradient, annealing controls*when*sparsity activates, and curriculum determines*what order*examples are presented\.

Table 2:232^\{3\}ablation\. T = Token SEAD, A = Annealing, C = Curriculum\. The AIMEΔ\\Deltacolumn highlights the supervision quality cascade: T alone hurts hard benchmarks; T\+C recovers via reliable entropy signals\. The full combination is super\-additive\.

## 4Analysis

### 4\.1Token\-Level Analysis

![Refer to caption](https://arxiv.org/html/2606.28562v1/x1.png)\(a\)Sorted per\-token FKL by zone\. The top 50% of tokens \(Zones B\+C\) concentrate 90% of total loss, while the remaining 50% \(Zone A\) carry only 10%\.
![Refer to caption](https://arxiv.org/html/2606.28562v1/x2.png)\(b\)Token\-level zone visualization on OLMo\-3\-7B output\. Background color indicates zone assignment; opacity is proportional to FKL magnitude\. Strategy forks \(dark red\) cluster at reasoning decision points\.
![Refer to caption](https://arxiv.org/html/2606.28562v1/x3.png)\(c\)Zone assignment in joint entropy space \(HTH\_\{T\}vs\.HSH\_\{S\}\)\. Jaccard overlap of top\-20% sets is 63\.7%, confirming complementary signals from teacher and student entropy\.
![Refer to caption](https://arxiv.org/html/2606.28562v1/x4.png)\(d\)Loss concentration by zone\. Zone C \(10% of tokens\) carries 31\.9% of FKL loss \(3\.19×3\.19\\timesconcentration\); Zone B\+C together achieve1\.81×1\.81\\timesover uniform\.

Figure 2:SEAD zone analysis on OLMo\-3\-7B / OLMo\-3\.1\-32B \(50/40/10 config\)\.\(a\) Sorted per\-token FKL reveals a heavy\-tailed loss distribution where 50% of tokens dominate gradient signal\. \(b\) Zone coloring on actual model output shows Zone C \(red\) tokens align with reasoning forks\. \(c\) Joint entropy space confirms teacher and student signals are partially complementary \(Jaccard = 63\.7%\)\. \(d\) Loss concentration validates that Zone A tokens are safe to skip \(0\.19×0\.19\\timesratio\) while Zone C tokens concentrate3\.19×3\.19\\timesmore loss per token\.#### Entropy distribution and zone assignment\.

Zone A tokens overwhelmingly correspond to deterministic computation steps and connectives, e\.g\., on OLMo\-3 7B/32B \(50/40/10 config\), 73\.2% of computation tokens and 54\.2% of syntax tokens are assigned to Zone A\. Zone B tokens cluster at reasoning transitions where the student holds partial knowledge \(17\.2% strategy forks, 52\.7% common tokens receiving moderate gradient\), and RKL sharpens the student toward the teacher’s peak mode\. Zone C tokens appear at genuine reasoning forks—63\.9% are*strategy fork*positions where both teacher and student entropy exceed the 75th percentile, representing critical decision points with multiple valid continuations where FKL’s mode\-covering property prevents catastrophic collapse\. Crucially, the 50/40/10 configuration routes*zero*strategy fork tokens to Zone A, ensuring every reasoning\-critical position receives active supervision \(Table[3](https://arxiv.org/html/2606.28562#S4.T3)\)\. Figure[2\(b\)](https://arxiv.org/html/2606.28562#S4.F2.sf2)visualizes this on actual model output: Zone C tokens \(red\) concentrate at branching points in the reasoning chain, while Zone A tokens \(green\) cover predictable syntax and arithmetic with near\-zero loss contribution\.

#### Selection overlap and signal complementarity\.

With a Jaccard index of 63\.7% between the top\-20% teacher entropy and top\-20% student entropy positions, approximately 36% of high\-entropy tokens are unique to one signal \(Figure[2\(c\)](https://arxiv.org/html/2606.28562#S4.F2.sf3)\)\. Teacher entropy identifies positions where guidance is most valuable \(multiple valid continuations\), while student entropy identifies positions the student has already mastered\. This partial overlap justifies joint conditioning: neither signal alone captures the full picture\.

#### Gradient concentration\.

Figure[2\(d\)](https://arxiv.org/html/2606.28562#S4.F2.sf4)quantifies the concentration effect: Zone A \(50% of tokens\) carries only 9\.7% of total FKL loss, while Zone B\+C \(50% of tokens\) concentrates 90\.3% of gradient signal\. The per\-zone ratios are stark: Zone C tokens carry3\.19×3\.19\\timestheir fair share of loss, Zone B tokens carry1\.46×1\.46\\times, and Zone A tokens carry only0\.19×0\.19\\times\. This means skipping Zone A discards half the tokens while retaining over 90% of the training signal, a highly efficient compute\-quality tradeoff\. Figure[2\(a\)](https://arxiv.org/html/2606.28562#S4.F2.sf1)shows this visually: the sorted FKL curve drops precipitously, with Zone B and C tokens \(orange, red\) dominating the high\-loss region while Zone A tokens \(green\) cluster near zero\.

Table 3:Token categorization: P\(zone\|\|category\)\. The 50/40/10 config routes zero strategy forks to Zone A, ensuring all reasoning\-critical tokens receive supervision\.

### 4\.2Training Dynamics

A known failure mode of vanilla OPD is premature entropy collapse, where the student model overly concentrates probability mass on the teacher’s peak mode before exploring alternative reasoning paths\. SEAD effectively mitigates this collapse\. As illustrated in Appendix[C\.2](https://arxiv.org/html/2606.28562#A3.SS2), vanilla OPD exhibits a clear decreasing trend in average token\-level entropy during training, whereas SEAD maintains stable, consistently higher entropy levels\. This preserves necessary distributional diversity while still sharpening the student’s outputs\.

## 5Related Work

#### On\-Policy Distillation\.

GKD\(Agarwalet al\.,[2024](https://arxiv.org/html/2606.28562#bib.bib1)\)established on\-policy distillation for LMs\. Extensions include RKL for reasoning\(Lu and Thinking Machines Lab,[2025](https://arxiv.org/html/2606.28562#bib.bib2)\), self\-distillation \(OPSD,Zhaoet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib6)\), and theoretical unification with dense KL\-constrained RL \(G\-OPD,Yanget al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib5)\)\. Recent systems work focuses on scaling\(Koet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib3); Wu and others,[2026](https://arxiv.org/html/2606.28562#bib.bib8)\)\.

#### Token\-Level Selection\.

EOPD\(Jinet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib4)\)switches FKL/RKL based on teacher entropy alone\. TIP\(Xuet al\.,[2026a](https://arxiv.org/html/2606.28562#bib.bib10)\)uses student entropy×\\timesdivergence for weighting with a single divergence\. SE\-KD\(Tavoret al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib11)\)and SCOPE\(Zhenget al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib12)\)explore other selection criteria\. SEAD differs via \(i\)*joint*teacher\-student partitioning, and \(ii\) per\-zone divergence switching\.

#### Divergence Scheduling\.

Reopold\(Koet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib3)\)uses a discrete two\-stage schedule; PACED\(Xuet al\.,[2026b](https://arxiv.org/html/2606.28562#bib.bib15)\)finds forward\-then\-reverse optimal; DRKL\(Luong and others,[2026](https://arxiv.org/html/2606.28562#bib.bib16)\)addresses entropy collapse\. Our annealing generalizes these to continuous transitions motivated by evolving zone composition\.

#### Curriculum for Reasoning\.

Curriculum has been studied for RL\-based reasoning\(Parasharet al\.,[2025](https://arxiv.org/html/2606.28562#bib.bib23); Wenet al\.,[2025](https://arxiv.org/html/2606.28562#bib.bib24)\)\. However,No prior work applies curriculum to OPD, despite it being flagged as an open problem\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib6); Liet al\.,[2026](https://arxiv.org/html/2606.28562#bib.bib18)\)\. Our contribution is showing*why*it is uniquely necessary: the modulated PL condition \(Assumption[1](https://arxiv.org/html/2606.28562#Thmassumption1)\) formalizes supervision quality degradation specific to OPD\.

## 6Conclusion and Limitations

In conclusion, SEAD addresses the dependence of teacher supervision on student competence in on\-policy distillation by utilizing entropy as a unified signal\. It establishes a mutually reinforcing framework operating across three levels: applying adaptive divergences to informative tokens, temporally annealing from exploration to refinement, and employing a competence\-gated prompt curriculum\. However, this work has several limitations\. First, the curriculum relies on static difficulty scores rather than computationally expensive adaptive re\-estimation\. Second, our method may apply to other domains \(e\.g\., code\) for future work\. Finally, SEAD’s efficacy on very long reasoning chains requires further investigation\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§1](https://arxiv.org/html/2606.28562#S1.p1.1),[§1](https://arxiv.org/html/2606.28562#S1.p2.1),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px1.p1.1)\.
- W\. Jin, T\. Min, Y\. Yang, S\. R\. Kadhe, Y\. Zhou, D\. Wei, N\. Baracaldo, and K\. Lee \(2026\)Entropy\-aware on\-policy distillation of language models\.arXiv preprint arXiv:2603\.07079\.Cited by:[1st item](https://arxiv.org/html/2606.28562#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2606.28562#S1.p2.1),[§2\.3](https://arxiv.org/html/2606.28562#S2.SS3.p1.9),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px2.p1.1)\.
- H\. Karimi, J\. Nutini, and M\. Schmidt \(2016\)Linear convergence of gradient and proximal\-gradient methods under the polyak\-łojasiewicz condition\.InJoint European conference on machine learning and knowledge discovery in databases,pp\. 795–811\.Cited by:[§A\.3](https://arxiv.org/html/2606.28562#A1.SS3.4.p4.5)\.
- J\. Ko, S\. Abdali, Y\. J\. Kim, T\. Chen, and P\. Cameron \(2026\)Scaling reasoning efficiently via relaxed on\-policy distillation\.arXiv preprint arXiv:2603\.11137\.Cited by:[2nd item](https://arxiv.org/html/2606.28562#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2606.28562#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.28562#S2.SS1.p1.10),[§2\.4](https://arxiv.org/html/2606.28562#S2.SS4.p1.4),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[3rd item](https://arxiv.org/html/2606.28562#S1.I1.i3.p1.1),[item 3](https://arxiv.org/html/2606.28562#S1.I2.i3.p1.1),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px4.p1.1)\.
- K\. Lu and Thinking Machines Lab \(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.External Links:[Document](https://dx.doi.org/10.64434/tml.20251026),[Link](https://thinkingmachines.ai/blog/on-policy-distillation)Cited by:[§1](https://arxiv.org/html/2606.28562#S1.p2.1),[§3\.1](https://arxiv.org/html/2606.28562#S3.SS1.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px1.p1.1)\.
- H\. Luonget al\.\(2026\)Diversity\-aware reverse Kullback\-Leibler divergence for large language model distillation\.arXiv preprint arXiv:2604\.00223\.Cited by:[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px3.p1.1)\.
- S\. Parashar, S\. Gui, X\. Li, H\. Ling, S\. Vemuri, B\. Olson, E\. Li, Y\. Zhang, J\. Caverlee, D\. Kalathil,et al\.\(2025\)Curriculum reinforcement learning from easy to hard tasks improves llm reasoning\.arXiv preprint arXiv:2506\.06632\.Cited by:[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px4.p1.1)\.
- E\. A\. Platanios, O\. Stretcu, G\. Neubig, B\. Poczos, and T\. Mitchell \(2019\)Competence\-based curriculum learning for neural machine translation\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 1162–1172\.Cited by:[§B\.2](https://arxiv.org/html/2606.28562#A2.SS2.p1.13),[§2\.5](https://arxiv.org/html/2606.28562#S2.SS5.SSS0.Px2.p1.3)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3\.1](https://arxiv.org/html/2606.28562#S3.SS1.SSS0.Px4.p1.1)\.
- M\. Song and M\. Zheng \(2026\)A survey of on\-policy distillation for large language models\.arXiv preprint arXiv:2604\.00626\.Cited by:[§1](https://arxiv.org/html/2606.28562#S1.p1.1)\.
- A\. Tavor, I\. Ebenspanger, N\. Cnaan, and M\. Geva \(2026\)Rethinking selective knowledge distillation\.arXiv preprint arXiv:2602\.01395\.Cited by:[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px2.p1.1)\.
- L\. Wen, Y\. Cai, F\. Xiao, X\. He, Q\. An, Z\. Duan, Y\. Du, J\. Liu, T\. Tanglifu, X\. Lv,et al\.\(2025\)Light\-r1: curriculum sft, dpo and rl for long cot from scratch and beyond\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 318–327\.Cited by:[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px4.p1.1)\.
- Y\. Wuet al\.\(2026\)Lightning OPD: efficient post\-training for large reasoning models with offline on\-policy distillation\.arXiv preprint arXiv:2604\.13010\.Cited by:[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. Geramifard \(2026a\)TIP: token importance in on\-policy distillation\.arXiv preprint arXiv:2604\.14084\.Cited by:[§2\.3](https://arxiv.org/html/2606.28562#S2.SS3.p1.9),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px2.p1.1)\.
- Y\. Xu, H\. Sang, Z\. Zhou, R\. He, and Z\. Wang \(2026b\)PACED: distillation at the frontier of student competence\.arXiv preprint arXiv:2603\.11178\.Cited by:[2nd item](https://arxiv.org/html/2606.28562#S1.I1.i2.p1.1),[§2\.4](https://arxiv.org/html/2606.28562#S2.SS4.p1.4),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px3.p1.1)\.
- W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. Lin \(2026\)Learning beyond teacher: generalized on\-policy distillation with reward extrapolation\.arXiv preprint arXiv:2602\.12125\.Cited by:[§1](https://arxiv.org/html/2606.28562#S1.p2.1),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px1.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§3\.1](https://arxiv.org/html/2606.28562#S3.SS1.SSS0.Px2.p1.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[3rd item](https://arxiv.org/html/2606.28562#S1.I1.i3.p1.1),[item 3](https://arxiv.org/html/2606.28562#S1.I2.i3.p1.1),[§3\.1](https://arxiv.org/html/2606.28562#S3.SS1.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px4.p1.1)\.
- B\. Zheng, X\. Ma, Y\. Liang, J\. Ruan, X\. Fu, K\. Lin, B\. Zhu, K\. Zeng, and X\. Cai \(2026\)SCOPE: signal\-calibrated on\-policy distillation enhancement with dual\-path adaptive weighting\.arXiv preprint arXiv:2604\.10688\.Cited by:[§5](https://arxiv.org/html/2606.28562#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix ATheoretical Proofs

### A\.1Full Assumptions for Theorem[1](https://arxiv.org/html/2606.28562#Thmtheorem1)

###### Assumption 2\(Bounded score functions\)\.

There exists a constantG\>0G\>0such that for every token positionttand vocabulary elementv∈𝒱v\\in\\mathcal\{V\},∥∇θlogπθ\(v∣𝐜t\)∥≤G\\\|\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\\|\\leq G\.

###### Assumption 3\(Low\-entropy agreement in Zone A\)\.

For every token positiont∈𝒜t\\in\\mathcal\{A\}, the teacher and student distributions satisfyHte​\(t\)≤τH\_\{\\mathrm\{te\}\}\(t\)\\leq\\tauandHθ​\(t\)≤τH\_\{\\theta\}\(t\)\\leq\\tau, andTV\(πte\(⋅∣𝐜t\),πθ\(⋅∣𝐜t\)\)≤δ\(τ\)\\mathrm\{TV\}\(\\pi\_\{\\mathrm\{te\}\}\(\\cdot\\mid\\mathbf\{c\}\_\{t\}\),\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{c\}\_\{t\}\)\)\\leq\\delta\(\\tau\), whereδ​\(τ\)→0\\delta\(\\tau\)\\to 0asτ→0\\tau\\to 0\. A sufficient condition is that both distributions place mass≥1−τ/2\\geq 1\-\\sqrt\{\\tau/2\}on a common modevt⋆v^\{\\star\}\_\{t\}\.

###### Assumption 4\(Bounded log\-density ratio\)\.

For allv∈𝒱v\\in\\mathcal\{V\}and token positionstt,\|logπθ\(v∣𝐜t\)−logπte\(v∣𝐜t\)\|≤Bmax\|\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\-\\log\\pi\_\{\\mathrm\{te\}\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\|\\leq B\_\{\\max\}\. In Zone A, the low\-entropy agreement \(Assumption[3](https://arxiv.org/html/2606.28562#Thmassumption3)\) guarantees that for the dominant modevt⋆v^\{\\star\}\_\{t\}:\|log⁡πθ​\(vt⋆\)−log⁡πte​\(vt⋆\)\|≤δ​\(τ\)/\(1−τ/2\)\|\\log\\pi\_\{\\theta\}\(v^\{\\star\}\_\{t\}\)\-\\log\\pi\_\{\\mathrm\{te\}\}\(v^\{\\star\}\_\{t\}\)\|\\leq\\delta\(\\tau\)/\(1\-\\sqrt\{\\tau/2\}\), while off\-mode tokens \(v≠vt⋆v\\neq v^\{\\star\}\_\{t\}\) satisfy\|log⁡πθ​\(v\)−log⁡πte​\(v\)\|≤log⁡\|𝒱\|\|\\log\\pi\_\{\\theta\}\(v\)\-\\log\\pi\_\{\\mathrm\{te\}\}\(v\)\|\\leq\\log\|\\mathcal\{V\}\|trivially since all probabilities are at least1/\|𝒱\|1/\|\\mathcal\{V\}\|under temperature\-bounded sampling\.

### A\.2Proof of Theorem[1](https://arxiv.org/html/2606.28562#Thmtheorem1)

###### Proof\.

Write the full gradient as a convex decomposition over zones:

g=\|𝒜\|N​g¯𝒜\+\|ℬ∪𝒞\|N​g^,g=\\frac\{\|\\mathcal\{A\}\|\}\{N\}\\,\\bar\{g\}\_\{\\mathcal\{A\}\}\+\\frac\{\|\\mathcal\{B\}\\cup\\mathcal\{C\}\|\}\{N\}\\,\\hat\{g\},whereg¯𝒜=1\|𝒜\|​∑t∈𝒜∇θℒ​\(t\)\\bar\{g\}\_\{\\mathcal\{A\}\}=\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{t\\in\\mathcal\{A\}\}\\nabla\_\{\\theta\}\\mathcal\{L\}\(t\)\. Since\|ℬ∪𝒞\|/N=1−s\|\\mathcal\{B\}\\cup\\mathcal\{C\}\|/N=1\-s:

g−\(1−s\)​g^=s​g¯𝒜\.g\-\(1\-s\)\\,\\hat\{g\}=s\\,\\bar\{g\}\_\{\\mathcal\{A\}\}\.It suffices to bound‖∇θℒ​\(t\)‖\\\|\\nabla\_\{\\theta\}\\mathcal\{L\}\(t\)\\\|for eacht∈𝒜t\\in\\mathcal\{A\}\.

#### Forward KL gradient at a Zone A token\.

For forward KL,∇θLFKL​\(t\)=−∑v∈𝒱πte​\(v∣𝐜t\)​∇θlog⁡πθ​\(v∣𝐜t\)\\nabla\_\{\\theta\}\\,L\_\{\\mathrm\{FKL\}\}\(t\)=\-\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\mathrm\{te\}\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\. Using the score\-function identity∑vπθ​\(v∣𝐜t\)​∇θlog⁡πθ​\(v∣𝐜t\)=𝟎\\sum\_\{v\}\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)=\\mathbf\{0\}:

∇θLFKL​\(t\)\\displaystyle\\nabla\_\{\\theta\}\\,L\_\{\\mathrm\{FKL\}\}\(t\)=−∑v\[πte​\(v∣𝐜t\)−πθ​\(v∣𝐜t\)\]​∇θlog⁡πθ​\(v∣𝐜t\)\.\\displaystyle=\-\\sum\_\{v\}\\bigl\[\\pi\_\{\\mathrm\{te\}\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\-\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\bigr\]\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\.\(8\)By Assumption[2](https://arxiv.org/html/2606.28562#Thmassumption2),‖∇θLFKL​\(t\)‖≤2​G​TV​\(πte,πθ\)≤2​G​δ​\(τ\)\\\|\\nabla\_\{\\theta\}\\,L\_\{\\mathrm\{FKL\}\}\(t\)\\\|\\leq 2\\,G\\,\\mathrm\{TV\}\(\\pi\_\{\\mathrm\{te\}\},\\pi\_\{\\theta\}\)\\leq 2\\,G\\,\\delta\(\\tau\)\.

WhenHte​\(t\)≤τH\_\{\\mathrm\{te\}\}\(t\)\\leq\\tau, the teacher places mass≥1−τ/2\\geq 1\-\\sqrt\{\\tau/2\}on a single tokenv⋆v^\{\\star\}\(via the binary entropy inequalityh​\(p\)=−p​log⁡p−\(1−p\)​log⁡\(1−p\)≥2​p2h\(p\)=\-p\\log p\-\(1\{\-\}p\)\\log\(1\{\-\}p\)\\geq 2p^\{2\}for smallpp, applied to the off\-mode mass\)\. Similarly for the student\. Then from \([8](https://arxiv.org/html/2606.28562#A1.E8)\):‖∇θLFKL​\(t\)‖≤G​\(2​τ\+2​δ​\(τ\)\)\\\|\\nabla\_\{\\theta\}\\,L\_\{\\mathrm\{FKL\}\}\(t\)\\\|\\leq G\(\\sqrt\{2\\tau\}\+2\\,\\delta\(\\tau\)\)\.

#### Reverse KL gradient at a Zone A token\.

The gradient ofLRKL\(t\)=KL\(πθ\(⋅∣𝐜t\)∥πte\(⋅∣𝐜t\)\)L\_\{\\mathrm\{RKL\}\}\(t\)=\\mathrm\{KL\}\(\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{c\}\_\{t\}\)\\\|\\pi\_\{\\mathrm\{te\}\}\(\\cdot\\mid\\mathbf\{c\}\_\{t\}\)\)is:

∇θLRKL​\(t\)\\displaystyle\\nabla\_\{\\theta\}L\_\{\\mathrm\{RKL\}\}\(t\)=∑v∈𝒱∇θ\[πθ​\(v∣𝐜t\)​\(log⁡πθ​\(v∣𝐜t\)−log⁡πte​\(v∣𝐜t\)\)\]\\displaystyle=\\sum\_\{v\\in\\mathcal\{V\}\}\\nabla\_\{\\theta\}\\\!\\left\[\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\bigl\(\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\-\\log\\pi\_\{\\mathrm\{te\}\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\bigr\)\\right\]=∑v∈𝒱πθ​\(v∣𝐜t\)​\[log⁡πθ​\(v∣𝐜t\)−log⁡πte​\(v∣𝐜t\)\+1\]​∇θlog⁡πθ​\(v∣𝐜t\)\\displaystyle=\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\bigl\[\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\-\\log\\pi\_\{\\mathrm\{te\}\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\+1\\bigr\]\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)=∑v∈𝒱πθ​\(v∣𝐜t\)​\[log⁡πθ​\(v∣𝐜t\)−log⁡πte​\(v∣𝐜t\)\]​∇θlog⁡πθ​\(v∣𝐜t\),\\displaystyle=\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\bigl\[\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\-\\log\\pi\_\{\\mathrm\{te\}\}\(v\\mid\\mathbf\{c\}\_\{t\}\)\\bigr\]\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\\mathbf\{c\}\_\{t\}\),\(9\)where the last equality uses the score function identity∑vπθ​\(v\)​∇θlog⁡πθ​\(v\)=𝟎\\sum\_\{v\}\\pi\_\{\\theta\}\(v\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\)=\\mathbf\{0\}\.

In Zone A, both distributions concentrate mass≥1−τ/2\\geq 1\-\\sqrt\{\\tau/2\}on a common modev⋆v^\{\\star\}\(Assumption[3](https://arxiv.org/html/2606.28562#Thmassumption3)\)\. Splitting the sum into on\-mode \(v=v⋆v=v^\{\\star\}\) and off\-mode \(v≠v⋆v\\neq v^\{\\star\}\) contributions:

- •On\-mode:πθ​\(v⋆\)≥1−τ/2\\pi\_\{\\theta\}\(v^\{\\star\}\)\\geq 1\-\\sqrt\{\\tau/2\}and\|log⁡πθ​\(v⋆\)−log⁡πte​\(v⋆\)\|≤δ​\(τ\)/\(1−τ/2\)\|\\log\\pi\_\{\\theta\}\(v^\{\\star\}\)\-\\log\\pi\_\{\\mathrm\{te\}\}\(v^\{\\star\}\)\|\\leq\\delta\(\\tau\)/\(1\-\\sqrt\{\\tau/2\}\)\.
- •Off\-mode:Total mass≤τ/2\\leq\\sqrt\{\\tau/2\}, and each log\-ratio is bounded bylog⁡\|𝒱\|\\log\|\\mathcal\{V\}\|\(Assumption[4](https://arxiv.org/html/2606.28562#Thmassumption4)\)\.

By Assumption[2](https://arxiv.org/html/2606.28562#Thmassumption2):

‖∇θLRKL​\(t\)‖\\displaystyle\\\|\\nabla\_\{\\theta\}L\_\{\\mathrm\{RKL\}\}\(t\)\\\|≤G​\[\(1−τ/2\)⋅δ​\(τ\)1−τ/2\+τ/2⋅log⁡\|𝒱\|\]\\displaystyle\\leq G\\left\[\(1\-\\sqrt\{\\tau/2\}\)\\cdot\\frac\{\\delta\(\\tau\)\}\{1\-\\sqrt\{\\tau/2\}\}\+\\sqrt\{\\tau/2\}\\cdot\\log\|\\mathcal\{V\}\|\\right\]=G​\(δ​\(τ\)\+τ/2​log⁡\|𝒱\|\)=𝒪​\(G​τ​log⁡\|𝒱\|\)\.\\displaystyle=G\\bigl\(\\delta\(\\tau\)\+\\sqrt\{\\tau/2\}\\,\\log\|\\mathcal\{V\}\|\\bigr\)=\\mathcal\{O\}\\\!\\bigl\(G\\sqrt\{\\tau\}\\,\\log\|\\mathcal\{V\}\|\\bigr\)\.\(10\)

#### Aggregation\.

The FKL bound givesG​\(2​τ\+2​δ​\(τ\)\)G\(\\sqrt\{2\\tau\}\+2\\,\\delta\(\\tau\)\)and the RKL bound givesG​\(δ​\(τ\)\+τ/2​log⁡\|𝒱\|\)G\(\\delta\(\\tau\)\+\\sqrt\{\\tau/2\}\\,\\log\|\\mathcal\{V\}\|\)\. Taking the maximum:‖g¯𝒜‖≤G​\(2​τ\+2​δ​\(τ\)\+τ/2​log⁡\|𝒱\|\)\\\|\\bar\{g\}\_\{\\mathcal\{A\}\}\\\|\\leq G\(\\sqrt\{2\\tau\}\+2\\,\\delta\(\\tau\)\+\\sqrt\{\\tau/2\}\\,\\log\|\\mathcal\{V\}\|\)\. Therefore‖g−\(1−s\)​g^‖=s​‖g¯𝒜‖≤s​G​\(2​τ\+2​δ​\(τ\)\+τ/2​log⁡\|𝒱\|\)=O​\(s​G​τ​log⁡\|𝒱\|\)\\\|g\-\(1\-s\)\\hat\{g\}\\\|=s\\\|\\bar\{g\}\_\{\\mathcal\{A\}\}\\\|\\leq s\\,G\(\\sqrt\{2\\tau\}\+2\\,\\delta\(\\tau\)\+\\sqrt\{\\tau/2\}\\,\\log\|\\mathcal\{V\}\|\)=O\(s\\,G\\sqrt\{\\tau\}\\,\\log\|\\mathcal\{V\}\|\)\. In practicelog⁡\|𝒱\|=O​\(log⁡105\)≈11\.5\\log\|\\mathcal\{V\}\|=O\(\\log 10^\{5\}\)\\approx 11\.5is a moderate constant, so the bound simplifies toO​\(s​G​τ\)O\(s\\,G\\sqrt\{\\tau\}\)up to logarithmic factors\. ∎

### A\.3Full Assumptions and Proof of Theorem[2](https://arxiv.org/html/2606.28562#Thmtheorem2)

The analysis requires four assumptions\. Assumptions[5](https://arxiv.org/html/2606.28562#Thmassumption5)and[6](https://arxiv.org/html/2606.28562#Thmassumption6)are standard; the novelty is Assumption[1](https://arxiv.org/html/2606.28562#Thmassumption1)\.

###### Assumption 5\(Smoothness\)\.

Eachℓi\\ell\_\{i\}isβ\\beta\-smooth:‖∇ℓi​\(θ\)−∇ℓi​\(θ′\)‖≤β​‖θ−θ′‖\\\|\\nabla\\ell\_\{i\}\(\\theta\)\-\\nabla\\ell\_\{i\}\(\\theta^\{\\prime\}\)\\\|\\leq\\beta\\\|\\theta\-\\theta^\{\\prime\}\\\|\.

###### Assumption 6\(Bounded stochastic gradients\)\.

𝔼​\[gi​\(θ\)\]=∇ℓi​\(θ\)\\mathbb\{E\}\[g\_\{i\}\(\\theta\)\]=\\nabla\\ell\_\{i\}\(\\theta\)and𝔼​‖gi​\(θ\)−∇ℓi​\(θ\)‖2≤σ2\\mathbb\{E\}\\\|g\_\{i\}\(\\theta\)\-\\nabla\\ell\_\{i\}\(\\theta\)\\\|^\{2\}\\leq\\sigma^\{2\}\.

###### Assumption 7\(Stage mastery\)\.

For each stagess, afterTsT\_\{s\}SGD steps on𝒬≤s\\mathcal\{Q\}\_\{\\leq s\}, the student achievespi​\(θ\)≥αminp\_\{i\}\(\\theta\)\\geq\\alpha\_\{\\min\}for allqi∈𝒬≤sq\_\{i\}\\in\\mathcal\{Q\}\_\{\\leq s\}, whereϕ​\(αmin\)≥c\>0\\phi\(\\alpha\_\{\\min\}\)\\geq c\>0\.

###### Proof of Theorem[2](https://arxiv.org/html/2606.28562#Thmtheorem2)\.

Step 1: Per\-step descent\.Byβ\\beta\-smoothness andη≤1/β\\eta\\leq 1/\\beta:

𝔼g​\[ℓi​\(θt\)−ℓi​\(θt\+1\)\]≥η2​‖∇ℓi​\(θt\)‖2−η2​β​σ22\.\\mathbb\{E\}\_\{g\}\[\\ell\_\{i\}\(\\theta\_\{t\}\)\-\\ell\_\{i\}\(\\theta\_\{t\+1\}\)\]\\geq\\frac\{\\eta\}\{2\}\\\|\\nabla\\ell\_\{i\}\(\\theta\_\{t\}\)\\\|^\{2\}\-\\frac\{\\eta^\{2\}\\beta\\sigma^\{2\}\}\{2\}\.\(11\)
Step 1b: Lifting to stage\-level loss\.At each step, the SGD algorithm samples a promptiiuniformly from the eligible set𝒟s\\mathcal\{D\}\_\{s\}at stagess\. Define the stage\-level average lossLs​\(θ\)=1\|𝒟s\|​∑i∈𝒟sℓi​\(θ\)L\_\{s\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{D\}\_\{s\}\|\}\\sum\_\{i\\in\\mathcal\{D\}\_\{s\}\}\\ell\_\{i\}\(\\theta\)\. Taking expectation overi∼Uniform​\(𝒟s\)i\\sim\\mathrm\{Uniform\}\(\\mathcal\{D\}\_\{s\}\)in \([11](https://arxiv.org/html/2606.28562#A1.E11)\):

𝔼i,g​\[Ls​\(θt\)−Ls​\(θt\+1\)\]≥η2​𝔼i​\[‖∇ℓi​\(θt\)‖2\]−η2​β​σ22\.\\mathbb\{E\}\_\{i,g\}\[L\_\{s\}\(\\theta\_\{t\}\)\-L\_\{s\}\(\\theta\_\{t\+1\}\)\]\\geq\\frac\{\\eta\}\{2\}\\,\\mathbb\{E\}\_\{i\}\\\!\\left\[\\\|\\nabla\\ell\_\{i\}\(\\theta\_\{t\}\)\\\|^\{2\}\\right\]\-\\frac\{\\eta^\{2\}\\beta\\sigma^\{2\}\}\{2\}\.\(12\)Crucially,𝔼i​\[‖∇ℓi‖2\]≥‖𝔼i​\[∇ℓi\]‖2=‖∇Ls​\(θt\)‖2\\mathbb\{E\}\_\{i\}\[\\\|\\nabla\\ell\_\{i\}\\\|^\{2\}\]\\geq\\\|\\mathbb\{E\}\_\{i\}\[\\nabla\\ell\_\{i\}\]\\\|^\{2\}=\\\|\\nabla L\_\{s\}\(\\theta\_\{t\}\)\\\|^\{2\}by Jensen, so the bound on the*average*of squared norms is at least as strong as what a global PL condition onLsL\_\{s\}would require\. We proceed using𝔼i​\[‖∇ℓi‖2\]\\mathbb\{E\}\_\{i\}\[\\\|\\nabla\\ell\_\{i\}\\\|^\{2\}\]directly\.

Step 2: Modulated PL\.By Assumption[1](https://arxiv.org/html/2606.28562#Thmassumption1):‖∇ℓi​\(θt\)‖2≥ϕ​\(pi​\(θt\)\)​μ​\(ℓi​\(θt\)−ℓi⋆\)\\\|\\nabla\\ell\_\{i\}\(\\theta\_\{t\}\)\\\|^\{2\}\\geq\\phi\(p\_\{i\}\(\\theta\_\{t\}\)\)\\,\\mu\\,\(\\ell\_\{i\}\(\\theta\_\{t\}\)\-\\ell\_\{i\}^\{\\star\}\)\. Averaging overi∼Uniform​\(𝒟s\)i\\sim\\mathrm\{Uniform\}\(\\mathcal\{D\}\_\{s\}\)and substituting into \([12](https://arxiv.org/html/2606.28562#A1.E12)\):

𝔼i,g​\[Ls​\(θt\)−Ls​\(θt\+1\)\]≥η​μ2​𝔼i​\[ϕ​\(pi​\(θt\)\)​\(ℓi​\(θt\)−ℓi⋆\)\]−η2​β​σ22\.\\mathbb\{E\}\_\{i,g\}\[L\_\{s\}\(\\theta\_\{t\}\)\-L\_\{s\}\(\\theta\_\{t\+1\}\)\]\\geq\\frac\{\\eta\\mu\}\{2\}\\,\\mathbb\{E\}\_\{i\}\\\!\\left\[\\phi\(p\_\{i\}\(\\theta\_\{t\}\)\)\\,\(\\ell\_\{i\}\(\\theta\_\{t\}\)\-\\ell\_\{i\}^\{\\star\}\)\\right\]\-\\frac\{\\eta^\{2\}\\beta\\sigma^\{2\}\}\{2\}\.\(13\)
Step 3: Curriculum analysis\.Under curriculum,ϕ​\(pi​\(θ\)\)≥c\\phi\(p\_\{i\}\(\\theta\)\)\\geq con all eligible prompts \(Assumption[7](https://arxiv.org/html/2606.28562#Thmassumption7)\)\. From Step 2, usingϕ≥c\\phi\\geq cuniformly:𝔼i​\[ϕ​\(pi\)​\(ℓi−ℓi⋆\)\]≥c​\(Ls​\(θt\)−Ls⋆\)\\mathbb\{E\}\_\{i\}\[\\phi\(p\_\{i\}\)\(\\ell\_\{i\}\-\\ell\_\{i\}^\{\\star\}\)\]\\geq c\\,\(L\_\{s\}\(\\theta\_\{t\}\)\-L\_\{s\}^\{\\star\}\)\. This yields a standard PL descent with effective constantμeff=c​μ\\mu\_\{\\mathrm\{eff\}\}=c\\mu\. Applying the SGD convergence rate under PL\[Karimiet al\.,[2016](https://arxiv.org/html/2606.28562#bib.bib21)\]with optimal step sizeη=c​μ​ϵ/\(β​σ2\)\\eta=c\\mu\\epsilon/\(\\beta\\sigma^\{2\}\):

TC=O​\(β​σ2\(c​μ\)2​ϵ\+1c​μ​log⁡1ϵ\)\.T^\{\\textup\{C\}\}=O\\\!\\left\(\\frac\{\\beta\\sigma^\{2\}\}\{\(c\\,\\mu\)^\{2\}\\,\\epsilon\}\+\\frac\{1\}\{c\\,\\mu\}\\log\\frac\{1\}\{\\epsilon\}\\right\)\.The first term dominates in the stochastic regime \(σ2\>0\\sigma^\{2\}\>0\)\.

Step 4: Uniform analysis\.Under uniform sampling over allnnprompts, defineϕ¯U=1n​∑i=1nϕ​\(pi​\(θ\)\)\\bar\{\\phi\}^\{\\,\\textup\{U\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\phi\(p\_\{i\}\(\\theta\)\)\. The expected descent from Step 2 becomes:

𝔼i​\[ϕ​\(pi\)​\(ℓi−ℓi⋆\)\]\.\\mathbb\{E\}\_\{i\}\[\\phi\(p\_\{i\}\)\\,\(\\ell\_\{i\}\-\\ell\_\{i\}^\{\\star\}\)\]\.Since competenceϕ​\(pi\)\\phi\(p\_\{i\}\)and suboptimality\(ℓi−ℓi⋆\)\(\\ell\_\{i\}\-\\ell\_\{i\}^\{\\star\}\)are negatively correlated \(easy prompts have highϕ\\phibut low suboptimality\), by Chebyshev’s sum inequality:

𝔼i​\[ϕ​\(pi\)​\(ℓi−ℓi⋆\)\]≤ϕ¯U⋅\(L​\(θ\)−L⋆\)\.\\mathbb\{E\}\_\{i\}\[\\phi\(p\_\{i\}\)\\,\(\\ell\_\{i\}\-\\ell\_\{i\}^\{\\star\}\)\]\\leq\\bar\{\\phi\}^\{\\,\\textup\{U\}\}\\cdot\(L\(\\theta\)\-L^\{\\star\}\)\.Therefore the effective PL constant under uniform sampling is*at most*ϕ¯U​μ\\bar\{\\phi\}^\{\\,\\textup\{U\}\}\\mu\(the true constant is even smaller due to the negative correlation, making uniform sampling*strictly worse*than this upper bound suggests\)\. Applying the same SGD convergence machinery:

TU≥Ω​\(β​σ2\(ϕ¯U​μ\)2​ϵ\+1ϕ¯U​μ​log⁡1ϵ\)\.T^\{\\textup\{U\}\}\\geq\\Omega\\\!\\left\(\\frac\{\\beta\\sigma^\{2\}\}\{\(\\bar\{\\phi\}^\{\\,\\textup\{U\}\}\\,\\mu\)^\{2\}\\,\\epsilon\}\+\\frac\{1\}\{\\bar\{\\phi\}^\{\\,\\textup\{U\}\}\\,\\mu\}\\log\\frac\{1\}\{\\epsilon\}\\right\)\.Because hard prompts contributeϕ≈0\\phi\\approx 0early in training,ϕ¯U<c\\bar\{\\phi\}^\{\\,\\textup\{U\}\}<c\.

Step 5: Speed\-up\.In the stochastic regime, the ratio of convergence times is:TU/TC≥Ω​\(\(c/ϕ¯U\)2\)T^\{\\textup\{U\}\}/T^\{\\textup\{C\}\}\\geq\\Omega\\\!\\left\(\(c/\\bar\{\\phi\}^\{\\textup\{U\}\}\)^\{2\}\\right\)\. Worst case: the student is initially competent only on stage 1 \(1/S1/Sfraction of prompts\), soϕ¯U≤c/S\\bar\{\\phi\}^\{\\textup\{U\}\}\\leq c/S, giving speedup≥Ω​\(S2\)\\geq\\Omega\(S^\{2\}\)\. Even in the log\-dominated \(deterministic\) regime,TU/TC=Θ​\(c/ϕ¯U\)=Θ​\(S\)T^\{\\textup\{U\}\}/T^\{\\textup\{C\}\}=\\Theta\(c/\\bar\{\\phi\}^\{\\textup\{U\}\}\)=\\Theta\(S\)\.

Step 6: OPD specificity\.

- •Off\-policy KD:ϕ≡1\\phi\\equiv 1\(teacher traces always coherent\)\. No curriculum advantage\.
- •RL:Binary reward gives unbiased \(if high\-variance\) gradients regardless of competence\. OPD is different: teacher logits on incoherent rollouts are*biased*, not just noisy\. Theϕ\\phi\-modulation captures gradient*norm*degradation, not just variance\.

∎

## Appendix BAdditional Experiment Setup

### B\.1Licenses for Existing Assets

We use only existing models, datasets, benchmarks, and software frameworks whose terms permit research use\. Table[4](https://arxiv.org/html/2606.28562#A2.T4)summarizes the main assets used in this work\. We use these assets for training, evaluation, and comparison only, and we do not redistribute third\-party model weights, benchmark data, or dataset contents unless their licenses explicitly permit redistribution\. When releasing code, we will include links to the original asset sources, preserve copyright and license notices, and specify the exact versions used for all reproducibility\-critical dependencies\.

Table 4:Existing assets used in the paper\. For assets with provider\-specific or competition\-specific terms, we use them only for evaluation and do not redistribute their contents\.All third\-party assets are used consistently with their intended research or evaluation purposes\. For benchmarks derived from mathematical competitions or external providers, we report aggregate evaluation results only and do not claim ownership over the underlying problems\. The proposed method itself does not require any proprietary dataset or private user data\.

### B\.2Training Details\.

During each iteration, we sampleB=96B=96prompts and generate one rollout per prompt using a sampling temperature of 0\.7 and a maximum generation length of 16,384 tokens\. Following teacher scoring, we performK=1K=1gradient update with a clipping parameterϵ=0\.2\\epsilon=0\.2\. We use a peak learning rate of5×10−95\\times 10^\{\-9\}with a linear warmup and decay schedule over a single training epoch\. For the SEAD configuration, the regularization weights are set toρA=50\\rho\_\{A\}=50,ρB=40\\rho\_\{B\}=40, andρC=10\\rho\_\{C\}=10\. The annealing factor follows a cosine schedule, decaying smoothly fromαstart=0\.8\\alpha\_\{\\text\{start\}\}=0\.8toαend=0\.0\\alpha\_\{\\text\{end\}\}=0\.0\. For curriculum learning, we adopt a smooth competence\-based ordering inspired byPlatanioset al\.\[[2019](https://arxiv.org/html/2606.28562#bib.bib22)\]\. We first estimate per\-problem difficulty asdi=1−a^id\_\{i\}=1\-\\hat\{a\}\_\{i\}, wherea^i\\hat\{a\}\_\{i\}is the pass rate computed fromK=8K=8independent rollouts of the student model prior to training\. Examples are then sorted by difficulty in ascending order and presented easy\-to\-hard in a single pass without replacement, with small Gaussian noise \(σ=0\.02\\sigma=0\.02\) added to difficulty scores for stochastic tie\-breaking\. This provides a smooth, continuous curriculum that naturally increases problem difficulty as training progresses, without requiring discrete phase boundaries or explicit competence thresholds\.

## Appendix CAdditional Experimental Results

### C\.1Results on Nemotron

We present the results on Nemotron model pair in Table[5](https://arxiv.org/html/2606.28562#A3.T5)\.

Table 5:Results on Nemotron model pair \(Nemotron\-8B student, Nemotron\-49B teacher\)\.
### C\.2Training Dynamics

A known failure mode of OPD is premature entropy collapse: if the student is trained with reverse KL from the beginning, it is encouraged to concentrate probability mass on the teacher’s current peak mode before it has explored alternative valid reasoning paths\. SEAD addresses this through the interaction of token selection and KL annealing\. Early in training, the coefficientα\\alphais high, so Zone C tokens receive forward\-KL supervision; later,α\\alphadecays toward zero, shifting the active objective toward reverse KL refinement\.

The key distinction from a global annealing baseline is that SEAD does not apply this uniformly across all tokens\. Roughly 50% of tokens are assigned to Zone A and skipped, because both teacher and student are already confident on them\. Consequently, annealing only affects the subset of tokens where divergence choice matters: Zone C reasoning forks, where forward KL preserves the teacher’s support, and Zone B partially mastered tokens, where reverse KL sharpens the student toward the teacher’s preferred continuation\.

Figure[3](https://arxiv.org/html/2606.28562#A3.F3)shows the average entropy for different methods during the training\. Among these methods, vanilla OPD shows a clear decreasing trend in entropy, where all our SEAD\-related methods keep the entropy at a similar level and higher than vanilla OPD, which proves the ability of our method to prevent entropy collapse\.

![Refer to caption](https://arxiv.org/html/2606.28562v1/x5.png)Figure 3:Average token\-level entropy over training steps\. Vanilla OPD \(red\) suffers progressive entropy collapse \(0\.39→0\.310\.39\\to 0\.31\), while SEAD’s token selection \(blue\), annealing \(green\), and their combination \(purple\) maintain stable entropy throughout training, preserving the diversity for token distribution\.

## Appendix DBroader Impacts

This work aims to improve the efficiency of distilling reasoning capabilities from larger language models into smaller student models\. By reducing the amount of token\-level supervision required during on\-policy distillation, SEAD may lower the computational cost of post\-training and make strong reasoning models more accessible to researchers and practitioners with limited hardware resources\. This could have positive impacts by reducing energy consumption, lowering deployment costs, and enabling broader participation in research on reasoning\-oriented language models\.

At the same time, improving the efficiency of reasoning\-model distillation may also make capable models easier to train, adapt, and deploy\. Such models could be misused for generating misleading content, automating deceptive reasoning, assisting with harmful planning, or increasing the scale of low\-cost model deployment without sufficient safety evaluation\. Although our experiments focus on mathematical reasoning benchmarks, the same distillation principles may transfer to other domains where stronger reasoning capabilities require careful governance\.

Our method does not directly address model safety, factuality, robustness, bias, privacy, or misuse prevention\. In particular, SEAD optimizes the efficiency and effectiveness of teacher–student distillation, but it does not guarantee that the student inherits only desirable behaviors from the teacher\. If the teacher model exhibits unsafe, biased, or unreliable behavior, the student may also learn such behavior\. Therefore, practical deployment of models trained with SEAD should include standard safety evaluations, red\-teaming, domain\-specific risk assessment, and monitoring for unintended behavior\.

The empirical scope of this paper is limited to mathematical reasoning benchmarks and two teacher–student model families\. As a result, the broader societal implications of applying SEAD to open\-domain assistants, code\-generation systems, or high\-stakes decision\-support systems remain uncertain\. We encourage future work to evaluate competence\-aware distillation under broader safety, robustness, and alignment criteria before deployment in sensitive settings\.

## Appendix EDeclaration of LLM Usage

This paper studies large language models as the primary subject of research\. The proposed method uses teacher and student language models during on\-policy distillation, as described in the main text\.

In addition, the authors used large language model tools to assist with non\-substantive writing tasks, including improving clarity, editing grammar, drafting checklist responses, and revising explanatory text\. All technical ideas, algorithmic design choices, theoretical claims, proofs, experimental results, tables, and conclusions were produced, checked, and approved by the authors\. The authors take full responsibility for the content of the paper, including any text that was edited with LLM assistance\.