Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD

# Not Every Token Is Worth Distilling:Selective Supervision for Direct-OPD
Source: [https://arxiv.org/html/2609.29142](https://arxiv.org/html/2609.29142)
Yibo Zhao††thanks:Equal contribution\. The first two authors may list their names in either order on their CVs\.Yunshi LanXiang Li††thanks:Corresponding author:xiangli@dase\.ecnu\.edu\.cn\.Affiliation:School of Data Science and EngineeringAffiliation:East China Normal UniversityAffiliation:[![[Uncaptioned image]](https://arxiv.org/html/2609.29142v1/figure/hf-logo.png)Hugging Face](https://huggingface.co/SelectiveDOPD)[Github](https://github.com/Luli3220/S2D-OPD)

###### Abstract

Direct On\-Policy Distillation \(Direct\-OPD\) transfers reinforcement\-learning\-induced policy improvements from a small model to a larger student by using the token\-level log\-ratio between post\-RL and pre\-RL checkpoints as dense supervision on the student’s own rollouts\. This transfer rewards the policy shift at every state, yet the log\-ratio measures only relative change: it can stay fixed even as the probability mass that both checkpoints assign to the student’s candidate tokens vanishes\. Through an exact construction, we show that the Direct\-OPD reward and its update can remain unchanged while the Jensen–Shannon divergence \(JSD\) and both KL directions between the checkpoints vanish with this mass, and we note that a small JSD bounds how much the teacher’s behavior changed\. Motivated by this analysis, we propose Selective Supervision for Direct\-OPD \(S2D\-OPD\), which ranks student\-sampled states by their teacher–reference JSD and masks Direct\-OPD supervision at low\-divergence states, retaining only the top 10% of states per response\. Across two teacher pairs and four student models ranging from 1\.7B to 8B parameters, S2D\-OPD improves held\-out accuracy over dense Direct\-OPD on AIME and HMMT benchmarks in seven of eight settings and matches it in the eighth, without extra forward passes\. Our code is available at[https://anonymous\.4open\.science/r/S2D\-OPD\-8868](https://anonymous.4open.science/r/S2D-OPD-8868)\.

> “No Free Lunch for Supervised Machine Learning\.”— David H\. Wolpert

## 1Introduction

Guided by scaling laws\([Kaplan et al\., 2020](https://arxiv.org/html/2609.29142#bib.bib7);[Hoffmann et al\., 2022](https://arxiv.org/html/2609.29142#bib.bib8);[Pearce and Song, 2024](https://arxiv.org/html/2609.29142#bib.bib9)\), recent large language models have continued to scale up pre\-training\([DeepSeek\-AI et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib5);[Team et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib4);[GLM\-5\-Team et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib6)\), which broadens their knowledge and latent capabilities\. Post\-training, most prominently reinforcement learning \(RL\)\([Yue et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib10);[Shao et al\., 2024](https://arxiv.org/html/2609.29142#bib.bib11);[Zhao et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib12)\), is then needed to elicit and refine these capabilities\. As model size grows, RL demands more rollout generation, training compute, memory, and infrastructure\([Wu et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib13)\), and stable optimization remains difficult to achieve\([Wang et al\., 2026a](https://arxiv.org/html/2609.29142#bib.bib14)\)\. Consequently, the models with the greatest post\-training potential are also the most expensive to improve through RL\.

One line of work makes large\-scale RL more stable through improved training algorithms\([Yu et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib15);[Zheng et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib16);[MiniMax et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib17);[Ma et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib1);[Hou et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib3)\)and more efficient through training and inference systems\([Sheng et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib18);[Fu et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib19);[Kwon et al\., 2023](https://arxiv.org/html/2609.29142#bib.bib20);[Narayanan et al\., 2021](https://arxiv.org/html/2609.29142#bib.bib21);[Zhu et al\., 2025](https://arxiv.org/html/2609.29142#bib.bib2)\)\. These advances make RL on large models more practical, but its cost still grows with model size\. Another line of work uses on\-policy distillation \(OPD\)\([Agarwal et al\., 2024](https://arxiv.org/html/2609.29142#bib.bib30);[Gu et al\., 2024](https://arxiv.org/html/2609.29142#bib.bib31);[Lu and Lab, 2025](https://arxiv.org/html/2609.29142#bib.bib29)\), in which a stronger teacher provides token\-level supervision on states sampled by the student\. OPD transfers capabilities efficiently to a smaller student\([Li et al\., 2026b](https://arxiv.org/html/2609.29142#bib.bib22);[Fu et al\., 2026c](https://arxiv.org/html/2609.29142#bib.bib44)\), but it relies on a teacher stronger than the student, which is unavailable when the target is already the strongest model\. Together, these approaches leave a challenge open: how to improve a large model without paying the cost of RL at its scale\.

Direct\-OPD\([Feng et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib23)\)and Proxy\-OPD\([Fu et al\., 2026a](https://arxiv.org/html/2609.29142#bib.bib24)\)recently proposed a weak\-to\-strong route around this challenge: running RL on a small model and transferring the result to a larger one\. Both methods extract the RL\-induced policy shift as the token\-level log\-ratio between the post\-RL teacher and its pre\-RL reference, and use it as an OPD reward on states sampled by the larger student\. This turns outcome\-level rewards into dense token\-level supervision for the large model, without running RL at its scale or requiring a teacher stronger than the student\.

However, this apparent free lunch leaves unexamined*whether every token\-level reward reflects a meaningful change in the teacher’s behavior*\. Because the log\-ratio reward measures only relative change, it can stay fixed even as the probability mass that both checkpoints assign to the student’s top candidates vanishes\. Direct\-OPD can therefore reward a state where both checkpoints barely support these candidates as strongly as one where RL clearly changed the teacher’s behavior\. This raises the question that motivates this work:*is every state’s policy shift worth distilling for free?*

We make this probability\-mass mismatch exact in Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1): as the mass that both checkpoints assign to the student’s top candidates vanishes, the Direct\-OPD reward and gradient can stay fixed, whereas the Jensen–Shannon divergence \(JSD\) and both directions of KL divergence between the checkpoints vanish with it\. Because JSD accounts for the probability mass that the log\-ratio ignores, we proposeSelectiveSupervision forDirect\-OPD\(S2D\-OPD\), which ranks student\-sampled states by teacher–reference JSD and masks Direct\-OPD supervision at low\-divergence states\. Empirically, the answer to our question is no: retaining only the top 10% of states per response, S2D\-OPD improves held\-out accuracy over dense Direct\-OPD in seven of eight teacher–student settings and matches it in the eighth \(mean gain 0\.95 points; 95% CI 0\.40–1\.54\), without extra forward passes\.

In summary, our contributions are threefold:

- •A Probability\-Mass Mismatch in Direct\-OPD\.Through an exact construction, we show that the log\-ratio reward and its local gradient on the student can remain fixed while the probability mass behind the policy shift vanishes, and with it the teacher–reference JSD and both KL directions\. We further show that JSD bounds how much the teacher’s behavior can change at every state, which motivates selecting states by divergence rather than by the reward itself\.
- •Stable and Effective Selective Transfer\.We propose S2D\-OPD, which masks Direct\-OPD supervision at low\-divergence states during policy transfer\. Across four student scales and two teacher pairs, it improves held\-out accuracy over Direct\-OPD in seven of eight settings and yields smoother late\-stage validation curves under the JustRL teacher pair\.
- •Understanding the Gains from Selective Transfer\.Within a fixed teacher–student setting, performance broadly rises with JSD percentile: the top bin outperforms a uniformly sampled 10% subset, whereas the lowest bin degrades the student below its initialization and eventually collapses\. Across teacher pairs, the pair with lower overall JSD benefits more from masking, consistent with low\-divergence filtering being a source of the improvement over dense Direct\-OPD\.

## 2Related Work

On\-Policy Distillation\.OPD trains a student on prefixes sampled from its own policy, using the teacher’s next\-token distributions as dense supervision at every position\([Agarwal et al\., 2024](https://arxiv.org/html/2609.29142#bib.bib30);[Gu et al\., 2024](https://arxiv.org/html/2609.29142#bib.bib31);[Lu and Lab, 2025](https://arxiv.org/html/2609.29142#bib.bib29)\)\. Subsequent work refines it through alternative objectives\([Jin et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib34);[Jia et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib38)\), stabilization strategies\([Li et al\., 2026b](https://arxiv.org/html/2609.29142#bib.bib22);[Fu et al\., 2026b](https://arxiv.org/html/2609.29142#bib.bib37)\), and privileged\-context self\-distillation\([Zhao et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib36);[Pan et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib39)\)\. Despite their differences, these methods primarily learn from the teacher’s policy itself, limiting transfer to improvements already present in the teacher; recent work therefore targets the teacher’s policy shift instead\. ExOPD\([Yang et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib25)\)extrapolates the teacher’s improvement over its reference model to construct a target beyond the teacher\. Direct\-OPD\([Feng et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib23)\)and Proxy\-OPD\([Fu et al\., 2026a](https://arxiv.org/html/2609.29142#bib.bib24)\)instead transfer the log\-ratio between a reward\-optimized checkpoint and its pre\-RL reference, analogous to the logit shifts induced by fine\-tuning studied in CMC\([Wu et al\., 2024](https://arxiv.org/html/2609.29142#bib.bib45)\)\. This targets the RL\-induced policy shift and can provide useful supervision even for students already stronger than the post\-RL teacher\. However, token\-level log\-ratios capture relative changes but are insensitive to the absolute probability mass supporting these changes\. We examine this probability\-mass mismatch in Direct\-OPD and use teacher–reference divergence to select supervision positions\.

Token Selection in Policy Distillation\.Selective distillation asks which positions are worth training on, and existing criteria differ mainly in which distributions they read\. Some read the student alone, prioritizing positions where it is uncertain\([Tavor et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib33);[Ko et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib32)\); others the teacher alone, weighting by its confidence or local margin\([Jin et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib34);[Zhou et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib35)\); a third group compares the two, emphasizing teacher–student disagreement, which TIP\([Xu et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib28)\)organizes through an entropy–divergence taxonomy and TA\-OPD\([Wang et al\., 2026b](https://arxiv.org/html/2609.29142#bib.bib26)\)restricts to the student’s predictive support\. We consider the policy change from a pre\-RL reference to a post\-RL teacher at student\-visited prefixes\. A related approach, OPD2\([Heo et al\., 2026](https://arxiv.org/html/2609.29142#bib.bib27)\), gates sampled\-token delta updates by sign agreement between the centered teacher–base and teacher–student log\-ratios\. Our selection criterion instead ranks positions by teacher–reference JSD, accounting for the probability mass underlying the policy shift while retaining the Direct\-OPD update at selected positions\.

## 3Preliminaries

Setting\.We consider three policies: a pre\-RL referenceπref\\pi\_\{\\mathrm\{ref\}\}, a post\-RL teacherπT\\pi\_\{\\mathrm\{T\}\}obtained fromπref\\pi\_\{\\mathrm\{ref\}\}by outcome\-based RL such as GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.29142#bib.bib11)\), and a larger studentπθ\\pi\_\{\\theta\}initialized atπstu\\pi\_\{\\mathrm\{stu\}\}\. In our experiments, both teacher\-side checkpoints are publicly released \(Sec\.[5\.1](https://arxiv.org/html/2609.29142#S5.SS1)\), so we run no RL ourselves\. Given a prompt𝒙∼𝒟\{\\bm\{x\}\}\\sim\\mathcal\{D\}and a response𝒚=\(y1,…,y\|𝒚\|\)\{\\bm\{y\}\}=\(y\_\{1\},\\ldots,y\_\{\|\{\\bm\{y\}\}\|\}\)sampled from the student, positiontthas state𝒔t=\(𝒙,𝒚<t\)\{\\bm\{s\}\}\_\{t\}=\(\{\\bm\{x\}\},\{\\bm\{y\}\}\_\{<t\}\)\.

Direct\-OPDtreats the teacher’s RL\-induced policy shift as a dense reward for the student\. For any tokenvv, the reward at state𝒔t\{\\bm\{s\}\}\_\{t\}is the teacher–reference log\-ratio

Δt​\(v∣𝒔t\)=log⁡πT​\(v∣𝒔t\)πref​\(v∣𝒔t\),\\Delta\_\{t\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)=\\log\\frac\{\\pi\_\{\\mathrm\{T\}\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)\},\(1\)which is positive where RL increased the probability ofvvand negative where it decreased it\.

Direct\-OPD maximizes this reward on states visited by the student, with KL regularization:

JDirect\-OPD\(θ\)=𝔼𝒙∼𝒟,\(𝒔t,yt\)∼πθ\[Δt\(yt∣𝒔t\)−αDKL\[πθ\(⋅∣𝒔t\)∥πstu\(⋅∣𝒔t\)\]\]\.J\_\{\\text\{Direct\-OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{\{\\bm\{x\}\}\\sim\\mathcal\{D\},\\,\(\{\\bm\{s\}\}\_\{t\},y\_\{t\}\)\\sim\\pi\_\{\\theta\}\}\\left\[\\Delta\_\{t\}\(y\_\{t\}\\mid\{\\bm\{s\}\}\_\{t\}\)\-\\alpha D\_\{\\mathrm\{KL\}\}\\left\[\\pi\_\{\\theta\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\\|\\pi\_\{\\mathrm\{stu\}\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\right\]\\right\]\.\(2\)Here,α\>0\\alpha\>0controls KL regularization, and the expectation covers all valid response positions\.

Top\-KKimplementation\.In practice, Direct\-OPD evaluates the reward on the student’s top\-KKcandidates at each state rather than only on the sampled token:

𝒱K\(𝒔t\)=TopK\(πθ\(⋅∣𝒔t\),K\),p¯t\(v\)=πθ​\(v∣𝒔t\)∑u∈𝒱K​\(𝒔t\)πθ​\(u∣𝒔t\),\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\_\{t\}\)=\\operatorname\{TopK\}\(\\pi\_\{\\theta\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\),K\),\\quad\\bar\{p\}\_\{t\}\(v\)=\\frac\{\\pi\_\{\\theta\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)\}\{\\sum\_\{u\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\_\{t\}\)\}\\pi\_\{\\theta\}\(u\\mid\{\\bm\{s\}\}\_\{t\}\)\},\(3\)whereTopK⁡\(⋅,K\)\\operatorname\{TopK\}\(\\cdot,K\)returns the set ofKKtokens with the largest probabilities\. Each candidate receives the teacher–reference rewardΔt​\(v∣𝒔t\)\\Delta\_\{t\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\), weighted by its renormalized student probabilityp¯t​\(v\)\\bar\{p\}\_\{t\}\(v\)\. With the state and candidate set held fixed, the implemented local reward\-gradient contribution is:

gtR=∑v∈𝒱K​\(𝒔t\)sg⁡\[p¯t​\(v\)​log⁡πT​\(v∣𝒔t\)πref​\(v∣𝒔t\)\]​∇θ​log​πθ​\(v∣𝒔t\),g\_\{t\}^\{R\}=\\sum\_\{v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\_\{t\}\)\}\\operatorname\{sg\}\\left\[\\bar\{p\}\_\{t\}\(v\)\\log\\frac\{\\pi\_\{\\mathrm\{T\}\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)\}\\right\]\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\),\(4\)wheresg\\operatorname\{sg\}denotes stop\-gradient\. The student\-anchor KL term supplies a separate regularization gradient\. Appendix[A](https://arxiv.org/html/2609.29142#A1)describes its implementation and adaptive coefficient\.

## 4Method

Direct\-OPD applies its log\-ratio reward at every valid position of a student response\. Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1)shows that this reward can ignore the probability mass behind the teacher’s policy shift, and Sec\.[4\.2](https://arxiv.org/html/2609.29142#S4.SS2)introduces S2D\-OPD, which selects states by teacher–reference divergence\.

### 4\.1Theoretical Motivation: Probability\-Mass Mismatch

At each state, Direct\-OPD weights the teacher–reference rewardsΔt​\(v∣𝒔t\)\\Delta\_\{t\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)by the student’s renormalized probabilitiesp¯t​\(v\)\\bar\{p\}\_\{t\}\(v\)over its top\-KKcandidates in Eq\.[4](https://arxiv.org/html/2609.29142#S3.E4)\. These weights reflect the student’s preferences, whereas the rewards encode relative changes in the teacher’s policy\. Neither depends on how much probability mass the teacher and reference place on these candidates: rescaling both by a common factor leaves every log\-ratio unchanged\. The following exact construction makes this precise: the teacher–reference divergence can vanish while the Direct\-OPD update stays fixed\.

#### An exact construction\.

Fix a state𝒔\{\\bm\{s\}\}and a student checkpointθ0\\theta\_\{0\}with top\-KKcandidate set𝒱K=𝒱K​\(𝒔\)\\mathcal\{V\}\_\{K\}=\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\. Let𝐭≠𝐪\\mathbf\{t\}\\neq\\mathbf\{q\}be strictly positive probability vectors on𝒱K\\mathcal\{V\}\_\{K\}, let𝐛\\mathbf\{b\}be a strictly positive probability vector on its complement, and for0<ϵ<10<\\epsilon<1define the teacher and reference:

Pϵ​\(v\)=\{ϵ​tv,v∈𝒱K,\(1−ϵ\)​bv,v∉𝒱K,Qϵ​\(v\)=\{ϵ​qv,v∈𝒱K,\(1−ϵ\)​bv,v∉𝒱K\.P\_\{\\epsilon\}\(v\)=\\begin\{cases\}\\epsilon t\_\{v\},&v\\in\\mathcal\{V\}\_\{K\},\\\\ \(1\-\\epsilon\)b\_\{v\},&v\\notin\\mathcal\{V\}\_\{K\},\\end\{cases\}\\qquad Q\_\{\\epsilon\}\(v\)=\\begin\{cases\}\\epsilon q\_\{v\},&v\\in\\mathcal\{V\}\_\{K\},\\\\ \(1\-\\epsilon\)b\_\{v\},&v\\notin\\mathcal\{V\}\_\{K\}\.\\end\{cases\}\(5\)Here,ϵ\\epsilonis the probability mass that each checkpoint assigns to the student’s candidates, while the student, candidate set, and conditional distributions stay fixed\. Then, for every candidatev∈𝒱Kv\\in\\mathcal\{V\}\_\{K\},

Δϵ​\(v∣𝒔\)=log⁡tvqv,DJS​\(Pϵ,Qϵ\)=ϵ​DJS​\(𝐭,𝐪\),\\Delta\_\{\\epsilon\}\(v\\mid\{\\bm\{s\}\}\)=\\log\\frac\{t\_\{v\}\}\{q\_\{v\}\},\\qquad D\_\{\\mathrm\{JS\}\}\(P\_\{\\epsilon\},Q\_\{\\epsilon\}\)=\\epsilon\\,D\_\{\\mathrm\{JS\}\}\(\\mathbf\{t\},\\mathbf\{q\}\),\(6\)and both directions of KL scale withϵ\\epsilonin the same way\. Asϵ→0\\epsilon\\to 0, all three divergences vanish, whereas every candidate reward, and hence the Direct\-OPD update of Eq\.[4](https://arxiv.org/html/2609.29142#S3.E4), stays fixed and nonzero\. The construction thus isolates a single degree of freedom, the probability mass behind the policy shift, to which the Direct\-OPD update is insensitive but the divergences are not\. We state this result formally in Prop\.[1](https://arxiv.org/html/2609.29142#Thmproposition1)and prove it in App\.[B](https://arxiv.org/html/2609.29142#A2), including why the update is nonzero\.

#### Divergence bounds the behavioral change\.

The construction shows that the Direct\-OPD reward can ignore divergence; conversely, divergence bounds how much the teacher’s behavior can change\. For any two distributionsPPandQQon a finite set and any eventAA,

\|P⁡\(A\)−Q⁡\(A\)\|≤DTV​\(P,Q\)≤2​DJS​\(P,Q\),\|P\(A\)\-Q\(A\)\|\\;\\leq\\;D\_\{\\mathrm\{TV\}\}\(P,Q\)\\;\\leq\\;\\sqrt\{2\\,D\_\{\\mathrm\{JS\}\}\(P,Q\)\},\(7\)whereDTV​\(P,Q\)=maxA⁡\|P⁡\(A\)−Q⁡\(A\)\|D\_\{\\mathrm\{TV\}\}\(P,Q\)=\\max\_\{A\}\|P\(A\)\-Q\(A\)\|is the total variation distance andDJSD\_\{\\mathrm\{JS\}\}is measured in nats\. The second inequality follows from Pinsker’s inequality applied to each term ofDJS\(P,Q\)=12DKL\(P∥M\)\+12DKL\(Q∥M\)D\_\{\\mathrm\{JS\}\}\(P,Q\)=\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(P\\\|M\)\+\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(Q\\\|M\)withM=\(P\+Q\)/2M=\(P\+Q\)/2, sinceDTV​\(P,M\)=DTV​\(Q,M\)=12​DTV​\(P,Q\)D\_\{\\mathrm\{TV\}\}\(P,M\)=D\_\{\\mathrm\{TV\}\}\(Q,M\)=\\frac\{1\}\{2\}D\_\{\\mathrm\{TV\}\}\(P,Q\)\. Unlike the construction, this bound holds at every state: wherever the teacher–reference JSD is small, the teacher assigns nearly the same probability as the reference to every token and every set of tokens\. Together, the two results characterize what low\-divergence masking removes: states at which the teacher’s behavior provably changed little, yet at which the Direct\-OPD update can be as large as anywhere else\. They do not show that removing these states improves transfer, which Sec\.[5\.3](https://arxiv.org/html/2609.29142#S5.SS3)tests by training on JSD percentile bins\.

### 4\.2Divergence\-Guided State Selection

Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1)shows that the Direct\-OPD reward is insensitive to the probability mass behind a policy shift, whereas the teacher–reference JSD bounds how much the teacher’s behavior changed at a state\. S2D\-OPD therefore scores each student\-sampled state by this divergence and retains Direct\-OPD supervision only at the highest\-scoring states within each response \(Fig\.[1](https://arxiv.org/html/2609.29142#S4.F1)\)\. We use JSD as the default score because, unlike KL, it is symmetric, bounded bylog⁡2\\log 2, and finite when either checkpoint assigns a token zero probability; the procedure applies unchanged to other divergences\.

![Refer to caption](https://arxiv.org/html/2609.29142v1/method.png)Figure 1:Overview of S2D\-OPD\. At each student\-sampled state, teacher–reference JSD is computed over the student’s top\-KKcandidates, with the remaining probability mass grouped into a residual tokenvov\_\{\\mathrm\{o\}\}\. The highest\-JSD states within each response are retained for the Direct\-OPD objective\.#### Scoring on the student’s candidates\.

Rather than the full vocabulary, we evaluate the divergence on the top\-KKcandidate set𝒱K​\(𝒔t\)\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\_\{t\}\), where Direct\-OPD already evaluates both checkpoints, and add a residual tokenvov\_\{\\mathrm\{o\}\}that collects all remaining probability mass\. ForM∈\{T,ref\}M\\in\\\{\\mathrm\{T\},\\mathrm\{ref\}\\\}, define

π~M​\(v∣𝒔t\)=\{πM​\(v∣𝒔t\),v∈𝒱K​\(𝒔t\),1−∑u∈𝒱K​\(𝒔t\)πM​\(u∣𝒔t\),v=vo\.\\widetilde\{\\pi\}\_\{M\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)=\\begin\{cases\}\\pi\_\{M\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\),&v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\_\{t\}\),\\\\\[3\.0pt\] 1\-\\displaystyle\\sum\_\{u\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\_\{t\}\)\}\\pi\_\{M\}\(u\\mid\{\\bm\{s\}\}\_\{t\}\),&v=v\_\{\\mathrm\{o\}\}\.\\end\{cases\}\(8\)This keeps each candidate’s probability and the total residual mass without renormalization\. By the data\-processing inequality, the coarsening cannot increase JSD:

d~t≜DJS\(π~T\(⋅∣𝒔t\)∥π~ref\(⋅∣𝒔t\)\)≤DJS\(πT\(⋅∣𝒔t\)∥πref\(⋅∣𝒔t\)\)\.\\tilde\{d\}\_\{t\}\\triangleq D\_\{\\mathrm\{JS\}\}\\\!\\left\(\\widetilde\{\\pi\}\_\{\\mathrm\{T\}\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\,\\middle\\\|\\,\\widetilde\{\\pi\}\_\{\{\\mathrm\{ref\}\}\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\right\)\\leq D\_\{\\mathrm\{JS\}\}\\\!\\left\(\{\\pi\}\_\{\\mathrm\{T\}\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\,\\middle\\\|\\,\{\\pi\}\_\{\{\\mathrm\{ref\}\}\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\right\)\.\(9\)Fig\.[1](https://arxiv.org/html/2609.29142#S4.F1)illustrates two ways in which the score can be small\. Either both checkpoints place little mass on the student’s candidates, so that both compressed distributions concentrate onvov\_\{\\mathrm\{o\}\}, which is the regime of the construction in Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1); or both place substantial mass on the candidates and agree on how it is distributed\. The score treats these cases alike, and it need not separate them: because Eq\.[7](https://arxiv.org/html/2609.29142#S4.E7)holds for any pair of distributions, including the compressed ones, a small score implies in either case that the teacher barely changed its behavior on the student’s candidates\. Conversely, the score is large only when the checkpoints disagree on the individual candidate probabilities or on the total mass assigned to the candidate set\. The score thus measures behavioral change at the resolution of the student’s candidates: differences among tail tokens outside the candidate set do not affect it\.

#### Per\-response selection\.

We select states within each response, so that rollouts with different overall divergence levels all contribute supervision\. For valid response positions𝒯\\mathcal\{T\}and a common retention ratioρ∈\(0,1\]\\rho\\in\(0,1\], we retain thekkstates with the largest scores:

k=max⁡\{1,⌈ρ​\|𝒯\|⌉\},ℐ=TopKt∈𝒯⁡\(d~t,k\),k=\\max\\left\\\{1,\\,\\left\\lceil\\rho\|\\mathcal\{T\}\|\\right\\rceil\\right\\\},\\qquad\\mathcal\{I\}=\\operatorname\{TopK\}\_\{t\\in\\mathcal\{T\}\}\(\\tilde\{d\}\_\{t\},k\),\(10\)where retaining at least one state ensures that every rollout is represented in the objective\. Selecting within each response also makes each mask independent of the rest of the batch\.

#### Masked objective\.

We apply the maskmt=𝟏\{t∈ℐ\}m\_\{t\}=\\mathbf\{1\}\\\{t\\in\\mathcal\{I\}\\\}to both terms of the Direct\-OPD objective:

JS2​D​\-​OPD\(θ\)=𝔼𝒙∼𝒟,\(𝒔t,yt\)∼πθ\[Δt\(yt∣𝒔t\)−αDKL\[πθ\(⋅∣𝒔t\)∥πstu\(⋅∣𝒔t\)\]\|mt=1\]\.J\_\{\\mathrm\{S^\{2\}D\\text\{\-\}OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{\{\\bm\{x\}\}\\sim\\mathcal\{D\},\\,\(\{\\bm\{s\}\}\_\{t\},y\_\{t\}\)\\sim\\pi\_\{\\theta\}\}\\left\[\\Delta\_\{t\}\(y\_\{t\}\\mid\{\\bm\{s\}\}\_\{t\}\)\-\\alpha D\_\{\\mathrm\{KL\}\}\\\!\\left\[\\pi\_\{\\theta\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\\|\\pi\_\{\\mathrm\{stu\}\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\right\]\\,\\middle\|\\,m\_\{t\}=1\\right\]\.\(11\)In practice, we hold the selection fixed during optimization and average over all retained states in the batch, which gives the update

𝐠S2​D​\-​OPD=1\|ℐℬ\|∑t∈ℐℬ\[gtR−α∇θDKL\[πθ\(⋅∣𝒔t\)∥πstu\(⋅∣𝒔t\)\]\],\\mathbf\{g\}^\{\\mathrm\{S^\{2\}D\\text\{\-\}OPD\}\}=\\frac\{1\}\{\|\\mathcal\{I\}\_\{\\mathcal\{B\}\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\_\{\\mathcal\{B\}\}\}\\Big\[g\_\{t\}^\{R\}\-\\alpha\\,\\nabla\_\{\\theta\}D\_\{\\mathrm\{KL\}\}\\\!\\left\[\\pi\_\{\\theta\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\\|\\pi\_\{\\mathrm\{stu\}\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\)\\right\]\\Big\],\(12\)whereℐℬ\\mathcal\{I\}\_\{\\mathcal\{B\}\}collects the retained states of all responses in the batch andgtRg\_\{t\}^\{R\}is the reward gradient of Eq\.[4](https://arxiv.org/html/2609.29142#S3.E4)\. Masked states thus receive neither the reward nor the student anchor, andρ=1\\rho=1recovers Direct\-OPD\. The adaptive KL coefficient is still computed from all valid positions; App\.[A](https://arxiv.org/html/2609.29142#A1)gives the remaining optimization details\. Because the score reuses the teacher and reference probabilities that Direct\-OPD already computes, S2D\-OPD requires no extra forward passes\.

Table 1:Main results \(Avg@32\) across four student models and two teacher–reference pairs, comparing dense Direct\-OPD with S2D\-OPD using top\-10% JSD selection\. Checkpoints are selected on AIME24/25 and evaluated on held\-out AIME26 and HMMT \(Nov\. 2025 and Feb\. 2026\)\.Test Avg\.is the average accuracy over 93 held\-out problems; parentheses denote gains over the initial student\.R1\-Distill\-1\.5B→\\rightarrowJustRL\-1\.5BNemotron\-1\.5B→\\rightarrowQuestA\-1\.5BCheckpoint SelectionHeld\-out EvaluationCheckpoint SelectionHeld\-out EvaluationStudent / MethodAIME24AIME25AIME26HMMTTest Avg\.AIME24AIME25AIME26HMMTTest Avg\.Qwen3\-1\.7B49\.336\.737\.728\.231\.349\.336\.737\.728\.231\.3\+ Direct\-OPD59\.742\.945\.432\.036\.4\(\+5\.1\)58\.643\.446\.831\.636\.5\(\+5\.2\)\+ S2D\-OPD61\.444\.547\.432\.937\.6\(\+6\.3\)59\.543\.848\.133\.338\.1\(\+6\.8\)Qwen3\-4B73\.265\.264\.845\.952\.073\.265\.264\.845\.952\.0\+ Direct\-OPD78\.170\.366\.847\.153\.5\(\+1\.5\)77\.670\.470\.447\.154\.6\(\+2\.6\)\+ S2D\-OPD77\.070\.968\.746\.553\.7\(\+1\.7\)78\.068\.370\.749\.356\.2\(\+4\.2\)Qwen3\-8B77\.366\.067\.549\.755\.477\.366\.067\.549\.755\.4\+ Direct\-OPD77\.572\.269\.850\.256\.5\(\+1\.1\)75\.971\.769\.748\.655\.4\(\+0\.0\)\+ S2D\-OPD78\.073\.470\.750\.056\.7\(\+1\.3\)78\.171\.770\.950\.657\.1\(\+1\.7\)R1\-Distill\-7B56\.740\.548\.229\.535\.656\.740\.548\.229\.535\.6\+ Direct\-OPD63\.645\.756\.832\.840\.5\(\+4\.9\)60\.742\.349\.531\.937\.6\(\+2\.0\)\+ S2D\-OPD64\.647\.155\.033\.640\.5\(\+4\.9\)60\.642\.352\.432\.038\.6\(\+3\.0\)

## 5Experiments

Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1)shows which states low\-divergence masking removes, but not whether removing them improves transfer\. We test this through three research questions \(further analyses in App\.[C](https://arxiv.org/html/2609.29142#A3),[D](https://arxiv.org/html/2609.29142#A4),[E](https://arxiv.org/html/2609.29142#A5)\):

- RQ1:Does masking low\-JSD states improve transfer? \(Sec\.[5\.2](https://arxiv.org/html/2609.29142#S5.SS2)\)
- RQ2:How does JSD relate to supervision utility and selective masking gains? \(Sec\.[5\.3](https://arxiv.org/html/2609.29142#S5.SS3)\)
- RQ3:Are the gains robust across divergence measures and selection schemes? \(Sec\.[5\.4](https://arxiv.org/html/2609.29142#S5.SS4)\)

### 5\.1Experimental Setup

We evaluate S2D\-OPD with two teacher pairs, R1\-Distill\-1\.5B→\\rightarrowJustRL\-1\.5B\([He et al\., 2025a](https://arxiv.org/html/2609.29142#bib.bib42)\)and Nemotron\-1\.5B→\\rightarrowQuestA\-1\.5B\([Li et al\., 2026a](https://arxiv.org/html/2609.29142#bib.bib43)\), and transfer each policy shift to four students: Qwen3\-1\.7B, Qwen3\-4B, Qwen3\-8B111https://huggingface\.co/Qwen/Qwen3\-\{1\.7,4,8\}B, and R1\-Distill\-7B222https://huggingface\.co/deepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-7B\. All students are trained on Skywork\-OR1\-RL\-Data\([He et al\., 2025b](https://arxiv.org/html/2609.29142#bib.bib41)\)\. Following the observation of[Meng et al\. \(2026\)](https://arxiv.org/html/2609.29142#bib.bib40)that RL\-induced policy shifts are sparse, with large divergence concentrated in a small fraction of tokens, we retain the top 10% of states per response \(ρ=0\.1\\rho=0\.1\) by default\. We select checkpoints on AIME 2024 and AIME 2025 and evaluate the selected checkpoint on three held\-out benchmarks: AIME 2026, HMMT November 2025, and HMMT February 2026\. All three postdate the release of every student model and are therefore absent from its training data, including undisclosed post\-training data\. We report Avg@32 on all benchmarks, define validation accuracy as the mean over AIME 2024 and AIME 2025, and give further training hyperparameter details in App\.[F](https://arxiv.org/html/2609.29142#A6)\.

### 5\.2Masking Low\-Divergence States Improves Transfer

Retaining 10% of states improves held\-out transfer\.Tab\.[1](https://arxiv.org/html/2609.29142#S4.T1)compares S2D\-OPD with dense Direct\-OPD across four students and two teacher pairs\. Retaining 10% of states achieves comparable validation accuracy and improves the held\-out Test Avg\. in seven of eight settings, with a tie in the eighth\. We assess statistical significance over 93 held\-out problems, first averaging each problem’s 32 responses and then averaging its paired differences across the eight settings\. Using a paired problem bootstrap and an exact one\-sided sign\-flip test, we find that S2D\-OPD improves mean held\-out accuracy from 46\.36% to 47\.31%, a gain of 0\.95 points \(95% CI: 0\.40–1\.54;p=6\.4×10−4p=6\.4\\times 10^\{\-4\}\)\.

Figure 2:Validation accuracy \(Avg@32\) under the JustRL teacher pair across four students\. Dashed lines mark the initial student; dotted lines mark the JustRL\-1\.5B teacher\.\(a\)Performance across JSD percentile bins\.\(b\)Mean JSD tracks transfer gains\.
Figure 3:Which states are retained, not how many, determines transfer \(Qwen3\-1\.7B, JustRL teacher pair\)\.\(a\)Validation accuracy when training on each JSD percentile bin or on a random 10% of states\.\(b\)Gain in peak validation accuracy over the initial student versus the mean training JSD of the retained states; the line is a log\-linear fit\.Selective supervision preserves early learning while improving late\-stage stability\.Fig\.[2](https://arxiv.org/html/2609.29142#S5.F2)shows the validation trajectories under the JustRL teacher pair\. Despite masking low\-divergence states, S2D\-OPD keeps pace with dense Direct\-OPD during early training, matching or exceeding its accuracy over the first 60 steps for all four students\. Discarding 90% of states thus does not slow early learning, suggesting that the supervision driving early improvement is concentrated in the retained high\-divergence subset\. Later in training, S2D\-OPD maintains higher accuracy for all four students and is more stable on Qwen3\-4B and Qwen3\-8B, where dense Direct\-OPD repeatedly falls back from its peaks and, on Qwen3\-4B, even drops below the initial student by step 300\. These results answer RQ1: masking low\-JSD states improves held\-out transfer without slowing early learning\.

### 5\.3Low\-Divergence Supervision Is Redundant and Harmful

Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1)shows that low\-JSD states are those at which the teacher’s behavior changed little, yet the Direct\-OPD update there can be as large as anywhere else\. To test what such supervision contributes, we fix the JustRL teacher pair, the Qwen3\-1\.7B student, the training data, and the hyperparameters, and vary only which states enter the Direct\-OPD objective\. Within each response, we rank states by JSD, split them into ten equal\-sized percentile bins \(0–10, …, 90–100\), and train a separate student on each bin\. As a control, we train on a uniformly sampled 10% of states, whose update is an unbiased estimate of the dense Direct\-OPD update \(Fig\.[3\(a\)](https://arxiv.org/html/2609.29142#S5.F3.sf1)\)\.

Dense supervision is largely redundant\.The random 10% control reaches a peak validation accuracy of 51\.2, on par with dense Direct\-OPD \(51\.3; Tab\.[1](https://arxiv.org/html/2609.29142#S4.T1)\)\. Discarding 90% of states at random thus loses little, supporting the view of[Fu et al\. \(2026c\)](https://arxiv.org/html/2609.29142#bib.bib44)that OPD is “data\-overfed but algorithm\-starved\.”

Which states are retained determines transfer\.At the same 10% budget, performance rises broadly with JSD percentile, and the top bin reaches 52\.9, above both the random control and dense Direct\-OPD\. The effect is graded: across all eleven subsets, the gain in peak validation accuracy over the initial student is approximately linear in the logarithm of the subset’s mean training JSD \(R2=0\.97R^\{2\}=0\.97; Fig\.[3\(b\)](https://arxiv.org/html/2609.29142#S5.F3.sf2)\)\. The random control lies on the same line, so within this setting the mean divergence of the retained states predicts transfer whether they are selected by rank or at random\.

Low\-divergence supervision is harmful, not merely weak\.Trained in isolation, the three lowest bins end below the initial student, and the two lowest collapse\. This matches the regime identified in Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1), where the Direct\-OPD update persists although the teacher barely changed\. Top\-KKoverlap suggests a mechanism \(App\.[C](https://arxiv.org/html/2609.29142#A3)\): the top bin moves the student toward the teacher while preserving its overlap with the reference, whereas lower bins move it away from the reference without a commensurate gain in teacher alignment\.

Divergence and reward magnitude rank states differently\.Fig\.[5](https://arxiv.org/html/2609.29142#S5.F5)shows a state from an initial Qwen3\-1\.7B rollout under the JustRL pair\. RL lowers the probability of the sampled token5from 0\.12 to2×10−52\\times 10^\{\-5\}, giving this state the largest Direct\-OPD reward magnitude among the response’s 8,192 states, although the teacher’s distribution moves by only 0\.12 in total variation\. Its JSD \(0\.044\) falls below the retention cutoff \(0\.067\), so S2D\-OPD masks it while retaining a state with a smaller log\-ratio but a larger redistribution of mass \(JSD 0\.262\)\.

\(a\)All\-position and selected\-position JSD\.\(b\)Held\-out gains over Direct\-OPD\.
Figure 4:Teacher\-pair comparison with top\-10% retention across four students\.\(a\)Mean teacher–reference JSD over all valid response states \(solid\) and retained states \(dashed\) during S2D\-OPD training\.\(b\)Held\-out Test Avg\. gains of S2D\-OPD over dense Direct\-OPD under each teacher pair, in percentage points\.![Refer to caption](https://arxiv.org/html/2609.29142v1/case06.png)Figure 5:JSD and token log\-ratio rank states differently\.The state with the largest sampled\-token log\-ratio in the response is masked, while a larger redistribution of probability mass is retained\.Masking helps more under the lower\-divergence teacher pair\.Mean JSD is higher for JustRL than for QuestA over both all states and the retained top 10% \(Fig\.[4\(a\)](https://arxiv.org/html/2609.29142#S5.F4.sf1)\), yet the held\-out gains of masking over dense Direct\-OPD are larger under QuestA: 1\.0–1\.7 points versus 0\.0–1\.2 for JustRL across the four students \(Fig\.[4\(b\)](https://arxiv.org/html/2609.29142#S5.F4.sf2)\)\. If the benefit came from concentrating supervision on high absolute divergence, this ordering would be reversed; together with the harm of low\-JSD bins, it suggests that masking helps mainly by removing supervision that undermines learning\. These results answer RQ2: JSD ranks supervision utility within a teacher–student setting, and masking pays off by filtering low\-divergence supervision rather than by maximizing the divergence of what remains\.

### 5\.4Selective Masking Is Robust Across Design Choices

We vary three design choices while keeping the rest of S2D\-OPD fixed: the divergence used for scoring, the scope over which states are ranked, and the retention rate \(Fig\.[6](https://arxiv.org/html/2609.29142#S5.F6)\)\.

Divergence measure\.The construction in Sec\.[4\.1](https://arxiv.org/html/2609.29142#S4.SS1)shows that both KL directions vanish with the candidate mass just as JSD does, so either should flag the same low\-mass states\. Under the JustRL pair, selection by forward KL,DKL\(πT∥πref\)D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\mathrm\{T\}\}\\\|\\pi\_\{\\mathrm\{ref\}\}\), or reverse KL,DKL\(πref∥πT\)D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\mathrm\{ref\}\}\\\|\\pi\_\{\\mathrm\{T\}\}\), attains higher mean validation accuracy than dense Direct\-OPD from step 160 onward on all three Qwen3 students, and avoids the late decline of dense Direct\-OPD on Qwen3\-4B \(final accuracy 71\.6–74\.1 versus 67\.2\)\. Forward KL closely tracks JSD at all three scales, whereas reverse KL is weaker on Qwen3\-4B\.

Selection scope\.Ranking states across the whole batch, so that a response may retain none, matches response\-level selection on Qwen3\-1\.7B under both teacher pairs \(peak validation accuracy 52\.3 versus 52\.9 under JustRL and 51\.7 versus 51\.7 under QuestA\)\. The per\-response rule of Sec\.[4\.2](https://arxiv.org/html/2609.29142#S4.SS2)is thus not needed for accuracy; we keep it so that each mask is independent of other responses\.

Retention rate\.Varyingρ\\rhofrom 5% to 20% on Qwen3\-1\.7B changes peak validation accuracy by less than one point \(52\.2–53\.0\), and every rate stays above dense Direct\-OPD \(51\.3\) and the random 10% control \(51\.2; App\.[D](https://arxiv.org/html/2609.29142#A4)\)\. These results answer RQ3: within the tested range, the gains do not hinge on the divergence measure, selection scope, or retention rate\.

\(a\)Divergence measure\.\(b\)Selection scope\.
Figure 6:Robustness of selective masking across design choices\.\(a\)Top\-10% selection by JSD, reverse KL, and forward KL on Qwen3\-1\.7B, 4B, and 8B under the JustRL teacher pair\.\(b\)Response\-level versus batch\-level selection on Qwen3\-1\.7B under both teacher pairs\.

## 6Conclusion

Direct\-OPD transfers the policy shift that RL induces in a small teacher to a larger student by rewarding every student\-sampled state with the teacher–reference log\-ratio\. We showed that this reward ignores the probability mass behind the shift: it can stay fixed while the mass on the student’s candidates, and with it the teacher–reference divergence, vanishes\. Building on this observation, we introduced S2D\-OPD, which retained Direct\-OPD supervision only at the highest\-JSD states of each response and required no extra forward passes\. Retaining 10% of states improved held\-out accuracy over dense Direct\-OPD in seven of eight teacher–student settings and tied in the eighth\. Controlled analyses further showed that dense supervision was largely redundant, that transfer depended on which states were retained rather than how many, and that low\-divergence supervision could harm the student when trained on alone; the gains persisted across divergence measures, selection scopes, and retention rates\. These findings suggest that weak\-to\-strong policy transfer depends not only on what is transferred but also on where it is applied\. Extending state selection to adaptive retention rates and to domains beyond mathematical reasoning is a natural next step\.

### AI Use Statement

We used generative AI tools solely to improve the readability and clarity of the manuscript, including language editing and suggestions on presentation\. All AI\-assisted revisions were reviewed by the authors, who take full responsibility for the final content of this work\.

### Reproducibility Statement

To facilitate reproducibility, we describe the selective supervision procedure in Section[4](https://arxiv.org/html/2609.29142#S4)and report the teacher–reference pairs, datasets, training configurations, and evaluation protocols in Section[5\.1](https://arxiv.org/html/2609.29142#S5.SS1)and Appendix[F](https://arxiv.org/html/2609.29142#A6)\. We provide our code at[https://anonymous\.4open\.science/r/S2D\-OPD\-8868](https://anonymous.4open.science/r/S2D-OPD-8868)\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1),[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- DeepSeek\-AIet al\.\(2026\)DeepSeek\-AI, A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling, C\. Lu, C\. Zhao, C\. Deng, C\. Hou, C\. Xu,et al\.DeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Fenget al\.\(2026\)S\. Feng, H\. Gao, H\. Chi, H\. Wu, Z\. Zhang, Z\. Jiang, B\. He, W\. Ma, Y\. Zhang, and H\. ZhouWeak\-to\-strong generalization via direct on\-policy distillation\.External Links:2607\.05394,[Link](https://arxiv.org/abs/2607.05394)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p3.1),[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Fuet al\.\(2026a\)D\. Fu, R\. Wu, Y\. Yang, J\. Mei, L\. Wen, P\. Cai, X\. Yang, Y\. Liu, B\. Shi, and Y\. QiaoProxy opd: on\-policy distillation with transferable relative proxy update\.External Links:2607\.11505,[Link](https://arxiv.org/abs/2607.11505)Cited by:[Appendix G](https://arxiv.org/html/2609.29142#A7.p1.1),[§1](https://arxiv.org/html/2609.29142#S1.p3.1),[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Fuet al\.\(2025\)W\. Fu, J\. Gao, X\. Shen, C\. Zhu, Z\. Mei, C\. He, S\. Xu, G\. Wei, J\. Mei, W\. JIASHU, T\. Yang, B\. Yuan, and Y\. WuAREAL: a large\-scale asynchronous reinforcement learning system for language reasoning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=X9diEuva9R)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Fuet al\.\(2026b\)Y\. Fu, H\. Huang, K\. Jiang, J\. Liu, Z\. Jiang, Y\. Zhu, and D\. ZhaoRevisiting on\-policy distillation: empirical failure modes and simple fixes\.External Links:2603\.25562,[Link](https://arxiv.org/abs/2603.25562)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Fuet al\.\(2026c\)Z\. Fu, B\. He, Y\. Zuo, H\. Huang, J\. Zhang, R\. Xiao, C\. Qian, Q\. Luo, H\. Gao, Y\. Wang, Z\. Liu, N\. Ding, and C\. XiaoRethinking on\-policy distillation of large language models ii: one training example\.External Links:2609\.04172,[Link](https://arxiv.org/abs/2609.04172)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1),[§5\.3](https://arxiv.org/html/2609.29142#S5.SS3.p2.1)\.
- GLM\-5\-Teamet al\.\(2026\)GLM\-5\-Team, A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie, C\. Zhu, C\. Yin, C\. Wang, G\. Pan, H\. Zeng,et al\.GLM\-5: from vibe coding to agentic engineering\.External Links:2602\.15763,[Link](https://arxiv.org/abs/2602.15763)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1),[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Heet al\.\(2025a\)B\. He, Z\. Qu, Z\. Liu, Y\. Chen, Y\. Zuo, C\. Qian, K\. Zhang, W\. Chen, C\. Xiao, G\. Cui, N\. Ding, and Z\. LiuJustRL: scaling a 1\.5b llm with a simple rl recipe\.External Links:2512\.16649,[Link](https://arxiv.org/abs/2512.16649)Cited by:[§5\.1](https://arxiv.org/html/2609.29142#S5.SS1.p1.1)\.
- Heet al\.\(2025b\)J\. He, J\. Liu, C\. Y\. Liu, R\. Yan, C\. Wang, P\. Cheng, X\. Zhang, F\. Zhang, J\. Xu, W\. Shen, S\. Li, L\. Zeng, T\. Wei, C\. Cheng, B\. An, Y\. Liu, and Y\. ZhouSkywork open reasoner 1 technical report\.External Links:2505\.22312,[Link](https://arxiv.org/abs/2505.22312)Cited by:[Appendix F](https://arxiv.org/html/2609.29142#A6.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2609.29142#S5.SS1.p1.1)\.
- Heoet al\.\(2026\)B\. Heo, J\. Hwang, S\. Yun, and D\. HanOn\-policy delta distillation\.External Links:2607\.15161,[Link](https://arxiv.org/abs/2607.15161)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p2.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc,et al\.Training compute\-optimal large language models\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Houet al\.\(2026\)Z\. Hou, Y\. Li, J\. Tang, and Y\. DongSingle\-rollout asynchronous optimization for agentic reinforcement learning\.External Links:2607\.07508,[Link](https://arxiv.org/abs/2607.07508)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Jiaet al\.\(2026\)N\. Jia, H\. Yang, X\. Ma, J\. Lian, S\. Zhang, W\. Zhang, K\. Zeng, X\. Cai, and Z\. SunAsymmetric on\-policy distillation: bridging exploitation and imitation at the token level\.External Links:2605\.06387,[Link](https://arxiv.org/abs/2605.06387)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Jinet al\.\(2026\)W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. LeeEntropy\-aware on\-policy distillation of language models\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=J5i09faOOf)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p1.1),[§2](https://arxiv.org/html/2609.29142#S2.p2.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Koet al\.\(2026\)J\. Ko, S\. Abdali, Y\. J\. Kim, T\. Chen, and P\. CameronScaling reasoning efficiently via relaxed on\-policy distillation\.External Links:2603\.11137,[Link](https://arxiv.org/abs/2603.11137)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p2.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Liet al\.\(2026a\)J\. Li, H\. Lin, H\. Lu, K\. Wen, Z\. Yang, J\. Gao, Y\. Wu, and J\. ZhangQuestA: expanding reasoning capacity in LLMs via question augmentation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3MifB0f7qR)Cited by:[§5\.1](https://arxiv.org/html/2609.29142#S5.SS1.p1.1)\.
- Liet al\.\(2026b\)Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. DingRethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.External Links:2604\.13016,[Link](https://arxiv.org/abs/2604.13016)Cited by:[Appendix C](https://arxiv.org/html/2609.29142#A3.p1.1),[Appendix C](https://arxiv.org/html/2609.29142#A3.p3.1),[Appendix C](https://arxiv.org/html/2609.29142#A3.p4.1),[§1](https://arxiv.org/html/2609.29142#S1.p2.1),[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Lu and Lab \(2025\)K\. Lu and T\. M\. LabOn\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/on\-policy\-distillationExternal Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1),[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Maet al\.\(2025\)W\. Ma, H\. Zhang, L\. Zhao, Y\. Song, Y\. Wang, Z\. Sui, and F\. LuoStabilizing moe reinforcement learning by aligning training and inference routers\.External Links:2510\.11370,[Link](https://arxiv.org/abs/2510.11370)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Menget al\.\(2026\)H\. Meng, K\. Huang, S\. Wei, C\. Ma, S\. Yang, X\. Wang, G\. Wang, B\. Ding, and J\. ZhouSparse but critical: a token\-level analysis of distributional shifts in RLVR fine\-tuning of LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8vWIXno8LW)Cited by:[§5\.1](https://arxiv.org/html/2609.29142#S5.SS1.p1.1)\.
- MiniMaxet al\.\(2025\)MiniMax, A\. Chen, A\. Li, B\. Gong, B\. Jiang, B\. Fei, B\. Yang, B\. Shan, C\. Yu, C\. Wang, C\. Zhu, C\. Xiao, C\. Du, C\. Zhang, C\. Qiao, C\. Zhang,et al\.MiniMax\-m1: scaling test\-time compute efficiently with lightning attention\.External Links:2506\.13585,[Link](https://arxiv.org/abs/2506.13585)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Narayananet al\.\(2021\)D\. Narayanan, M\. Shoeybi, J\. Casper, P\. LeGresley, M\. Patwary, V\. Korthikanti, D\. Vainbrand, P\. Kashinkunti, J\. Bernauer, B\. Catanzaro, A\. Phanishayee, and M\. ZahariaEfficient large\-scale language model training on gpu clusters using megatron\-lm\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,SC ’21,New York, NY, USA\.External Links:ISBN 9781450384421,[Link](https://doi.org/10.1145/3458817.3476209),[Document](https://dx.doi.org/10.1145/3458817.3476209)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Panet al\.\(2026\)L\. Pan, S\. Tao, Y\. Zhai, L\. Zhang, Z\. Liu, B\. Ding, A\. Liu, and L\. WenRLCSD: reinforcement learning with contrastive on\-policy self\-distillation\.External Links:2606\.11709,[Link](https://arxiv.org/abs/2606.11709)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Pearce and Song \(2024\)T\. Pearce and J\. SongReconciling kaplan and chinchilla scaling laws\.Transactions on Machine Learning Research\.Note:Reproducibility CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=NLoaLyuUUF)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1),[§3](https://arxiv.org/html/2609.29142#S3.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,EuroSys ’25,New York, NY, USA,pp\. 1279–1297\.External Links:ISBN 9798400711961,[Link](https://doi.org/10.1145/3689031.3696075),[Document](https://dx.doi.org/10.1145/3689031.3696075)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Tavoret al\.\(2026\)A\. Tavor, I\. Ebenspanger, N\. Cnaan, and M\. GevaRethinking selective knowledge distillation\.External Links:2602\.01395,[Link](https://arxiv.org/abs/2602.01395)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p2.1)\.
- Teamet al\.\(2026\)K\. Team, T\. Bai, Y\. Bai, Y\. Bao, S\. H\. Cai, Y\. Cao, Y\. Charles, H\. S\. Che, C\. Chen, G\. Chen, H\. Chen, J\. Chen, J\. Chen, J\. Chen, J\. Chen, K\. Chen,et al\.Kimi k2\.5: visual agentic intelligence\.External Links:2602\.02276,[Link](https://arxiv.org/abs/2602.02276)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Wanget al\.\(2026a\)X\. Wang, H\. Zhang, H\. Wang, Y\. Shi, R\. Li, K\. Han, C\. Tong, H\. Deng, A\. K\. Taylor, R\. Sun, Y\. Zhu, J\. Cong, Y\. Sun, and W\. WangARLArena: a unified framework for stable agentic reinforcement learning\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=90kxFi9VGP)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Wanget al\.\(2026b\)Y\. Wang, S\. Lu, Y\. Gu, P\. Wang, Y\. Yang, Z\. Yan, C\. Xie, J\. Wu, and H\. YangNot all disagreement is learnable: token teachability in on\-policy distillation\.External Links:2605\.26844,[Link](https://arxiv.org/abs/2605.26844)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p2.1)\.
- Wuet al\.\(2025\)B\. Wu, S\. Wang, Y\. Tang, J\. Ding, E\. Helenowski, L\. Tan, T\. Xu, T\. Gowda, Z\. Chen, C\. Zhu, X\. Tang, Y\. Qian, B\. Zhu, and R\. HouLlamaRL: a distributed asynchronous reinforcement learning framework for efficient large\-scale llm training\.External Links:2505\.24034,[Link](https://arxiv.org/abs/2505.24034)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Wuet al\.\(2024\)J\. Wu, H\. Sun, H\. Cai, L\. Su, S\. Wang, D\. Yin, X\. Li, and M\. GaoCross\-model control: improving multiple large language models in one\-time training\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=YPqHSTSoFs)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Xuet al\.\(2026\)Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. GeramifardTIP: token importance in on\-policy distillation\.External Links:2604\.14084,[Link](https://arxiv.org/abs/2604.14084)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p2.1)\.
- Yanget al\.\(2026\)W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. LinLearning beyond teacher: generalized on\-policy distillation with reward extrapolation\.External Links:2602\.12125,[Link](https://arxiv.org/abs/2602.12125)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong,et al\.DAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Yueet al\.\(2025\)Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. HuangDoes reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=4OsgYD7em5)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Zhaoet al\.\(2025\)R\. Zhao, A\. Meterez, S\. M\. Kakade, C\. Pehlevan, S\. Jelassi, and E\. MalachEcho chamber: RL post\-training amplifies behaviors learned in pretraining\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=dp4KWuSDzj)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p1.1)\.
- Zhaoet al\.\(2026\)S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. GroverSelf\-distilled reasoner: on\-policy self\-distillation for large language models\.External Links:2601\.18734,[Link](https://arxiv.org/abs/2601.18734)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p1.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.External Links:2507\.18071,[Link](https://arxiv.org/abs/2507.18071)Cited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.
- Zhouet al\.\(2026\)Y\. Zhou, L\. Zhang, Y\. Wu, M\. Wang, B\. Peng, J\. Liu, X\. Fan, and Z\. ZhaoSAGE\-opd: selective agent\-guided intervention for multi\-turn on\-policy distillation\.External Links:2606\.19659,[Link](https://arxiv.org/abs/2606.19659)Cited by:[§2](https://arxiv.org/html/2609.29142#S2.p2.1)\.
- Zhuet al\.\(2025\)Z\. Zhu, C\. Xie, X\. Lv, and slime ContributorsSlime: an llm post\-training framework for rl scaling\.Note:[https://github\.com/THUDM/slime](https://github.com/THUDM/slime)GitHub repository\. Corresponding author: Xin LvCited by:[§1](https://arxiv.org/html/2609.29142#S1.p2.1)\.

## Appendix AOptimization Details

This section details the adaptive KL control inherited from Direct\-OPD and the selective position\-wise aggregation used in S2D\-OPD\.

### A\.1Adaptive KL Control

The KL coefficient controls the strength of the student anchor relative to the policy\-shift reward\. We update this coefficient using Direct\-OPD’s sign\-based controller:

αm\+1=clip⁡\(αm​\[1\+ϵ​sgn⁡\(r¯m\)\],αmin,αmax\),\\alpha\_\{m\+1\}=\\operatorname\{clip\}\\\!\\left\(\\alpha\_\{m\}\\left\[1\+\\epsilon\\,\\operatorname\{sgn\}\(\\bar\{r\}\_\{m\}\)\\right\],\\alpha\_\{\\min\},\\alpha\_\{\\max\}\\right\),\(13\)wherer¯m\\bar\{r\}\_\{m\}denotes the mean student\-weighted policy shiftp¯t​\(v\)​Δt​\(v∣𝒔t\)\\bar\{p\}\_\{t\}\(v\)\\Delta\_\{t\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)over valid response positions and their top\-KKcandidates at iterationmm\. The controller increasesα\\alphawhen the mean weighted policy shift is positive and decreases it when negative, subject to the bounds\. We useα0=2\.5\\alpha\_\{0\}=2\.5,ϵ=0\.01\\epsilon=0\.01, and\[αmin,αmax\]=\[0\.5,2\.5\]\[\\alpha\_\{\\min\},\\alpha\_\{\\max\}\]=\[0\.5,2\.5\]\. The updated coefficientαm\+1\\alpha\_\{m\+1\}is used in the subsequent actor update\.

### A\.2Detailed Loss Aggregation

Direct\-OPD averages the local update over all valid response positions\. S2D\-OPD keeps the local update unchanged but restricts this average to selected positions\. Letℐ\\mathcal\{I\}collect the positions selected independently within each response by Eq\.[10](https://arxiv.org/html/2609.29142#S4.E10)across the batch\. Giving each retained position equal weight yields the local ascent direction

gθS2​D​\-​OPD=1\|ℐ\|​∑t∈ℐ\[∑v∈𝒱K​\(𝒔t\)sg⁡\[p¯t​\(v\)​Δt​\(v∣𝒔t\)\]​∇θ​log​πθ​\(v∣𝒔t\)−αm\+1​∇θd^t​\(θ\)\]\.g\_\{\\theta\}^\{\\mathrm\{S^\{2\}D\\text\{\-\}OPD\}\}=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\}\\Biggl\[\\sum\_\{v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\_\{t\}\)\}\\operatorname\{sg\}\\\!\\left\[\\bar\{p\}\_\{t\}\(v\)\\Delta\_\{t\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)\\right\]\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\{\\bm\{s\}\}\_\{t\}\)\-\\alpha\_\{m\+1\}\\nabla\_\{\\theta\}\\widehat\{d\}\_\{t\}\(\\theta\)\\Biggr\]\.\(14\)Here,sg\\operatorname\{sg\}denotes stop\-gradient, andd^t​\(θ\)\\widehat\{d\}\_\{t\}\(\\theta\)is the per\-position KL penalty estimator toward the initial studentπstu\\pi\_\{\\mathrm\{stu\}\}, computed usingverl’slow\_var\_klimplementation\. Both terms are applied to the same selected response positions\. The sampled prefixes, candidate sets, selection mask, and KL coefficient are held fixed during differentiation\.

## Appendix BProof of the Mass\-Invariance Construction

###### Proposition 1\(Mass invariance of the Direct\-OPD update\)\.

Fix a state𝐬\{\\bm\{s\}\}and a student checkpointθ0\\theta\_\{0\}, and writep⁡\(v\)=πθ0​\(v∣𝐬\)\>0p\(v\)=\\pi\_\{\\theta\_\{0\}\}\(v\\mid\{\\bm\{s\}\}\)\>0for allv∈𝒱v\\in\\mathcal\{V\}\. Let𝒱K​\(𝐬\)⊊𝒱\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\\subsetneq\\mathcal\{V\}be the top\-KKcandidate set, withK≥2K\\geq 2, and definep¯​\(v\)=p⁡\(v\)/∑u∈𝒱K​\(𝐬\)p⁡\(u\)\\bar\{p\}\(v\)=p\(v\)/\\sum\_\{u\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\}p\(u\)forv∈𝒱K​\(𝐬\)v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\. Let𝐭=\(tv\)v∈𝒱K​\(𝐬\)\\mathbf\{t\}=\(t\_\{v\}\)\_\{v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\}and𝐪=\(qv\)v∈𝒱K​\(𝐬\)\\mathbf\{q\}=\(q\_\{v\}\)\_\{v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\}be distinct, strictly positive probability vectors on the candidate set, and let𝐛=\(bv\)v∈𝒱∖𝒱K​\(𝐬\)\\mathbf\{b\}=\(b\_\{v\}\)\_\{v\\in\\mathcal\{V\}\\setminus\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\}be a common strictly positive probability vector on its complement\. For0<ϵ<10<\\epsilon<1, define the teacher and reference distributions:

Pϵ​\(v\)=\{ϵ​tv,v∈𝒱K​\(𝒔\),\(1−ϵ\)​bv,v∉𝒱K​\(𝒔\),Qϵ​\(v\)=\{ϵ​qv,v∈𝒱K​\(𝒔\),\(1−ϵ\)​bv,v∉𝒱K​\(𝒔\)\.P\_\{\\epsilon\}\(v\)=\\begin\{cases\}\\epsilon t\_\{v\},&v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\),\\\\ \(1\-\\epsilon\)b\_\{v\},&v\\notin\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\),\\end\{cases\}\\qquad Q\_\{\\epsilon\}\(v\)=\\begin\{cases\}\\epsilon q\_\{v\},&v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\),\\\\ \(1\-\\epsilon\)b\_\{v\},&v\\notin\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\.\\end\{cases\}\(15\)Here,ϵ\\epsilonis the total probability mass that each checkpoint assigns to the student’s candidates\. Both checkpoints vary withϵ\\epsilon, while the student, candidate set, and conditional distributions remain fixed\.

Then, for everyϵ∈\(0,1\)\\epsilon\\in\(0,1\):

1. \(i\)*Reward invariance\.*For every candidatev∈𝒱K​\(𝒔\)v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\), the Direct\-OPD reward \(Eq\.[1](https://arxiv.org/html/2609.29142#S3.E1)\) withπT=Pϵ\\pi\_\{\\mathrm\{T\}\}=P\_\{\\epsilon\}andπref=Qϵ\\pi\_\{\\mathrm\{ref\}\}=Q\_\{\\epsilon\}satisfies Δϵ​\(v∣𝒔\)=log⁡Pϵ​\(v\)Qϵ​\(v\)=log⁡tvqv\.\\Delta\_\{\\epsilon\}\(v\\mid\{\\bm\{s\}\}\)=\\log\\frac\{P\_\{\\epsilon\}\(v\)\}\{Q\_\{\\epsilon\}\(v\)\}=\\log\\frac\{t\_\{v\}\}\{q\_\{v\}\}\.\(16\)
2. \(ii\)*Fixed, nonzero update\.*The local reward gradient of Direct\-OPD \(Eq\.[4](https://arxiv.org/html/2609.29142#S3.E4)\) atθ0\\theta\_\{0\}, 𝐠ϵ=∑v∈𝒱K​\(𝒔\)p¯​\(v\)​log⁡tvqv​∇θ​log⁡πθ​\(v∣𝒔\)\|θ=θ0=𝐠,\\mathbf\{g\}\_\{\\epsilon\}=\\sum\_\{v\\in\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\}\\bar\{p\}\(v\)\\log\\frac\{t\_\{v\}\}\{q\_\{v\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v\\mid\{\\bm\{s\}\}\)\\Big\|\_\{\\theta=\\theta\_\{0\}\}=\\mathbf\{g\},\(17\)is independent ofϵ\\epsilonand nonzero with respect to the student’s logits\.
3. \(iii\)*Divergences scale with the mass\.*The teacher–reference divergences satisfy DJS\(Pϵ,Qϵ\)=ϵDJS\(𝐭,𝐪\),DKL\(Pϵ∥Qϵ\)=ϵDKL\(𝐭∥𝐪\),DKL\(Qϵ∥Pϵ\)=ϵDKL\(𝐪∥𝐭\)\.D\_\{\\mathrm\{JS\}\}\(P\_\{\\epsilon\},Q\_\{\\epsilon\}\)=\\epsilon D\_\{\\mathrm\{JS\}\}\(\\mathbf\{t\},\\mathbf\{q\}\),\\ D\_\{\\mathrm\{KL\}\}\(P\_\{\\epsilon\}\\\|Q\_\{\\epsilon\}\)=\\epsilon D\_\{\\mathrm\{KL\}\}\(\\mathbf\{t\}\\\|\\mathbf\{q\}\),\\ D\_\{\\mathrm\{KL\}\}\(Q\_\{\\epsilon\}\\\|P\_\{\\epsilon\}\)=\\epsilon D\_\{\\mathrm\{KL\}\}\(\\mathbf\{q\}\\\|\\mathbf\{t\}\)\.\(18\)

Consequently, asϵ→0\\epsilon\\to 0, all three divergences vanish, whereas the reward and its gradient remain fixed and nonzero\.

###### Proof\.

\(i\) The common factorϵ\\epsiloncancels in the ratio\. \(ii\) By \(i\), each term of𝐠ϵ\\mathbf\{g\}\_\{\\epsilon\}is independent ofϵ\\epsilon, since𝒱K​\(𝒔\)\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)andp¯\\bar\{p\}depend only onθ0\\theta\_\{0\}\. For softmax logits𝐳\\mathbf\{z\},∇𝐳logπ\(v∣𝒔\)=𝐞v−π\(⋅∣𝒔\)\\nabla\_\{\\mathbf\{z\}\}\\log\\pi\(v\\mid\{\\bm\{s\}\}\)=\\mathbf\{e\}\_\{v\}\-\\pi\(\\cdot\\mid\{\\bm\{s\}\}\), so𝐠=𝐰−\(∑uwu\)​p\\mathbf\{g\}=\\mathbf\{w\}\-\(\\sum\_\{u\}w\_\{u\}\)\\,p, wherewv=p¯​\(v\)​log⁡\(tv/qv\)w\_\{v\}=\\bar\{p\}\(v\)\\log\(t\_\{v\}/q\_\{v\}\)on𝒱K​\(𝒔\)\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)andwv=0w\_\{v\}=0elsewhere\. Since𝒱K​\(𝒔\)⊊𝒱\\mathcal\{V\}\_\{K\}\(\{\\bm\{s\}\}\)\\subsetneq\\mathcal\{V\}andp\>0p\>0,𝐠=𝟎\\mathbf\{g\}=\\mathbf\{0\}would require𝐰=𝟎\\mathbf\{w\}=\\mathbf\{0\}, i\.e\.,𝐭=𝐪\\mathbf\{t\}=\\mathbf\{q\}\. \(iii\) Off the candidate set,PϵP\_\{\\epsilon\},QϵQ\_\{\\epsilon\}, and their mixture coincide, so the shared tail contributes zero to each divergence; on the candidate set, the factorϵ\\epsiloncancels inside each logarithm and factors out of the sum\. ∎

## Appendix CToken Overlap Across Divergence Ranks

Prior work associates successful OPD with increasing overlap between the student’s and teacher’s high\-probability token sets\([Li et al\., 2026b](https://arxiv.org/html/2609.29142#bib.bib22)\)\. Direct\-OPD instead transfers a teacher–reference log\-ratio, motivating us to examine overlap with both checkpoints\. We track how these overlaps evolve when supervision is restricted to different teacher–reference JSD deciles\.

For a student policyπS\\pi\_\{S\}and a comparison policyπC\\pi\_\{C\}at statests\_\{t\}, let

𝒱kS​\(𝒔t\)\\displaystyle\\mathcal\{V\}\_\{k\}^\{S\}\(\{\\bm\{s\}\}\_\{t\}\)=TopK\(πS\(⋅∣𝒔t\),k\),\\displaystyle=\\operatorname\{TopK\}\\bigl\(\\pi\_\{S\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\),k\\bigr\),\(19\)𝒱kC​\(𝒔t\)\\displaystyle\\mathcal\{V\}\_\{k\}^\{C\}\(\{\\bm\{s\}\}\_\{t\}\)=TopK\(πC\(⋅∣𝒔t\),k\),\\displaystyle=\\operatorname\{TopK\}\\bigl\(\\pi\_\{C\}\(\\cdot\\mid\{\\bm\{s\}\}\_\{t\}\),k\\bigr\),\(20\)so that the per\-state overlap ratio is

Overlapk⁡\(S,C,𝒔t\)=\|𝒱kS​\(𝒔t\)∩𝒱kC​\(𝒔t\)\|k\.\\operatorname\{Overlap\}\_\{k\}\(S,C;\{\\bm\{s\}\}\_\{t\}\)=\\frac\{\\left\|\\mathcal\{V\}\_\{k\}^\{S\}\(\{\\bm\{s\}\}\_\{t\}\)\\cap\\mathcal\{V\}\_\{k\}^\{C\}\(\{\\bm\{s\}\}\_\{t\}\)\\right\|\}\{k\}\.\(21\)which we report withk=16k=16against both the post\-RL teacher \(C=TC=T\) and the reference \(C=TrefC=T\_\{\\mathrm\{ref\}\}\)\. Every bin starts from the same student initialization, so the two panels record how training on a given decile moves the student relative to each checkpoint\.

Figure 7:Top\-16 token overlap over training for Qwen3\-1\.7B under the JustRL teacher pair, using JSD deciles or random 10% selection\. Panels compare the student with the post\-RL teacher \(left\) and pre\-RL reference \(right\)\. Overlap is the fraction of shared tokens between top\-16 sets\.Higher\-JSD bins generally attain greater overlap with the post\-RL teacher \(Figure[7](https://arxiv.org/html/2609.29142#A3.F7), left\), broadly tracking the performance ordering in Figure[3\(a\)](https://arxiv.org/html/2609.29142#S5.F3.sf1)\. This resembles the overlap growth observed in successful standard OPD\([Li et al\., 2026b](https://arxiv.org/html/2609.29142#bib.bib22)\), here under an objective that transfers the teacher’s policy shift\. The reference panel adds a complementary observation: the highest\-JSD bin increases teacher overlap while maintaining reference overlap above its initial level\. Reference overlap alone does not follow the performance ordering; for example, the 10–20 bin recovers close to its initial reference overlap despite much smaller gains in teacher overlap\.

The strongest transfer occurs without a trade\-off between overlap with the two checkpoints: the highest\-JSD bin increases teacher overlap while sustaining reference overlap\. This suggests that transferring the RL\-induced policy shift need not entail moving from the reference’s high\-probability candidate set\. The teacher–reference log\-ratio can favor a token that remains highly ranked under both checkpoints, allowing transfer through changes in relative preference among shared candidates\. Related evidence from standard OPD shows that supervision restricted to tokens shared by the student and teacher recovers nearly the full benefit of student top\-kksupervision\([Li et al\., 2026b](https://arxiv.org/html/2609.29142#bib.bib22)\)\. Our JSD criterion identifies positions with substantial changes in candidate probabilities or total mass relative to the tail \(Eq\.[8](https://arxiv.org/html/2609.29142#S4.E8)\)\. Together, these observations suggest a view of selective transfer as learning substantial changes in probability allocation within largely overlapping candidate spaces\.

## Appendix DSensitivity to Retention Ratio

Our main experiments retain the 10% of positions with the highest teacher–reference JSD within each response\. Figure[8](https://arxiv.org/html/2609.29142#A4.F8)compares retention ratios of 5%, 10%, 15%, and 20% for Qwen3\-1\.7B under the JustRL teacher pair\. All four settings yield broadly similar learning curves and substantial gains over the base model on AIME 2024 and AIME 2025\. No retention ratio consistently dominates across both benchmarks and training checkpoints\.

Figure 8:Sensitivity to the per\-response retention ratio for Qwen3\-1\.7B under the JustRL teacher pair\. The four settings retain the top 20%, 15%, 10%, and 5% of positions ranked by teacher–reference JSD\. Dashed lines indicate base\-model accuracy\.Together with the decile analysis in Section[5\.3](https://arxiv.org/html/2609.29142#S5.SS3), these results reveal an asymmetry in supervision selection\. Training on low\-JSD bins can substantially degrade transfer, whereas varying the retained fraction within the high\-JSD region has a comparatively small effect\. Moreover, the most selective setting does not consistently outperform the broader subsets\. This pattern supports using JSD to screen supervision, without requiring that larger divergence always imply greater transfer utility\. It also accords with our theoretical motivation: small divergence bounds the observable policy change, while large divergence alone does not establish the value of transferring that change\. The similar performance across retention ratios suggests that effective selection admits a broad operating range in this setting, and we adopt 10% as a common default\.

## Appendix ECase Study: What Policy Changes Does Selection Preserve?

To examine what the mask keeps and removes, we inspect states from 20 rollouts of the initial Qwen3\-1\.7B student on AIME 2026 under the JustRL pair\. At each state, we compare the teacher and reference next\-token distributions over the student’s top\-KKcandidates and the residual token, and we mark the state as retained if its JSD falls within the top 10% of its response, following the per\-response rule of Sec\.[4\.2](https://arxiv.org/html/2609.29142#S4.SS2)\. We writeΔ​p\\Delta pfor the change in a token’s probability from the reference to the teacher\. We selected the four cases below by hand for interpretability, so they illustrate how the score behaves rather than estimate how often each pattern occurs\.

![Refer to caption](https://arxiv.org/html/2609.29142v1/case17.png)Figure 9:Same sampled token, different mask decisions\.The mask filters a calculation on which the teacher and reference agree but retains supervision for a boundary correction\.![Refer to caption](https://arxiv.org/html/2609.29142v1/case20.png)Figure 10:Retained shifts encode continuation preferences\.At two retained states, the teacher shifts probability away fromAlternatively; the later state favorsLet,Given, andSince\.Selection acts on states, not on sampled tokens\.Fig\.[9](https://arxiv.org/html/2609.29142#A5.F9)contrasts two states at which the student samples3\. At the first,

3completes a valid calculation, and the teacher and reference distributions are nearly identical \(JSD=0\.0001\\mathrm\{JSD\}=0\.0001\)\. At the second,

3sets an incorrect boundary:k=3k=3still permitsA=B=1A=B=1, so excluding their simultaneous occurrence requiresk≥4k\\geq 4\. Here RL moves probability from3to4\(Δ​p≈−0\.55\\Delta p\\approx\-0\.55and\+0\.56\+0\.56\), and the state is retained \(JSD=0\.1735\\mathrm\{JSD\}=0\.1735\)\. The same token thus receives opposite mask decisions, because the score depends on how the teacher’s distribution changed at the state rather than on which token was sampled\.

![Refer to caption](https://arxiv.org/html/2609.29142v1/case09.png)Figure 11:A later reconsideration cue does not undo an earlier adverse shift\.The mask retains a shift toward an incorrect parity judgment, followed by a teacher preference for reconsideration\.Retained shifts include preferences over how reasoning continues\.Fig\.[10](https://arxiv.org/html/2609.29142#A5.F10)shows two retained states at which the student opens a new passage with

Alternatively\. At both, RL lowers the probability of this token; at the later state the decrease is large \(Δ​p≈−0\.71\\Delta p\\approx\-0\.71\), and the teacher instead favorsLet,Given, andSince\. Together with the arithmetic cases, this shows that the retained supervision covers both local corrections and preferences over how the reasoning proceeds, consistent with a score that measures behavioral change rather than correctness\.

JSD does not judge correctness\.In Fig\.[11](https://arxiv.org/html/2609.29142#A5.F11), the student describesR1 = 6as

odd\. RL reinforces this error, raising the probability ofodd\(Δ​p≈\+0\.41\\Delta p\\approx\+0\.41\) and lowering that ofeven\(Δ​p≈−0\.12\\Delta p\\approx\-0\.12\)\. The JSD of 0\.1197 exceeds the response’s retention cutoff of 0\.0744, so this adverse shift is retained\. At the next state, conditioned on the erroneous statement, the teacher favorsWaitoverTherefore\(from 0\.01 to 0\.71 and from 0\.49 to 0\.02\), a preference compatible with reconsideration, but this later shift does not cancel the reward foroddat the earlier state\. The score thus selects states by the size of the teacher’s change, not by whether the change is correct\.

Summary\.These cases separate where supervision is applied from what it encourages\. JSD decides which states are retained, while Direct\-OPD’s log\-ratio rewards decide which tokens are encouraged at those states\. Because the score measures how much the teacher’s distribution changed on the student’s candidates, the retained states include numerical corrections, continuation preferences, and local errors alike\. Masking therefore concentrates Direct\-OPD supervision on the states where RL changed the teacher most, without filtering shifts by their correctness\.

## Appendix FTraining Details

#### Data and prompt\.

All runs use the math subset of Skywork\-OR1\-RL\-Data\([He et al\., 2025b](https://arxiv.org/html/2609.29142#bib.bib41)\)and apply the following prompt template\.

> Solve the following math problem step by step\. The last line of your response should be of the form Answer: $Answer \(without quotes\) where $Answer is the answer to the problem\. \{Question\} Remember to put your answer on its own line after "Answer:"\.

#### Training and evaluation\.

All experiments are implemented withverland run on 8 NVIDIA H200 GPUs\. Table[2](https://arxiv.org/html/2609.29142#A6.T2)summarizes the default training and evaluation configuration used throughout our experiments\. Owing to limited compute, each configuration is trained with a single run; the confidence interval in Sec\.[5\.2](https://arxiv.org/html/2609.29142#S5.SS2)resamples held\-out problems and does not reflect variation across training seeds\.

Table 2:Default training and evaluation configuration\.SettingTrainingEvaluationFrameworkverl–Hardware8×8\\timesNVIDIA H200–Global batch size128–Mini\-batch size128–Rolloutnn4–Max\. prompt length1,024–Max\. response length2,04831,744Samples per problem–32Sampling temperature1\.00\.7OPD support size Top\-KK16–Top\-ppsampling1\.00\.95Learning rate1×10−61\\times 10^\{\-6\}–Training steps300–KL coefficientα\\alphaAdaptive–Controllerϵ\\epsilon0\.01–\[αmin,αmax\]\[\\alpha\_\{\\min\},\\alpha\_\{\\max\}\]\[0\.5,2\.5\]\[0\.5,\\,2\.5\]–Checkpoint selection–AIME 24/25Held\-out evaluation–AIME 26, HMMT Nov\.25 / Feb\.26

## Appendix GLimitations

Our work has four main limitations\. First, JSD is a proxy for how much RL changed the teacher, not a judge of whether the change is correct\. Because it scores only the magnitude of the teacher–reference difference, S2D\-OPD retains a large shift toward an error as readily as a large correction \(Fig\.[11](https://arxiv.org/html/2609.29142#A5.F11)\), so the quality of the retained supervision still depends on the teacher’s RL\. Selectors that also account for the direction or correctness of a shift, for example through verifier or outcome signals, could filter such states\. Second, JSD is blind to what the student needs\. The score depends on the student only through its top\-KKcandidate set: a state where the student already follows the teacher’s shift consumes the same budget as one where it does not, and a low\-divergence shift that the student lacks is discarded\. Combining teacher–reference divergence with student\-side signals, in the spirit of the student\-side calibration in Proxy\-OPD\([Fu et al\., 2026a](https://arxiv.org/html/2609.29142#bib.bib24)\), is a natural extension\. Third, our evidence is limited in scale and scope\. We transfer from 1\.5B teachers to students of up to 8B parameters, whereas weak\-to\-strong transfer is most valuable for much larger students, where RL is most costly\. All experiments use mathematical reasoning with verifiable rewards; whether selective supervision helps in code generation or agentic tasks, where RL\-induced shifts may be distributed differently across states, remains untested\. Finally, owing to limited compute, each configuration is trained once, so our confidence interval reflects variation across held\-out problems rather than across training seeds; repeated runs would strengthen the per\-setting comparisons\.