SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation

# SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation
Source: [https://arxiv.org/html/2608.14647](https://arxiv.org/html/2608.14647)
###### Abstract

Dirty\-history rollouts make multi\-turn on\-policy self\-distillation \(OPSD\) brittle: once a student emits an erroneous intermediate reply, later turns are conditioned on that reply, and uniform distillation can spend loss on tokens that carry little corrective signal\. We introduceSMOPD\(Selective Masking for On\-Policy Distillation\), a loss\-only stabilization method for multi\-turn OPSD\. For each generated middle\-turn reply,SMOPDranks token positions by student entropy and removes the lowest\-entropy 20% from the clipped generalized Jensen\-Shannon distillation loss; final\-answer and FULL\-preservation losses are unchanged\. This design targets token\-level uncertainty rather than coarse trajectory outcomes, adds no parameters, and has zero inference\-time overhead\. We compareSMOPDwith a correctness\-scaling variant that multiplies a common detached reliability proxy using final\-answer correctness\. On LiC with Qwen3 models,SMOPDimproves SHARDED\-view accuracy by 1\.0–2\.5 percentage points in single\-seed 1\.7B, 4B, and 8B comparisons, and a small 4B multi\-seed check shows a \+1\.7pp mean SHARDED gain over baseline \(two\-tailedp=0\.022p=0\.022\)\. Adding the outcome scalar is harmful without masking at 1\.7B \(−\-4\.0pp\) and remains scale\-dependent when combined with masking \(\+1\.3pp at 4B, neutral at 1\.7B, and−\-0\.5pp at 8B\)\. These archived aggregate results suggest that token\-level uncertainty is a more reliable stabilization signal than scalar final\-answer correctness in this evaluated dirty\-history OPSD setting, while leaving causal mechanism tests and broader benchmark validation to future work\.

## 1Introduction

Large language models \(LLMs\) are increasingly deployed in multi\-turn interactive settings—chat, tool use, and collaborative problem solving—where they must maintain coherence across multiple exchanges\. Training LLMs for such settings is challenging: supervised fine\-tuning requires costly human demonstrations at every turn, while single\-turn evaluation is a poor proxy for multi\-turn performance\(Dubois et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib16); Madaan et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib15)\)\.

On\-Policy Self\-Distillation \(OPSD\) has emerged as an effective paradigm for multi\-turn training\. The model generates its own trajectories \(on\-policy rollouts\), and a frozen EMA teacher provides distillation targets under clean, single\-turn context\. MAIGO\(Zheng et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib1)\)implements this via a history\-cleaning mechanism that isolates the teacher from student\-generated errors, plus a single\-turn preservation branch to prevent catastrophic forgetting\.

However, a fundamental challenge remains:dirty history\. During multi\-turn rollouts, the student accumulates errors across turns\. Each intermediate response conditions the next prompt, creating a compounding distortion that pulls the student away from the reference distribution\. Prior work has proposed various techniques for single\-turn settings—RL stabilization \(PPO clipping\(Schulman et al\.,[2017](https://arxiv.org/html/2608.14647#bib.bib6)\)\), preference optimization \(DPO\(Rafailov et al\.,[2023](https://arxiv.org/html/2608.14647#bib.bib7)\), KTO\(Ethayarajh et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib8)\)\), and distillation\-specific stabilization \(REOPOLD\(Ko et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib4)\), EOPD\(Jin et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib2)\)\)—but whether any of these transfer to multi\-turn OPSD is an open question\.

Against this backdrop, we define a specific, reusable stabilization component for dirty\-history OPSD: percentile\-based student\-entropy selection inside the middle\-turn loss\. We call this entropy\-masked variantSMOPD\(Selective Masking for On\-Policy Distillation\)\. The selection rule is closest to REOPOLD’s entropy\-guided dynamic sampling\(Ko et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib4)\); our contribution is to place it in a multi\-turn clean\-teacher / dirty\-student GJS distillation loss and compare it directly with coarse trajectory\-outcome weighting\.

We study two interventions:

1. 1\.Token\-level entropy masking: Mask low\-entropy \(overconfident\) tokens from the GJS distillation loss\. The intuition: when the student assigns high confidence to a token, that token contributes little learning signal and may represent overconfident errors formed under dirty history\.
2. 2\.Outcome\-guided correctness scaling: Multiply the common reliability proxy by a scalar that up\-weights middle turns when the final answer is incorrect and down\-weights them when correct\. The intuition: failed trajectories may carry more informative signal about where dirty\-history corruption occurs; successful trajectories have less to learn from\.

We evaluate both via systematic ablation across three model scales \(Qwen3\-1\.7B, 4B, 8B\) on the LiC multi\-turn benchmark\. Our findings are:

- •Entropy masking is the most consistent positive signal in our evidence: It improves SHARDED\-view accuracy by 1\.0–2\.5pp across the evaluated model scales, with zero additional parameters and zero inference overhead\.
- •Outcome correctness scaling degrades without masking and is mixed when combined: Without masking, it lowers SHARDED by 4\.0pp\. Combined with masking, results are scale\-dependent: neutral at 1\.7B, \+1\.3pp at 4B \(two\-tailedp=0\.101p=0\.101; directional one\-tailedp=0\.0504p=0\.0504\), and−\-0\.5pp at 8B\. The combination does not reliably outperform masking alone\.
- •Reliability evidence is strongest for entropy masking at 4B: The relative gain from entropy masking remains positive at 200 steps, and the 4B multi\-seed comparison shows a \+1\.7pp SHARDED mean gain with low observed seed\-to\-seed spread\.

Our results suggest a practical starting point for multi\-turn OPSD: prioritize token\-level uncertainty \(entropy\-based token selection\) before adding coarse outcome signals\. The simplicity of entropy masking makes it straightforward to test in OPSD pipelines\.

In summary, our contributions are:

- •We introduceSMOPD, a loss\-only selective masking component that adapts percentile\-based token\-level entropy selection to multi\-turn OPSD with dirty\-history rollouts and a GJS middle\-turn loss\.
- •We analyze token\-level confidence as a motivation for loss masking and separate this mechanism hypothesis from the archived aggregate accuracy evidence\.
- •Through ablation across three model scales, we find that entropy masking improves multi\-turn accuracy by 1\.0–2\.5pp in our evaluated settings, while correctness\-based outcome scaling—effective in single\-turn RL\-style settings—does not transfer reliably in these experiments\.
- •We provide a small multi\-seed check at 4B that supports the entropy\-masking signal and indicates that the outcome scalar’s effect remains uncertain across scales\.
- •We identify a methodological concern—spurious large effects from single\-seed OPSD experiments—and offer practical evaluation recommendations\.

## 2Related Work

Table 1:Positioning relative to closely related OPD/OPSD stabilization methods\.Table[1](https://arxiv.org/html/2608.14647#S2.T1)is the main positioning device for this paper\. The key distinction is scope:SMOPDis a narrow loss\-side intervention inside a MAIGO\-style clean\-teacher / dirty\-student training family, not a replacement for broader OPD frameworks or richer outcome\-guided credit\-assignment methods\.

#### Multi\-turn OPSD\.

OPSD trains on student\-generated trajectories while an EMA teacher supplies reference logits\(Zhao et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib24)\)\. MAIGO\(Zheng et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib1)\)extends this idea to multi\-turn LiC by cleaning teacher history and preserving FULL\-view competence\. We use the same clean\-teacher / dirty\-student motivation, but average all eligible middle\-turn losses and study selective masking inside the GJS loss\. Adjacent self\-improvement and self\-distillation lines include self\-rewarding and self\-play training\(Yuan et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib13); Chen et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib14)\), SDFT\(Shenfeld et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib28)\), context\-conditioned OPD\(Ye et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib22)\), empirical OPD failure analyses\(Fu et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib3)\), reflective OPSD\(Zhao et al\.,[2026b](https://arxiv.org/html/2608.14647#bib.bib29)\), and reliability\-aware self\-distillation frameworks\(Song and Zheng,[2026](https://arxiv.org/html/2608.14647#bib.bib25); Jin et al\.,[2026b](https://arxiv.org/html/2608.14647#bib.bib33)\)\.

#### Stabilization signals\.

Single\-turn RLHF and preference optimization rely on KL\-style constraints or preference\-derived rewards\(Ouyang et al\.,[2022](https://arxiv.org/html/2608.14647#bib.bib9); Rafailov et al\.,[2023](https://arxiv.org/html/2608.14647#bib.bib7); Ethayarajh et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib8)\); G\-OPD and ExOPD connect OPD to dense KL\-constrained RL\(Yang et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib23)\)\. Token\-level OPD methods are closer to our mechanism: REOPOLD selects high\-entropy student tokens during refinement\(Ko et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib4)\), EOPD switches KL direction using entropy\(Jin et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib2)\), AOPD changes token\-level learning mode by advantage region\(Jia et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib32)\), and DRKL studies reverse\-KL overconfidence\(Luong et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib27)\)\. We adapt percentile\-based student\-entropy selection to dirty\-history middle turns, with the percentile computed within each generated reply rather than over a batch\.

#### Outcome and benchmark context\.

Process rewards provide step\-level feedback but require extra supervision\(Lightman et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib12)\)\. Outcome\-guided methods such as OGLS\-SD and SCOPE use richer steering or dual\-path weighting\(Yang et al\.,[2026b](https://arxiv.org/html/2608.14647#bib.bib30); Zheng et al\.,[2026b](https://arxiv.org/html/2608.14647#bib.bib31)\); CREDIT argues that coarse trajectory outcomes are insufficient for OPD credit assignment\(Shen et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib34)\)\. Our outcome variant is intentionally weaker: a scalar correctness\-dependent multiplier on a common reliability\-weighted middle\-turn loss\. We evaluate on LiC\(Laban et al\.,[2025](https://arxiv.org/html/2608.14647#bib.bib17)\), whose paired FULL/SHARDED protocol directly measures the multi\-turn degradation that MAIGO targets\. MT\-Bench offers complementary multi\-turn evaluation\(Zheng et al\.,[2024](https://arxiv.org/html/2608.14647#bib.bib18)\), but lacks this paired\-view control\.

#### Entropy signal\.

Entropy regularization is standard in reinforcement learning\(Haarnoja et al\.,[2018](https://arxiv.org/html/2608.14647#bib.bib19); Schulman et al\.,[2017](https://arxiv.org/html/2608.14647#bib.bib6)\), and recent LLM work links entropy dynamics to reasoning diversity and exploration\(Cui et al\.,[2025](https://arxiv.org/html/2608.14647#bib.bib20); Cheng et al\.,[2025](https://arxiv.org/html/2608.14647#bib.bib21)\)\. In contrast,SMOPDuses entropy as a per\-token selection signal for middle\-turn distillation, not as a global regularizer\.

## 3Motivation: Token\-Level Confidence in Multi\-Turn OPSD

Dirty\-history OPSD averages a per\-token GJS loss over student\-generated middle turns, but the tokens in a reply are not equally informative\. Low\-entropy positions often correspond to predictable continuations, formatting, or locally committed choices; high\-entropy positions more often mark decision points such as operators, variable bindings, or logical connectors\. Under dirty history, confidence can also become miscalibrated after earlier student errors enter the context\.

SMOPDtherefore treats entropy masking as a stabilization hypothesis: retain the uncertain token positions where correction is more likely to matter, and remove the lowest\-entropy positions from the middle\-turn loss\. We use a per\-reply percentile rather than a global threshold because entropy scales shift across tasks, turns, and training stages\. Throughout the reported experiments,β=0\.8\\beta=0\.8retains the highest\-entropy 80% of tokens within each generated middle\-turn reply\. The aggregate results below test whether this rule improves accuracy; they do not by themselves prove a token\-level causal mechanism\.

## 4Method

We adopt a MAIGO\-inspired clean\-teacher / dirty\-student framework\(Zheng et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib1)\): the student modelπθ\\pi\_\{\\theta\}generates multi\-turn rollouts with dirty history, while a frozen EMA teacherπEMA\\pi\_\{\\text\{EMA\}\}produces token\-level logits under clean, single\-turn context—middle turns see only user turnsu1,…,utu\_\{1\},\\ldots,u\_\{t\}, and the answer turn sees the canonical FULL\-view specificationff\. Our implementation deliberately differs from MAIGO in two places: it averages over all eligible middle turns in a rollout rather than sampling one eligible turn as a single\-turn estimator, and it uses the detached reliability proxy in §[4\.3](https://arxiv.org/html/2608.14647#S4.SS3)as the common middle\-turn weight\. Outcome\-guided variants multiply that proxy by a final\-answer correctness scalar\. Figure[1](https://arxiv.org/html/2608.14647#S4.F1)summarizes where entropy masking enters the training loop\.

![Refer to caption](https://arxiv.org/html/2608.14647v1/x1.png)Figure 1:Entropy masking in multi\-turn OPSD\.The student generates dirty\-history rollouts; the EMA teacher is evaluated under clean context on the same generated response tokens\. Entropy masking is applied only to middle\-turn GJS losses, while answer\-turn and FULL\-preservation RKL losses remain unmasked\.### 4\.1Preliminaries

Given a student\-generated reply at turnttwith tokensy1,…,yNy\_\{1\},\\ldots,y\_\{N\}, the per\-token loss at positioniiis computed as a divergence between the student’s distributionpstudenti=πθ\(⋅∣cidirty\)p\_\{\\text\{student\}\}^\{i\}=\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{i\}^\{\\text\{dirty\}\}\)and the teacher’s distributionpteacheri=πEMA\(⋅∣ciclean\)p\_\{\\text\{teacher\}\}^\{i\}=\\pi\_\{\\text\{EMA\}\}\(\\cdot\\mid c\_\{i\}^\{\\text\{clean\}\}\), wherecidirtyc\_\{i\}^\{\\text\{dirty\}\}andcicleanc\_\{i\}^\{\\text\{clean\}\}denote the dirty\-history and clean\-history contexts, respectively\.

Token alignment and teacher forcing\. For each generated student reply, both student and teacher distributions are evaluated on the same token sequencey1,…,yNy\_\{1\},\\ldots,y\_\{N\}\. The contexts differ, but the supervised positions do not: the student distribution is conditioned on the dirty prefix that includes prior student replies, while the teacher distribution is teacher\-forced on the student\-generated continuation under the corresponding clean context\. This avoids variable\-length alignment between independently generated teacher and student replies; divergence is computed position\-wise over the shared vocabulary at each generated token position\.

Generalized Jensen\-Shannon Divergence\. For middle turns, we use GJS with coefficientβmid=0\.5\\beta\_\{\\text\{mid\}\}=0\.5:

GJS​\(pstudenti,pteacheri\)=βmid​DKL​\(pteacheri∥m\)\+\(1−βmid\)​DKL​\(pstudenti∥m\)\\text\{GJS\}\(p\_\{\\text\{student\}\}^\{i\},p\_\{\\text\{teacher\}\}^\{i\}\)=\\beta\_\{\\text\{mid\}\}\\,D\_\{\\text\{KL\}\}\(p\_\{\\text\{teacher\}\}^\{i\}\\,\\\|\\,m\)\+\(1\-\\beta\_\{\\text\{mid\}\}\)\\,D\_\{\\text\{KL\}\}\(p\_\{\\text\{student\}\}^\{i\}\\,\\\|\\,m\)\(1\)wherem=βmid​pteacheri\+\(1−βmid\)​pstudentim=\\beta\_\{\\text\{mid\}\}\\,p\_\{\\text\{teacher\}\}^\{i\}\+\(1\-\\beta\_\{\\text\{mid\}\}\)\\,p\_\{\\text\{student\}\}^\{i\}is the interpolated mixture distribution\. GJS provides symmetric, bounded gradients suitable for intermediate reasoning steps, avoiding the mode\-seeking bias of pure reverse KL\.

Reverse KL Divergence\. For the answer turn, we use RKL with coefficientβfinal=1\.0\\beta\_\{\\text\{final\}\}=1\.0:

ℒanswer=1N​∑i=1ND~KL​\(pstudenti∥pteacheri\)\\mathcal\{L\}\_\{\\text\{answer\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\widetilde\{D\}\_\{\\text\{KL\}\}\(p\_\{\\text\{student\}\}^\{i\}\\,\\\|\\,p\_\{\\text\{teacher\}\}^\{i\}\)\(2\)RKL penalizes student samples outside the teacher’s support—appropriate for the final answer where precision matters\.

Pointwise clipping, preservation, and EMA\. Following MAIGO’s token\-level clipping setting, every per\-token GJS and RKL term used for training is clipped atϵclip=0\.5\\epsilon\_\{\\text\{clip\}\}=0\.5:

d~i=min⁡\(di,ϵclip\)\.\\widetilde\{d\}\_\{i\}=\\min\(d\_\{i\},\\epsilon\_\{\\text\{clip\}\}\)\.\(3\)We writeGJS~\\widetilde\{\\text\{GJS\}\}andD~KL\\widetilde\{D\}\_\{\\text\{KL\}\}for clipped divergences\. With probabilityρ=0\.2\\rho=0\.2, we also compute FULL\-branch RKL on the canonical single\-turn taskff\. After each optimizer step, the EMA model is updated asθEMA←λ​θEMA\+\(1−λ\)​θ\\theta\_\{\\text\{EMA\}\}\\leftarrow\\lambda\\theta\_\{\\text\{EMA\}\}\+\(1\-\\lambda\)\\thetawithλ=0\.99\\lambda=0\.99\.

### 4\.2Token\-Level Entropy Masking

For each middle turn, the GJS divergence produces a per\-token lossℓi\\ell\_\{i\}on the student’s generated reply\. We compute student entropy at each token position:

Hi=−∑v∈Vpθ​\(v\|x<i\)​log⁡pθ​\(v\|x<i\)H\_\{i\}=\-\\sum\_\{v\\in V\}p\_\{\\theta\}\(v\|x\_\{<i\}\)\\log p\_\{\\theta\}\(v\|x\_\{<i\}\)\(4\)
whereVVis the vocabulary\. Given a retention ratioβ∈\(0,1\]\\beta\\in\(0,1\], we retain the topβ\\betafraction of tokens by entropy and zero out the lowest\-entropy positions:

ℒmid,t=1\|𝒯t\|​∑i∈𝒯tGJS~​\(pstudenti,pteacheri\)\\mathcal\{L\}\_\{\\text\{mid\},t\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{t\}\|\}\\sum\_\{i\\in\\mathcal\{T\}\_\{t\}\}\\widetilde\{\\text\{GJS\}\}\(p\_\{\\text\{student\}\}^\{i\},p\_\{\\text\{teacher\}\}^\{i\}\)\(5\)
where𝒯t=\{i:Hi≥quantile​\(\{Hj\}j=1Nt,1−β\)\}\\mathcal\{T\}\_\{t\}=\\\{i:H\_\{i\}\\geq\\text\{quantile\}\(\\\{H\_\{j\}\\\}\_\{j=1\}^\{N\_\{t\}\},1\-\\beta\)\\\}is the retained token set within replytt\. We useβ=0\.8\\beta=0\.8\. The operation is loss\-side only: no extra parameters, no generation changes, and zero inference\-time overhead\.

### 4\.3Outcome\-Guided Turn Weighting

For each middle turntt, we compute a detached reliability proxy\. This reliability weighting is shared by all reported configurations; when outcome guidance is enabled, we additionally apply a correctness\-dependent scalar\. Letyiy\_\{i\}be theiith token in the student\-generated middle reply\. We measure the student–teacher discrepancy on the realized tokens as

δtproxy=1Nt​∑i=1Nt\|log⁡pstudenti​\(yi\)−log⁡pteacheri​\(yi\)\|\.\\delta\_\{t\}^\{\\text\{proxy\}\}=\\frac\{1\}\{N\_\{t\}\}\\sum\_\{i=1\}^\{N\_\{t\}\}\\left\|\\log p\_\{\\text\{student\}\}^\{i\}\(y\_\{i\}\)\-\\log p\_\{\\text\{teacher\}\}^\{i\}\(y\_\{i\}\)\\right\|\.\(6\)
Letδ~\\tilde\{\\delta\}be the median ofmax⁡\(δjproxy,ϵ\)\\max\(\\delta\_\{j\}^\{\\text\{proxy\}\},\\epsilon\)over eligible middle turns in the same rollout\. We define a MAIGO\-inspired reliability proxy

rt=δ~δ~\+max⁡\(δtproxy,ϵ\)\.r\_\{t\}=\\frac\{\\tilde\{\\delta\}\}\{\\tilde\{\\delta\}\+\\max\(\\delta\_\{t\}^\{\\text\{proxy\}\},\\epsilon\)\}\.\(7\)
This proxy follows MAIGO’s motivation\(Zheng et al\.,[2026a](https://arxiv.org/html/2608.14647#bib.bib1)\)of down\-weighting turns that have drifted far from the clean\-context reference, but it is not an exact reproduction of MAIGO’s adaptive estimator\. We then multiply it by a correctness\-dependent scalar:

wt=clip\[0,1\]⁡\(rt⋅\(1\+η⋅\(1−2​o\)\)\),w\_\{t\}=\\operatorname\{clip\}\_\{\[0,1\]\}\\left\(r\_\{t\}\\cdot\\bigl\(1\+\\eta\\cdot\(1\-2o\)\\bigr\)\\right\),\(8\)
whereo∈\{0,1\}o\\in\\\{0,1\\\}is the binary outcome \(1 = correct, 0 = incorrect\)\. We useη=0\.2\\eta=0\.2, so incorrect outcomes multiply the reliability proxy by 1\.2 and correct outcomes by 0\.8\.

The combined middle\-turn loss with both techniques becomes:

ℒmidcombined=1\|ℐ\|​∑t∈ℐwt​\(1\|𝒯t\|​∑i∈𝒯tGJS~​\(pstudenti,pteacheri\)\)\\mathcal\{L\}\_\{\\text\{mid\}\}^\{\\text\{combined\}\}=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\}w\_\{t\}\\left\(\\frac\{1\}\{\|\\mathcal\{T\}\_\{t\}\|\}\\sum\_\{i\\in\\mathcal\{T\}\_\{t\}\}\\widetilde\{\\text\{GJS\}\}\(p\_\{\\text\{student\}\}^\{i\},p\_\{\\text\{teacher\}\}^\{i\}\)\\right\)\(9\)
whereℐ\\mathcal\{I\}is the set of eligible non\-empty middle turns\. MAIGO’s Algorithm 1 samples one eligible middle turn as an estimator; our implementation averages all eligible middle\-turn losses in the rollout\.

### 4\.4Training Procedure

Algorithm[4\.4](https://arxiv.org/html/2608.14647#S4.SS4)summarizes the training loop\. Following the REOPOLD\-inspired schedule, entropy masking is disabled during the first 33% of steps by settingβ=1\.0\\beta=1\.0; unlike REOPOLD, we do not use a reward\-filtering phase\.

Algorithm 1: Entropy\-masked multi\-turn OPSD training\. Input:Studentπθ\\pi\_\{\\theta\}, EMA teacherπEMA\\pi\_\{\\text\{EMA\}\}, training set𝒟\\mathcal\{D\}, masking ratioβ=0\.8\\beta=0\.8, outcome sensitivityη\\eta, exploration fractionγ=0\.33\\gamma=0\.33\. Output:Trained studentπθ\\pi\_\{\\theta\}\.

1. 1\.InitializeπEMA←πθ\\pi\_\{\\text\{EMA\}\}\\leftarrow\\pi\_\{\\theta\}\.
2. 2\.For steps=1,…,Ss=1,\\ldots,S, sampleℬ∼𝒟\\mathcal\{B\}\\sim\\mathcal\{D\}and generate dirty\-history rollouts\{a1,…,aT\}∼πθ\\\{a\_\{1\},\\ldots,a\_\{T\}\\\}\\sim\\pi\_\{\\theta\}\.
3. 3\.Compute teacher logitsπEMA\(⋅∣clean context\)\\pi\_\{\\text\{EMA\}\}\(\\cdot\\mid\\text\{clean context\}\)for each turn\.
4. 4\.Ifs\>γ​Ss\>\\gamma S, computeHiH\_\{i\}\(Eq\.[4](https://arxiv.org/html/2608.14647#S4.E4)\) and retain the per\-reply top\-β\\betaentropy tokens; otherwise set𝒯t\\mathcal\{T\}\_\{t\}to all tokens\.
5. 5\.Compute reliability proxiesrtr\_\{t\}and setwtw\_\{t\}by Eq\.[8](https://arxiv.org/html/2608.14647#S4.E8), withwt=rtw\_\{t\}=r\_\{t\}whenη=0\\eta=0\.
6. 6\.Clip token losses \(Eq\.[3](https://arxiv.org/html/2608.14647#S4.E3)\) and computeℒmidcombined\\mathcal\{L\}\_\{\\text\{mid\}\}^\{\\text\{combined\}\}\(Eq\.[9](https://arxiv.org/html/2608.14647#S4.E9)\) on retained tokens\.
7. 7\.Add answer\-turn RKL and, with probabilityρ=0\.2\\rho=0\.2, FULL\-branch RKL\.
8. 8\.Updateπθ\\pi\_\{\\theta\}by gradient descent and update EMA using decayλ\\lambda\.

## 5Experimental Setup

### 5\.1Models and Optimization

We use Qwen3\-Instruct models\(Yang et al\.,[2025](https://arxiv.org/html/2608.14647#bib.bib5)\)at three scales: 1\.7B, 4B, and 8B parameters\. All models are fine\-tuned with LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2608.14647#bib.bib10)\)\(rankr=64r=64,α=128\\alpha=128, dropout0\.00\.0\) applied to all linear projections in the attention and feed\-forward layers\.

We train for 100 steps using AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2608.14647#bib.bib11)\)with learning rate5×10−65\\times 10^\{\-6\}\. EMA decay is set toλ=0\.99\\lambda=0\.99\. The FULL preservation branch is sampled with probabilityρ=0\.2\\rho=0\.2\. Following the REOPOLD\-inspired schedule described in §[4\.4](https://arxiv.org/html/2608.14647#S4.SS4), an exploration phase \(entropy masking disabled, without REOPOLD’s reward\-based filtering\) covers the first 33% of training steps\. All experiments use seed 42 unless noted otherwise\.

### 5\.2Benchmark and Metric

We use the LiC \(Language in Context\) combined benchmark\. The training set contains 12,165 multi\-turn examples spanning math, database, code, and action\-planning domains\. Evaluation uses 200 held\-out math examples with both FULL\-view \(single\-turn complete task\) and SHARDED\-view \(step\-by\-step turn revelation\) protocols\. The primary metric is*SHARDED accuracy*, which measures multi\-turn reasoning performance under dirty\-history conditions\.

### 5\.3Evidence Boundary and Provenance

All tabulated and plotted accuracy values are aggregate metrics traceable to archived run artifacts\. We exclude unarchived token\-level diagnostics, per\-turn slices, prompt\-level examples, and exploratory parsing outputs from the quantitative evidence base\. Accordingly,β=0\.8\\beta=0\.8is treated as a fixed operating point rather than a proven optimum\. The arXiv source package contains only manuscript source, style file, and figures; it excludes model weights, checkpoints, logs, and raw evaluation outputs\.

### 5\.4Configurations

We test four configurations:

- •baseline: Clean\-teacher / dirty\-student OPSD with the common reliability proxy, no entropy mask and no outcome scalar \(β=1\.0\\beta=1\.0,η=0\.0\\eta=0\.0\)\.
- •\+entropy: Entropy masking on top of the common reliability proxy \(β=0\.8\\beta=0\.8,η=0\.0\\eta=0\.0\)\.
- •\+outcome: Correctness scalar on top of the common reliability proxy \(β=1\.0\\beta=1\.0,η=0\.2\\eta=0\.2\)\.
- •combined: Both stabilization techniques active \(β=0\.8\\beta=0\.8,η=0\.2\\eta=0\.2\)\. Two archived training runs with identical hyperparameters are averaged to estimate run\-to\-run variation\.

### 5\.5Hardware

All experiments run on a single NVIDIA RTX PRO 6000 Blackwell GPU \(96GB VRAM\)\. Each 100\-step training run takes approximately 40–50 minutes; full evaluation \(200 samples×\\times2 views\) takes an additional 60–80 minutes\.

## 6Results

### 6\.1Main Ablation Study \(1\.7B\)

Table 2:Ablation study on Qwen3\-1\.7B \(100 steps, seed=42\)\.†Mean of two archived training runs with identical hyperparameters \(75\.0%/64\.5% and 78\.0%/67\.0%\)\. The 2\.5pp SHARDED range illustrates the scale of run\-to\-run variation in this setting\.

![Refer to caption](https://arxiv.org/html/2608.14647v1/x2.png)Figure 2:Ablation results on Qwen3\-1\.7B\.In this ablation, entropy masking raises SHARDED by 2\.5pp, the outcome scalar lowers it by 4\.0pp, and the combined mean does not improve over baseline\.Table[2](https://arxiv.org/html/2608.14647#S6.T2)presents the core ablation results\. Entropy masking gives the largest positive change in this ablation: it improves SHARDED accuracy by 2\.5pp over baseline alongside a comparable FULL improvement \(\+2\.5pp\)\. Adding the outcome scalar without masking lowers SHARDED by 4\.0pp, suggesting that binary outcome signals are too noisy to guide middle\-turn training in this setting\.

The combined configuration \(β=0\.8,η=0\.2\\beta=0\.8,\\eta=0\.2\) was run twice with identical hyperparameters\. The two runs differ by 2\.5pp SHARDED \(64\.5% vs\. 67\.0%\), illustrating the scale of run\-to\-run variation in this setting\. The mean result \(65\.8%\) is 0\.7pp below baseline, suggesting that combining the outcome scalar with entropy masking provides no clear benefit over masking alone at the 1\.7B scale and may slightly degrade performance\.

### 6\.2Cross\-Scale Validation \(4B, 8B\)

We probe whether the 1\.7B findings transfer to larger models by running baseline, \+entropy, and combined on Qwen3\-4B and Qwen3\-8B\.

Table 3:Cross\-scale validation of entropy masking \(100 steps, seed=42\)\.Δ\\DeltaSH = SHARDED gain over baseline at each scale\.![Refer to caption](https://arxiv.org/html/2608.14647v1/x3.png)Figure 3:Single\-seed cross\-scale comparisons\.Entropy masking has positive SHARDED deltas at all evaluated scales, with smaller observed gains at larger model sizes\.Table[3](https://arxiv.org/html/2608.14647#S6.T3)shows positive single\-seed SHARDED deltas for entropy masking at all evaluated scales: \+2\.5pp \(1\.7B\), \+1\.5pp \(4B\), and \+1\.0pp \(8B\)\. The gains are small and should be read descriptively, but their direction is consistent with the 1\.7B ablation\. Combined results remain scale\-dependent:−\-0\.7pp at 1\.7B, \+1\.3pp at 4B, and−\-0\.5pp at 8B\.

Table 4:Absolute SHARDED accuracy for combined\-configuration comparisons\. The 1\.7B combined value is the mean of two archived runs, the 4B row uses the multi\-seed means from Table[5](https://arxiv.org/html/2608.14647#S6.T5), and the 8B row is a single\-seed comparison\.Addressing a 4B outlier\. An initial 4B combined run produced a \+7\.0pp SHARDED gain \(71\.5% vs\. 64\.5%\), but replication reduced the estimate to \+1\.3pp mean gain\. We treat the outlier as evidence of run/seed variance rather than as the main effect\.

![Refer to caption](https://arxiv.org/html/2608.14647v1/x4.png)Figure 4:Effect breakdown \(1\.7B\)\.Entropy masking gives the largest positive SHARDED delta in this ablation; the outcome scalar without masking is negative; combined does not improve over baseline at this scale\.
### 6\.3Small Multi\-Seed Reliability Check

For 4B baseline and \+entropy, we combine the cross\-scale seed\-42 runs with seeds 123 and 456\. For combined, the initial seed\-42 outlier is reported separately; Table[5](https://arxiv.org/html/2608.14647#S6.T5)uses three subsequent reliability runs\.

Table 5:Multi\-seed validation on 4B, 100 steps\. Mean±\\pmstd across 3 seeds\. Tablepp\-values are from Welch’s two\-tailedtt\-test comparing SHARDED against baseline; directional one\-tailed values are reported in text\.‡The combined row uses three subsequent reliability runs\. The earlier combined seed\-42 outlier is reported separately in §[7\.3](https://arxiv.org/html/2608.14647#S7.SS3)\.

Entropy masking shows a \+1\.7pp SHARDED mean gain \(one\-tailedp=0\.011p=0\.011; two\-tailedp=0\.022p=0\.022\)\. Combined is weaker \(\+1\.3pp, one\-tailedp=0\.0504p=0\.0504; two\-tailedp=0\.101p=0\.101\) and inconsistent across scales\. With only three seeds, thesepp\-values are rough reliability checks rather than precise population estimates\.

![Refer to caption](https://arxiv.org/html/2608.14647v1/x5.png)Figure 5:4B multi\-seed reliability check\.Bars show mean SHARDED accuracy across three seeds, error bars show sample standard deviation, and points show individual seed results\. The \+entropy condition is the clearest positive signal; the combined condition has wider observed spread and weaker evidence\.Figure[5](https://arxiv.org/html/2608.14647#S6.F5)visualizes the same archived SHARDED values as Table[5](https://arxiv.org/html/2608.14647#S6.T5)\. At 200 steps, \+entropy remains \+1\.0pp over baseline \(68\.0% vs\. 67\.0%\), suggesting that the relative gain is not only a 100\-step artifact\.

### 6\.4Training Duration Analysis

We sweep training steps \(100–400\) on the 1\.7B baseline to validate our choice of 100 steps as the primary operating point\.

Table 6:Step count sweep on Qwen3\-1\.7B baseline \(seed=42\)\. SHARDED peaks at 200 steps, then decreases while the FULL\-SHARDED gap widens\.SHARDED peaks at 200 steps \(68\.5%\) and then falls, while FULL rises at 400 steps and the FULL\-SHARDED gap widens to 16\.5pp\. We use 100 steps as the primary operating point because it is near the SHARDED peak while halving training time for cross\-scale comparisons\.

We further validate at 200 steps on the 4B model, shown in Table[7](https://arxiv.org/html/2608.14647#S6.T7)\.

Table 7:Training stability at 200 steps, 4B, seed=42\.†The combined configuration’s 100\-step reference \(71\.5%\) was identified as the \+7\.0pp outlier discussed in §[7\.3](https://arxiv.org/html/2608.14647#S7.SS3)\. At 200 steps, combined achieves 67\.5%, consistent with the multi\-seed mean of 65\.5% at 100 steps \(Table[5](https://arxiv.org/html/2608.14647#S6.T5)\)\.

## 7Analysis

### 7\.1Mechanism Scope

The archived results support a narrow empirical claim: in the LiC/Qwen3 settings evaluated here, entropy masking is the most consistent positive intervention among the tested configurations\. They do not establish that entropy masking is generally optimal for OPSD, thatβ=0\.8\\beta=0\.8is the best masking ratio, or that overconfidence suppression is the unique cause of the gains\.

The results are compatible with three mechanisms:

1. 1\.Overconfidence suppression\. Masking may remove some confidently wrong low\-entropy updates induced by dirty history\.
2. 2\.Signal\-to\-noise filtering\. Low\-entropy positions may be dominated by routine tokens, so filtering them can concentrate GJS loss on more informative disagreements\.
3. 3\.Implicit curriculum\. High\-entropy tokens may mark harder decision points, such as operators, variable bindings, and logical connectors\.

These interpretations fit the aggregate pattern, including the smaller gains at larger scales, but they require token\-level calibration logs to verify\.

Alternative explanations remain plausible: reduced effective supervision, changed gradient scale from the1/\|𝒯t\|1/\|\\mathcal\{T\}\_\{t\}\|normalization, or token\-distribution artifacts unrelated to confidence\. Distinguishing these accounts requires at least four controls: random masking, divergence\-based masking, fixed\-denominator masking, and direct entropy\-error calibration\. We therefore treat the mechanism discussion as interpretation rather than causal evidence\.

### 7\.2Why Outcome Weighting Is Fragile

Outcome correctness scaling assumes that intermediate turn quality correlates with final\-answer correctness\. In multi\-turn OPSD this proxy is noisy: a good intermediate turn can occur in a failed trajectory, while a flawed step can be followed by recovery\. Unlike single\-turn preference optimization, an error at turnttchanges the input distribution at turnt\+1t\+1, so the final outcome is a delayed and entangled signal\. A single binary label then applies the same scalar to all middle turns, amplifying variance rather than localizing correction\. The EMA teacher already supplies clean\-context token targets, so the scalar outcome signal can also conflict with the teacher\-student divergence\.

This helps explain why the outcome scalar is harmful without masking at 1\.7B and inconsistent when combined with masking\. Outcome signals may still help at larger scales or with finer granularity, such as turn\-level correctness or process feedback, but our results align with credit\-assignment work showing that coarse trajectory outcomes are insufficient for OPD without more local signals\(Shen et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib34); Yang et al\.,[2026b](https://arxiv.org/html/2608.14647#bib.bib30)\)\.

### 7\.3The Outlier Problem in Small\-Scale OPSD

Single\-seed OPSD experiments can produce spuriously large effects\. The 4B combined outlier \(\+7\.0pp\) is far larger than the replicated \+1\.3pp mean, and the 1\.7B combined runs differ by 2\.5pp SHARDED despite identical hyperparameters\. This matches broader reports that OPD is sensitive to teacher choice, loss formulation, and privileged\-information design\(Zhu et al\.,[2026](https://arxiv.org/html/2608.14647#bib.bib26)\)\. Core OPSD comparisons should therefore include multi\-seed results or at least within\-configuration replication, and single\-seed gains comparable to observed run\-to\-run variation should remain provisional\.

## 8Conclusion

We presentedSMOPD, a selective token\-entropy masking variant for dirty\-history multi\-turn OPSD\. In LiC/Qwen3 experiments,SMOPDimproves SHARDED accuracy by 1\.0–2\.5pp across the evaluated 1\.7B–8B scales while adding no parameters and zero inference overhead\. Outcome\-guided correctness scaling is harmful without masking at 1\.7B \(−\-4\.0pp\) and inconsistent when combined with masking \(\+1\.3pp at 4B, neutral at 1\.7B,−\-0\.5pp at 8B\)\.

The practical implication is cautious but useful: start with token\-level uncertainty before adding scalar trajectory outcomes\. A small step sweep,β=0\.8\\beta=0\.8as an initial operating point, target\-scale validation for any outcome signal, and multi\-seed reporting are the most important evaluation practices suggested by these runs\.

### Practical Takeaways

1. 1\.Use entropy masking as the first stabilization check\. It is a drop\-in loss modification with zero inference cost\.
2. 2\.Start fromβ=0\.8\\beta=0\.8, not as a default law\. Treat the ratio as an operating point to sweep when compute allows\.
3. 3\.Do not rely on binary outcome scaling alone\. The 1\.7B degradation is practically meaningful, and the combined effect must be validated at the target scale\.
4. 4\.Report replication\. The 4B outlier shows that single\-seed OPSD gains can be misleading\.

### Limitations

The evidence is limited to LiC, Qwen3\-Instruct models, math\-heavy held\-out evaluation, LoRA rank 64, and mostly 100\-step training\. Cross\-scale results are single\-seed except for the 4B reliability check, andβ=0\.8\\beta=0\.8is a fixed operating point rather than an established optimum\. We also do not report unarchived token\-level diagnostics as quantitative evidence; stronger claims about entropy\-error calibration require preserved raw token logs and masking controls\.

### Data, Code, and Compute Availability

We use the public LiC benchmark and publicly released Qwen3\-Instruct base models\. This arXiv package includes manuscript source and figures only; model weights, checkpoints, raw logs, and raw evaluation outputs are not included\. Quantitative claims are based on archived aggregate artifacts, with fuller code/configuration release planned after cleanup\. Experiments used one NVIDIA RTX PRO 6000 Blackwell GPU \(96GB VRAM\), approximately 40–50 minutes per 100\-step run and 60–80 minutes per two\-view evaluation\.

### Broader Impact

If the result transfers, entropy masking could make multi\-turn OPSD slightly more data\-efficient without changing inference behavior\. We do not identify risks specific to the masking rule beyond the general risks of deploying more capable LLMs\.

Acknowledgment\. The authors thank the maintainers of the LiC benchmark for providing standardized multi\-turn evaluation protocols\.

## References

- Zheng et al\. \(2026a\)H\. Zheng, Y\. Zhu, S\. Yuan, S\. Chen, Q\. Wang, W\. Zhang, J\. Xiao, and Y\. Zhuang\. MAIGO: Mitigating Lost\-in\-Conversation with History\-Cleaned On\-Policy Self\-Distillation\.arXiv preprint arXiv:2605\.27186, 2026\.
- Jin et al\. \(2026a\)W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. Lee\. Entropy\-Aware On\-Policy Distillation of Language Models\.arXiv preprint arXiv:2603\.07079, 2026\.
- Fu et al\. \(2026\)Y\. Fu, H\. Huang, K\. Jiang, Y\. Zhu, and D\. Zhao\. Revisiting On\-Policy Distillation: Empirical Failure Modes and Simple Fixes\.arXiv preprint arXiv:2603\.25562, 2026\.
- Ko et al\. \(2026\)J\. Ko, S\. Abdali, Y\. J\. Kim, T\. Chen, and P\. Cameron\. Scaling Reasoning Efficiently via Relaxed On\-Policy Distillation\.arXiv preprint arXiv:2603\.11137, 2026\.
- Yang et al\. \(2025\)A\. Yang et al\. Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388, 2025\.
- Schulman et al\. \(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\. Proximal Policy Optimization Algorithms\.arXiv preprint arXiv:1707\.06347, 2017\.
- Rafailov et al\. \(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\. Direct Preference Optimization: Your Language Model is Secretly a Reward Model\. InNeurIPS, 2023\.
- Ethayarajh et al\. \(2024\)K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela\. KTO: Model Alignment as Prospect Theoretic Optimization\. InICML, 2024\.
- Ouyang et al\. \(2022\)L\. Ouyang et al\. Training Language Models to Follow Instructions with Human Feedback\. InNeurIPS, 2022\.
- Hu et al\. \(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\. LoRA: Low\-Rank Adaptation of Large Language Models\. InICLR, 2022\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. Hutter\. Decoupled Weight Decay Regularization\. InICLR, 2019\.
- Lightman et al\. \(2024\)H\. Lightman et al\. Let’s Verify Step by Step\. InICLR, 2024\.
- Yuan et al\. \(2024\)W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. Weston\. Self\-Rewarding Language Models\. InICML, 2024\.
- Chen et al\. \(2024\)Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu\. Self\-Play Fine\-Tuning Converts Weak Language Models to Strong Language Models\. InICML, 2024\.
- Madaan et al\. \(2024\)A\. Madaan et al\. Self\-Refine: Iterative Refinement with Self\-Feedback\. InNeurIPS, 2024\.
- Dubois et al\. \(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto\. Length\-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators\. InCOLM, 2024\.
- Laban et al\. \(2025\)P\. Laban, H\. Hayashi, Y\. Zhou, and J\. Neville\. LLMs Get Lost in Multi\-Turn Conversation\.arXiv preprint arXiv:2505\.06120, 2025\.
- Zheng et al\. \(2024\)L\. Zheng, W\.\-L\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, et al\. Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\. InNeurIPS, 2024\.
- Haarnoja et al\. \(2018\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\. Soft Actor\-Critic: Off\-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor\. InICML, 2018\.
- Cui et al\. \(2025\)G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen, et al\. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models\.arXiv preprint arXiv:2505\.22617, 2025\.
- Cheng et al\. \(2025\)D\. Cheng, S\. Huang, X\. Zhu, B\. Dai, W\. X\. Zhao, Z\. Zhang, and F\. Wei\. Reasoning with Exploration: An Entropy Perspective\.arXiv preprint arXiv:2506\.14758, 2025\.
- Ye et al\. \(2026\)T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei\. On\-Policy Context Distillation for Language Models\.arXiv preprint arXiv:2602\.12275, 2026\.
- Yang et al\. \(2026a\)W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. Lin\. Learning beyond Teacher: Generalized On\-Policy Distillation with Reward Extrapolation\.arXiv preprint arXiv:2602\.12125, 2026\.
- Zhao et al\. \(2026a\)S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover\. Self\-Distilled Reasoner: On\-Policy Self\-Distillation for Large Language Models\.arXiv preprint arXiv:2601\.18734, 2026\.
- Song and Zheng \(2026\)M\. Song and M\. Zheng\. A Survey of On\-Policy Distillation for Large Language Models\.arXiv preprint arXiv:2604\.00626, 2026\.
- Zhu et al\. \(2026\)S\. Zhu, X\. Ye, H\. Lu, W\. Shi, and G\. Liu\. The Many Faces of On\-Policy Distillation: Pitfalls, Mechanisms, and Fixes\.arXiv preprint arXiv:2605\.11182, 2026\.
- Luong et al\. \(2026\)H\.\-C\. Luong, D\. B\. Tran, and L\. Chen\. Diversity\-Aware Reverse Kullback\-Leibler Divergence for Large Language Model Distillation\.arXiv preprint arXiv:2604\.00223, 2026\.
- Shenfeld et al\. \(2026\)I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal\. Self\-Distillation Enables Continual Learning\.arXiv preprint arXiv:2601\.19897, 2026\.
- Zhao et al\. \(2026b\)Z\. Zhao, X\. Ma, L\. Yang, Y\. Feng, D\. Shi, J\. He, X\. Xin, Z\. Ren, and X\.\-M\. Wu\. ROSD: Reflective On\-Policy Self\-Distillation for Language Model Reasoning across Domains\.arXiv preprint arXiv:2605\.28014, 2026\.
- Yang et al\. \(2026b\)Y\. Yang, X\. Wang, and W\. Zhang\. OGLS\-SD: On\-Policy Self\-Distillation with Outcome\-Guided Logit Steering for LLM Reasoning\.arXiv preprint arXiv:2605\.12400, 2026\.
- Zheng et al\. \(2026b\)B\. Zheng, X\. Ma, Y\. Liang, J\. Ruan, X\. Fu, K\. Lin, B\. Zhu, K\. Zeng, and X\. Cai\. SCOPE: Signal\-Calibrated On\-Policy Distillation Enhancement with Dual\-Path Adaptive Weighting\.arXiv preprint arXiv:2604\.10688, 2026\.
- Jia et al\. \(2026\)N\. Jia, H\. Yang, X\. Ma, J\. Lian, S\. Zhang, W\. Zhang, K\. Zeng, X\. Cai, and Z\. Sun\. Asymmetric On\-Policy Distillation: Bridging Exploitation and Imitation at the Token Level\.arXiv preprint arXiv:2605\.06387, 2026\.
- Jin et al\. \(2026b\)Y\. Jin, Y\. Wang, L\. Fu, Y\. Xiao, Y\. Luo, H\. Liu, B\. A\. Prakash, J\. Hester, J\. Wang, and S\. Kumar\. UniSD: Towards a Unified Self\-Distillation Framework for Large Language Models\.arXiv preprint arXiv:2605\.06597, 2026\.
- Shen et al\. \(2026\)G\. Shen, L\. Huang, X\. Cheng, C\. Zhao, J\. Li, D\. Zhao, and X\. Yu\. From Generic Correlation to Input\-Specific Credit in On\-Policy Self Distillation\.arXiv preprint arXiv:2605\.11613, 2026\.

## Appendix

### A\. Reproducibility and Experimental Protocol

Table[8](https://arxiv.org/html/2608.14647#Sx1.T8)consolidates the experimental settings used for the reported runs\. These settings are collected here to make the arXiv version auditable without requiring readers to reconstruct the protocol from the method and results sections\.

Table 8:Protocol and hyperparameter summary for the reported experiments\.All reported accuracy values are aggregate evaluation results\. The arXiv source package contains the manuscript, style file, and figures needed for compilation, but not model checkpoints, model weights, raw logs, or raw evaluation outputs\.

### B\. Source\-to\-Claim Provenance

Table[9](https://arxiv.org/html/2608.14647#Sx1.T9)maps the paper’s main quantitative claims to the archived aggregate artifacts used for this arXiv draft\. This table is intended as a provenance checklist rather than a replacement for a future full reproducibility release\.

Table 9:Source\-to\-claim map for reported quantitative evidence\.The current manuscript deliberately excludes unarchived token\-level diagnostics, per\-turn slices, prompt\-level examples, qualitative case studies, and masking\-ratio sensitivity rows as quantitative evidence\. Those analyses may be useful in a later release, but they should be reported only after the corresponding raw artifacts are preserved\.

### C\. Statistical Interpretation Notes

The held\-out evaluation set contains 200 examples, so reported accuracies move in increments of 0\.5 percentage points\. A 1\.0pp SHARDED difference corresponds to two evaluation examples, and a 2\.5pp difference corresponds to five examples\. For this reason, the single\-seed cross\-scale comparisons in Table[3](https://arxiv.org/html/2608.14647#S6.T3)should be read as directional descriptive evidence, not as precise effect\-size estimates\.

The 4B reliability check uses three seeds per configuration\. Table[5](https://arxiv.org/html/2608.14647#S6.T5)reports Welch’s two\-tailedtt\-test for SHARDED accuracy against the baseline; the corresponding directional one\-tailed values are discussed in the main text because the intervention hypothesis is directional\. Withn=3n=3, thesepp\-values are rough indicators under limited power\. They support a cautious reliability claim for \+entropy in this setting, but they do not establish a general scaling law or an optimal masking ratio\.

The combined configuration is handled conservatively because an early 4B seed\-42 run produced a \+7\.0pp SHARDED outlier\. That run is reported in the outlier discussion and is not folded into the multi\-seed reliability mean\. The combined row in Table[5](https://arxiv.org/html/2608.14647#S6.T5)therefore estimates the subsequent reliability runs, while the outlier remains visible as a warning about single\-seed OPSD variance\.

Table 10:How the main empirical results should be interpreted\.
### D\. Implementation Notes Relative to Prior Work

The method is closest to the MAIGO/REOPOLD line of work, but it is not an exact reproduction of either algorithm\. Relative to MAIGO, we use a clean\-teacher / dirty\-student training family, but average over all eligible non\-empty middle turns in a rollout rather than sampling one eligible turn as an estimator\. The detached reliability proxy in Eq\.[7](https://arxiv.org/html/2608.14647#S4.E7)is common to the reported configurations; the outcome variant multiplies this proxy by a correctness scalar and is not MAIGO’s full adaptive reliability estimator\.

Relative to REOPOLD, the entropy selection rule is adapted to multi\-turn OPSD rather than single\-turn OPD refinement\. The threshold is computed within each generated middle\-turn reply, not as a batch\-level percentile\. The REOPOLD\-inspired schedule in Algorithm[4\.4](https://arxiv.org/html/2608.14647#S4.SS4)only delays entropy masking during the first 33% of training steps; it does not implement REOPOLD’s reward\-based filtering phase\.

All divergence terms are computed with token alignment through teacher forcing: the student and EMA teacher are evaluated on the same student\-generated token sequence, while their contexts differ\. This avoids aligning two independently generated responses\. Entropy masking applies only to middle\-turn GJS terms; answer\-turn RKL and FULL\-preservation RKL are left unmasked\.

### E\. Per\-Seed Breakdown for Multi\-Seed Validation

Table[11](https://arxiv.org/html/2608.14647#Sx1.T11)lists the exact rows used for the 4B reliability summary\. It is a provenance breakdown rather than a strictly paired\-seed design: baseline and \+entropy use the paper\-canonical Phase 3 seed\-42 rows, while the combined seed\-42 row is a subsequent reliability rerun and excludes the earlier \+7\.0pp outlier\.

Table 11:Per\-seed results for multi\-seed validation \(4B, 100 steps\)\.‡For baseline and \+entropy, seed 42 is the cross\-scale run used in Table[3](https://arxiv.org/html/2608.14647#S6.T3)\. The combined seed\-42 row is a subsequent reliability rerun; the earlier combined seed\-42 outlier is reported separately in §[7\.3](https://arxiv.org/html/2608.14647#S7.SS3)\.

### F\. Step Sweep Figure

![Refer to caption](https://arxiv.org/html/2608.14647v1/x6.png)Figure 6:Step count sweep on 1\.7B baseline\.SHARDED peaks at 200 steps then decreases, while FULL rises at 400 steps, suggesting possible overfitting to single\-turn performance\.