Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
Summary
This paper studies self-distillation with privileged information (PI) as a lone post-training objective for LLMs, reproducing reported gains on easy tasks but showing it fails on difficult reasoning tasks: per-token loss drops while validation accuracy stagnates or degrades. The authors trace the failure to PI bias, which pulls teacher targets toward a reference trajectory and trains students to be flatter and less decisive without improving reasoning.
arXiv:2608.04794v1 Announce Type: new
Abstract: Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.
View Cached Full Text
Cached at: 08/06/26, 07:43 AM
# How PI-Conditioned Teachers Break Self-Distillation
Source: [https://arxiv.org/html/2608.04794](https://arxiv.org/html/2608.04794)
## Privileged, but Biased: How PI\-Conditioned Teachers Break Self\-Distillation
Sarthak Harne Chinmay Karkar Yash Pandya Ahmed Awadallah Akshay Nambi Microsoft Research
###### Abstract
Self\-distillation \(SD\) has emerged as a compute\-efficient alternative to reinforcement learning with verifiable rewards: a self\-teacher, conditioned on privileged information \(PI\) about the answer such as a reference solution, supplies dense per\-token supervision to a student that never sees it\. Reported gains, however, come almost exclusively from narrow, low\-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO’s reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not\. Across question answering, mathematics, coding, and multi\-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per\-token loss falls steadily while validation accuracy does not improve and typically degrades\. We explain this failure through a single causal chain from the loss to the model it produces\. The chain begins with PI bias: having seen one particular reference solution, the teacher’s per\-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score\. Trained to match this target everywhere, the student’s objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low\-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires\. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success\.
## 1Introduction
Reinforcement learning with verifiable rewards \(RLVR\) is a standard recipe for post\-training large language models to reason\(Shaoet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib8); Lambertet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib11)\), but its supervision is coarse, sparse, and expensive: one scalar per rollout gives every token the same credit, many prompts yield no correct rollout, and each update needs a verifier and many on\-policy samples\. This motivates a denser signal suppliedduringthe rollout\. Distillation\(Koet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib12); Agarwalet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib13)\)supplies one but needs a separate, larger teacher; SD removes that requirement by building the teacher from the model being trained, conditioning it on privileged information \(PI\) the student is not given, such as a reference solution, a hint, or execution feedback\(Shenfeldet al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib10)\)\. Conditioned on the PI the model is a much stronger next\-token predictor of a correct trajectory, so matching student to teacher along the student’s rollout turns a single privileged example into a dense per\-token signal, with no separate teacher and no reward\. SDPO\(Hübotteret al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib1)\)and OPSD\(Zhaoet al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib2)\)use SD in RL post\-training and report matching or beating GRPO at lower sample and compute cost, but on comparatively easy tasks such as short multiple\-choice knowledge questions\.On difficult tasks, does SD as a lone objective, with no reward term, optimize task correctness, or a proxy that merely correlates with it easy tasks?
We find that it does not optimize task correctness\. We reproduce the reported SDPO behavior in its easy setting, then apply the identical setup to difficult tasks under four varied domains: general QA \(MMLU\-Pro\), mathematics \(DAPO\-Math\), coding \(CodeForces\), and multi\-turn agentic tool use\. We observe the opposite outcome: the per\-token loss decreases steadily while validation accuracy does not improve and typically degrades, by up to3\.513\.51points on average in the agentic domain and7\.07\.0on individual benchmarks, under both recipes \(Figure[2](https://arxiv.org/html/2608.04794#S5.F2), Section[5\.2](https://arxiv.org/html/2608.04794#S5.SS2)\)\. Since code, teacher, and clipping are unchanged, the difference lies in the objective, and not in the specific recipe\.
The cause is two design choices the methods share:PI\-conditioning and per\-token density jointly decouple the loss from correctness\.A teacher that has read one reference solution defines at each position a distribution shaped by that solution’s surface form, not by correctness, and the dense divergence targets every token, including the many that do not decide the answer\. Our central contribution makes this concrete as acausal chain: PI bias→\\rightarrowa loss blind to correctness→\\rightarroweffort spent on uninformative tokens→\\rightarrowexploration penalized→\\rightarrowa flattened student, every link measured on the same Qwen3\-8B runs \(Section[6](https://arxiv.org/html/2608.04794#S6), Figures[3](https://arxiv.org/html/2608.04794#S6.F3)and[4](https://arxiv.org/html/2608.04794#S6.F4)\)\.
\(1\) The teacher encodes one solution, not correctness\.OurPI Bias Score, a teacher\-minus\-student log\-probability ratio needing only forward passes, reaches0\.520\.52on the in\-context solution against≤0\.02\{\\leq\}0\.02on a*different correct*solution to the same problem\. This shows that the teacher barely separates from a solution to an unrelated problem\. What the privileged information transfers is a specific trajectory instead of correctness\.
\(2\) A target built from one trajectory says nothing about the trajectory correctness\.Split by verifier outcome, per\-token loss \(≈3\.5×10−4\{\\approx\}3\.5\\times 10^\{\-4\}\) and KL \(≈0\.03\{\\approx\}0\.03\) overlap for correct and incorrect rollouts at every step\. When they differ, it runs backwards, the teacher–student gap settles at0\.70\.7on correct rollouts against0\.450\.45on incorrect ones\. This indicates that the learning pressure is highest where the student is already right, so the loss falls without accuracy following\.
\(3\) Density spreads that signal over tokens that do not decide the answer\.Stopwords, uncertainty markers, punctuation and whitespace absorb55\.4%55\.4\\%of the per\-token loss, while the content words, numbers and math symbols that fix the answer absorb comparatively little\.
\(4\) Where the loss is high, it penalizes exploration\.Within*correct*rollouts, positions off the reference path carry KL out to≈0\.31\{\\approx\}0\.31against≈0\.08\{\\approx\}0\.08on\-path, a factor of four\. Having read the solution, the teacher cannot tell a productive detour from a mistake, so the search that reasoning requires is penalized shortening the model responses\.
\(5\) The student flattens rather than sharpens\.The teacher–student gap closes from≈3\.1\{\\approx\}3\.1to≈0\{\\approx\}0, but its±1\\pm 1standard\-deviation band does not contract and student entropy rises\. The endpoint is a flatter, earlier\-committing policy that solves no more problems\. This becomes the training signature of Figure[2](https://arxiv.org/html/2608.04794#S5.F2)\.
Prior work saw fragments of this from opposite ends: RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2608.04794#bib.bib4)\)derives a conditional mutual\-informationleakagebound but stops at theory, whileKimet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib3)\)document degraded reasoning but leave its origin open; we show these are two ends of one mechanism, on difficult tasks where the method is actually used\.
Our contributions are:\(1\)evidence that SD, effective on easy tasks, fails as a lone objective on difficult reasoning, math, coding, and agentic tasks, holding across response length, task difficulty, model scale, reasoning mode, PI form, and both recipes \(Sections[5](https://arxiv.org/html/2608.04794#S5),[7](https://arxiv.org/html/2608.04794#S7)\);\(2\)a causal chain of token\- and distribution\-level measurements explaining the failure \(Section[6](https://arxiv.org/html/2608.04794#S6)\); and\(3\)thePI Bias Score, which measures how strongly a PI\-conditioned teacher favors the in\-context solution over other correct ones from forward passes alone\. These reframe SD as an underspecified family of objectives rather than a method that does not work: the density that motivates it is intact; instead the target fails to encode correctness\.
## 2Background and Related Work
SD sits between two signals it tries to combine: RLVR gives a coarse trajectory\-level signal, while distillation gives a dense per\-token signal but needs an external teacher\. RLVR trainsπθ\\pi\_\{\\theta\}to maximize a binary rewardR\(x,y\)R\(x,y\)checking whether responseyyto promptxxis correct\(Lambertet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib11)\), and GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib8)\)estimates a group\-relative advantage fromGGrollouts per prompt with a PPO\-style clipped update; every token receives the same advantage, and it vanishes when all rollouts in a group share a reward\.
Self\-distillation: SDPO and OPSD\.SD replaces the reward with PI, side information about the answer available at training but not at test time; we use “PI” as an umbrella term for the feedbackffof SDPO\(Hübotteret al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib1)\), the reference solutiony⋆y^\{\\star\}of OPSD\(Zhaoet al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib2)\)and RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2608.04794#bib.bib4)\), and the conditioning contextccofKimet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib3)\)\. Training minimizes a stop\-gradient per\-token divergence between teacher and student along a student rollout \(Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\)\), so no external model or reward is required\. SDPO uses a per\-token KL to a teacher conditioned onff, ranging from a prior successful rollout to environment output such as failed unit tests and stabilized by an EMA or trust\-region teacher; it matches or exceeds GRPO at several\-fold lower compute on SciKnowEval, ToolAlpaca and LiveCodeBench\. OPSD uses a forwardKL\(pT∥pS\)\\mathrm\{KL\}\(p\_\{T\}\\\|p\_\{S\}\)with the teacher conditioned ony⋆y^\{\\star\}and fixed to the initial policy, observes that a few stylistic tokens carry far higher divergence than answer\-determining ones, casts the update as a dense token\-level rather than sequence\-level policy gradient, and reports gains over GRPO and off\-policy distillation on AIME\(Mathematical Association of America,[2024](https://arxiv.org/html/2608.04794#bib.bib19),[2025](https://arxiv.org/html/2608.04794#bib.bib20)\)and HMMT\(Dekonincket al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib28)\)\.
Critiques and augmentations\.RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2608.04794#bib.bib4)\)proves the objective decomposes asℒOPSD=ℒ∗\+I\(Yt;R∣X,Y<t\)\\mathcal\{L\}\_\{\\mathrm\{OPSD\}\}=\\mathcal\{L\}^\{\*\}\+I\(Y\_\{t\};R\\mid X,Y\_\{<t\}\), an irreducible conditional mutual information: the teacher fits a single PI\-conditioned trace rather than the marginal over valid solutions, so a teacher favoring one phrasing trains the student to adopt it; RLSD’s fix uses the teacher only to reweight a verifier\-driven advantage\.Kimet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib3)\)report that richer teacher conditioning makes reasoning more concise and confident, speeding in\-domain optimization but degrading out\-of\-distribution and on harder problems by up to∼\\sim40% across the Qwen3 and DeepSeek\-R1\-Distill families\. A further line keeps a verifiable reward primary and enters SD as a small\-weight auxiliary regularizer\(Wanget al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib5); Luet al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib6); Yanget al\.,[2026b](https://arxiv.org/html/2608.04794#bib.bib7)\)in multi\-turn and agentic settings\. Across all of this work SD as a*lone*objective on difficult tasks is never tested; that is the gap we take up, tying the leakage and uncertainty\-suppression accounts to one set of measured training runs\.
\(a\)Biology: loss
\(b\)Biology: accuracy
\(c\)Physics: loss
\(d\)Physics: accuracy
Figure 1:Reproducing SD in its original regime\.Qwen3\-8B on SciKnowEval with whole\-solution PI\. Loss decreases and validation*rises*on Biology and Physics, matching the reported SDPO trend\.
## 3The SD Objective and Two Sources of Bias
A policyπθ\\pi\_\{\\theta\}generatesy=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\)for a promptxxautoregressively\. The student is conditioned on the prompt alone and the self\-teacher is the same model additionally conditioned on the PIrr:pS\(⋅∣x,y<t\)=πθ\(⋅∣x,y<t\)p\_\{S\}\(\\cdot\\mid x,y\_\{<t\}\)=\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<t\}\)andpT\(⋅∣x,y<t\)=πθ′\(⋅∣x,r,y<t\)p\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\)=\\pi\_\{\\theta^\{\\prime\}\}\(\\cdot\\mid x,r,y\_\{<t\}\), whereθ′\\theta^\{\\prime\}is the student’s own parameters or a slow copy that tracks them\. Conditioning onrrmakespTp\_\{T\}a stronger next\-token predictor of a correct continuation; the student, which never seesrr, is trained to match it along a rollouty^∼pS\\hat\{y\}\\sim p\_\{S\}from the student itself:
ℒSD\(θ\)=𝔼\(x,r\)𝔼y^∼sg\(pS\)\[1\|y^\|∑t=1\|y^\|D\(sg\(pT\(⋅∣x,r,y^<t\)\)∥pS\(⋅∣x,y^<t\)\)\],\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SD\}\}\(\\theta\)=\\mathbb\{E\}\_\{\(x,r\)\}\\,\\mathbb\{E\}\_\{\\hat\{y\}\\sim\\mathrm\{sg\}\(p\_\{S\}\)\}\\\!\\left\[\\frac\{1\}\{\|\\hat\{y\}\|\}\\sum\_\{t=1\}^\{\|\\hat\{y\}\|\}D\\\!\\big\(\\mathrm\{sg\}\(p\_\{T\}\(\\cdot\\mid x,r,\\hat\{y\}\_\{<t\}\)\)\\,\\big\\\|\\,p\_\{S\}\(\\cdot\\mid x,\\hat\{y\}\_\{<t\}\)\\big\)\\right\],
\(1\)wheresg\\mathrm\{sg\}is stop\-gradient, so gradients flow only through the student\. SDPO and OPSD instantiateDDdifferently; we study Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\) as alone objective, with no reward term\. Two of its properties drive everything that follows\.
Property 1: a dense per\-token target\.Unlike a trajectory\-level reward, Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\) imposes a target at every position; both SDPO and OPSD analyze this as a dense token\-level policy gradient \(SDPO via its per\-token KL, their Proposition 2\.1, and OPSD by contrast with sequence\-level self taught reasoning\)\. Density is the design’s advantage, but it also places punctuation, stopwords, and style\-carrying discourse markers on equal footing with content\-bearing tokens\.
Property 2: a PI\-conditioned target\.With a whole reference solutionr=y⋆r=y^\{\\star\}the target reflects one particular solution, in both the answer it reaches and how it is phrased and ordered; matching it token by token asks the student to reproduce that behavior everywhere\. RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2608.04794#bib.bib4)\)formalizes the cost as an irreducibleI\(Yt;R∣X,Y<t\)I\(Y\_\{t\};R\\mid X,Y\_\{<t\}\), measuring how much the target depends on the PI beyond what correctness requires\. A dense loss aimed at one solution’s surface form therefore spreads learning signal across all tokens rather than the decisions that make an answer correct\. We explore this further link by link in Section[6](https://arxiv.org/html/2608.04794#S6)\.
## 4Experimental Setup
#### Domains and data\.
Prior work evaluates SD on short, knowledge\-recall tasks\. To test the objective where it is meant to be used, we assemble four harder domains: general QA uses MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib14)\), extending SDPO’s SciKnowEval format to ten options and reasoning over recall; mathematics uses the de\-duplicated DAPO\-Math\-17k\(Yuet al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib17); Open R1 Team,[2025b](https://arxiv.org/html/2608.04794#bib.bib18)\), whose spread of response lengths separates trace length from difficulty \(Section[7](https://arxiv.org/html/2608.04794#S7)\); coding uses CodeForces\(Open R1 Team,[2025a](https://arxiv.org/html/2608.04794#bib.bib22)\), demanding an algorithm rather than a short completion; and the agentic setting uses BFCL\(Patilet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib27)\), requiring state across turns and composed tool calls\. We train on 2,000 examples for general QA, math, and coding, and on the BFCL multiturn split, evaluating every domainin\-domainand onheld\-out transfer benchmarksfrom different sources, so a gain cannot come from fitting the training distribution\. Transfer benchmarks are SciKnowEval\(Fenget al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib16)\)and GPQA\-D\(Reinet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib15)\); AIME24\(Mathematical Association of America,[2024](https://arxiv.org/html/2608.04794#bib.bib19)\), AIME25\(Mathematical Association of America,[2025](https://arxiv.org/html/2608.04794#bib.bib20)\), Olympiad Bench\(Heet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib21)\); MBPP\+\(Austinet al\.,[2021](https://arxiv.org/html/2608.04794#bib.bib24)\), HumanEval\+\(Chenet al\.,[2021](https://arxiv.org/html/2608.04794#bib.bib25)\), CodeElo\(Quanet al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib26)\), LCBv6\(Jainet al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib29)\); and BFCLv4 multiturn\(Patilet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib27)\)\. Data details are in Appendix[A](https://arxiv.org/html/2608.04794#A1), prompts and templates in Appendix[D](https://arxiv.org/html/2608.04794#A4)\.
#### Models and privileged information\.
Our primary model is Qwen3\-8B\(Yanget al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib30)\)in both its think \(long chain\-of\-thought\) and instruct variants, so every domain is studied under both reasoning modes; for a size comparison we additionally train Qwen3\-32B on general QA\. Our default PI is the whole reference solutionr=y⋆r=y^\{\\star\}: followingHübotteret al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib1)\)we sample multiple rollouts per query and take one random correct rollout\. Section[6\.1](https://arxiv.org/html/2608.04794#S6.SS1.SSS0.Px3)additionally studies short hints \(one to two sentences\) and skills \(≈\\approx500\-token structured guides of relevant techniques\)\.
#### Objective, divergence, and training\.
We train with the lone objective of Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\), no reward or verifier in the loss, instantiatingDDas the symmetric Jensen–Shannon divergence,JSD\(p∥q\)=12KL\(p∥m\)\+12KL\(q∥m\)\\mathrm\{JSD\}\(p\\,\\\|\\,q\)=\\tfrac\{1\}\{2\}\\mathrm\{KL\}\(p\\,\\\|\\,m\)\+\\tfrac\{1\}\{2\}\\mathrm\{KL\}\(q\\,\\\|\\,m\)withm=12\(p\+q\)m=\\tfrac\{1\}\{2\}\(p\+q\), matching SDPO’s implementation; the OPSD variant is reported in Section[5\.2](https://arxiv.org/html/2608.04794#S5.SS2)\. Two ingredients stabilize training on harder data, both motivated by the loss concentration of Section[6\.3](https://arxiv.org/html/2608.04794#S6.SS3): aggressive per\-token clipping of the divergence\(Zhaoet al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib2)\), preventing a few high\-divergence stylistic tokens from dominating the gradient, and a slow EMA teacherθ′←\(1−α\)θ′\+αθ\\theta^\{\\prime\}\\leftarrow\(1\-\\alpha\)\\theta^\{\\prime\}\+\\alpha\\thetawithα=0\.001\\alpha\{=\}0\.001\(vs\.0\.010\.01in SDPO and0\.050\.05inKimet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib3)\)\), which removed the late\-training instabilities we first observed\. Hyperparameters: JSD clippingτ=0\.001\\tau\{=\}0\.001, learning rate1e−51\\mathrm\{e\}\{\-5\}, batch size3232,33epochs \(Appendix[B](https://arxiv.org/html/2608.04794#A2)\)\. We report task accuracy \(math, QA\), fraction of test cases passed \(coding\), and the BFCL success metric \(agentic\), each averaged over 4 sampled val trajectories per task at each validation step, and track the per\-token SD loss, validation accuracy, student entropyℍ\[pS\(⋅∣x,y^<t\)\]\\mathbb\{H\}\[p\_\{S\}\(\\cdot\\mid x,\\hat\{y\}\_\{<t\}\)\], and average response length, plus teacher–student KL and teacher perplexity for Section[6\.5](https://arxiv.org/html/2608.04794#S6.SS5)\.
#### Reproducing SD in its original regime\.
Using whole\-solution PI, we train Qwen3\-8B with Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\) on SciKnowEval with SDPO’s codebase\(Hübotteret al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib1)\)and reported hyperparameters, and recover the qualitative SDPO result: on Biology and Physics the loss decreases while validation accuracy rises and responses shorten \(Figure[1](https://arxiv.org/html/2608.04794#S2.F1)\), confirming that implementation is faithful\. We further add on to the SDPO codebase for all our experiments\. Our code is available as part of the supplementary material\.
## 5The Training Signature of Lone SD
Applying the same implementation to the four domains produces a consistent training signature: the loss is optimized, and nothing that determines task success improves\. Harder data also changes where the loss lives\. Unclipped, the top 50% of tokens consume 98\.5% of the loss on MMLU\-Pro, mirroring the skew OPSD\(Zhaoet al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib2)\)reports on math: a few high\-divergence tokens dominate the gradient, so the average falls while little signal reaches content tokens\. This is what the clipping and slow teacher of Section[4](https://arxiv.org/html/2608.04794#S4)address\.
### 5\.1Patterns observed during training
Figure[2](https://arxiv.org/html/2608.04794#S5.F2)shows the training dynamics for Qwen3\-8B, and the same pattern recurs across settings\.The loss is optimized:the per\-token JS divergence of Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\) decreases steadily, indicating closer matching to the teacher\.Validation accuracy does not follow:it remains flat or degrades, so optimization is decoupled from task performance\.Student entropy rises:ℍ\[pS\(⋅∣x,y^<t\)\]\\mathbb\{H\}\[p\_\{S\}\(\\cdot\\mid x,\\hat\{y\}\_\{<t\}\)\]increases, contrary to the sharpening expected of an improving model \(Section[6\.5](https://arxiv.org/html/2608.04794#S6.SS5)\)\.Response length falls:while desirable for short\-form generation, on reasoning tasks this is also consistent with reduced exploration before committing \(Section[6\.4](https://arxiv.org/html/2608.04794#S6.SS4)\)\. Any single trend admits a benign interpretation; together they indicate a systematic failure mode\. The pattern is consistent across all four domains and both reasoning modes\. The only exception is agentic instruct mode, where the small training set and short responses \(300 tokens\) yield a modest validation improvement\. Because the domains differ substantially in response length, credit structure, and reference\-solution format, the shared behavior is most naturally attributed to the objective rather than any particular dataset\. Training curves for all domains are provided in Figure[2](https://arxiv.org/html/2608.04794#S5.F2)\.
Figure[2](https://arxiv.org/html/2608.04794#S5.F2)shows the per\-domain training dynamics for self\-distillation used as the sole training signal on Qwen3\-8B\. Each row corresponds to one domain, and each column reports one diagnostic over training: the per\-token loss, the validation accuracy, the student entropy, and the mean response length\. Across all four domains the per\-token loss decreases while the validation accuracy does not improve, and the entropy and response length move in the directions described in the main paper\.
Figure 2:Per\-domain training dynamics for lone self\-distillation on Qwen3\-8B\. Rows, from top to bottom, correspond to general QA \(MMLU\-Pro\), mathematics \(DAPO\-Math\), coding \(CodeForces\), and multi\-turn agentic tool use \(BFCL\)\. Columns, from left to right, report the per\-token loss, the validation accuracy, the student entropy, and the mean response length over training\. Across all four domains the loss decreases while the validation accuracy does not improve\. For BFCL the second column reports the validation score rather than the accuracy; the score equals twice the accuracy\.Table 1:SD as a lone objective does not improve validation performance across various domains\.PositiveΔ\\Deltain green, negative in red; averages over four independent rollouts\.ThinkInstructDomainBenchmarkBaseSD𝚫\\mathbf\{\\Delta\}BaseSD𝚫\\mathbf\{\\Delta\}GeneralQAMMLU\-Pro70\.2±\\pm2\.269\.8±\\pm2\.2−0\.4\-0\.469\.5±\\pm2\.167\.5±\\pm2\.3−2\-2GPQA\-D59\.5±\\pm3\.057\.6±\\pm3\.2−1\.9\-1\.949\.2±\\pm2\.746\.0±\\pm2\.8−3\.2\-3\.2Bio \(SKE\)38\.5±\\pm3\.436\.7±\\pm3\.6−1\.8\-1\.833\.5±\\pm3\.532\.3±\\pm3\.4−1\.2\-1\.2Chem \(SKE\)51\.9±\\pm2\.051\.9±\\pm2\.0044\.6±\\pm1\.841\.5±\\pm2\.0−3\.1\-3\.1Mat\. \(SKE\)68\.0±\\pm3\.264\.6±\\pm3\.1−3\.4\-3\.454\.7±\\pm3\.154\.9±\\pm3\.40\.20\.2Phys\. \(SKE\)77\.5±\\pm2\.875\.6±\\pm3\.0−1\.9\-1\.964\.2±\\pm3\.264\.1±\\pm3\.3−0\.1\-0\.1Average60\.9360\.9359\.3759\.37−1\.56\-1\.5652\.6252\.6251\.0551\.05−1\.57\-1\.57MathDAPO79\.1±\\pm2\.080\.1±\\pm2\.01148\.8±\\pm2\.244\.8±\\pm2\.4−4\-4AIME2467\.5±\\pm7\.766\.0±\\pm7\.8−1\.5\-1\.524\.2±\\pm7\.023\.6±\\pm7\.0−0\.6\-0\.6AIME2553\.3±\\pm7\.452\.9±\\pm7\.4−0\.4\-0\.420\.0±\\pm6\.217\.2±\\pm6\.1−2\.8\-2\.8Olympiad53\.6±\\pm1\.953\.4±\\pm1\.8−0\.2\-0\.246\.3±\\pm1\.746\.5±\\pm1\.70\.20\.2Average63\.3863\.3863\.163\.1−0\.28\-0\.2834\.8234\.8233\.0233\.02−1\.8\-1\.8CodingCodeForces24\.5±\\pm1\.625\.0±\\pm1\.80\.50\.517\.3±\\pm1\.316\.4±\\pm1\.3−0\.9\-0\.9MBPP\+59\.3±\\pm2\.353\.1±\\pm2\.1−6\.2\-6\.243\.2±\\pm2\.055\.8±\\pm2\.312\.612\.6HumanEval\+80\.5±\\pm3\.180\.4±\\pm2\.9−0\.1\-0\.177\.4±\\pm2\.873\.6±\\pm3\.1−3\.8\-3\.8CodeElo23\.8±\\pm1\.525\.1±\\pm1\.91\.31\.313\.8±\\pm1\.112\.1±\\pm1\.3−1\.7\-1\.7LCBv650\.0±\\pm3\.953\.0±\\pm3\.73352\.3±\\pm2\.850\.8±\\pm3\.2−1\.5\-1\.5Average47\.6247\.6247\.3247\.32−0\.3\-0\.340\.840\.841\.7441\.740\.940\.94AgenticBase V386\.0±\\pm3\.283\.2±\\pm4\.6−2\.8\-2\.868\.0±\\pm4\.672\.5±\\pm4\.24\.54\.5Base V451\.3±\\pm2\.948\.8±\\pm2\.9−2\.5\-2\.523\.5±\\pm2\.831\.0±\\pm3\.27\.57\.5Miss Func51\.0±\\pm3\.044\.0±\\pm2\.9−7\-719\.5±\\pm2\.421\.5±\\pm2\.722Miss Param33\.0±\\pm2\.931\.36±\\pm2\.7−1\.64\-1\.6417\.0±\\pm2\.319\.88±\\pm2\.72\.882\.88Long Ctx35\.5±\\pm2\.931\.88±\\pm2\.7−3\.62\-3\.6216\.5±\\pm2\.523\.0±\\pm2\.96\.56\.5Average51\.3651\.3647\.8547\.85−3\.51\-3\.5128\.928\.933\.5833\.584\.684\.68
### 5\.2End\-task results
No improvement where reasoning is required\.Table[1](https://arxiv.org/html/2608.04794#S5.T1)evaluates each trained model against its own base, in\-domain and on transfer\. Across the reasoning\-heavy domains, in both modes, the objective produces no improvement, and degradation is largest where reasoning demand is highest: the think variant loses3\.513\.51points on average in the agentic domain, including77on BFCL Multi\-Turn Missing Function V4, and both modes lose1\.61\.6across general QA\. In\-domain and transfer move together, consistent with the objective failing to install a capability rather than installing a narrow one\.
Positive averages appear only under the instruct variant\.Two domain averages come out positive, neither a gain\. The instruct variant improves on the agentic BFCL benchmarks, but BFCL trains on a much smaller split \(100 tasks\) with short instruct rollouts \(mean 300 tokens\)\. In coding, the instruct gain on MBPP\+ \(\+12\.6\+12\.6\) is offset by losses on HumanEval\+ \(−3\.8\-3\.8\), CodeElo \(−1\.7\-1\.7\), LCBv6 \(−1\.5\-1\.5\) and CodeForces \(−0\.9\-0\.9\), so performance still declines averaged over benchmarks and modes\. Both cases are confined to the instruct model, where shorter responses limit the reach of the causal chain; under the think, flat or declining is the norm\.
The result is not specific to SDPO\.We repeat the evaluation under the OPSD recipe, a single rollout per step, fixed correct answers, and an initial\-policy teacher\. Averages change by−2\.0\-2\.0and−0\.7\-0\.7points on general QA and−4\.3\-4\.3and−0\.4\-0\.4on math \(think and instruct\), with 18 of 20 benchmark–mode cells non\-positive and the same signature throughout \(full table in Table[2](https://arxiv.org/html/2608.04794#S5.T2)\)\. Since the recipes differ in divergence direction, teacher schedule, rollout count, and clipping, the behavior tracks the shared structure of Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\) rather than one method’s implementation choices\.
Table 2:The same non\-improvement pattern holds under the OPSD recipe\.Results on General QA and Mathematics using the OPSD training recipe\. PositiveΔ\\Deltavalues are shown in green and negative values in red\. All values are averaged over four independent rollouts\.ThinkInstructDomainBenchmarkBaseSD𝚫\\mathbf\{\\Delta\}BaseSD𝚫\\mathbf\{\\Delta\}GeneralQAMMLU\-Pro70\.2±\\pm2\.270\.1±\\pm2\.2−0\.1\-0\.169\.5±\\pm2\.169\.1±\\pm2\.3−0\.4\-0\.4GPQA\-D59\.5±\\pm3\.057\.1±\\pm3\.0−2\.4\-2\.449\.2±\\pm2\.746\.6±\\pm2\.8−2\.6\-2\.6Bio \(SKE\)38\.5±\\pm3\.435\.1±\\pm3\.5−3\.4\-3\.433\.5±\\pm3\.532\.6±\\pm3\.4−0\.9\-0\.9Chem \(SKE\)51\.9±\\pm2\.051\.2±\\pm2\.0−0\.7\-0\.744\.6±\\pm1\.843\.0±\\pm2\.0−1\.6\-1\.6Mat\. \(SKE\)68\.0±\\pm3\.264\.2±\\pm3\.1−3\.8\-3\.854\.7±\\pm3\.156\.6±\\pm3\.41\.91\.9Phys\. \(SKE\)77\.5±\\pm2\.876\.1±\\pm3\.0−1\.4\-1\.464\.2±\\pm3\.263\.9±\\pm3\.3−0\.3\-0\.3Average60\.9360\.9358\.9758\.97−1\.96\-1\.9652\.6252\.6251\.9751\.97−0\.65\-0\.65MathDAPO79\.1±\\pm2\.076\.0±\\pm1\.9−3\.1\-3\.148\.8±\\pm2\.246\.8±\\pm2\.4−2\-2AIME2467\.5±\\pm7\.762\.2±\\pm7\.3−5\.3\-5\.324\.2±\\pm7\.023\.7±\\pm7\.0−0\.5\-0\.5AIME2553\.3±\\pm7\.448\.2±\\pm8\.2−5\.1\-5\.120\.0±\\pm6\.218\.6±\\pm6\.1−1\.4\-1\.4Olympiad53\.6±\\pm1\.950\.0±\\pm1\.8−3\.6\-3\.646\.3±\\pm1\.748\.4±\\pm1\.92\.12\.1Average63\.3863\.3859\.159\.1−4\.28\-4\.2834\.8234\.8234\.3834\.38−0\.44\-0\.44
#### What the signature rules out\.
Three explanations for falling loss and flat accuracy are ruled out above\.Optimization failure:the loss decreases smoothly under the stabilization of Section[4](https://arxiv.org/html/2608.04794#S4)\.Implementation error:the same code, teacher, and clipping reproduce the reported gains on SciKnowEval \(Figure[1](https://arxiv.org/html/2608.04794#S2.F1)\); what changes is task difficulty\. What remains is the objective itself\.
## 6A Mechanistic Account: From Privileged Information to a Misguided Student
We now measure each link of the predicted chain in causal order, beginning with the one we can intervene on: the bias is a property of the PI, and changing the PI’s specificity moves it as predicted, so the chain does not rest on correlation alone\. All measurements use Qwen3\-8B with whole\-solution PI; a rollout iscorrectif the student solved the problem\.
### 6\.1PI bias: the teacher targets one solution, not correctness
#### An instrument for the bias\.
To measure what the teacher moves the student toward, fix a rollout prefix\(x,y<t\)\(x,y\_\{<t\}\)and a target sequencew=\(w1,…,wK\)w=\(w\_\{1\},\\dots,w\_\{K\}\)\. The student and teacher scores are the average log\-probability each assigns toww, and thePI Bias Scoreis their difference:
PSt\(w\)=stT\(w\)−stS\(w\)=1K∑k=1KlogpT\(wk∣x,y⋆,y<t\)pS\(wk∣x,y<t\)\.\\mathrm\{PS\}\_\{t\}\(w\)=s^\{\\mathrm\{T\}\}\_\{t\}\(w\)\-s^\{\\mathrm\{S\}\}\_\{t\}\(w\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\log\\frac\{p\_\{T\}\(w\_\{k\}\\mid x,y^\{\\star\},y\_\{<t\}\)\}\{p\_\{S\}\(w\_\{k\}\\mid x,y\_\{<t\}\)\}\.\(2\)This discounts what the student already assigns toww, reporting only how much the PI moves mass toward it \(Appendix[E](https://arxiv.org/html/2608.04794#A5)\)\.
#### A design that separates correctness from imitation\.
We evaluate Eq\. \([2](https://arxiv.org/html/2608.04794#S6.E2)\) for four targets: the in\-context solutiony⋆y^\{\\star\}, a*different correct*solutiony′y^\{\\prime\}to the same query, a solutiony~\\tilde\{y\}to an unrelated problem, and an incorrect solutiony−y^\{\-\}, writtenPS⋆\\mathrm\{PS\}^\{\\star\},PS′\\mathrm\{PS\}^\{\\prime\},PS∼\\mathrm\{PS\}^\{\\sim\},PS−\\mathrm\{PS\}^\{\-\}\. The contrastPS⋆\\mathrm\{PS\}^\{\\star\}vs\.PS′\\mathrm\{PS\}^\{\\prime\}discriminates: a teacher encoding*correctness*would givePS⋆≈PS′\\mathrm\{PS\}^\{\\star\}\\approx\\mathrm\{PS\}^\{\\prime\}, one encoding*the trajectory it was shown*would spike onPS⋆\\mathrm\{PS\}^\{\\star\}alone\. We observe the second, in every domain \(Figure[3](https://arxiv.org/html/2608.04794#S6.F3)\):PS⋆≫PS′≈PS∼\>PS−\\mathrm\{PS\}^\{\\star\}\\gg\\mathrm\{PS\}^\{\\prime\}\\approx\\mathrm\{PS\}^\{\\sim\}\>\\mathrm\{PS\}^\{\-\}, the teacher barely separating a different correct solution from one to an unrelated problem\.The teacher does not encode correctness; it encodes one solution\.This is the empirical counterpart of the bound ofYanget al\.\([2026a](https://arxiv.org/html/2608.04794#bib.bib4)\), turning their irreducibleI\(Yt;R∣X,Y<t\)I\(Y\_\{t\};R\\mid X,Y\_\{<t\}\)into a measurable quantity; the gapPS⋆−PS′\\mathrm\{PS\}^\{\\star\}\-\\mathrm\{PS\}^\{\\prime\}persists throughout training\.
Figure 3:The teacher targets one solution, not correctness\.PI bias score \(Eq\. \([2](https://arxiv.org/html/2608.04794#S6.E2)\)\) per domain, whole\-solution PI; taller bar==more mass moved toward that target\. OnlyPS⋆\\mathrm\{PS\}^\{\\star\}, the in\-context solution, is tall; a different*correct*solutionPS′\\mathrm\{PS\}^\{\\prime\}scores like an unrelated onePS∼\\mathrm\{PS\}^\{\\sim\}, where a correctness\-encoding teacher would givePS⋆≈PS′\\mathrm\{PS\}^\{\\star\}\\approx\\mathrm\{PS\}^\{\\prime\}\.
#### Weaker PI gives a weaker signal, not a better one\.
If over\-specificity were the whole problem, a more general PI should repair it\. On DAPO\-Math we replace the whole solution with the two weaker forms of Section[4](https://arxiv.org/html/2608.04794#S4), shorthintsand detailedskills\. The bias moves as predicted, but not usefully: under both, the score collapses for every target, withPS⋆\\mathrm\{PS\}^\{\\star\},PS′\\mathrm\{PS\}^\{\\prime\},PS−\\mathrm\{PS\}^\{\-\}andPS∼\\mathrm\{PS\}^\{\\sim\}close together\. For hints as PI, the PS\* is 0\.0015, PS\- is 0\.0192 and PSĩs \-0\.0578, for skills as PI, the PS\* is \-0\.0021, PS\- is \-0\.0108 and PSĩs 0\.0071\. Note that for hints and skills, PS\* and PS’ do not differ in definition\. The teacher no longer singles out the in\-context solution, but now moves little mass toward*any*target\. Performance follows \(Table[3](https://arxiv.org/html/2608.04794#S6.T3)\): against a base average of63\.463\.4, hints lose5\.15\.1points and skills3\.53\.5, while the whole solution stays within half a point of base\. The two regimes fail for opposite reasons, which is the sense in which the bias is causal rather than incidental: PI specificity controls the bias and neither end of the dial produces learning, a whole solution tying the target to one trajectory while hints and skills are too weak to move the student\. Examples of hints, skills and how we construct them are detailed in Appendix[C](https://arxiv.org/html/2608.04794#A3)\.
Table 3:Neither over\-specific nor under\-specific PI produces learning\.SD on DAPO\-Math \(Qwen3\-8B think\) under three forms of privileged information; hints and skills reduce PI bias \(Figure[3](https://arxiv.org/html/2608.04794#S6.F3)\) but weaken the signal\.\(a\)Mean per\-token loss: curves overlap near3\.5×10−43\.5\\times 10^\{\-4\}\.
\(b\)Mean KL: curves overlap near0\.030\.03\.
\(c\)LP gap: correct∼0\.7\{\\sim\}0\.7, incorrect∼0\.45\{\\sim\}0\.45\.
\(d\)Loss by token type
\(e\)Off\- vs\. on\-path KL
\(f\)Signed LP gap
Figure 4:The causal chain\.\(a–c\) The loss does not track correctness: for correct \(green\) vs\. incorrect \(red\) rollouts, loss \(a\) and KL \(b\) overlap throughout training, while the absolute LP gap \(c\) separates them in the*opposite*direction, staying higher on rollouts the student already solves\.\(d\)The loss budget falls mostly on low\-information tokens rather than the content words, numbers and math symbols that determine the answer\.\(e\)Even within correct rollouts, off\-path exploratory tokens incur much larger divergence than on\-path ones\.\(f\)The average signed teacher–student gap shrinks to zero while its variance persists, indicating cancellation rather than convergence\.
### 6\.2The loss does not track correctness
A target defined by one trajectory carries no information about whether the student’s own trajectory reaches the correct answer, so Eq\.[1](https://arxiv.org/html/2608.04794#S3.E1)predicts a loss blind to correctness\. An objective that taught problem\-solving should assign greater loss to failures\. It does not\. Mean per\-token loss settles near3\.5×10−43\.5\\times 10^\{\-4\}and mean KL near0\.030\.03for both correct and incorrect rollouts, with overlapping four\-rollout bands throughout training \(Figures[4\(a\)](https://arxiv.org/html/2608.04794#S6.F4.sf1),[4\(b\)](https://arxiv.org/html/2608.04794#S6.F4.sf2)\); if anything, incorrect rollouts exhibit slightly*lower*KL\. Thus, at no scale does the objective concentrate effort where the student goes wrong\. Where the groups do separate, the direction is reversed: the*absolute*log\-probability gap\|logpT−logpS\|\|\\log p\_\{T\}\-\\log p\_\{S\}\|falls from roughly33to below11within ten steps, then stabilizes near0\.70\.7on correct rollouts versus0\.450\.45on incorrect ones \(Figure[4\(c\)](https://arxiv.org/html/2608.04794#S6.F4.sf3)\)\.The learning signal is strongest on rollouts the student already solves\.This directly explains the decoupling in Section[5](https://arxiv.org/html/2608.04794#S5): a loss that cannot distinguish success from failure can decrease without improving accuracy\.
### 6\.3The loss lands on uninformative tokens
Per\-token density predicts which tokens absorb the loss Section[5](https://arxiv.org/html/2608.04794#S5)showed to be concentrated\. Labelling every token by one of nine mutually exclusive types \(special token, whitespace, punctuation, uncertainty marker, stopword, number, math symbol, content word, or other\) and attributing each step’s loss accordingly, the tokens absorbing the most are predominantly low\-information: in a representative step, stopwords, uncertainty markers \(“wait,” “maybe,” “perhaps”\), punctuation and whitespace account for55\.38%of the total per\-token loss, while the content words, numbers and math symbols that determine the answer absorb comparatively little \(Figure[4\(d\)](https://arxiv.org/html/2608.04794#S6.F4.sf4)\); the ordering is stable across training\. Each token is assigned the first category it matches in the order listed above, with the uncertainty\-marker set taken fromZhaoet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib2)\)and a standard English stopword list\.The objective’s optimization capacity is spent primarily on tokens that do not determine correctness\.This unifies two prior observations:Zhaoet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib2)\)that stylistic tokens carry higher divergence than mathematical ones,Kimet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib3)\)that SD shifts epistemic\-marker probabilities by an order of magnitude more than the average token\. Both are overloaded into one loss budget in which uninformative tokens dominate throughout training, not only at initialization\.
#### Average per\-token loss by token type\.
Figure[5](https://arxiv.org/html/2608.04794#S6.F5)reports the average per\-token distillation loss for each category, computed as the summed per\-token loss of a category’s tokens divided by the number of those tokens\. Dividing by the token count rather than summing the loss mass means that a category cannot rank highly merely by being numerous\. Uncertainty markers carry the highest average per\-token loss, followed by stopwords and content words, while numbers and special tokens carry the lowest\. Because of this, even when the number of uncertainty markers and punctuations are less, their total loss share is high\.
Figure 5:Average per\-token distillation loss for each token category\. Bars are ordered by decreasing average loss and whiskers denote±1\\pm 1SEM\. Because the loss is divided by the number of tokens in each category, a category cannot rank highly merely by being numerous\.
### 6\.4The objective penalizes exploration
A reasoning model explores several partial paths before committing, so parts of even a correct rollout are legitimately off\-path, and a teacher conditioned ony⋆y^\{\\star\}cannot recognize such a detour as productive\. Ranking rollout positions by the student scorestS\(y⋆\)s^\{\\mathrm\{S\}\}\_\{t\}\(y^\{\\star\}\), we call the top quartile*on\-path*and the bottom quartile*off\-path*and compare their full KL distributions, restricting first to correct rollouts so that off\-path cannot be conflated with wrong\. Within correct rollouts, off\-path KL extends to≈0\.31\{\\approx\}0\.31against≈0\.08\{\\approx\}0\.08on\-path, a factor of four \(Figure[4\(e\)](https://arxiv.org/html/2608.04794#S6.F4.sf5)\)\. Whereas, within incorrect rollouts the asymmetry widens,≈0\.47\{\\approx\}0\.47against≈0\.10\{\\approx\}0\.10\. Because the teacher has ready⋆y^\{\\star\}, a detour on the way to a correct answer is indistinguishable from a mistake, and these positions absorb the largest updates\.Within rollouts the student gets right, the objective penalizes exploration most, treating the search that reasoning requires as error\.This supplies the mechanism behind an effectKimet al\.\([2026](https://arxiv.org/html/2608.04794#bib.bib3)\)report only as a phenomenon, and it is why responses shorten: a model penalized for deliberating commits sooner\.
### 6\.5The student flattens rather than sharpens
The accumulated pressure leaves a trace on the student’s distribution\. The mean teacher–student log\-probability gap falls from≈3\.1\{\\approx\}3\.1at initialization, undershoots to≈−0\.65\{\\approx\}\-0\.65around step 15, and settles at≈0\{\\approx\}0from roughly step 30 \(Figure[4\(f\)](https://arxiv.org/html/2608.04794#S6.F4.sf6)\), even with an EMA rate as small as0\.0010\.001\. Read alone, a gap of zero says the student has matched its teacher; two observations rule that out\. The±1\\pm 1standard\-deviation band does not contract with the mean, staying near±1\\pm 1: the student matches the teacher*on average*while disagreeing as much as ever at individual tokens, which indicates cancellation rather than convergence\. And over the same period student entropy rises in most domains, with the teacher’s perplexity on student rollouts rising alongside, whereas a student that had genuinely learned the teacher’s distribution would become sharper and easier to predict\.The shrinking gap is not convergence but collapse: the student ends flatter and less decisive than it began, closing the loop with Section[5](https://arxiv.org/html/2608.04794#S5): a flatter policy commits earlier \(falling length\) and solves no more problems \(flat validation\)\.
## 7Robustness
Response length\.A dense per\-token loss spreads over more positions as trajectories lengthen, so the failure could be dilution; if it were, the short split would improve\. Training separately on DAPO\-Math response\-length tertiles \(short\[1318,4364\]\[1318,4364\], medium\[4387,11939\]\[4387,11939\], long\[11972,16384\]\[11972,16384\]tokens\) all three fall below the base average of63\.463\.4, by1\.11\.1,0\.80\.8and1\.61\.6points; in\-domain DAPO\-Math rises in every split \(\+1\.4\+1\.4,\+1\.0\+1\.0,\+0\.6\+0\.6\) while eight of the nine transfer cells fall and the ninth gains0\.10\.1in TableLABEL:tab:resp\_len\. The short split, where supervision is least diluted, is no better than the long one, and its token\-type loss shares are unchanged\.
Table 4:Robustness to trajectory length\.Response\-length tertiles on DAPO\-Math \(short\[1318,4364\]\[1318,4364\], medium\[4387,11939\]\[4387,11939\], long\[11972,16384\]\[11972,16384\]tokens\); Qwen3\-8B think\. “SD” andΔ=SD−Base\\Delta=\\text\{SD\}\-\\text\{Base\}follow Table[1](https://arxiv.org/html/2608.04794#S5.T1),greenpositive andrednegative; base scores are omitted for space and recoverable asSD−Δ\\text\{SD\}\-\\Delta\.Task difficulty\.The failure could instead reflect problems the student cannot solve at all; if so, the easy split would improve\. Splitting MMLU\-Pro by the base model’s mean success rate \(MSR\) at 8 samples into easy \(MSR@8∈\[0\.75,0\.875\]\\mathrm\{MSR@8\}\\in\[0\.75,0\.875\]\), medium \(\[0\.375,0\.625\]\[0\.375,0\.625\]\) and hard \(\[0\.125,0\.25\]\[0\.125,0\.25\]\) subsets: all three fall below the base average of60\.960\.9, easy by3\.63\.6points, medium by1\.41\.4, hard by1\.61\.6as seen in TableLABEL:tab:diff\-scale\. In summary, the model performs no better on the easy split as compared to the hard split\.
Model scale\.SD’s benefits are reported to grow with model size\(Hübotteret al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib1)\), so a larger model should improve\. TableLABEL:tab:diff\-scaleshows how training Qwen3\-32B on MMLU\-Pro moves averages by−0\.6\-0\.6points in think and−1\.4\-1\.4in instruct, with the signature unchanged and the PI bias score undiminished\. The larger model starts from a higher base,64\.764\.7against60\.960\.9for Qwen3\-8B on the same benchmarks\. With the reasoning modes of Section[5](https://arxiv.org/html/2608.04794#S5)and the PI\-form intervention of Section[6\.1](https://arxiv.org/html/2608.04794#S6.SS1), the result is therefore unchanged across trajectory length, task difficulty, model scale, reasoning mode, and PI content\.
Table 5:Robustness to task difficulty and model scale\.Easy/medium/hard subsets of MMLU\-Pro \(left\) and Qwen3\-32B on full MMLU\-Pro \(right\); Qwen3\-8B think unless noted\. “SD” andΔ=SD−Base\\Delta=\\text\{SD\}\-\\text\{Base\}follow Table[1](https://arxiv.org/html/2608.04794#S5.T1),greenpositive andrednegative; base scores are omitted for space and recoverable asSD−Δ\\text\{SD\}\-\\Delta\.
## 8Discussion
What fails is the target, not the density\.Not optimization, implementation, data, or scale \(Sections[5](https://arxiv.org/html/2608.04794#S5),[7](https://arxiv.org/html/2608.04794#S7)\), but the target in Eq\. \([1](https://arxiv.org/html/2608.04794#S3.E1)\): a PI\-conditioned teacher predictsonecorrect continuation and is indifferent between it and the others, so it spreads trajectory information, not correctness, over every token\. Density cannot repair a target that cannot distinguish a correct continuation from a well\-phrased incorrect one\.
Why the objective works on easy tasks\.Our account predicts rather than contradicts prior work’s regime of success: where responses are short and the answer is a bounded choice, one reference solution nearly spans the space of correct ones, little exploration exists to suppress, and few tokens are irrelevant to correctness, so imitating one solution \(even at high PI bias\) is close to learning to be correct\. All three weaken as responses lengthen and answers open up; our results and SDPO/OPSD’s are consistent, differing only in where imitation/correctness cease to coincide\.
Implications for practice\.A falling SD loss is not evidence of learning; monitor validation accuracy, which separates the two cases\. The PI Bias Score screens a candidate PI before training at a cost negligible beside it, forward passes over a handful of prefixes and four targets: a largePS⋆−PS′\\mathrm\{PS\}^\{\\star\}\-\\mathrm\{PS\}^\{\\prime\}gap means the teacher will transfer a trajectory rather than a skill, a uniformly small score that it will transfer nothing\. Nor is PI a free hyperparameter with a benign middle setting: both ends of the specificity dial fail\.
Relation to methods that retain a reward, and the open problem\.The augmentation literature\(Yanget al\.,[2026a](https://arxiv.org/html/2608.04794#bib.bib4); Wanget al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib5); Luet al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib6)\)divides by what each term supplies: a verifiable reward carries correctness but nothing about form, the SD term form but almost nothing about correctness\. This is what RLSD’s fix accomplishes: reweighting a verifier\-driven advantage keeps the density and discards the term carrying the bias\. Whether combining them repairs the failure we document, our experiments cannot answer: we do not run that setting\. A dense target thatalsoencodes correctness remains open; because the bias enters at the first link, the teacher’s commitment to a single trajectory, the constructions with the clearest claim on the source make the target reflect thesetof correct continuations\.
Limitations\.Our study is observational, characterizing the objective rather than proposing a remedy\. We evaluate one model family \(Qwen3\) at two scales across four domains and make no cross\-architecture claim; results average four rollouts, and some per\-domain differences are small relative to run\-to\-run variation\. The PI Bias Score is a log\-probability ratio at sampled positions, depends on the choice ofy′,y~,y−y^\{\\prime\},\\tilde\{y\},y^\{\-\}, and complements the conditional mutual information analysis ofYanget al\.\([2026a](https://arxiv.org/html/2608.04794#bib.bib4)\)without estimating it\. Hints and skills are model\-generated, so the weak\-PI results may partly reflect their construction\. We do not claim self\-distillation is ineffective in general, only that in our settings a PI\-conditioned per\-token objective alone gives no learning signal aligned with task correctness\.
## 9Conclusion
We asked whether a dense PI\-conditioned target carries information about correctness when it is the sole training signal\. It does not\. Across four reasoning\-heavy domains, loss decreases while accuracy stays flat or degrades, under both recipes, both reasoning modes, and across trajectory length, task difficulty, and model scale\. The PI Bias Score shows why: a teacher conditioned on a single reference solution learns that trajectory rather than correctness\. The resulting loss cannot distinguish correct from incorrect rollouts, concentrates on already\-correct trajectories, low\-information tokens, and off\-path exploration, and leaves the student flatter rather than sharper\. Weakening the PI fails too, so neither overly specific nor overly weak PI signals correctness\. The limitation is not target density but a teacher anchored to one reference trajectory rather than the task objective\.
## References
- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. Bachem \(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21246–21263\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5be69a584901a26c521c2b51e40a4c20-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton \(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p3.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p3.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Dekoninck, N\. Jovanović, T\. Gehrunger, K\. Rögnvaldsson, I\. Petrov, C\. Sun, and M\. Vechev \(2026\)Beyond benchmarks: matharena as an evaluation platform for mathematics with LLMs\.In3rd AI for Math Workshop: Toward Self\-Evolving Scientific Agents,External Links:[Link](https://openreview.net/forum?id=DmPE4byHuN)Cited by:[§2](https://arxiv.org/html/2608.04794#S2.p2.6)\.
- K\. Feng, K\. Ding, W\. Wang, X\. Zhuang, Z\. Wang, M\. Qin, Y\. Zhao, J\. Yao, Q\. Zhang, and H\. Chen \(2024\)SciKnowEval: evaluating multi\-level scientific knowledge of large language models\.arXiv preprint arXiv:2406\.09098\.Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p1.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun \(2024\)OlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p2.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. Kleine Buening, C\. Guestrin, and A\. Krause \(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[Table 7](https://arxiv.org/html/2608.04794#A2.T7),[Table 7](https://arxiv.org/html/2608.04794#A2.T7.14.2),[Appendix B](https://arxiv.org/html/2608.04794#A2.p1.1),[§1](https://arxiv.org/html/2608.04794#S1.p1.1),[§2](https://arxiv.org/html/2608.04794#S2.p2.6),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px2.p1.2),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2608.04794#S7.p3.4)\.
- N\. Jain, Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Vol\.2025\.Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p3.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Kim, X\. Luo, M\. Kim, S\. Lee, D\. Kim, J\. Jeon, D\. Li, and Y\. Yang \(2026\)Why does self\-distillation \(sometimes\) degrade the reasoning capability of LLMs?\.arXiv preprint arXiv:2603\.24472\.Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p9.1),[§2](https://arxiv.org/html/2608.04794#S2.p2.6),[§2](https://arxiv.org/html/2608.04794#S2.p3.2),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px3.p1.12),[§6\.3](https://arxiv.org/html/2608.04794#S6.SS3.p1.1),[§6\.4](https://arxiv.org/html/2608.04794#S6.SS4.p1.7)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DISTILLM: towards streamlined distillation for large language models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p1.1)\.
- N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. Hajishirzi \(2024\)Tülu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p1.1),[§2](https://arxiv.org/html/2608.04794#S2.p1.5)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:Introduces HumanEval\+ and MBPP\+Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p3.1)\.
- Z\. Lu, Z\. Yao, Z\. Han, Z\. Wang, J\. Wu, Q\. Gu, X\. Cai, W\. Lu, J\. Xiao, Y\. Zhuang, and Y\. Shen \(2026\)Self\-distilled agentic reinforcement learning\.arXiv preprint arXiv:2605\.15155\.Cited by:[§2](https://arxiv.org/html/2608.04794#S2.p3.2),[§8](https://arxiv.org/html/2608.04794#S8.p4.1)\.
- Mathematical Association of America \(2024\)AIME 2024: american invitational mathematics examination\.Note:[https://huggingface\.co/datasets/Maxwell\-Jia/AIME\_2024](https://huggingface.co/datasets/Maxwell-Jia/AIME_2024)Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p2.1),[§2](https://arxiv.org/html/2608.04794#S2.p2.6),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- Mathematical Association of America \(2025\)AIME 2025: american invitational mathematics examination\.Note:[https://huggingface\.co/datasets/opencompass/AIME2025](https://huggingface.co/datasets/opencompass/AIME2025)Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p2.1),[§2](https://arxiv.org/html/2608.04794#S2.p2.6),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- Open R1 Team \(2025a\)CodeForces dataset\.Note:[https://huggingface\.co/datasets/open\-r1/codeforces](https://huggingface.co/datasets/open-r1/codeforces)Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p3.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- Open R1 Team \(2025b\)DAPO\-Math\-17k\-Processed\.Note:[https://huggingface\.co/datasets/open\-r1/DAPO\-Math\-17k\-Processed](https://huggingface.co/datasets/open-r1/DAPO-Math-17k-Processed)Filtered and de\-duplicated English subset of DAPO\-Math\-17kCited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p2.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- S\. G\. Patil, H\. Mao, C\. Cheng\-Jie Ji, F\. Yan, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2024\)The berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p4.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- S\. Quan, J\. Yang, B\. Yu, B\. Zheng, D\. Liu, A\. Yang, X\. Ren, B\. Gao, Y\. Miao, Y\. Feng, Z\. Wang, J\. Yang, Z\. Cui, Y\. Fan, Y\. Zhang, B\. Hui, and J\. Lin \(2025\)CodeElo: benchmarking competition\-level code generation of LLMs with human\-comparable elo ratings\.arXiv preprint arXiv:2501\.01257\.Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p3.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)GPQA: a graduate\-level google\-proof q&a benchmark\.InConference on Language Modeling \(COLM\),Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p1.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p1.1),[§2](https://arxiv.org/html/2608.04794#S2.p1.5)\.
- I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal \(2026\)Self\-distillation enables continual learning\.arXiv preprint arXiv:2601\.19897\.Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p1.1)\.
- H\. Wang, G\. Wang, H\. Xiao, Y\. Zhou, Y\. Pan, J\. Wang, K\. Xu, Y\. Wen, X\. Ruan, X\. Chen, and H\. Qi \(2026\)Skill\-SD: skill\-conditioned self\-distillation for multi\-turn LLM agents\.arXiv preprint arXiv:2604\.10674\.Cited by:[§2](https://arxiv.org/html/2608.04794#S2.p3.2),[§8](https://arxiv.org/html/2608.04794#S8.p4.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen \(2024\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p1.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.Cited by:[Appendix B](https://arxiv.org/html/2608.04794#A2.p1.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px2.p1.2)\.
- C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. Duan \(2026a\)Self\-distilled RLVR\.arXiv preprint arXiv:2604\.03128\.Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p9.1),[§2](https://arxiv.org/html/2608.04794#S2.p2.6),[§2](https://arxiv.org/html/2608.04794#S2.p3.2),[§3](https://arxiv.org/html/2608.04794#S3.p3.2),[§6\.1](https://arxiv.org/html/2608.04794#S6.SS1.SSS0.Px2.p1.15),[§8](https://arxiv.org/html/2608.04794#S8.p4.1),[§8](https://arxiv.org/html/2608.04794#S8.p5.1)\.
- S\. Yang, J\. Wu, Z\. Lu, Y\. Shen, F\. Zhang, L\. Feng, S\. Zhang, H\. Luo, Z\. Lian, Z\. Wen, and J\. Tao \(2026b\)OPID: on\-policy skill distillation for agentic reinforcement learning\.External Links:2606\.26790Cited by:[§2](https://arxiv.org/html/2608.04794#S2.p3.2)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang \(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§A\.1](https://arxiv.org/html/2608.04794#A1.SS1.p2.1),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px1.p1.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§1](https://arxiv.org/html/2608.04794#S1.p1.1),[§2](https://arxiv.org/html/2608.04794#S2.p2.6),[§4](https://arxiv.org/html/2608.04794#S4.SS0.SSS0.Px3.p1.12),[§5](https://arxiv.org/html/2608.04794#S5.p1.1),[§6\.3](https://arxiv.org/html/2608.04794#S6.SS3.p1.1)\.
## Contents
ADatasets, Splits, and PI Construction\.[A](https://arxiv.org/html/2608.04794#A1)
A\.1Sources and splits\.[A\.1](https://arxiv.org/html/2608.04794#A1.SS1)
A\.2PI construction\.[A\.2](https://arxiv.org/html/2608.04794#A1.SS2)
BHyperparameters \(Implementation Details\)\.[B](https://arxiv.org/html/2608.04794#A2)
CQualitative Examples \(Full Solution vs Hints vs Skills\)\.[C](https://arxiv.org/html/2608.04794#A3)
DPrompts and Templates\.[D](https://arxiv.org/html/2608.04794#A4)
D\.1Per\-domain problem prompts\.[D\.1](https://arxiv.org/html/2608.04794#A4.SS1)
D\.2Self\-distillation templates\.[D\.2](https://arxiv.org/html/2608.04794#A4.SS2)
D\.3Privileged\-information generation prompts\.[D\.3](https://arxiv.org/html/2608.04794#A4.SS3)
D\.4Skill derivation prompt\.[D\.4](https://arxiv.org/html/2608.04794#A4.SS4)
EPI Bias Implementation Details\.[E](https://arxiv.org/html/2608.04794#A5)
This supplement provides implementation details, additional results, and qualitative material for the main paper\.
## Appendix ADatasets, Splits, and PI Construction
The study spans four domains: general question answering, mathematics, coding, and multi\-turn agentic tool use\. For each domain, one dataset provides training and in\-domain evaluation, and a set of held\-out datasets measures transfer\. Table[6](https://arxiv.org/html/2608.04794#A1.T6)lists them\. This section describes their sources, the train, validation, and test splits, and the construction of the three forms of privileged information \(PI\)\.
Table 6:Domains, training data, and evaluation benchmarks used in the study\.### A\.1Sources and splits
General question answering uses MMLU\-Pro\[Wanget al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib14)\]for training and in\-domain evaluation\. Transfer is measured on GPQA\-Diamond\[Reinet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib15)\]and on four SciKnowEval\[Fenget al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib16)\]subjects \(biology, chemistry, materials, and physics\)\. The SciKnowEval subjects are divided into training and test portions by a ninety\-ten split\.
Mathematics uses the processed release of DAPO\-Math\[Yuet al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib17), Open R1 Team,[2025b](https://arxiv.org/html/2608.04794#bib.bib18)\]for training and in\-domain evaluation, and measures transfer on AIME 2024\[Mathematical Association of America,[2024](https://arxiv.org/html/2608.04794#bib.bib19)\], AIME 2025\[Mathematical Association of America,[2025](https://arxiv.org/html/2608.04794#bib.bib20)\], and OlympiadBench\[Heet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib21)\]\.
Coding uses CodeForces\[Open R1 Team,[2025a](https://arxiv.org/html/2608.04794#bib.bib22)\]problems, restricted to those with a problem description, for training and in\-domain evaluation, and measures transfer on MBPP\+ and HumanEval\+\[Austinet al\.,[2021](https://arxiv.org/html/2608.04794#bib.bib24), Chenet al\.,[2021](https://arxiv.org/html/2608.04794#bib.bib25), Liuet al\.,[2023](https://arxiv.org/html/2608.04794#bib.bib23)\], CodeElo\[Quanet al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib26)\], and LiveCodeBench v6\[Jainet al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib29)\]\.
The multi\-turn agentic domain uses the Berkeley Function\-Calling Leaderboard\[Patilet al\.,[2024](https://arxiv.org/html/2608.04794#bib.bib27)\]\. Training and in\-domain evaluation use the version\-three multi\-turn base v3 category, which comprises 100 tasks\. Transfer is measured on the version\-four multi\-turn v4 categories: base, missing function, missing parameter, and long context\.
For the in\-domain training datasets, a stratified validation subset of 300 examples is drawn, stratified over pass\-rate bins\. For the transfer datasets, the validation set is the full dataset\. At training time, each in\-domain training set is subsampled to a random subset of 2,000 examples drawn without replacement\. The multi\-turn agentic domain is the exception and uses its full training set\.
### A\.2PI construction
Three forms of privileged information are used\. Any one of them can be supplied to training in place of the others\.
#### Whole solution\.
For each question, one correct solution is selected from the model’s own successful attempts, with its reasoning trace removed\. The selected solution is presented to the teacher through the templates of Appendix[D](https://arxiv.org/html/2608.04794#A4)\.
#### Hint\.
A short insight of one to two sentences is generated from the question and a correct solution by a separate model, and is constrained not to reveal the final answer\. The generation prompt is given in Appendix[D](https://arxiv.org/html/2608.04794#A4)\.
#### Skill\.
Problems are grouped into clusters, and one structured skill card of roughly 500 words is derived per cluster and shared across its member problems\. Each card names a general technique together with its tools, common pitfalls, and the cues that indicate when it applies, and is constrained not to reveal any specific answer\. The derivation prompt is given in Appendix[D](https://arxiv.org/html/2608.04794#A4)\.
## Appendix BHyperparameters \(Implementation Details\)
We report the training configuration used for the self\-distillation runs, grouped following the format used in the appendix of the original SDPO paper\[Hübotteret al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib1)\]\. The configuration for the primary Qwen3\-8B runs is shared across the four domains; we note where the coding and multi\-turn agentic domains and the Qwen3\-32B size study differ\. The base model is Qwen3\-8B\[Yanget al\.,[2025](https://arxiv.org/html/2608.04794#bib.bib30)\], and every domain is studied in both its thinking \(long chain\-of\-thought\) and instruct variants; the configuration below corresponds to the thinking variant\. All experiments are run on a single node of eight NVIDIA B200 GPUs\.
Two coefficients with similar names are distinguished here to avoid confusion\. The divergence used by the loss is controlled by a mixing coefficient for which0denotes forward KL,11denotes reverse KL, and0\.50\.5denotes the symmetric Jensen\-Shannon divergence; every primary run uses0\.50\.5\. The rate at which the teacher tracks the student through an exponential moving average is a separate quantity, set to0\.0010\.001; this is the teacher EMA rate referred to in the main paper\.
Table 7:Training configuration for the primary Qwen3\-8B self\-distillation runs, grouped following the format used by the original SDPO paper\[Hübotteret al\.,[2026](https://arxiv.org/html/2608.04794#bib.bib1)\]\. The configuration is shared across the four domains; the deviations noted below the table apply to the coding, multi\-turn agentic, and Qwen3\-32B size\-study runs\.The following domain\-specific deviations from Table[7](https://arxiv.org/html/2608.04794#A2.T7)apply\. The coding and multi\-turn agentic runs use a learning rate of1×10−61\\times 10^\{\-6\}\. The Qwen3\-32B size study uses a maximum response length of 10240, a maximum model context of 12288, and tensor\-parallel size 4\.
## Appendix CQualitative Examples \(Full Solution vs Hints vs Skills\)
This section illustrates the three forms of privileged information on a shared training question\. Figure[6](https://arxiv.org/html/2608.04794#A3.F6)shows a representative mathematics problem together with the whole solution, the derived skill, and the derived hint, each of which can serve as the privileged information supplied to the teacher in a separate training configuration\.
Figure 6:A representative mathematics training question \(top\) shown with the three forms of privileged information derived for it\. The whole solution presents a complete correct answer, the skill presents a structured technique card for the question’s cluster, and the hint presents a short insight\. In training, each form is supplied on its own to the privileged\-information\-conditioned teacher, following the templates of Appendix[D](https://arxiv.org/html/2608.04794#A4)\.#### Whole\-solution form\.
A single correct solution to the question, with its reasoning trace removed, is presented to the teacher under a header that labels it as the correct solution, followed by the instruction to solve the original question with the student’s own approach, following the mathematics reprompt template of Appendix[D](https://arxiv.org/html/2608.04794#A4)\.
#### Hint form\.
A one\-to\-two\-sentence insight is presented to the teacher through the hint template of Appendix[D](https://arxiv.org/html/2608.04794#A4)\. The insight names the governing idea, for example the relevant theorem or identity, without stating the final answer\.
#### Skill form\.
A structured skill card for the question’s cluster is presented to the teacher as five fields\. The layout is
Skill: <name\>
General approach: <prose description of the technique\>
Key tools & formulas: <theorems, identities, algorithms\>
Pitfalls & checks: <common mistakes, edge cases, checks\>
When it applies: <signals indicating this skill\>
Listing 1: Rendered skill\-card fields\.
The card describes a general technique for the cluster and does not reveal the answer to any single problem\.
## Appendix DPrompts and Templates
This section reproduces the prompt and template strings used in training and in the construction of privileged information\. Long lines are wrapped for display\. Placeholders in braces, for example`\{problem\}`and`\{hint\}`, are filled at run time\. Each block is labelled with a numbered listing caption\.
### D\.1Per\-domain problem prompts
The student is shown a domain\-specific prompt that wraps each problem\.
\{problem\}
Please reason step by step, and put your final answer within \\boxed\{\}\.
Listing 2: General reasoning and mathematics problem prompt\.
\{internallinenumbers\*\}You are a coding expert\. You will be given a coding problem, and you need to write a correct Python program that matches the specification and passes all tests\. The time limit is 1 second\. You may start by outlining your thought process\. In the end, please provide the complete code in a code block enclosed with \`\`\` \`\`\`\.
\{problem\}
Listing 3: Coding problem prompt\.
\{internallinenumbers\*\}The following are multiple choice questions \(with answers\) about \{subject\}\. Think step by step and then finish your answer with "the answer is \(X\)" where X is the correct letter choice\.
Question:
\{question\}
Options:
A\. \{option\_A\}
B\. \{option\_B\}
\.\.\.
Answer: Let’s think step by step\.
Listing 4: MMLU\-Pro multiple\-choice prompt\.
\{internallinenumbers\*\}Return your final response within \\boxed\{\} and only include the letter choice \(A, B, C, or D\) as your final response\.
Problem: \{problem\}
Options: \{options\}
Answer:
Listing 5: GPQA prompt\.
\{internallinenumbers\*\}Given a question and four options, please select the right answer\. Respond in the following format:
<reasoning\>
\.\.\.
</reasoning\>
<answer\>
\.\.\.
</answer\>
\{internallinenumbers\*\}For the answer, only output the letter corresponding to the correct option \(A, B, C, or D\), and nothing else\. Do not restate the answer text\. For example, if the answer is "A", just output:
<answer\>
A
</answer\>
Listing 6: SciKnowEval system prompt\.
### D\.2Self\-distillation templates
During self\-distillation the teacher prompt is assembled from a reprompt template that combines the problem, a solution section carrying the privileged information, and an optional feedback section\. The privileged information enters through the solution section \(a whole solution\) or through the hint template \(a hint or skill\)\. The default templates are shown first, followed by the per\-domain overrides used for mathematics and the multi\-turn agentic setting; the overrides label the injected privileged information as a correct solution and close the reprompt with an instruction to use the student’s own approach\.
\{prompt\}\{solution\}\{feedback\}
Correctly solve the original question\.
Listing 7: Default reprompt template\.
Correct solution:
\{successful\_previous\_attempt\}
Listing 8: Default solution section\.
The following is feedback from your unsuccessful earlier attempt:
\{feedback\_raw\}
Listing 9: Default feedback section\.
Here is a hint for solving this problem:
\{hint\}
Listing 10: Default hint template\.
hint\_template:
Here is the correct solution for the above question:
\{hint\}
reprompt\_template:
\{prompt\}\{solution\}\{feedback\}
Now, using your own approach, try to correctly solve the original question\.
Listing 11: Mathematics reprompt and hint templates\.
reprompt\_template:
\{prompt\}\{solution\}\{feedback\}
\{internallinenumbers\*\}After understanding this, please try to solve this problem using your own approach below\.
solution\_template:
Here is the correct solution for the problem:
\`\`\`
\{successful\_previous\_attempt\}
\`\`\`
Listing 12: Multi\-turn agentic reprompt and solution templates\.
### D\.3Privileged\-information generation prompts
Hints and compressed solutions used as privileged information are generated by a separate model\.
\{internallinenumbers\*\}You are given a math problem and its correct solution\. Your job is to extract the core mathematical trick, principle, or insight that unlocks the solution\.
Rules:
\- 1\-3 sentences maximum\.
\- Do NOT reveal the final answer or any numerical result\.
\- Do NOT give step\-by\-step instructions\.
\- Do NOT start with "The key insight is" or similar preambles\.
\- Only state the idea, theorem, formula, or trick itself\.
Respond in the following format:
<hint\>
\[your hint here\]
</hint\>
Problem: \`\`\`\{problem\}\`\`\`
Solution:
\`\`\`
\{solution\}
\`\`\`
Listing 13: Hint generation prompt\.
\{internallinenumbers\*\}You are given a math problem and its correct \(but verbose\) solution\. Rewrite the solution as a short, dense derivation\.
Rules:
\- Include ALL key steps and the final answer\.
\- Remove all verbose explanations, motivations, headers, and formatting\.
\- Use mathematical notation directly instead of prose where possible\.
\- Target length: 3\-6 sentences or equivalent equations\.
\- The compressed solution must be self\-contained and verifiable\.
\- Do NOT omit the final boxed answer\.
Respond in the following format:
<compressed\>
\[your compressed solution here\]
</compressed\>
Problem: \`\`\`\{problem\}\`\`\`
Solution:
\`\`\`
\{solution\}
\`\`\`
Listing 14: Compressed\-solution generation prompt\.
### D\.4Skill derivation prompt
The skill form of privileged information is derived once per problem cluster\.
\{internallinenumbers\*\}You are analyzing a cluster of related \{domain\} to extract ONE reusable problem\-solving \*skill\* that applies across the whole cluster\.
\{internallinenumbers\*\}A skill is the Goldilocks middle ground between a one\-line hint and a full solution: rich enough to materially guide a solver on ANY problem in this class, yet general enough that it never solves a specific instance\. It will be shown to a model as privileged context alongside a long \(multi\-thousand\-token\) solution attempt, so it must carry enough substance to actually shape the reasoning\.
Hard rules:
\{internallinenumbers\*\}\- Describe a GENERAL, reusable technique/strategy that plausibly applies to MANY problems in this cluster, not just one\.
\- Do NOT reveal or compute any specific numeric answer, final result, or option\.
\- Do NOT walk through the solution of any single problem instance\.
\- Do NOT give a numbered step\-by\-step procedure or a fixed solution template\.
\{internallinenumbers\*\}\- Be concrete and substantive: name the actual reasoning moves, representations, formulas, theorems, and checks a strong solver would use\.
Produce a structured skill card with these fields:
\- name: a short 3\-7 word handle for the technique\.
\- approach: the general line of attack, in flowing prose\.
\{internallinenumbers\*\}\- tools: the key formulas, theorems, identities, representations, or algorithms this technique relies on\.
\- pitfalls: common mistakes, tricky edge cases, and sanity\-checks\.
\- cues: the textual or structural signals in a problem that indicate this skill applies\.
\{internallinenumbers\*\}Aim for a substantial, information\-dense card \(roughly 250\-400 words total across the fields\)\.
Listing 15: Skill derivation prompt\.
## Appendix EPI Bias Implementation Details
This section documents how the PI Bias Score and the on\-path score used in the main paper are computed\. For a student rollout, let the student prefix up to positionttbe its firstttresponse tokens\. For a target token sequenceww, define the on\-path score
st\(w\)=1\|w\|∑k=0\|w\|−1logPmodel\(wk∣c,prefix<t,w<k\),s\_\{t\}\(w\)=\\frac\{1\}\{\|w\|\}\\sum\_\{k=0\}^\{\|w\|\-1\}\\log P\_\{\\text\{model\}\}\\\!\\left\(w\_\{k\}\\mid c,\\,\\text\{prefix\}\_\{<t\},\\,w\_\{<k\}\\right\),\(3\)the mean log\-probability the model assigns to the tokens ofwwwhenwwis appended after the student prefix\. The contextccis either the teacher prompt, which contains the privileged information, givingstT\(w\)s^\{\\mathrm\{T\}\}\_\{t\}\(w\), or the student prompt, which does not, givingstS\(w\)s^\{\\mathrm\{S\}\}\_\{t\}\(w\)\. The PI Bias Score is the difference
PSt\(w\)=stT\(w\)−stS\(w\),\\mathrm\{PS\}\_\{t\}\(w\)=s^\{\\mathrm\{T\}\}\_\{t\}\(w\)\-s^\{\\mathrm\{S\}\}\_\{t\}\(w\),\(4\)so that a positive value indicates that the privileged information present in the teacher prompt raises the model’s tendency to continue towardww\.
#### Targets\.
Four targetswware scored at each evaluated position:
- •y⋆y^\{\\star\}, the in\-context correct solution, that is the privileged information itself, taken from the teacher prompt\. For hint\-style privileged information that does not embed a full solution, a correct solution to the same question is used with its reasoning trace removed\.
- •y′y^\{\\prime\}, a different correct solution to the same question, which requires at least two distinct correct solutions\.
- •y−y^\{\-\}, an incorrect solution to the same question, taken from an incorrect attempt\.
- •y~\\tilde\{y\}, an unrelated correct solution drawn from the other questions in the same training step\.
#### Positions and lookahead\.
Each target is scored in full: the lookahead length is set toK=2000K=2000tokens, which exceeds the length of every target, so no target is truncated\. Positions are evaluated at every token, over assistant tokens \(and not user tokens\) only\. Prompts longer than 8000 tokens are skipped\. When the student is inside a reasoning block at positiontt, a closing marker is inserted before the target so that the appended continuation is well formed\.
#### Models\.
The teacher scoresTs^\{\\mathrm\{T\}\}is computed with the teacher checkpoint and the student scoresSs^\{\\mathrm\{S\}\}with the student checkpoint at the corresponding training step\. Before the first checkpoint, the base model is used for both roles\. The two passes are computed separately and merged by position\.
#### Scoring\.
The score is obtained from a single forward pass over the assembled prompt and target\. At each target position the language\-model head produces a distribution over the vocabulary, from which the log\-probability of the target token is read, and the on\-path score is the mean of these log\-probabilities over the target\.Similar Articles
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
Adaptive Teacher Exposure for Self-Distillation (ATESD) improves LLM reasoning by dynamically adjusting how much of the reference reasoning the teacher shows the student during training, using a learnable policy controller and a discounted learning-progress reward. Experiments on math benchmarks show consistent improvements over existing self-distillation and RL baselines.
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
This paper identifies that teacher token reliability in reasoning distillation is trajectory-structured and proposes Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies increasing position weights to improve performance without additional teacher computation.
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
This paper introduces Motab, a new pipeline for LLM reasoning distillation that mitigates both off-policy and on-policy exposure biases by dynamically monitoring student generation and backtracking to safe states with teacher intervention, achieving ~3% average improvement.
Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization
This paper proposes PTD-PO, a privileged tutoring distillation framework that provides dense token-level supervision for reinforcement learning with verifiable rewards in multimodal reasoning tasks, without exposing the answer. It uses structured hints and a Top-K JS divergence objective to stabilize training, consistently outperforming existing methods on 2B-8B LVLMs.
PHF: Privileged Hidden Flow for On-Policy Self-Distillation
PHF proposes a method to distill hidden state trajectories from a privileged teacher to a student during on-policy self-distillation, improving reasoning performance on language models.