When Context Returns: Toward Robust Internalization in On-Policy Distillation
Summary
The paper identifies that reintroducing privileged context to a distilled student model degrades performance (context-induced degradation), and proposes a lightweight consistency regularizer that anchors no-context outputs to mitigate this issue, improving robustness across 12 configurations.
arXiv:2606.11627v1 Announce Type: new
Abstract: Recent work has shown that on-policy distillation can internalize privileged context, such as system prompts or task hints, into a student model so that the context is no longer needed at inference time. Although this approach successfully improves the student's no-context performance, we identify an interesting and previously unstudied phenomenon: in many settings, reintroducing the original privileged context to the distilled student actually degrades its performance, even on instances it already solves correctly without context. We term this context-induced degradation and argue that robust internalization demands not only matching the teacher's context-conditioned behavior, but also remaining stable when the context is reintroduced, a property we call context removability. Motivated by this observation, we propose a lightweight consistency regularizer that first anchors the student's no-context output via stop-gradient, then penalizes the context-conditioned output for deviating from it via forward KL divergence. This simple addition requires only one extra forward pass per training step, yet it effectively mitigates context-induced degradation and, in many cases, even improves no-context performance. Across 12 configurations spanning diverse domains and model families, our method improves context-conditioned accuracy in the majority of settings, reduces context-induced harm in 11 out of 12 settings, and effectively eliminates response-length inflation. A mechanistic case study further confirms that context removability is achieved at the representation level, with hidden states remaining nearly identical regardless of whether the context is present.
View Cached Full Text
Cached at: 06/11/26, 01:50 PM
# Toward Robust Internalization in On-Policy Distillation
Source: [https://arxiv.org/html/2606.11627](https://arxiv.org/html/2606.11627)
## When Context Returns: Toward Robust Internalization in On\-Policy Distillation
Xun Wang1, Ruishuo Chen1, Zhuoran Li1, Yu Chen1, and Longbo Huang1\{\}^\{1\\,\\text\{\\faIcon\{envelope\}\}\} 1IIIS, Tsinghua UniversityCorresponding Author: longbohuang@tsinghua\.edu\.cn
###### Abstract
Recent work has shown that on\-policy distillation can internalize privileged context, such as system prompts or task hints, into a student model so that the context is no longer needed at inference time\. Although this approach successfully improves the student’s no\-context performance, we identify an interesting and previously unstudied phenomenon: in many settings, reintroducing the original privileged context to the distilled student actually*degrades*its performance, even on instances it already solves correctly without context\. We term this*context\-induced degradation*and argue that robust internalization demands not only matching the teacher’s context\-conditioned behavior, but also remaining stable when the context is reintroduced, a property we call*context removability*\. Motivated by this observation, we propose a lightweight consistency regularizer that first anchors the student’s no\-context output via stop\-gradient, then penalizes the context\-conditioned output for deviating from it via forward KL divergence\. This simple addition requires only one extra forward pass per training step, yet it effectively mitigates context\-induced degradation and, in many cases, even improves no\-context performance\. Across 12 configurations spanning diverse domains and model families, our method improves context\-conditioned accuracy in the majority of settings, reduces context\-induced harm in 11 out of 12 settings, and effectively eliminates response\-length inflation\. A mechanistic case study further confirms that context removability is achieved at the representation level, with hidden states remaining nearly identical regardless of whether the context is present\.
## 1\. Introduction
Language models are frequently augmented with privileged context, such as system prompts encoding behavioral constraints, chain\-of\-thought traces providing expert reasoning\(Weiet al\.,[2022](https://arxiv.org/html/2606.11627#bib.bib1)\), or game\-state descriptions summarizing decision\-relevant information, that boosts performance via in\-context learning\(Brownet al\.,[2020](https://arxiv.org/html/2606.11627#bib.bib2); Donget al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib3)\)but inflates latency, increases serving cost, and risks exposing sensitive instructions to end users\. To eliminate these drawbacks, on\-policy self\-distillation\(Zhaoet al\.,[2026a](https://arxiv.org/html/2606.11627#bib.bib4); Shenfeldet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib5); Hübotteret al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib6)\)trains the model to internalize the context by distilling its own context\-conditioned outputs into the context\-free pathway\.Yeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\)generalize this idea to on\-policy context distillation by allowing distinct teacher and student models, enabling the student to perform well at deployment*without*requiring the context\.
A natural question arises: if the student has truly internalized the privileged information, what happens when the context is*reintroduced*? Intuitively, re\-presenting already\-internalized information should be an idempotent operation\(Liuet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib8)\): a student who has memorized the textbook should not perform worse when allowed to consult it\. However, we find that this is often not the case: across the majority of settings, reintroducing the distilled context not only inflates the model’s output length but also*degrades*its accuracy \([Figure˜1](https://arxiv.org/html/2606.11627#S1.F1), left\)\. We call this phenomenon*context\-induced degradation*\.
Figure 1:Left: OPD trains the student to match the teacher’s context\-conditioned output without access tocc\. At test time, the student performs well without context, but its performance can drop whenccis reintroduced\.Middle: Training curves averaged across 12 settings\. OPD’s w/ ctx accuracy \(gray dashed\) consistently lags behind w/o ctx accuracy \(gray solid\), while NCA \(colored\) closes this gap\.Right: Final accuracy comparison\. NCA improves both no\-context \(\+2\.1 pp\) and context\-conditioned \(\+3\.0 pp\) accuracy\.We trace this to the fact that OPD optimizes exclusively for what we call*privileged fidelity*, i\.e\., the student without context should match the teacher with context, but does not constrain the student’s behavior*with*context\. We argue that robust internalization requires an additional property,*context removability*: if the model has genuinely absorbed the privileged information into its parameters, the context becomes redundant, and the student’s output should remain stable regardless of whether it is present\.
Interestingly, instead of requiring fundamental changes to the training pipeline, we find that adding a single consistency KL regularizer effectively mitigates this degradation with almost no loss in privileged fidelity\. Moreover, in many settings this regularizer even*improves*the student’s no\-context performance, the primary training objective\.
Concretely, we propose*No\-Context Anchoring*\(NCA\), which treats the student’s no\-context output as a stop\-gradient anchor and penalizes the context\-conditioned output for deviating from it, requiring only one additional forward pass per training step\. Across 12 configurations spanning diverse domains and model families, NCA improves context\-conditioned accuracy in the majority of settings, reduces context\-induced harm in 11 out of 12 settings, and effectively eliminates response\-length inflation \([Figure˜1](https://arxiv.org/html/2606.11627#S1.F1), middle and right\)\. Further mechanistic analysis confirms that NCA achieves context removability not only at the output level but also at the representation level, producing nearly identical hidden states regardless of whether the context is present\.
In summary, our main contributions are:
- •We study the phenomenon that distilled students can degrade when the privileged context is reintroduced, which we call*context\-induced degradation*, and propose*context removability*as a desirable property for robust internalization\.
- •We propose No\-Context Anchoring \(NCA\), a simple yet effective consistency regularizer that enforces context removability while simultaneously improving no\-context performance, requiring only one additional forward pass per training step\.
- •Across 12 configurations spanning diverse domains and model families, NCA improves context\-conditioned accuracy in the majority of settings, reduces context\-induced harm in 11 out of 12 settings, and effectively eliminates response\-length inflation\. A mechanistic case study further confirms that NCA achieves context removability at the representation level\.
## 2\. Related Work
##### Knowledge Distillation and On\-Policy Distillation\.
Knowledge distillation\(Hintonet al\.,[2015](https://arxiv.org/html/2606.11627#bib.bib9)\)transfers knowledge from a teacher to a student by matching output distributions, and has been widely applied to LLMs for compressing reasoning\(Guoet al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib10)\), instruction following\(Ouyanget al\.,[2022](https://arxiv.org/html/2606.11627#bib.bib11)\), and general capabilities\(Guet al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib12); Koet al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib14); Xuet al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib13)\)\. Recently, on\-policy distillation \(OPD\)\(Agarwalet al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib15); DeepSeek\-AI,[2026](https://arxiv.org/html/2606.11627#bib.bib16); Xiaoet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib17); Fuet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib20); Liet al\.,[2026b](https://arxiv.org/html/2606.11627#bib.bib19)\)generates training data from the student’s own distribution, mitigating the train\-inference distribution mismatch of off\-policy approaches\. When the teacher is the model itself conditioned on privileged context, this becomes on\-policy self\-distillation\(Zhaoet al\.,[2026a](https://arxiv.org/html/2606.11627#bib.bib4); Shenfeldet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib5); Hübotteret al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib6)\)\.Yeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\)further relax this setting by allowing the privileged\-context teacher and the student to be instantiated by different models\.Zhaoet al\.\([2026b](https://arxiv.org/html/2606.11627#bib.bib18)\)extend this idea by treating the reasoning harness itself as privileged information to be distilled\. For a broader discussion of OPD and its variants, we refer readers toSong and Zheng \([2026](https://arxiv.org/html/2606.11627#bib.bib21)\)\.
##### Extensions and Analysis of On\-Policy Distillation\.
A growing body of work analyzes the mechanisms behind on\-policy distillation\. For example,Liet al\.\([2026b](https://arxiv.org/html/2606.11627#bib.bib19)\)identify conditions governing its success and failure, whileFuet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib20)\)catalog three failure modes of sampled\-token OPD\. Complementary to these analyses, several recent works combine on\-policy self\-distillation with reinforcement learning from verifiable rewards\(Yanget al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib22); Liet al\.,[2026a](https://arxiv.org/html/2606.11627#bib.bib23)\), aiming to further improve training efficiency and reasoning performance\. Our focus is orthogonal to both lines of work: rather than explaining when OPD succeeds, diagnosing its training failures, or improving its optimization signal, we study what happens after successful distillation when the privileged context is reintroduced\. We show that optimizing for privileged fidelity alone is insufficient without an explicit context\-removability objective\.
##### Consistency Regularization\.
Consistency regularization encourages a model to produce similar outputs for semantically equivalent inputs\(Berthelotet al\.,[2019](https://arxiv.org/html/2606.11627#bib.bib24); Sohnet al\.,[2020](https://arxiv.org/html/2606.11627#bib.bib25)\)\. Recent applications include robustness across paraphrased instructions\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib26)\)and self\-rewarding alignment\(Wanget al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib27)\)in LLMs, and idempotent knowledge retention in continual learning\(Liuet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib8)\)\. These works enforce consistency across diverse aspects of model behavior\. In contrast, we apply the consistency principle to context distillation, ensuring that a distilled model remains stable when the privileged context used during training is reintroduced at inference time\.
## 3\. Problem Formulation
Consider a student modelgθg\_\{\\theta\}parameterized byθ\\theta, an inputxx, and a fixed privileged contextcc\(e\.g\., a system prompt, task hint, or game\-state scaffold\)\.111Becauseccis given and fixed for each task, the KL in[Equation1](https://arxiv.org/html/2606.11627#S3.E1)contains no irreducible entropy term\(Yanget al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib22)\)\. Yet even in this favorable setting, optimizing privileged fidelity alone does not guarantee context removability\.We denote the two student views asqx≜gθ\(y∣x\)q\_\{x\}\\triangleq g\_\{\\theta\}\(y\\mid x\)andqc≜gθ\(y∣x,c\)q\_\{c\}\\triangleq g\_\{\\theta\}\(y\\mid x,c\)\. On\-policy distillation with privileged context\(Yeet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib7)\)trains the student by minimizing:
ℒOPD=𝔼x∼𝒟\[KL\(qx∥f\(y∣x,c\)\)\],\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\left\[\\mathrm\{KL\}\\\!\\left\(q\_\{x\}\\,\\middle\\\|\\,f\(y\\mid x,c\)\\right\)\\right\],\(1\)wheref\(y∣x,c\)f\(y\\mid x,c\)is the teacher distribution\. In practice, the KL is computed at the token level and summed over the sequence\.
Note that[Equation˜1](https://arxiv.org/html/2606.11627#S3.E1)only constrainsqxq\_\{x\}\. The context\-conditioned viewqcq\_\{c\}receives no learning signal and is not directly constrained\. We argue that a necessary property for successful internalization is*context removability*: if the model has truly absorbedccinto its parameters, conditioning onccbecomes redundant and the two views should agree, i\.e\.,qx≈qcq\_\{x\}\\approx q\_\{c\}\. Although one might expect a sufficiently expressive model to satisfy this property after training, we find that this is not the case in the majority of settings\.
Figure 2:Context interaction after OPD\. x\-axis: accuracy drop upon context reintroduction; y\-axis:Harm\\mathrm\{Harm\}rate\.##### Three regimes of context interaction\.
[Figure˜2](https://arxiv.org/html/2606.11627#S3.F2)plots each of our 12 settings by the accuracy drop when context is reintroduced \(Accx−Accx,c\\mathrm\{Acc\}\_\{x\}\-\\mathrm\{Acc\}\_\{x,c\}\) againstHarm=P\(qcwrong∣qxcorrect\)\\mathrm\{Harm\}=P\(q\_\{c\}\\text\{ wrong\}\\mid q\_\{x\}\\text\{ correct\}\), i\.e\., how often a correct no\-context prediction becomes incorrect with context\. Three regimes emerge\. InRegime A\(7/12 settings\), context acts as a perturbation: the student already performs well without it, but reintroducing it causes both accuracy drops and high harm rates\. InRegime B, context still serves as a useful scaffold \(Accx,c\>Accx\\mathrm\{Acc\}\_\{x,c\}\>\\mathrm\{Acc\}\_\{x\}\), indicating incomplete internalization\. InRegime C, the two views already agree and harm is minimal\. The prevalence of Regime A motivates the need for an explicit context\-removability objective\.
## 4\. Method: No\-Context Anchoring
### 4\.1\. From Constrained to Regularized Objective
The analysis in[Section˜3](https://arxiv.org/html/2606.11627#S3.SS0.SSS0.Px1)shows that the standard training objective does not enforce context removability\. A natural fix is to explicitly constrain the divergence between the two views while optimizing privileged fidelity:
minθℒOPD\(θ\)s\.t\.D\(qx,qc\)≤ϵ,\\min\_\{\\theta\}\\;\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)\\quad\\text\{s\.t\.\}\\quad D\\\!\\left\(q\_\{x\},\\,q\_\{c\}\\right\)\\leq\\epsilon,\(2\)whereDDis a divergence between the two views\. In practice, we relax this into a penalized form with a fixed weightβ\>0\\beta\>0:
ℒ=ℒOPD\+β𝔼x∼𝒟\[D\(qx,qc\)\]\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\+\\beta\\,\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\\!\\left\[D\(q\_\{x\},q\_\{c\}\)\\right\]\.\(3\)
### 4\.2\. Choice of Divergence
The divergenceDDin[Equation˜3](https://arxiv.org/html/2606.11627#S4.E3)can take many forms\. We choose the forward KL divergenceD=KL\(qx∥qc\)D=\\mathrm\{KL\}\(q\_\{x\}\\\|q\_\{c\}\), which is mode\-covering and thus preventsqcq\_\{c\}from collapsing modes ofqxq\_\{x\}\. An additional benefit is that the expectation falls underqxq\_\{x\}, allowing us to reuse the on\-policy samples already generated by OPD without a separate rollout fromqcq\_\{c\}\. To further avoid the gradient of this term interfering with the OPD objective throughqxq\_\{x\}, we apply a stop\-gradient to anchor the no\-context view\. We call the resulting regularizer*No\-Context Anchoring*\(NCA\):
ℒNCA≜KL\(sg\[gθ\(⋅∣x\)\]∥gθ\(⋅∣x,c\)\),\\mathcal\{L\}\_\{\\mathrm\{NCA\}\}\\triangleq\\mathrm\{KL\}\\\!\\left\(\\operatorname\{sg\}\\\!\\left\[g\_\{\\theta\}\(\\cdot\\mid x\)\\right\]\\,\\middle\\\|\\,g\_\{\\theta\}\(\\cdot\\mid x,c\)\\right\),\(4\)wheresg\[⋅\]\\operatorname\{sg\}\[\\cdot\]denotes the stop\-gradient operator\.
This ensures thatqxq\_\{x\}receives gradients only from the OPD loss, whileqcq\_\{c\}receives gradients only fromℒNCA\\mathcal\{L\}\_\{\\mathrm\{NCA\}\}, eliminating gradient competition between the two objectives\. We compare against alternative divergence choices in our design analysis \([Section˜5\.4](https://arxiv.org/html/2606.11627#S5.SS4)\)\.
### 4\.3\. Computational Cost
The only additional cost of NCA is one forward pass per training step forqc=gθ\(⋅∣x,c\)q\_\{c\}=g\_\{\\theta\}\(\\cdot\\mid x,c\): thesg\[qx\]\\operatorname\{sg\}\[q\_\{x\}\]logits are detached from the forward pass that OPD already performs, and the on\-policy sequences from OPD are reused to evaluateℒNCA\\mathcal\{L\}\_\{\\mathrm\{NCA\}\}without a separate rollout\. The full procedure is summarized in[Algorithm˜1](https://arxiv.org/html/2606.11627#alg1)\.
Algorithm 1OPD with No\-Context Anchoring \(NCA\)0:Dataset
𝒟\\mathcal\{D\}, student
gθg\_\{\\theta\}, teacher
ff, context
cc, weight
β\\beta
1:foreach batch
\{xi\}∼𝒟\\\{x\_\{i\}\\\}\\sim\\mathcal\{D\}do
2:Sample
yi∼gθ\(⋅∣xi\)y\_\{i\}\\sim g\_\{\\theta\}\(\\cdot\\mid x\_\{i\}\)\(on\-policy rollout\)
3:
ℒOPD←1\|ℬ\|∑i∑tKL\(gθ\(⋅∣xi,yi,<t\)∥f\(⋅∣xi,c,yi,<t\)\)\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{i\}\\sum\_\{t\}\\mathrm\{KL\}\\\!\\left\(g\_\{\\theta\}\(\\cdot\\mid x\_\{i\},y\_\{i,<t\}\)\\,\\middle\\\|\\,f\(\\cdot\\mid x\_\{i\},c,y\_\{i,<t\}\)\\right\)
4:
ℒNCA←1\|ℬ\|∑i∑tKL\(sg\[gθ\(⋅∣xi,yi,<t\)\]∥gθ\(⋅∣xi,c,yi,<t\)\)\\mathcal\{L\}\_\{\\mathrm\{NCA\}\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{i\}\\sum\_\{t\}\\mathrm\{KL\}\\\!\\left\(\\operatorname\{sg\}\\\!\\left\[g\_\{\\theta\}\(\\cdot\\mid x\_\{i\},y\_\{i,<t\}\)\\right\]\\,\\middle\\\|\\,g\_\{\\theta\}\(\\cdot\\mid x\_\{i\},c,y\_\{i,<t\}\)\\right\)
5:Update
θ\\thetavia
∇θ\(ℒOPD\+β⋅ℒNCA\)\\nabla\_\{\\theta\}\(\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\+\\beta\\cdot\\mathcal\{L\}\_\{\\mathrm\{NCA\}\}\)
6:endfor
## 5\. Experiments
We evaluate No\-Context Anchoring \(NCA\) across 12 configurations spanning two domains \(system\-prompt QA and text games\), four tasks, and six models, aiming to answer: \(1\) How broadly does context\-induced degradation occur? \(2\) Can NCA mitigate it while preserving no\-context performance? \(3\) How does the choice of divergence and the regularization strengthβ\\betaaffect NCA? \(4\) What mechanisms underlie NCA’s effect on the model’s internal representations?
### 5\.1\. Experimental Setup
Tasks\.Our experiments cover two types of tasks\.*System\-prompt QA*: the privileged context is a task\-specific system prompt that the teacher uses but the student must internalize\. We use MedMCQA\(Palet al\.,[2022](https://arxiv.org/html/2606.11627#bib.bib28)\), a medical multiple\-choice QA dataset \(500 test samples\), and a Safety dataset combining Tweet Eval\(Barbieriet al\.,[2020](https://arxiv.org/html/2606.11627#bib.bib30)\), Hatecheck\(Röttgeret al\.,[2021](https://arxiv.org/html/2606.11627#bib.bib31)\), and Ethos\(Mollaset al\.,[2022](https://arxiv.org/html/2606.11627#bib.bib32)\)for harmful content classification \(498 test samples\)\.*Text games*: the privileged context is a game\-state scaffold \(e\.g\., optimal move hints\)\. We use Sokoban \(6×\\times6 grid, 1 box, spatial reasoning\) and FrozenLake \(3×\\times3 grid, 2 holes, navigation\) from TextArena\(Guertleret al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib33)\), both with a maximum of 5 game steps\. We use the QA datasets and text\-game configurations fromYeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\)\.
Models\.For system\-prompt QA, we evaluate four models: Llama\-3\.1\-8B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib34)\), Llama\-3\.2\-3B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2606.11627#bib.bib34)\), Qwen2\.5\-7B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib36)\), and Qwen3\-8B\(Yanget al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib35)\)\(no\-thinking mode\), each serving as both teacher and student\. For text games, Qwen3\-4B\-Instruct\-2507\(Yanget al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib35)\)serves as the teacher, with the same model as the student for self\-distillation and Qwen3\-1\.7B\(Yanget al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib35)\)\(no\-thinking mode\) for cross\-model distillation\.
Training\.All models are trained with full parameter fine\-tuning for 100 steps with learning rate5×10−65\\times 10^\{\-6\}\. The batch size for system\-prompt QA is 128, and for text games is 64\. We setβ=0\.5\\beta=0\.5for all settings except Qwen3\-4B\-Instruct\-2507→\\toQwen3\-1\.7B where we useβ=0\.1\\beta=0\.1\. For system\-prompt QA, since different models have varying in\-context learning abilities, we use MetaSPO\(Choiet al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib29)\)to optimize suitable system prompts for each base model\. For text games, we adopt the prompt templates fromYeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\)and provide game\-specific experience as the privileged context for each scenario\. The detailed system prompts and game\-specific experiences are provided in Appendix[A](https://arxiv.org/html/2606.11627#A1)\. The maximum response length is 512 tokens for system\-prompt QA and 1024 tokens for text games\.
Evaluation\.We evaluate every 10 training steps under both views \(no context and context reintroduced\) using the following metrics:
- •Accx\\mathrm\{Acc\}\_\{x\}/WRx\\mathrm\{WR\}\_\{x\}andAccx,c\\mathrm\{Acc\}\_\{x,c\}/WRx,c\\mathrm\{WR\}\_\{x,c\}: task performance \(accuracy for QA, win rate for games\) under the no\-context and context\-reintroduced views, respectively\.
- •Harm\\mathrm\{Harm\}: conditional probability of context flipping a correct prediction to incorrect, i\.e\.,P\(qcwrong∣qxcorrect\)P\(q\_\{c\}\\text\{ wrong\}\\mid q\_\{x\}\\text\{ correct\}\)\.
- •BothCorrect\\mathrm\{BothCorrect\}/BothWon\\mathrm\{BothWon\}: fraction of instances correct \(or games won\) under both views\.
- •Δlen\\Delta\_\{\\mathrm\{len\}\}: relative response length change \(%\) of the context\-reintroduced view compared to the OPD no\-context baseline \(QA only\)\.
For system\-prompt QA, we use greedy decoding and evaluate on the full test set\. For text games, we evaluate on 128 game instances, each with a single stochastic rollout \(T=0\.7T\{=\}0\.7, top\-p=0\.8p\{=\}0\.8, top\-k=20k\{=\}20\)\. We report the average performance over the last three checkpoints\. Throughout the experiments, we use OPD to refer specifically to on\-policy distillation with privileged context\. Our baseline is implemented based on the codebase ofYeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\), and we compare OPD against OPD \+ NCA \(Ours\), with additional divergence andβ\\betacomparisons in[Section˜5\.4](https://arxiv.org/html/2606.11627#S5.SS4)\.
Implementation\.Our code is built on top of the codebase ofYeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\), using veRL\(Shenget al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib37)\)as the training engine with FSDP for distributed training and vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2606.11627#bib.bib38)\)for on\-policy rollout generation\. All KL divergences are computed over the top\-k=256k\{=\}256logits with renormalization to prevent reward hacking from low\-probability tokens\. All experiments are conducted on a single server equipped with 8×\\timesNVIDIA H100 GPUs\.
### 5\.2\. Evaluation Results
#### 5\.2\.1System\-Prompt QA
Table 1:System\-prompt QA results on MedMCQA and Safety\. All metrics in %\. Best across Base/OPD/Ours per cell inbold\. Ours deltas vs\. OPD ingreen \(improvement\)/red \(degradation\)\. Values averaged over last three checkpoints\.†No\-thinking mode\.
[Table˜1](https://arxiv.org/html/2606.11627#S5.T1)summarizes system\-prompt QA results, which reveal several interesting patterns\.
Context\-induced degradation is pervasive\.Under OPD, all eight settings exhibit non\-trivialHarm\\mathrm\{Harm\}rates\. The most affected is Llama\-3\.2\-3B on MedMCQA, whereHarm\\mathrm\{Harm\}reaches 15\.7% andAccx,c\\mathrm\{Acc\}\_\{x,c\}drops 9\.5 points belowAccx\\mathrm\{Acc\}\_\{x\}, followed by Llama\-3\.1\-8B on both tasks, all falling in Regime A\. Representing a Regime C case, Qwen3\-8B already achieves strong base performance, and OPD actually*hurts*it relative to the base \(Accx\\mathrm\{Acc\}\_\{x\}:71\.2→69\.371\.2\\to 69\.3on MedMCQA,79\.3→78\.579\.3\\to 78\.5on Safety\), likely because further fine\-tuning leads to mild overfitting rather than improvement\. Despite this, NCA partially recoversAccx\\mathrm\{Acc\}\_\{x\}from69\.369\.3to69\.969\.9on MedMCQA and from78\.578\.5to79\.279\.2on Safety, suggesting that the consistency constraint acts as a useful regularizer even when the distillation signal itself provides limited benefit\.
NCA consistently reduces harm and improves stability\.NCA reducesHarm\\mathrm\{Harm\}in all eight settings and improvesBothCorrect\\mathrm\{BothCorrect\}across the board, meaning the student maintains correct answers under both views more reliably\. The largest improvements appear in Regime A settings: Llama\-3\.2\-3B MedMCQA seesHarm\\mathrm\{Harm\}drop by 14\.1 percentage points andBothCorrect\\mathrm\{BothCorrect\}improve by 10\.4 points\.
Response\-length inflation is largely eliminated\.Under OPD, seven of eight settings show substantial length inflation when context is reintroduced, withΔlen\\Delta\_\{\\mathrm\{len\}\}reaching up to \+39\.5% for Llama\-3\.2\-3B MedMCQA, indicating that the model generates notably more verbose responses under the context\-conditioned view\. NCA reduces this in seven of eight settings, five of which achieveΔlen≤3%\\Delta\_\{\\mathrm\{len\}\}\\leq 3\\%\. The most notable reduction is Llama\-3\.2\-3B on MedMCQA, where NCA reducesΔlen\\Delta\_\{\\mathrm\{len\}\}by 37\.4 percentage points\.
NCA often improves no\-context accuracy\.Interestingly, although NCA only regularizes the context\-conditioned viewqcq\_\{c\}, in seven of eight settings it also improvesAccx\\mathrm\{Acc\}\_\{x\}\(by 0\.6–3\.6 points\)\. This suggests that the consistency constraint provides an indirect regularization benefit to the no\-context pathway through shared parameters: by preventingqcq\_\{c\}from drifting, the parameter updates become more focused on genuine task learning rather than accommodating conflicting behaviors across views\. The largest gain is Qwen2\.5\-7B on MedMCQA \(\+3\.6\), where the consistency pressure forces more robust internalization\.
Boundary case\.On Qwen2\.5\-7B MedMCQA, a Regime B setting where context still carries useful information that OPD has not fully internalized,Accx,c\\mathrm\{Acc\}\_\{x,c\}drops by 2\.0 points under NCA\. Nevertheless, NCA still improvesAccx\\mathrm\{Acc\}\_\{x\}by 3\.6 points and reducesHarm\\mathrm\{Harm\}by 4\.2 points on this setting, suggesting that enforcing context removability has potential benefits for internalization even when it comes at some cost toAccx,c\\mathrm\{Acc\}\_\{x,c\}\.
#### 5\.2\.2Text Games
Table 2:Text game results on Sokoban and FrozenLake\. All metrics in %\. Best across Base/OPD/Ours per cell inbold\. Ours deltas vs\. OPD ingreen \(improvement\)/red \(degradation\)\. Values averaged over last three checkpoints\.†No\-thinking mode\.
[Table˜2](https://arxiv.org/html/2606.11627#S5.T2)shows text game results\. Unlike QA, text games require multi\-step reasoning and sequential decision\-making, where the interaction between context and learned policy is richer\.
Self\-distillation\.Under OPD, Qwen3\-4B\-Instruct\-2507 self\-distillation showsHarm\\mathrm\{Harm\}of 41\.2% on FrozenLake and 22\.1% on Sokoban, considerably higher than in QA, indicating that context\-induced degradation is more severe in this domain\. NCA substantially reducesHarm\\mathrm\{Harm\}in both cases \(by 21\.5 and 14\.0 percentage points respectively\) while improving win rates substantially on FrozenLake \(WRx\\mathrm\{WR\}\_\{x\}\+5\.7,WRx,c\\mathrm\{WR\}\_\{x,c\}\+12\.8\) and significantly boostingWRx,c\\mathrm\{WR\}\_\{x,c\}on Sokoban \(\+9\.9\) with almost no loss inWRx\\mathrm\{WR\}\_\{x\}\.
Cross\-model distillation\.For Qwen3\-4B\-Instruct\-2507→\\toQwen3\-1\.7B, the two games exhibit contrasting behaviors\. On Sokoban, OPD produces severeHarm\\mathrm\{Harm\}of35\.7%35\.7\\%, yet NCA brings it down to2\.4%2\.4\\%while boosting bothWRx\\mathrm\{WR\}\_\{x\}\(\+1\.6\) andWRx,c\\mathrm\{WR\}\_\{x,c\}\(\+20\.8\)\. On FrozenLake, however, the situation is reversed, as we discuss next\.
Boundary case\.FrozenLake with Qwen3\-4B\-Instruct\-2507→\\toQwen3\-1\.7B falls into Regime B and is the only setting where NCA increasesHarm\\mathrm\{Harm\}\(from5\.6%5\.6\\%to13\.3%13\.3\\%\)\. An unusual observation is that the Qwen3\-1\.7B student under OPD achievesWRx,c=86\.5%\\mathrm\{WR\}\_\{x,c\}=86\.5\\%, surpassing even the Qwen3\-4B\-Instruct\-2507 teacher with context \(75\.8%75\.8\\%\), suggesting that on\-policy training has led the student into a policy mode that the teacher evaluates as superior to its own\. NCA suppresses this amplified benefit \(WRx,c\\mathrm\{WR\}\_\{x,c\}−\-16\.4\) while only marginally improvingWRx\\mathrm\{WR\}\_\{x\}\(\+1\.1\)\. The mechanism behind this phenomenon remains unclear and warrants further investigation\. This case also inspires a promising future direction: dynamically adjusting the anchor based on the degree of internalization, rather than always anchoring to the no\-context view\.
Figure 3:Overall effect of NCA across all 12 settings\. Left: OPD\. Right: NCA\. Each point is one \(model, task\) setting, colored by regime\. NCA moves most points toward the origin, reducing both accuracy drop and harm rate\.[Figure˜3](https://arxiv.org/html/2606.11627#S5.F3)provides a bird’s\-eye view of NCA’s effect across all 12 settings\. Under OPD \(left\), Regime A points are scattered in the high\-harm, high\-accuracy\-drop region\. After applying NCA \(right\), nearly all points cluster near the origin, confirming that NCA broadly reduces context\-induced degradation across diverse models and tasks\.
### 5\.3\. Training Dynamics
\(a\)Llama\-3\.1\-8B\-Instruct, MedMCQA \(Regime A\)
\(b\)Qwen3\-4B\-Ins→\\to1\.7B, Sokoban \(Regime A\)
\(c\)Qwen3\-4B\-Ins→\\to1\.7B, FrozenLake \(Regime B\)
\(d\)Qwen3\-8B, Safety \(Regime C\)
Figure 4:Training dynamics across three regimes\. Each row shows accuracy \(left\), harm rate \(middle\), and response length \(right\)\. \(a–b\): In Regime A, NCA closes the accuracy gap and reduces harm\. \(c\): In Regime B, NCA trades someAccx,c\\mathrm\{Acc\}\_\{x,c\}for improvedAccx\\mathrm\{Acc\}\_\{x\}\. \(d\): In Regime C, both methods behave similarly with low harm\.To understand when and how NCA’s effect takes hold beyond the final metrics, we examine how the two views evolve over training\.[Figure˜4](https://arxiv.org/html/2606.11627#S5.F4)shows representative curves across the three regimes \(complete curves for all 12 settings are in[Appendix˜B](https://arxiv.org/html/2606.11627#A2)\)\.
In Regime A \([Figure˜4](https://arxiv.org/html/2606.11627#S5.F4)a–b\), OPD progressively improvesAccx\\mathrm\{Acc\}\_\{x\}butAccx,c\\mathrm\{Acc\}\_\{x,c\}stagnates or drops, widening the gap\. NCA keeps the two views tightly coupled\. On Llama\-3\.1\-8B MedMCQA \([Figure˜4](https://arxiv.org/html/2606.11627#S5.F4)a\), the gap narrows steadily from step 20 withHarm\\mathrm\{Harm\}suppressed throughout\. On Sokoban cross\-model \([Figure˜4](https://arxiv.org/html/2606.11627#S5.F4)b\), NCA prevents the sharpHarm\\mathrm\{Harm\}spike that OPD exhibits in the first 30 steps\. In Regime C \([Figure˜4](https://arxiv.org/html/2606.11627#S5.F4)d\), Qwen3\-8B Safety shows minimal differences, with both methods maintaining lowHarm\\mathrm\{Harm\}throughout, confirming that NCA does not hurt when the gap is already small\.
Regime B \([Figure˜4](https://arxiv.org/html/2606.11627#S5.F4)c\) reveals a more nuanced dynamic that may explain the anomalous student\-surpasses\-teacher behavior noted in[Section˜5\.2\.2](https://arxiv.org/html/2606.11627#S5.SS2.SSS2)\. TheHarm\\mathrm\{Harm\}curve shows a dramatic early spike \(∼70%\{\\sim\}70\\%at step 10–20\) followed by a sharp recovery to∼4%\{\\sim\}4\\%\. It suggests that the student, despite never seeing context during training, stumbles into a strong policy mode that both achieves high win rates and is highly favored by the context\-conditioned teacher\. NCA suppresses this spike \(∼30%\{\\sim\}30\\%\) but also blocks the recovery, ending with higherHarm\\mathrm\{Harm\}\(∼15%\{\\sim\}15\\%\)\.
Response\-length inflation \(right column\) is reduced by NCA in most settings, with Regime A showing the most dramatic compression\.
### 5\.4\. Design Choice Analysis
To examine the effect of divergence choice and the sensitivity toβ\\beta, we conduct experiments on Llama\-3\.1\-8B\-Instruct MedMCQA\. Results are shown in[Figure˜5](https://arxiv.org/html/2606.11627#S5.F5)\.
\(a\)Divergence comparison\.
\(b\)β\\betasensitivity\.
Figure 5:Design choice analysis on Llama\-3\.1\-8B\-Instruct MedMCQA\.Alternative divergences\.In[Figure˜5\(a\)](https://arxiv.org/html/2606.11627#S5.F5.sf1), we compare NCA against two additional choices:
- •DualOPD: directly alignsqcq\_\{c\}to the teacher viaKL\(qc∥f\(y∣x,c\)\)\\mathrm\{KL\}\(q\_\{c\}\\\|f\(y\\mid x,c\)\), without explicitly enforcingqx≈qcq\_\{x\}\\approx q\_\{c\}\.
- •SymKL: symmetrically aligns the two views via12KL\(sg\[qx\]∥qc\)\+12KL\(sg\[qc\]∥qx\)\\frac\{1\}\{2\}\\mathrm\{KL\}\(\\operatorname\{sg\}\[q\_\{x\}\]\\\|q\_\{c\}\)\+\\frac\{1\}\{2\}\\mathrm\{KL\}\(\\operatorname\{sg\}\[q\_\{c\}\]\\\|q\_\{x\}\), where both distributions are stop\-gradiented when serving as the reference\.
DualOPD learnsAccx\\mathrm\{Acc\}\_\{x\}most slowly and suffers a severe earlyAccx,c\\mathrm\{Acc\}\_\{x,c\}crash \(dropping to∼68%\{\\sim\}68\\%at step 20\) before gradually recovering\. Since the teacher distribution is typically multi\-modal, separately aligningqxq\_\{x\}andqcq\_\{c\}to it does not guarantee their mutual consistency: the two views can converge to different modes, and the additional distillation signal toqcq\_\{c\}can interfere with the primary OPD objective rather than help\. SymKL consistently underperforms inAccx\\mathrm\{Acc\}\_\{x\}, likely because its bidirectional gradient pullsqxq\_\{x\}away from the teacher\. NCA maintains the most stable performance across both views: itsAccx\\mathrm\{Acc\}\_\{x\}remains competitive with OPD while itsAccx,c\\mathrm\{Acc\}\_\{x,c\}stays high throughout, supporting our choice of forward KL with stop\-gradient onqxq\_\{x\}\.
Sensitivity toβ\\beta\.In[Figure˜5\(b\)](https://arxiv.org/html/2606.11627#S5.F5.sf2), we evaluateβ∈\{0\.1,0\.5,1\.0\}\\beta\\in\\\{0\.1,0\.5,1\.0\\\}\. All three converge to similarAccx\\mathrm\{Acc\}\_\{x\}\(∼77%\{\\sim\}77\\%\) but differ notably inAccx,c\\mathrm\{Acc\}\_\{x,c\}dynamics\.β=0\.1\\beta=0\.1is too weak to prevent early context\-induced drift, causingAccx,c\\mathrm\{Acc\}\_\{x,c\}to drop sharply to∼70%\{\\sim\}70\\%at step 20 before slowly recovering\.β=1\.0\\beta=1\.0maintains highAccx,c\\mathrm\{Acc\}\_\{x,c\}but slightly hurtsAccx\\mathrm\{Acc\}\_\{x\}\.β=0\.5\\beta=0\.5strikes the best balance, with its advantage over OPD becoming more pronounced in the later stages of training\. Notably, all threeβ\\betavalues improveAccx,c\\mathrm\{Acc\}\_\{x,c\}over OPD by the end of training, suggesting that NCA’s benefit to context\-conditioned performance is robust across a wide range ofβ\\beta\.
### 5\.5\. Mechanistic Case Study
\(a\)Per\-layer cosine similarity between context and no\-context hidden states\.
\(b\)Per\-layer gradient cosine similarity for OPD \(left\) and NCA \(right\)\.
Figure 6:Representation and gradient analysis on Qwen3\-4B\-Instruct\-2507, Sokoban\.To understand*how*NCA achieves context removability at the representation level, we conduct a case study on 16 randomly sampled Sokoban instances using Qwen3\-4B\-Instruct\-2507, comparing the base model against the OPD and NCA checkpoints at step 100\.
Hidden\-state similarity\.For each instance, we generate a responseyywithout context and then feed the same\(x,y\)\(x,y\)pair through both views \(with and without context\), computing the per\-token cosine similarity of hidden states at each layer and averaging over all tokens\.[Figure˜6\(a\)](https://arxiv.org/html/2606.11627#S5.F6.sf1)plots the mean and 95% confidence interval across the 16 instances\. The base model shows a gradual decline starting around layer 10, reaching a trough of approximately0\.9530\.953at layers 22–25, with a further sharp drop at the final layer\. OPD narrows this gap \(trough approximately0\.9850\.985\) but a noticeable divergence persists in middle and late layers\. NCA maintains near\-perfect similarity \(≥0\.997\{\\geq\}0\.997\) across all 36 layers, confirming context removability at the representation level\.
Gradient alignment\.For both the OPD\-only and NCA checkpoints, we generate on\-policy rollouts from the 16 instances and compute the gradients ofℒOPD\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}andℒNCA\\mathcal\{L\}\_\{\\mathrm\{NCA\}\}at each layer and parameter module, then measure their cosine similarity averaged across instances \([Figure˜6\(b\)](https://arxiv.org/html/2606.11627#S5.F6.sf2)\)\. Under the OPD\-only checkpoint \(left panel\), the two gradients are approximately orthogonal \(near\-zero similarity\) across most modules, indicating that OPD has converged to a point where it provides no learning signal along the direction that NCA targets\. This suggests that NCA fills a gap left by OPD, addressing context removability without interfering with the privileged fidelity that OPD has already achieved\. Under the jointly trained NCA checkpoint \(right panel\), the heatmap becomes predominantly negative \(blue\), indicating that the two objectives reach an equilibrium at the trained parameters: OPD pushes the layers to leverage context for distillation, while NCA pushes them to remain invariant to context, and the opposing gradients confirm that both forces are actively balanced\.
\(a\)Relative Frobenius norm‖ΔW‖F/‖Wbase‖F\\\|\\Delta W\\\|\_\{F\}/\\\|W\_\{\\mathrm\{base\}\}\\\|\_\{F\}of parameter updates\.
\(b\)Rank structure of parameter updates, normalized by the maximum possible rank\.
Figure 7:Parameter\-level analysis on Qwen3\-4B\-Instruct\-2507, Sokoban\.Parameter updates\.We compute the relative Frobenius norm‖ΔW‖F/‖Wbase‖F\\\|\\Delta W\\\|\_\{F\}/\\\|W\_\{\\mathrm\{base\}\}\\\|\_\{F\}for each parameter matrix at each layer \([Figure˜7\(a\)](https://arxiv.org/html/2606.11627#S5.F7.sf1)\)\. Both OPD and NCA concentrate parameter changes in the first∼20\{\\sim\}20layers, particularly in k\_proj and q\_proj across all layers\. Within the MLP, down\_proj and up\_proj updates are more prominent in shallow layers, while gate\_proj updates are stronger in deeper layers\. This observation is consistent with recent layer\-pruning analyses associating shallow layers more strongly with knowledge\- and retrieval\-oriented behaviors, and middle\-to\-deeper layers with generation\-based reasoning\(Songet al\.,[2026](https://arxiv.org/html/2606.11627#bib.bib39)\)\. This suggests that context internalization primarily modifies shallow\-layer knowledge representations, with smaller but non\-negligible changes propagating to middle and deeper layers that shape generation and reasoning behavior\. The transition zone around layers 15–20, where parameter updates begin to taper off, coincides with where the base model’s hidden\-state similarity decline accelerates in[Figure˜6\(a\)](https://arxiv.org/html/2606.11627#S5.F6.sf1), suggesting that context removability is largely determined by whether these early\-to\-middle layers receive sufficient modification to prevent the representational divergence from forming\.
The NCA\-vs\-OPD difference reveals that the additional updates from NCA are concentrated in shallow layers and k\_proj, and notably, these differences are often*larger*than either method’s deviation from the base model in the same regions\. This indicates that NCA and OPD push these parameters in substantially different directions, suggesting that context removability is primarily achieved through shallow\-layer modifications that the OPD loss alone does not produce\.
Rank structure of parameter updates\.To assess whether NCA changes the nature of the learned updates, we perform SVD on each parameter update matrixΔW∈ℝm×n\\Delta W\\in\\mathbb\{R\}^\{m\\times n\}as shown in[Figure˜7\(b\)](https://arxiv.org/html/2606.11627#S5.F7.sf2)\. The Rank99 ratio measures the minimum number of singular values needed to capture 99% of the total squared singular value mass, normalized bymin\(m,n\)\\min\(m,n\), which is the maximum possible rank\. The effective\-rank ratio is the exponential of the entropy of the normalized squared singular value distribution, similarly normalized\. Both OPD and NCA show high Rank99 ratios of approximately0\.80\.8–0\.950\.95but much lower effective\-rank ratios of approximately0\.20\.2–0\.30\.3, indicating that the updates are not simple low\-rank perturbations yet are dominated by a few principal directions with a heavy\-tailed singular value distribution\. Despite producing substantially different update directions \([Figure˜7\(a\)](https://arxiv.org/html/2606.11627#S5.F7.sf1)\), the two methods maintain nearly identical rank profiles, suggesting that NCA changes*where*and*in what direction*parameters are modified, but not*how*the updates are structured\.
### 5\.6\. Limitations
We acknowledge several limitations of the current study\. First, due to computational constraints, all experiments use a single training seed, and multi\-seed verification would strengthen the findings\. Second, the mechanistic analysis in[Section˜5\.5](https://arxiv.org/html/2606.11627#S5.SS5)examines a single Sokoban case study on one model, and extending it to more instances, tasks, and architectures would strengthen the representational findings\. Third, the method is designed for the OPD setting and may require adaptation for other distillation paradigms\.
## 6\. Conclusion
In this paper, we identify context\-induced degradation, an interesting and previously unstudied phenomenon in on\-policy distillation with privileged context, and propose No\-Context Anchoring \(NCA\), a lightweight consistency regularizer that enforces context removability with only one additional forward pass\. Across 12 configurations, NCA reduces context\-induced harm in 11 settings and often improves no\-context performance as well\. Our analysis reveals three regimes of context interaction that govern the method’s effect, and a mechanistic case study confirms that NCA achieves context removability at the representation level\. We hope this work encourages the community to evaluate privileged\-context distillation not only by no\-context performance but also by robustness to context reintroduction\.
## Acknowledgments
We thank Tianzhu Ye for helpful discussions and guidance on using the released code for reproduction\.
## References
- On\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 21246–21263\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5be69a584901a26c521c2b51e40a4c20-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- F\. Barbieri, J\. Camacho\-Collados, L\. Espinosa Anke, and L\. Neves \(2020\)TweetEval: unified benchmark and comparative evaluation for tweet classification\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1644–1650\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.148/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.148)Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p1.2)\.
- D\. Berthelot, N\. Carlini, I\. Goodfellow, N\. Papernot, A\. Oliver, and C\. Raffel \(2019\)MixMatch: a holistic approach to semi\-supervised learning\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/1cd138d0499a68f4bb72bee04bbec2d7-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2606.11627#S1.p1.1)\.
- Y\. Choi, J\. Baek, and S\. J\. Hwang \(2025\)System prompt optimization with meta\-learning\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 55451–55488\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/5000f096bed9360a060d835c2a1703bb-Paper-Conference.pdf)Cited by:[§A\.1](https://arxiv.org/html/2606.11627#A1.SS1.p1.1),[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p3.4)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- Q\. Dong, L\. Li, D\. Dai, C\. Zheng, J\. Ma, R\. Li, H\. Xia, J\. Xu, Z\. Wu, B\. Chang, X\. Sun, L\. Li, and Z\. Sui \(2024\)A survey on in\-context learning\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1107–1128\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.64/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.64)Cited by:[§1](https://arxiv.org/html/2606.11627#S1.p1.1)\.
- Y\. Fu, H\. Huang, K\. Jiang, J\. Liu, Z\. Jiang, Y\. Zhu, and D\. Zhao \(2026\)Revisiting on\-policy distillation: empirical failure modes and simple fixes\.arXiv preprint arXiv:2603\.25562\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p2.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 32694–32717\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/8ac015d409635f196f9e3e9dcfb9a94e-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Guertler, B\. Cheng, S\. Yu, B\. Liu, L\. Choshen, and C\. Tan \(2025\)Textarena\.arXiv preprint arXiv:2504\.11442\.Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p1.2)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang \(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](https://doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§1](https://arxiv.org/html/2606.11627#S1.p1.1),[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: towards streamlined distillation for large language models\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 24872–24895\.External Links:[Link](https://proceedings.mlr.press/v235/ko24c.html)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p5.2)\.
- G\. Li, T\. Yang, J\. Fang, M\. Song, M\. Zheng, H\. Guo, D\. Zhang, J\. Wang, and T\. Chua \(2026a\)Unifying group\-relative and self\-distillation policy optimization via sample routing\.arXiv preprint arXiv:2604\.02288\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026b\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.External Links:2604\.13016,[Link](https://arxiv.org/abs/2604.13016)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, Y\. Li, H\. Gao, Y\. Li, L\. Kong, L\. Sun, and W\. Huang \(2026\)IDER: IDempotent experience replay for reliable continual learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Vr5f3kRvLD)Cited by:[§1](https://arxiv.org/html/2606.11627#S1.p2.1),[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px3.p1.1)\.
- I\. Mollas, Z\. Chrysopoulou, S\. Karlos, and G\. Tsoumakas \(2022\)ETHOS: a multi\-label hate speech detection dataset\.Complex & Intelligent Systems8\(6\),pp\. 4663–4678\.External Links:ISSN 2198\-6053,[Link](https://doi.org/10.1007/s40747-021-00608-2),[Document](https://dx.doi.org/10.1007/s40747-021-00608-2)Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p1.2)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Pal, L\. K\. Umapathi, and M\. Sankarasubbu \(2022\)MedMCQA: a large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.InProceedings of the Conference on Health, Inference, and Learning,G\. Flores, G\. H\. Chen, T\. Pollard, J\. C\. Ho, and T\. Naumann \(Eds\.\),Proceedings of Machine Learning Research, Vol\.174,pp\. 248–260\.External Links:[Link](https://proceedings.mlr.press/v174/pal22a.html)Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p1.2)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p2.1)\.
- P\. Röttger, B\. Vidgen, D\. Nguyen, Z\. Waseem, H\. Margetts, and J\. Pierrehumbert \(2021\)HateCheck: functional tests for hate speech detection models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 41–58\.External Links:[Link](https://aclanthology.org/2021.acl-long.4/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.4)Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p1.2)\.
- I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal \(2026\)Self\-distillation enables continual learning\.arXiv preprint arXiv:2601\.19897\.Cited by:[§1](https://arxiv.org/html/2606.11627#S1.p1.1),[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)HybridFlow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,EuroSys ’25,New York, NY, USA,pp\. 1279–1297\.External Links:ISBN 9798400711961,[Link](https://doi.org/10.1145/3689031.3696075),[Document](https://dx.doi.org/10.1145/3689031.3696075)Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p5.2)\.
- K\. Sohn, D\. Berthelot, N\. Carlini, Z\. Zhang, H\. Zhang, C\. Raffel, E\. D\. Cubuk, A\. Kurakin, and C\. Li \(2020\)FixMatch: simplifying semi\-supervised learning with consistency and confidence\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 596–608\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/06964dce9addb1c5cb5d6e3d9838f733-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Song and M\. Zheng \(2026\)A survey of on\-policy distillation for large language models\.arXiv preprint arXiv:2604\.00626\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Song, K\. Wang, P\. Li, L\. Yin, and S\. Liu \(2026\)Demystifying the roles of llm layers in retrieval, knowledge, and reasoning\.InICASSP 2026 \- 2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 16792–16796\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP55912.2026.11462157)Cited by:[§5\.5](https://arxiv.org/html/2606.11627#S5.SS5.p4.2)\.
- Z\. Wang, W\. He, Z\. Liang, X\. Zhang, C\. Bansal, Y\. Wei, W\. Zhang, and H\. Yao \(2025\)CREAM: consistency regularized self\-rewarding language models\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 57017–57039\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/8f9d459c19b59b5400ce396e0f8c23e0-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, b\. ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2606.11627#S1.p1.1)\.
- B\. Xiao, B\. Xia, B\. Yang, B\. Gao, B\. Shen, C\. Zhang, C\. He, C\. Lou, F\. Luo, G\. Wang,et al\.\(2026\)Mimo\-v2\-flash technical report\.arXiv preprint arXiv:2601\.02780\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Xu, M\. Li, C\. Tao, T\. Shen, R\. Cheng, J\. Li, C\. Xu, D\. Tao, and T\. Zhou \(2024\)A survey on knowledge distillation of large language models\.arXiv preprint arXiv:2402\.13116\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p2.1)\.
- C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. Duan \(2026\)Self\-distilled rlvr\.arXiv preprint arXiv:2604\.03128\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px2.p1.1),[footnote 1](https://arxiv.org/html/2606.11627#footnote1)\.
- T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei \(2026\)On\-policy context distillation for language models\.arXiv preprint arXiv:2602\.12275\.Cited by:[§A\.1](https://arxiv.org/html/2606.11627#A1.SS1.p1.1),[§A\.2](https://arxiv.org/html/2606.11627#A1.SS2.p1.1),[§1](https://arxiv.org/html/2606.11627#S1.p1.1),[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.11627#S3.p1.6),[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p1.2),[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p3.4),[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p4.4),[§5\.1](https://arxiv.org/html/2606.11627#S5.SS1.p5.2)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026a\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§1](https://arxiv.org/html/2606.11627#S1.p1.1),[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhao, L\. Yan, W\. Sun, G\. Xing, S\. Wang, C\. Meng, Z\. Cheng, Z\. Ren, and D\. Yin \(2024\)Improving the robustness of large language models via consistency alignment\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 8931–8941\.External Links:[Link](https://aclanthology.org/2024.lrec-main.782/)Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Zhao, L\. Ma, and W\. Zhang \(2026b\)Training with harnesses: on\-policy harness self\-distillation for complex reasoning\.arXiv preprint arXiv:2605\.08741\.Cited by:[§2](https://arxiv.org/html/2606.11627#S2.SS0.SSS0.Px1.p1.1)\.
Appendix
## Appendix AImplementation Details
### A\.1\. System Prompts
We list the system prompts used as privileged context for each model and task\. These prompts are optimized using MetaSPO\(Choiet al\.,[2025](https://arxiv.org/html/2606.11627#bib.bib29)\)for each base model\. The Llama system prompts are taken directly fromYeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\)\. For the no\-context \(w/o ctx\) condition, we clear all content between the system\-prompt delimiter tokens while retaining the delimiter tokens themselves\.
MedMCQA\.
*Llama\-3\.1\-8B / Llama\-3\.2\-3B:*
> You are a knowledgeable and analytical assistant specializing in medical topics\. Your task is to accurately respond to medical inquiries by utilizing established medical knowledge, guidelines, and evidence\-based reasoning\. When presented with a question, carefully analyze the options provided and select the most appropriate answer\. Ensure that your responses are clear, concise, and well\-structured, including a rationale that explains your reasoning and cites relevant medical principles\. Prioritize accuracy and logical coherence in all your responses\.
*Qwen2\.5\-7B:*
> You are a knowledgeable and helpful assistant who critically analyzes questions and evaluates candidate answers\. For every query, thoroughly assess each option to determine the most accurate and relevant answer, relying on scientific evidence where applicable\. Before your selected answer, provide a clear and concise justification to explain your reasoning, highlighting any supporting evidence or context necessary to support your choice\. Then, always present your response in the specified format: <answer\>Answer Here</answer\>\.
*Qwen3\-8B:*
> You are a helpful assistant\.
Safety\.
*Llama\-3\.1\-8B / Llama\-3\.2\-3B:*
> You are an analytical assistant\. Your task is to evaluate statements, questions, and objections based on contextual understanding, evidence, and relevant criteria\. When providing responses, please adhere to the following guidelines: 1\. Factual Accuracy: Assess the factual correctness and relevance of the statement in question\. Provide context where necessary\. 2\. Opinion vs\. Fact: Clearly differentiate between opinion\-based claims and factual assertions\. Explain why a statement is considered an opinion or a fact\. 3\. Emotional Tone Recognition: Identify and analyze emotional tones, especially in cases involving sarcasm, hate speech, or any emotionally charged language\. Discuss the implications of tone in your assessment\. 4\. Balanced Perspective: Explore multiple sides of an argument when applicable\. Offer a well\-rounded analysis that considers contrasting viewpoints and broader implications\. 5\. Clarity and Structure: Format your final answer using <answer\>yes</answer\> or <answer\>no</answer\>, followed by a comprehensive explanation that includes reasoning, evidence, and relevant context\. By applying these guidelines, you will provide contextually aware, nuanced, and accurate evaluations in your responses\.
*Qwen2\.5\-7B:*
> You are a thoughtful and analytical assistant\. For each user query, evaluate the statement against various criteria, including truthfulness, the presence of sarcasm, and whether it qualifies as hate speech or offensive content\. Consider context, intent, and societal implications in your analysis\. Before your selected answer, provide a brief, clear explanation to support your answer\. Then, provide your response in the format <answer\>yes</answer\> or <answer\>no</answer\>\.
*Qwen3\-8B:*
> You are a thoughtful and analytical assistant\. For each user query, evaluate the statement against various criteria, including truthfulness, the presence of sarcasm, and whether it qualifies as hate speech or offensive content\. Consider context, intent, and societal implications in your analysis\. Provide your response in the format <answer\>yes</answer\> or <answer\>no</answer\>, and include a brief, clear explanation to support your answer\.
### A\.2\. Text Game Privileged Context
For text games, the privileged context is constructed using the experience template fromYeet al\.\([2026](https://arxiv.org/html/2606.11627#bib.bib7)\):
> You are an agent playing a game on a grid, acting as a reasoning engine\.Your decisions are based on the experience you have learned about the game’s rules or strategies\. This experience is only a guess of how the game works, and the rules and strategies may be incomplete or incorrect\.Given experience for rules or strategies you have learned:\{EXPERIENCES\}
We provide the teacher with the following game\-specific experience items\.
Sokoban experience:
> \- EXPERIENCE ITEM: In this Sokoban environment, ’P’ is the controllable player, ’X’ is a box, ’O’ is the box target, ’\#’ is a wall, and ’\_’ is empty floor\. The task is to push ’X’ onto ’O’; ’P’ itself reaching ’O’ is not sufficient\.\- EXPERIENCE ITEM: A move into ’\_’ is normal walking\. A move into ’X’ is a push only if the cell beyond ’X’ in the same direction is ’\_’ or ’O’\. A move into ’\#’ is always illegal\. The player can push boxes but cannot pull them\.\- EXPERIENCE ITEM: Evaluate moves by whether they help move ’X’ toward ’O’, not by whether ’P’ moves closer to ’O’\. In Sokoban, ’P’ often needs to move around the box to stand on the correct pushing side\.\- EXPERIENCE ITEM: Before pushing ’X’, check the destination cell of the box\. Push only if the new box position keeps a path toward ’O’; avoid pushing ’X’ into corners, walls, or corridors where it can no longer be moved to ’O’\.\- EXPERIENCE ITEM: After every push, ensure ’P’ can still reach the side of ’X’ needed for the next useful push\. A box position may look closer to ’O’ but still be bad if the player cannot stand behind it to continue pushing\.\- EXPERIENCE ITEM: For the board shown, the useful plan is to push ’X’ downward twice, then move ’P’ to the left side of ’X’ and push it right onto ’O’\. One valid action sequence is ’s, s, a, s, d’\.
FrozenLake experience:
> \- EXPERIENCE ITEM: In Textgame\-FrozenLake, ’P’ is the player, ’G’ is the goal, ’H’ is a hole or failure tile, and blank cells are safe movable spaces\. The objective is to move ’P’ to ’G’ in the shortest number of steps without ever stepping onto ’H’\.\- EXPERIENCE ITEM: Before choosing an action, check the adjacent cell in that direction\. Moving into ’H’ causes failure, and moving outside the grid is invalid, so only choose actions that stay inside the board and enter a safe blank cell or ’G’\.\- EXPERIENCE ITEM: Prioritize shortest safe paths to ’G’, not just moves that reduce distance immediately\. A direct move toward the goal is only good if the destination cell is not ’H’ and does not force a later dead end\.\- EXPERIENCE ITEM: Treat ’H’ tiles as absolute obstacles\. Do not step onto them, and when planning a route, mentally replace them with blocked cells while searching for the shortest path through blank cells to ’G’\.\- EXPERIENCE ITEM: When multiple safe moves exist, prefer the move that minimizes Manhattan distance to ’G’ while preserving a valid safe route\. If a move gets closer to ’G’ but leads into a blocked region or toward holes, choose the safer alternative\.\- EXPERIENCE ITEM: For each board, first eliminate illegal or losing actions, then compare the remaining safe actions by shortest\-path distance to ’G’\. The final answer must be one valid action wrapped in square brackets, such as ’\[up\]’, ’\[down\]’, ’\[left\]’, or ’\[right\]’\.
## Appendix BComplete Training Curves
We provide training curves \(accuracy, harm rate, and response length\) for all 12 settings, grouped by regime\. Each row shows one setting with three metrics \(columns: accuracy, harm rate, response length\)\.
\(a\)Llama\-3\.1\-8B\-Instruct, MedMCQA
\(b\)Llama\-3\.2\-3B\-Instruct, MedMCQA
\(c\)Llama\-3\.1\-8B\-Instruct, Safety
\(d\)Qwen2\.5\-7B\-Instruct, Safety
Figure 8:Regime A settings \(part 1\)\. NCA consistently closes the accuracy gap and reduces harm rate\.
\(a\)Qwen3\-4B\-Instruct\-2507 \(self\), Sokoban
\(b\)Qwen3\-4B\-Ins→\\to1\.7B, Sokoban
\(c\)Qwen3\-4B\-Instruct\-2507 \(self\), FrozenLake
Figure 9:Regime A settings \(part 2\)\. NCA consistently closes the accuracy gap and reduces harm rate\.
\(a\)Qwen2\.5\-7B\-Instruct, MedMCQA
\(b\)Llama\-3\.2\-3B\-Instruct, Safety
\(c\)Qwen3\-4B\-Ins→\\to1\.7B, FrozenLake
Figure 10:Regime B settings\. Context still provides useful scaffolding, and NCA trades someAccx,c\\mathrm\{Acc\}\_\{x,c\}for improvedAccx\\mathrm\{Acc\}\_\{x\}\.
\(a\)Qwen3\-8B, MedMCQA
\(b\)Qwen3\-8B, Safety
Figure 11:Regime C settings\. Both views already agree, and NCA shows minimal additional effect\.Similar Articles
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation
This paper introduces EDGE-OPD, a modification of on-policy self-distillation for LLMs that uses guided rollouts and evidence masks to internalize privileged context without degrading general capabilities, showing success in rare-token identity settings.
Context Distillation as Latent Memory Management
This paper formulates context distillation as a latent memory management problem, proposing a framework that stores distilled contexts as independent LoRA adapters with retrieval, routing, and self-gating to improve robustness and efficiency.
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
This paper systematically studies on-policy distillation in LLM post-training, clarifying its role as an exploration catalyst and identifying pathologies like Student-Teacher Mismatch and Length Exploitation, proposing light-weight signal regulations.
One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
PromptSD is a novel on-policy self-distillation method where the teacher differs from the student only by a learnable soft prompt, enabling multi-task knowledge absorption without weight drift. It matches or exceeds full fine-tuning on four tasks while preserving general capabilities.
The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.