On-Policy Distillation Teaches New Skills but Not New Knowledge

# On-Policy Distillation Teaches New Skills but Not New Knowledge
Source: [https://arxiv.org/html/2610.09639](https://arxiv.org/html/2610.09639)
††Correspondence to:ytangch@connect\.ust\.hk,imyiyang@ust\.hkYi YangAffiliation:The Hong Kong University of Science and Technology

###### Abstract

Abstract:On\-policy distillation \(OPD\) strengthens language\-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi\-step reasoning remains unknown\. We separate these capabilities using a controlled synthetic framework that measures the student’s initial capabilities and independently controls the teacher’s additional facts, compositional skill, or both\. Across four models from three families, reverse\-KL OPD reliably transfers compositional skill across unseen reasoning structures, but transfers minimal factual knowledge\. Decoupling the distillation recipe reveals the source of this asymmetry: replacing reverse KL with forward KL restores factual transfer, whereas student rollouts specifically improve the execution of multi\-step reasoning\. Experiments on recent factual QA and competition mathematics show a similar asymmetry under reverse\-KL OPD, yielding notable reasoning gains without factual memory expansion\. Together, these results demonstrate that on\-policy distillation does not expand a model’s parametric knowledge, but instead teaches it to organize and compose the knowledge it already possesses\.

## 1Introduction

Knowledge distillation transfers capabilities from a large teacher model to a smaller student\[[10](https://arxiv.org/html/2610.09639#bib.bib3),[15](https://arxiv.org/html/2610.09639#bib.bib5),[4](https://arxiv.org/html/2610.09639#bib.bib33)\]\. This allows smaller models to approach the performance of capable teachers at lower cost\[[34](https://arxiv.org/html/2610.09639#bib.bib34)\]\. Standard distillation, however, trains the student on fixed teacher outputs, so supervision never covers the student’s own generation mistakes; this mismatch is known as exposure bias\. On\-policy distillation \(OPD\) addresses this limitation by supervising the student along its own generation trajectories\[[1](https://arxiv.org/html/2610.09639#bib.bib4),[9](https://arxiv.org/html/2610.09639#bib.bib1),[20](https://arxiv.org/html/2610.09639#bib.bib2)\]\. Because supervision targets the prefixes the student actually produces, OPD strengthens complex reasoning and is now widely used in post\-training pipelines such as Gemini and Qwen\[[27](https://arxiv.org/html/2610.09639#bib.bib16),[29](https://arxiv.org/html/2610.09639#bib.bib11)\]\.

Yet these empirical gains leave open what the student learns from the teacher\. In multi\-step reasoning, reaching an answer requires both recalling domain facts and coordinating intermediate results\[[22](https://arxiv.org/html/2610.09639#bib.bib17)\]\. Higher benchmark scores can therefore reflect newly acquired facts, improved compositional skill, or both, but end\-to\-end evaluation cannot separate them\. These capabilities address distinct computational demands: factual knowledge relies on stored associations in memory, whereas compositional skill determines how intermediate steps are combined\[[21](https://arxiv.org/html/2610.09639#bib.bib8),[16](https://arxiv.org/html/2610.09639#bib.bib32),[33](https://arxiv.org/html/2610.09639#bib.bib9)\]\. A model can possess the relevant facts yet fail to compose them, or master multi\-step reasoning yet fail when querying unfamiliar facts\. Understanding what distillation transfers therefore requires evaluating factual knowledge and compositional skill as independent dimensions\.

This distinction matters for how on\-policy distillation can be applied in practice\. If factual knowledge transfers, distillation offers a way to expand a model’s parametric memory with new facts\. If compositional skill transfers, it teaches the student to use its existing factual knowledge across new reasoning patterns\. We therefore askwhether on\-policy distillation transfers a teacher’s factual knowledge, compositional skill, or both\.

Existing work offers conflicting perspectives on this question\.[Ding and Zhang \[6\]](https://arxiv.org/html/2610.09639#bib.bib7)show that suppressing low\-probability student tokens can match OPD’s reasoning gains without teacher supervision\. In contrast,[Li et al\. \[19\]](https://arxiv.org/html/2610.09639#bib.bib12)argue that successful OPD requires the teacher to offer new capabilities beyond what the student has seen during training\. Yet neither line of work clarifies whether the student gains new capabilities from the teacher, and whether such gains reflect factual knowledge or compositional skill\. Answering these questions requires measuring the student’s initial capabilities, controlling the teacher’s additional facts and compositional skill, and evaluating both dimensions independently\. Such control is difficult with pretrained language models: their training corpora are too vast to audit what they already know, and natural\-language tasks often require factual recall and reasoning together\.

To achieve this control, we construct a synthetic environment that explicitly separates factual knowledge from compositional skill \(Fig\.[1](https://arxiv.org/html/2610.09639#S1.F1)\)\. In natural language, factual knowledge often involves specific associations that cannot be inferred from syntax alone\[[21](https://arxiv.org/html/2610.09639#bib.bib8)\]\. Thus, we operationalize factual knowledge as a set of arbitrary input–output associations stored in a lookup table\. Each table maps a pair of input digits to an output digit\. To ensure that these associations must be memorized rather than inferred, we sample all table entries independently and uniformly at random, with no underlying algebraic rule\. As a result, the answer to an unseen table entry cannot be derived from other entries\. The model can answer correctly only if it has learned that specific mapping\. This construction also lets us control exactly which facts are available to the student during training\. Beyond recalling individual facts, multi\-step reasoning also requires the model to organize and reuse them across successive computations\. We operationalize compositional skill as the ability to combine facts across multi\-step reasoning structures\. In a sequential chain, each step directly uses the result of the previous step\. In a branching tree, the model must solve separate branches, retain intermediate results across intervening steps, and combine them later in the correct order\. By varying the tree structure while holding the underlying facts and number of operations fixed, we test whether the model acquires a general compositional skill rather than memorizing a specific computation pattern\.

Within this controlled setup, we establish an initial student,S0S\_\{0\}, by training it on an initial set of facts using single\-step queries and sequential chains\. This student reliably recalls the initial facts, but achieves near\-zero accuracy on held\-out facts and on branching trees\. Here,*initial*and*held\-out*refer to whether a fact is included during student initialization\. Starting fromS0S\_\{0\}, we then train three specialized teachers:TfT\_\{f\}is additionally trained on the held\-out facts,TsT\_\{s\}on branching trees using only the initial facts, andTf​sT\_\{fs\}on both\. We also retain the unchangedS0S\_\{0\}as a baseline teacher,T∅T\_\{\\emptyset\}, which has neither additional knowledge nor additional compositional skill\. We then distill each teacher into a student initialized fromS0S\_\{0\}and separately evaluate whether the student acquires the teacher’s additional knowledge or compositional skill, including whether the latter generalizes to previously unseen tree structures\.

Across four models from three families, our experiments reveal a consistent asymmetry in what is transferred: on\-policy distillation with reverse KL transfers compositional skill while failing to transfer factual knowledge\. Students distilled fromTsT\_\{s\}reach 79\.10% accuracy under greedy decoding on unseen tree structure, compared with 0\.72% under the baseline teacherT∅T\_\{\\emptyset\}\. In contrast, students distilled fromTfT\_\{f\}reach only 13\.12% on single\-step queries about held\-out facts and 0\.54% on chains of such facts\. To identify the source of this asymmetry, we separate two components of the distillation procedure: the source of training prefixes and the direction of the KL objective\. Replacing reverse KL with forward KL substantially restores factual transfer under both student\-generated and teacher\-generated prefixes, whereas student\-generated prefixes primarily improve multi\-step compositional execution\. Real\-world benchmarks on recent factual question answering and competition mathematics show the same pattern: reverse\-KL OPD improves mathematical reasoning by 6\.47 percentage points but produces no improvement in factual recall over the initial student\.

These findings suggest that reverse\-KL based OPD primarily teaches models how to use what they already know, rather than reliably adding what they do not know\. For practitioners, this distinction suggests that OPD is better suited for strengthening compositional reasoning than for factual knowledge updating, whereas forward\-KL objectives are substantially more effective when transferring new factual information is the goal\.

Figure 1:Overview of the setup\. \(a\) Random lookup tables represent facts; chains and trees specify how their query results are combined\. \(b\) Starting fromS0S\_\{0\}, teachers add exposure to held\-out facts \(Δ​F\\Delta F\), branching trees \(Δ​S\\Delta S\), or both, with frozenS0S\_\{0\}as the baseline\. \(c\) A fixed teacher supervises student rollouts with reverse KL\. \(d\) Evaluation separates factual recall on held\-out facts from compositional skill over initial facts\.
## 2Related Work

#### Knowledge Distillation

Knowledge distillation trains a student to match a teacher’s output distribution, allowing a compact model to capture behaviors from a larger counterpart\[[10](https://arxiv.org/html/2610.09639#bib.bib3)\]\. In autoregressive language modeling, sequence\-level distillation trains the student with cross\-entropy on teacher\-generated sequences\[[15](https://arxiv.org/html/2610.09639#bib.bib5)\]\. Beyond raw task accuracy, distillation can transfer underlying inductive biases\[[23](https://arxiv.org/html/2610.09639#bib.bib15)\]and recover memorized associations that the student never directly observed from a teacher’s soft labels\[[3](https://arxiv.org/html/2610.09639#bib.bib14)\]\. However, because sequence\-level distillation trains only on teacher\-generated sequences, the student receives no supervision along its own generation paths\.

#### On\-Policy Distillation

On\-policy distillation addresses this mismatch by evaluating teacher supervision on student\-generated trajectories\. Generalized Knowledge Distillation \(GKD\) formalizes this approach across different divergence objectives\[[1](https://arxiv.org/html/2610.09639#bib.bib4)\], while MiniLLM uses reverse KL to discourage the student from assigning excessive probability to sequences that are unlikely under the teacher\[[9](https://arxiv.org/html/2610.09639#bib.bib1)\]\. Empirical studies report contrasting findings on what students acquire through this supervision\. Some studies report evidence of capability expansion: the Qwen3 technical report shows that on\-policy distillation improves Pass@64 on math reasoning, whereas reinforcement learning from the same checkpoint leaves Pass@64 unchanged\[[29](https://arxiv.org/html/2610.09639#bib.bib11)\]\. In addition, SDFT demonstrates factual knowledge acquisition through on\-policy self\-distillation from a teacher conditioned on additional information\[[26](https://arxiv.org/html/2610.09639#bib.bib13)\]\. In contrast, other studies argue that reasoning gains mainly reflect more efficient sampling of existing capabilities\. At larger sampling budgets, pre\-OPD models can match or outperform their distilled counterparts, suggesting improved access to existing reasoning capabilities rather than a broader capability boundary\[[8](https://arxiv.org/html/2610.09639#bib.bib10),[32](https://arxiv.org/html/2610.09639#bib.bib6)\]\. Furthermore,[Ding and Zhang \[6\]](https://arxiv.org/html/2610.09639#bib.bib7)show that suppressing low\-probability student tokens without teacher supervision can match OPD’s reasoning gains\. These findings concern different sources of teacher information and different measures of student capability\. To address these confounding factors, we independently control the additional knowledge and compositional skill available to the teacher and evaluate whether each is transferred to the student\.

#### Synthetic Tasks for Factual Knowledge and Compositional Skill

Controlled synthetic tasks make it possible to precisely control training exposure and measure generalization\.[Morris et al\. \[21\]](https://arxiv.org/html/2610.09639#bib.bib8)measure memorization capacity using uniformly random sequences with known information content\. To study compositional skill,[Yuan et al\. \[33\]](https://arxiv.org/html/2610.09639#bib.bib9)construct string\-transformation tasks in which models first learn atomic functions and then learn to compose them through reinforcement learning\. Synthetic tasks have also been used to study distillation:[Behrens and Zdeborová \[3\]](https://arxiv.org/html/2610.09639#bib.bib14)show that teacher soft labels can convey memorized associations that students never directly observe\. We use controlled synthetic tasks to examine factual knowledge and compositional skill transfer under on\-policy distillation\. Each random lookup table represents a piece of factual knowledge, while chains and branching trees determine how known facts must be combined during multi\-step reasoning\. By independently varying the additional knowledge and skill available to the teacher, we test what the student acquires through distillation\.

## 3Experimental Setting: Separating Knowledge and Skill

To identify what is transferred during on\-policy distillation, we need to control both the student’s initial capabilities and the teacher’s additional knowledge and skill\. We therefore construct a controlled setting in which these capabilities can be varied independently\.

### 3\.1Synthetic Tasks for Knowledge and Reasoning

#### Factual Knowledge as Random Lookup Tables

Isolating factual transfer requires distinguishing memorization from rule\-based generalization\[[21](https://arxiv.org/html/2610.09639#bib.bib8)\]\. If a task contains learnable patterns, a model can deduce unseen answers rather than recall the specified mapping\. We therefore operationalize each fact as a two\-argument mapping represented by one random lookup table\. Because the entries have no underlying algebraic structure, the model must learn these associations to answer queries about the fact\. Therefore, withholding selected tables during student initialization controls which facts the student encounters\.

We instantiate each fact as a lookup table𝒯k:\{0,…,9\}2→\{0,…,9\}\\mathcal\{T\}\_\{k\}:\\\{0,\\ldots,9\\\}^\{2\}\\to\\\{0,\\ldots,9\\\}\. Each table contains 100 entries, each mapping an input digit pair to an output digit\. We usea∘kba\\circ\_\{k\}bto denote the result of applying table𝒯k\\mathcal\{T\}\_\{k\}independently at each digit position:

\(a∘kb\)j=𝒯k​\(aj,bj\),j∈\{1,2,3,4\}\.\(a\\circ\_\{k\}b\)\_\{j\}=\\mathcal\{T\}\_\{k\}\(a\_\{j\},b\_\{j\}\),\\qquad j\\in\\\{1,2,3,4\\\}\.\(1\)A single\-step fact query therefore requires four digit lookups with no carry across positions\. For example, fact𝒯3\\mathcal\{T\}\_\{3\}is identified by\[OP\_3\], and the query1234\[OP\_3\]5678evaluates to9025\. The lookup table itself is not included in the prompt, so answering correctly requires the corresponding associations to have been learned during training\. We generate 25 facts,𝒯=\{𝒯1,…,𝒯25\}\\mathcal\{T\}=\\\{\\mathcal\{T\}\_\{1\},\\ldots,\\mathcal\{T\}\_\{25\}\\\}, partitioned into 20*initial facts*,𝒯init\\mathcal\{T\}\_\{\\mathrm\{init\}\}, and five*held\-out facts*,𝒯held\\mathcal\{T\}\_\{\\mathrm\{held\}\}\. Initial facts are included during student initialization, whereas held\-out facts are excluded\. This partition remains fixed throughout teacher training and distillation\. The tokens\[OP\_1\]through\[OP\_25\]identify the facts in the serialized input and output\.

#### Compositional Skill across Reasoning Structures

Compositional skill is the ability to order fact queries and supply intermediate results according to a computation graph\[[22](https://arxiv.org/html/2610.09639#bib.bib17),[33](https://arxiv.org/html/2610.09639#bib.bib9)\]\. To isolate this ability from factual recall, we construct sequential chains and branching trees that query the same set of facts and require the same number of operations\. In a chain, each step simply takes the result of the immediately preceding step and applies the next fact\. In a branching tree, independent branches are evaluated before their results are combined\. Although the output is still generated sequentially, the model must retrieve earlier intermediate results across intervening steps to combine independent branches\. In our setting, chains are used during student initialization, while branching trees test compositional skill without introducing new facts\.

A simple three\-step example illustrates how intermediate results are passed in each structure:

Chain:\(\(a∘1b\)∘2c\)∘3dTree:\(a∘1b\)∘3\(c∘2d\)r1=a∘1br1=a∘1br2=r1∘2cr2=c∘2dr3=r2∘3dr3=r1∘3r2\.\\begin\{array\}\[\]\{ll\}\\text\{Chain: \}\(\(a\\circ\_\{1\}b\)\\circ\_\{2\}c\)\\circ\_\{3\}d&\\text\{Tree: \}\(a\\circ\_\{1\}b\)\\circ\_\{3\}\(c\\circ\_\{2\}d\)\\\\\[3\.0pt\] r\_\{1\}=a\\circ\_\{1\}b&r\_\{1\}=a\\circ\_\{1\}b\\\\ r\_\{2\}=r\_\{1\}\\circ\_\{2\}c&r\_\{2\}=c\\circ\_\{2\}d\\\\ r\_\{3\}=r\_\{2\}\\circ\_\{3\}d&r\_\{3\}=r\_\{1\}\\circ\_\{3\}r\_\{2\}\.\\end\{array\}Here,a,b,c,da,b,c,dare four\-digit inputs, and∘i\\circ\_\{i\}denotes a query to fact𝒯i\\mathcal\{T\}\_\{i\}\. In the chain, step 3 uses the immediately preceding resultr2r\_\{2\}and inputdd\. In the tree, step 3 combinesr1r\_\{1\}andr2r\_\{2\}, requiring retrieval ofr1r\_\{1\}across the intervening computation ofr2r\_\{2\}\. Thus, the two structures use the same facts but differ in how intermediate results must be organized and reused\. In our main experiments, each query contains seven operations over eight inputs\. Each completion includes an explicit step\-by\-step execution trace followed by the final answer\. We measure the student’s initial performance on both chain and branching\-tree structures in Section[3\.2](https://arxiv.org/html/2610.09639#S3.SS2.SSS0.Px1)\. Appendix[A](https://arxiv.org/html/2610.09639#A1)provides complete prompt and completion examples\.

### 3\.2Student Initialization and Teacher Design

We establish an initial student,S0S\_\{0\}, by training it on an initial set of facts and sequential chains\. We then verify that it reliably recalls these facts and executes chains, but performs poorly on held\-out facts and branching trees\.

#### Initial Student

We use four base models: Qwen3\-0\.6B and Qwen3\-4B\[[29](https://arxiv.org/html/2610.09639#bib.bib11)\], Gemma\-2\-2B\[[28](https://arxiv.org/html/2610.09639#bib.bib18)\], and OLMo3\-7B\[[24](https://arxiv.org/html/2610.09639#bib.bib19)\]\. All distillation experiments begin from the initial student,S0S\_\{0\}\. We obtainS0S\_\{0\}by fine\-tuning a base LLM on queries about the initial facts,𝒯init\\mathcal\{T\}\_\{\\mathrm\{init\}\}, using single\-step queries and sequential chains of varying lengths\. This initialization teaches the output format and factual recall while excluding data about held\-out facts,𝒯held\\mathcal\{T\}\_\{\\mathrm\{held\}\}, and branching trees\. Evaluation confirms thatS0S\_\{0\}reliably executes chains over initial facts but rarely solves branching trees or queries about held\-out facts\. Appendix[B](https://arxiv.org/html/2610.09639#A2)reports the training mixture and baseline measurements\.

#### Controlled Teacher Design

Starting fromS0S\_\{0\}, we train three specialized teachers by augmenting its training distribution with data about held\-out facts \(TfT\_\{f\}\), branching trees over initial facts \(TsT\_\{s\}\), or both \(Tf​sT\_\{fs\}\), corresponding toΔ​F\\Delta FandΔ​S\\Delta Sin Fig\.[1](https://arxiv.org/html/2610.09639#S1.F1)\(b\)\. Together with the frozen baseline teacherT∅=S0T\_\{\\emptyset\}=S\_\{0\}, they form a2×22\\times 2factorial design crossing exposure to held\-out facts with exposure to branching trees\. Here,ffdenotes factual knowledge andssdenotes compositional skill\. All teachers share the student’s base architecture and initialization\. Table[1](https://arxiv.org/html/2610.09639#S3.T1)\(a\) summarizes the factual knowledge and compositional skill of the initial student and teachers\. Evaluations confirm the intended capability separation\. For example, using Qwen3\-4B as base model,TfT\_\{f\}reaches 100% accuracy on held\-out factual knowledge queries but only 0\.60% on unseen branching trees, whereasTsT\_\{s\}reaches 100% on unseen trees but 0% on held\-out factual queries \(Table[5](https://arxiv.org/html/2610.09639#A2.T5)\)\.

Table 1:Student and teacher capabilities and distillation conditions\.\(a\) Student and teacher capabilitiesModelFactual knowledgeCompositional skillS0S\_\{0\}Initial factsChain executionT∅=S0T\_\{\\emptyset\}=S\_\{0\}Initial factsChain executionTfT\_\{f\}Initial \+ held\-out factsChain executionTsT\_\{s\}Initial factsChain and branching\-tree executionTf​sT\_\{fs\}Initial \+ held\-out factsChain and branching\-tree execution\(b\) On\-policy distillationFixed teacherStudent starts fromDistillation promptsTfT\_\{f\}S0S\_\{0\}Chains querying held\-out factsTsT\_\{s\}S0S\_\{0\}Branching trees over initial factsTf​sT\_\{fs\}S0S\_\{0\}Branching trees querying held\-out factsT∅T\_\{\\emptyset\}S0S\_\{0\}Same queries as theTfT\_\{f\}runT∅T\_\{\\emptyset\}S0S\_\{0\}Same queries as theTsT\_\{s\}run

### 3\.3On\-Policy Distillation and Evaluation

#### Training Objective

All distillation runs start from the same studentS0S\_\{0\}and keep the selected teacherTTfixed\. Given a promptxx, the current student policyPθP\_\{\\theta\}samples a completiony∼Pθ\(⋅∣x\)y\\sim P\_\{\\theta\}\(\\cdot\\mid x\), which is called a rollout\. At each positiontt, the teacher’s next\-token distributionPTP\_\{T\}is conditioned on the same prompt and the student’s preceding tokensy<ty\_\{<t\}\. We train the student to match this distribution using token\-level reverse KL\[[1](https://arxiv.org/html/2610.09639#bib.bib4),[9](https://arxiv.org/html/2610.09639#bib.bib1)\]\. For a batchℬ\\mathcal\{B\}of prompt\-completion pairs, the loss is

ℒ^OPD\(θ;ℬ\)=1Zℬ∑\(x,y\)∈ℬ∑t=1\|y\|DKL\(Pθ\(⋅∣x,y<t\)∥PT\(⋅∣x,y<t\)\),\\widehat\{\\mathcal\{L\}\}\_\{\\mathrm\{OPD\}\}\(\\theta;\\mathcal\{B\}\)=\\frac\{1\}\{Z\_\{\\mathcal\{B\}\}\}\\sum\_\{\(x,y\)\\in\\mathcal\{B\}\}\\sum\_\{t=1\}^\{\|y\|\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(P\_\{\\theta\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\\|\\,P\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\)\\right\),\(2\)whereZℬZ\_\{\\mathcal\{B\}\}counts completion tokens, excluding prompt and padding positions\. Sampled completions are held fixed during each update, and gradients pass only throughPθP\_\{\\theta\}\. Optimization details appear in Appendix[B](https://arxiv.org/html/2610.09639#A2)\.

#### Distillation Conditions

To test factual knowledge transfer, we distill fromTfT\_\{f\}using multi\-step chains that contain held\-out facts\. To test compositional skill transfer, we distill fromTsT\_\{s\}using multi\-step trees over initial facts\. The joint condition usesTf​sT\_\{fs\}with trees that also contain held\-out facts\. As a baseline, we useT∅=S0T\_\{\\emptyset\}=S\_\{0\}on the same prompts to provide a control with no additional teacher knowledge or skill\. Dataset sizes and mixtures appear in Appendix[B\.2](https://arxiv.org/html/2610.09639#A2.SS2)\.

#### Evaluation

We evaluate factual recall with single\-step queries and multi\-step chains over held\-out facts\. Compositional skill is evaluated on multi\-step trees over initial facts\. To measure generalization, we divide tree structures into three groups:*Shared*structures appear in both teacher training and distillation\.*Teacher\-only*structures appear only in teacher training\.*Unseen*structures appear in neither\. Distillation fromTsT\_\{s\}andTf​sT\_\{fs\}uses only shared structures, while evaluation covers all three groups using fresh queries\. Appendix[B\.3](https://arxiv.org/html/2610.09639#A2.SS3)provides the split details, test definitions, and metrics\.

## 4Synthetic Experiment Results

### 4\.1On\-Policy Distillation Teaches New Skills but Not New Knowledge

Figure 2:Student performance after on\-policy distillation under each teacher\. Top: greedy final\-answer accuracy; bottom: final\-answer Pass@32 \(at least one of 32 sampled completions correct\)\. Numbers above each group show the unweighted mean across four models\. All values are percentages\.Figure[2](https://arxiv.org/html/2610.09639#S4.F2)shows that on\-policy distillation transfers compositional skill beyond the tree structures used during training\. Averaged across the four models, students distilled fromTsT\_\{s\}achieve 80\.96% greedy accuracy on teacher\-only tree structures and 79\.10% on unseen structures, compared with just 0\.72% on the same unseen structures under the matchedT∅T\_\{\\emptyset\}control\. The high accuracy on unseen tree structures shows that the acquired compositional skill generalizes beyond the structures encountered during training\. Importantly, both the teacher\-only and unseen tests use only facts thatS0S\_\{0\}had already learned during initialization\. The improvement therefore reflects a new ability to organize and combine existing knowledge\.

Factual knowledge transfer is far more limited and varies across models\. Students distilled fromTfT\_\{f\}achieve a mean greedy accuracy of 13\.12% on single\-step queries about held\-out facts, compared with 0\.03% under the matchedT∅T\_\{\\emptyset\}control\. Gemma\-2\-2B reaches 32\.95%, showing that factual recall can improve in certain architectures\. However, accuracy drops to just 0\.54% on multi\-step chains where every step queries a held\-out fact\.

This asymmetry between factual knowledge and compositional skill transfer persists under a larger sampling budget\. Pass@32 counts a query as solved when at least one of 32 sampled completions has the correct final answer\. Students distilled fromTsT\_\{s\}achieve a mean Pass@32 of 88\.11% on unseen tree structures\. In contrast, students distilled fromTfT\_\{f\}reach only 15\.74% Pass@32 on single\-step queries and 2\.58% on chains, both querying held\-out facts\. The jointTf​sT\_\{fs\}condition exhibits the same disparity: students acquire robust compositional skills across tree structures while unable to reliably recall or chain new facts\. Rollout accuracy and reverse\-KL training curves are reported in Appendix[C](https://arxiv.org/html/2610.09639#A3)\(Figure[5](https://arxiv.org/html/2610.09639#A3.F5)\)\.

Insight 1: Transfer Asymmetry in Standard OPDOn\-policy distillation transfers compositional skill, but fails to transfer factual knowledge\.

### 4\.2Distilling the Distillation Recipe

Figure 3:Effects of prefix source and KL direction\. Rows show Qwen3\-4B and Gemma\-2\-2B\. Factual knowledge is evaluated after distillation fromTfT\_\{f\}; compositional skill after distillation fromTsT\_\{s\}\. Accuracy measures greedy final answers; NLL is reported for reference completions in bits per completion\. Reference prefixes come from teacher demonstrations\.Why does on\-policy distillation transfer compositional skill but fail on factual knowledge? In sequence distillation, the training procedure combines two independent design choices\[[5](https://arxiv.org/html/2610.09639#bib.bib21),[1](https://arxiv.org/html/2610.09639#bib.bib4)\]:*where the training prefixes come from*, and*how the student matches the teacher’s next\-token distribution*\. Standard OPD couples student rollouts with reverse KL, whereas offline distillation couples teacher demonstration prefixes with forward KL\. We therefore separate these two factors to determine which one drives the observed transfer asymmetry\.

The first choice is the*prefix source*\. To predict tokentt, the student conditions on a prefix of earlier tokensy<ty\_\{<t\}\. In offline teacher\-forcing, this prefix is drawn from ground\-truth teacher demonstrations \(y<t=y<t∗y\_\{<t\}=y^\{\*\}\_\{<t\}\), training the student exclusively along correct execution paths\. In on\-policy distillation, the student generates its own prefixes through autoregressive rollouts \(y<t∼Pθy\_\{<t\}\\sim P\_\{\\theta\}\), extending supervision to intermediate states that follow its own generation mistakes\.

The second choice is the*divergence objective*\. At each prefixy<ty\_\{<t\}, the frozen teacher supplies a target next\-token distributionPT\(⋅∣y<t\)P\_\{T\}\(\\cdot\\mid y\_\{<t\}\), and the student predictsPθ\(⋅∣y<t\)P\_\{\\theta\}\(\\cdot\\mid y\_\{<t\}\)\. Standard OPD minimizes reverse KL,DKL\(Pθ∥PT\)D\_\{\\mathrm\{KL\}\}\(P\_\{\\theta\}\\\|P\_\{T\}\)\[[9](https://arxiv.org/html/2610.09639#bib.bib1),[20](https://arxiv.org/html/2610.09639#bib.bib2)\], which is mode\-seeking and weights prediction errors by the student’s own probabilities\. In contrast, offline distillation typically minimizes forward KL,DKL\(PT∥Pθ\)D\_\{\\mathrm\{KL\}\}\(P\_\{T\}\\\|P\_\{\\theta\}\), which is mean\-covering and penalizes the student whenever it assigns low probability to tokens favored by the teacher\.

#### Evaluation

Crossing these two choices yields a2×22\\times 2factorial design\. We evaluate all four combinations on Qwen3\-4B and Gemma\-2\-2B\. We test factual recall on single\-step queries about held\-out facts after distillation fromTfT\_\{f\}, and compositional skill on unseen tree structures after distillation fromTsT\_\{s\}\. All runs share the initial checkpointS0S\_\{0\}, training data, and optimization hyperparameters\. We also report negative log\-likelihood on teacher demonstrations \(NLL, in bits per completion\) to measure the probability mass the student assigns to correct multi\-step executions independently of greedy decoding choices\.

#### Factual Transfer Depends on the Divergence Objective

Figure[3](https://arxiv.org/html/2610.09639#S4.F3)shows that factual recall depends almost entirely on the divergence objective, not the prefix source\. Under student prefixes, replacing reverse KL with forward KL substantially increases single\-step greedy accuracy: from 10\.95% to 54\.10% in Qwen3\-4B, and from 32\.95% to 94\.10% in Gemma\-2\-2B\. Under teacher prefixes, forward KL reaches similar accuracy \(54\.05% in Qwen3\-4B and 93\.00% in Gemma\-2\-2B\), showing that on\-policy rollouts provide no distinct advantage for learning new facts\. Reference\-completion NLL confirms this pattern, dropping substantially under forward KL across both prefix sources\.

These results show that OPD’s failure on factual knowledge stems from the divergence objective, not from on\-policy exploration\. Because reverse KL evaluates discrepancies under the student’s distribution, it only penalizes mistakes on tokens the student already produces with substantial probability\. When the student does not know a fact, it assigns negligible probability to the correct target, incurring almost no penalty for the omission\. Forward KL, by contrast, evaluates discrepancies under the teacher’s distribution, heavily penalizing the student whenever it misses the teacher’s target\.

#### Student Rollouts Strengthen Compositional Skill

Student rollouts strengthen compositional skill on unseen tree structures\. Under both KL directions, training on student prefixes yields lower reference\-completion NLL in both models\. Greedy accuracy also improves in three of the four comparisons\. For Qwen3\-4B, switching from teacher to student prefixes reduces NLL from 2\.40 to 1\.77 bits under reverse KL and from 2\.13 to 1\.15 bits under forward KL\. For Gemma\-2\-2B, the corresponding reductions are from 28\.13 to 25\.65 bits and from 17\.40 to 13\.65 bits\. Thus, training on the student’s own trajectories improves the average log\-likelihood of correct reference completions, even when the evaluated tree structures were absent from training\.

Insight 2: Decoupling the Distillation RecipeThe KL direction determines factual knowledge transfer, while on\-policy exploration drives compositional skill transfer\.

## 5Real\-World Validation

To test whether the transfer asymmetry observed in synthetic tasks holds in natural language settings, we evaluate on\-policy distillation on recent factual knowledge acquisition and mathematical reasoning\. These domains represent two different capabilities: factual question answering requires acquiring information that the student does not initially know, whereas mathematical reasoning requires applying and composing existing knowledge across multiple steps\.

#### Benchmark and Teacher Design

We conduct experiments on Qwen3\-4B\-Base\. We first fine\-tune this model on general\-instruction data from Tulu\-3\-SFT\[[17](https://arxiv.org/html/2610.09639#bib.bib22)\]to establish our initial studentS0S\_\{0\}\. We curate two complementary evaluation suites:

- •For recent factual knowledge,Knowledge\-2026aggregates 809 question\-answer pairs that entered public discourse in 2026, combining items from RealTimeQA\[[14](https://arxiv.org/html/2610.09639#bib.bib23)\], SealQA\[[25](https://arxiv.org/html/2610.09639#bib.bib24)\], FreshQA\[[31](https://arxiv.org/html/2610.09639#bib.bib25)\], and newly edited Wikipedia articles extracted via the WikiBigEdit pipeline\[[30](https://arxiv.org/html/2610.09639#bib.bib26)\]\. Because these items postdate the training cutoff of Qwen3\-4B\-Base,S0S\_\{0\}achieves only 11\.25% greedy accuracy\.
- •For mathematical reasoning,Math\-2026contains 201 competition problems drawn from MathArena\[[2](https://arxiv.org/html/2610.09639#bib.bib27)\]and NuminaMath\-CoT\[[18](https://arxiv.org/html/2610.09639#bib.bib28)\], whereS0S\_\{0\}achieves 17\.41% greedy accuracy\. The 201 problems are held out from a curated pool of 1,000\. The remaining 799 serve as distillation prompts\.

Starting fromS0S\_\{0\}, we train two specialized teachers by supervised fine\-tuning\. The knowledge teacherTKT\_\{K\}is fine\-tuned on the full set of 809 factual question\-answer pairs, and its 84\.55% onKnowledge\-2026is measured on this same set\. The reasoning teacherTRT\_\{R\}is fine\-tuned on competition mathematics with verified step\-by\-step solutions\[[7](https://arxiv.org/html/2610.09639#bib.bib29)\], reaching 43\.78% onMath\-2026\(Table[2](https://arxiv.org/html/2610.09639#S5.T2)\)\. Further benchmark and teacher details appear in Appendix[D](https://arxiv.org/html/2610.09639#A4)\.

#### On\-Policy Distillation Protocol and Results

To test factual transfer, we distillS0S\_\{0\}fromTKT\_\{K\}using these same factual question prompts for student rollouts\. To test reasoning transfer, we distillS0S\_\{0\}fromTRT\_\{R\}over these 799 problem prompts without demonstration traces\. Table[2](https://arxiv.org/html/2610.09639#S5.T2)compares performance before and after distillation\.

Table 2:Greedy accuracy \(%\) before and after on\-policy distillation on real\-world benchmarks\. Shaded cells indicate each teacher’s target capability\.The real\-world outcomes closely mirror our findings in the synthetic setting\. For recent factual knowledge, on\-policy distillation fails to transfer the teacher’s knowledge: student accuracy drops slightly from 11\.25% to 9\.64%, remaining near its initial level despite the teacher reaching 84\.55%\. Thus, even when the student is explicitly prompted with missing facts during distillation, factual recall remains near its initial level\.

For mathematical reasoning, on\-policy distillation successfully transfers the teacher’s multi\-step capability: distillation fromTRT\_\{R\}increases greedy accuracy onMath\-2026from 17\.41% to 23\.88% \(a 6\.47 percentage point gain\)\. This acquired skill generalizes effectively to competition problems excluded from distillation\. Neither distillation run produces cross\-capability transfer: distilling fromTRT\_\{R\}leaves factual recall at 10\.75% \(matchingS0S\_\{0\}’s 11\.25%\), while distilling fromTKT\_\{K\}shifts math accuracy only to 19\.40%\. Therefore, real\-world tasks show the same transfer asymmetry: standard OPD improves reasoning skills but transfers factual knowledge only weakly\.

## 6Additional Analysis

### 6\.1Execution Traces of Factual and Compositional Failures

\(a\)Chain querying only held\-out facts: correct structure, wrong values\(\(\(\(\(\(3486 \[OP\_24\] 2845\) \[OP\_24\] 0536\) \[OP\_22\] 7919\) \[OP\_24\] 1381\) \[OP\_22\] 2146\) \[OP\_25\] 4658\) \[OP\_25\] 3844 = ?Student distilled fromTfT\_\{f\}with reverse KL3486\[OP\_24\]2845=0135; 0135\[OP\_24\]0536=0426; 0426\[OP\_22\]7919=5020; 5020\[OP\_24\]1381=3609; 3609\[OP\_22\]2146=1948; 1948\[OP\_25\]4658=1332; 1332\[OP\_25\]3844=2752; \[ANS\]2752\[/ANS\]Student distilled fromTfT\_\{f\}with forward KL \(matches teacher demonstration\)3486\[OP\_24\]2845=8135; 8135\[OP\_24\]0536=9422; 9422\[OP\_22\]7919=0726; 0726\[OP\_24\]1381=3007; 3007\[OP\_22\]2146=1326; 1326\[OP\_25\]4658=3440; 3440\[OP\_25\]3844=5149; \[ANS\]5149\[/ANS\]

\(b\)Unseen tree over initial facts: correct values, flattened structure\(\(2250 \[OP\_8\] 0227\) \[OP\_6\] \(3754 \[OP\_20\] 0476\)\) \[OP\_16\] \(\(\(9734 \[OP\_1\] 6055\) \[OP\_12\] 7785\) \[OP\_4\] 8848\) = ?Student distilled fromTfT\_\{f\}with forward KL2250\[OP\_8\]0227=8071; 8071\[OP\_6\]3754=6519; 6519\[OP\_20\]0476=7364; 9734\[OP\_1\]6055=0619; 0619\[OP\_12\]7785=1563; 1563\[OP\_4\]8848=7255; 7255\[OP\_16\]7364=8812;\[ANS\]8812\[/ANS\]Student distilled fromTsT\_\{s\}with reverse KL \(matches teacher demonstration\)2250\[OP\_8\]0227=8071; 3754\[OP\_20\]0476=8437; 8071\[OP\_6\]8437=0493; 9734\[OP\_1\]6055=0619; 0619\[OP\_12\]7785=1563; 1563\[OP\_4\]8848=7255; 0493\[OP\_16\]7255=2271; \[ANS\]2271\[/ANS\]

Figure 4:Greedy execution traces from Qwen3\-4B students after distillation\. \(a\) Seven\-step chain over held\-out facts: reverse KL maintains correct execution syntax but outputs incorrect lookup values \(red\), whereas forward KL restores correct lookups\. \(b\) Unseen tree over initial facts: forward KL recalls all facts correctly but flattens the tree into a linear sweep \(red\), whereas reverse KL fromTsT\_\{s\}executes the valid postorder\. Red highlights incorrect values or misrouted operands; green marks correct final answers\.To examine how distillation failures manifest during generation, we inspect greedy execution traces from Qwen3\-4B on held\-out chains and unseen trees \(Figure[4](https://arxiv.org/html/2610.09639#S6.F4)\)\.

In Figure[4](https://arxiv.org/html/2610.09639#S6.F4)\(a\), the student is distilled fromTfT\_\{f\}with reverse KL and evaluated on a chain over held\-out facts\. The student follows the step\-by\-step format without error, passing intermediate outputs to the next query and closing with an answer tag\. What fails is the lookup facts: each step outputs an incorrect number from the queried table, such as0135instead of8135\. Across evaluation chains, the execution format never breaks, but most individual lookups are incorrect\. Distillation from the same teacher with forward KL provides the target entries and matches the teacher demonstration\.

In Figure[4](https://arxiv.org/html/2610.09639#S6.F4)\(b\), the student is distilled fromTfT\_\{f\}with forward KL and evaluated on an unseen tree over initial facts\. Here, the student recalls every queried fact correctly, but flattens the tree into a left\-to\-right chain\. After evaluating the first pair of leaves, it takes the next number3754as an operand rather than resolving the subtree3754 \[OP\_20\] 0476first\. BecauseTfT\_\{f\}does not have tree skill, the student never learns how to navigate branching dependencies during its own rollouts\. Distillation fromTsT\_\{s\}with reverse KL corrects the evaluation order and solves the tree\. In summary, these examples show that knowledge failures produce wrong values, whereas skill failures produce wrong reasoning order\.

### 6\.2Weight Updates under Factual and Compositional Distillation

We next examine how factual and compositional distillation reshape the student’s parameters by analyzing the merged LoRA updates relative toS0S\_\{0\}in Qwen3\-4B, Gemma\-2\-2B, and OLMo3\-7B \(Table[3](https://arxiv.org/html/2610.09639#S6.T3)\)\. Across all three models, two clear mechanistic patterns emerge\.

First, factual and compositional distillation occupy near\-orthogonal parameter subspaces\. Across matched prefix sources and divergence objectives, the cosine similarity between updates fromTfT\_\{f\}andTsT\_\{s\}is near zero, ranging from 0\.004 to 0\.046\. In contrast, changing the divergence objective while holding the task fixed preserves substantial directional alignment, reaching cosine similarities of 0\.64–0\.82 underTsT\_\{s\}\. This demonstrates that storing facts and acquiring compositional skills recruit distinct directions in parameter space rather than competing for the same weights\.

Second, the two capabilities modify distinct network depths\. In Qwen3\-4B, updates fromTfT\_\{f\}concentrate in the final layers, increasing monotonically with depth and peaking at layer 35 where final vocabulary predictions are formed\. Conversely, updates fromTsT\_\{s\}concentrate in the early and middle layers, peaking at layers 3 and 11 before decaying toward the output\. This divergence aligns with functional roles in Transformer architectures, where late layers mediate token\-level factual associations while earlier layers route and assemble intermediate representations across multi\-step dependencies\. Appendix[E](https://arxiv.org/html/2610.09639#A5)provides full breakdowns across individual projections, singular value distributions, and embedding drift \(Table[6](https://arxiv.org/html/2610.09639#A5.T6)\)\.

Insight 3: Orthogonal Subspaces and Depth SpecializationFactual and compositional updates occupy near\-orthogonal parameter directions and separate across network depths, with facts concentrating in final layers and compositional skill reshaping early routing layers\.

Table 3:LoRA weight updates relative toS0S\_\{0\}\. \(a\) Update magnitude and directional alignment across models\. \(b\) Layer\-wise norm distribution across depth quartiles in Qwen3\-4B\. Full breakdowns across all projections and embedding drift appear in Appendix[E](https://arxiv.org/html/2610.09639#A5)\(Table[6](https://arxiv.org/html/2610.09639#A5.T6)\)\.

### 6\.3Transfer under Alternative Distillation Objectives

We extend the comparison in Section[4\.2](https://arxiv.org/html/2610.09639#S4.SS2)to objectives that combine the effects of forward and reverse KL, keeping student rollouts as the prefix source\. On Qwen3\-4B, we evaluate GKD with Jensen–Shannon divergence \(β=0\.5\\beta=0\.5\)\[[1](https://arxiv.org/html/2610.09639#bib.bib4)\], ToDi with a sigmoid\-weighted combination of forward and reverse KL for each vocabulary token\[[13](https://arxiv.org/html/2610.09639#bib.bib30)\], and EOPD, which adds forward KL to a reverse\-KL objective at positions with high teacher entropy\[[12](https://arxiv.org/html/2610.09639#bib.bib31)\]\. Each objective is evaluated withTfT\_\{f\}andTsT\_\{s\}, starting fromS0S\_\{0\}\.

Table 4:Greedy final\-answer accuracy \(%\) on Qwen3\-4B with alternative distillation objectives\. All distillation runs use student prefixes\.Table[4](https://arxiv.org/html/2610.09639#S6.T4)shows that ToDi achieves the highest accuracy on both target tests: 64\.3% factual recall after distillation fromTfT\_\{f\}and 86\.4% on unseen tree structures after distillation fromTsT\_\{s\}\. These exceed forward KL by 10\.2 and 10\.0 percentage points, respectively\. GKD and EOPD retain the transfer asymmetry, reaching 70\.6% and 78\.7% on unseen tree structures but only 9\.1% and 13\.7% factual recall\. Thus, hybrid divergence objectives can improve both factual and compositional transfer, with performance varying substantially across objectives\. Across all objectives, distillation fromTfT\_\{f\}leaves compositional accuracy near zero, while distillation fromTsT\_\{s\}leaves held\-out factual recall at its initial level\.

## 7Conclusion

This paper investigates what language models acquire through on\-policy distillation by separating factual knowledge from multi\-step compositional skill\. Across synthetic tasks and real\-world benchmarks, we observe a consistent transfer asymmetry: standard reverse\-KL OPD transfers compositional skill strongly but factual knowledge only weakly\. By separating the source of training prefixes from the divergence objective, we show that factual transfer is primarily determined by the KL objective, whereas student rollouts improve multi\-step compositional execution\. Forward KL substantially improves factual transfer under both student\- and teacher\-generated prefixes\. Overall, these findings suggest that standard on\-policy distillation is better suited for improving how language models organize and use existing knowledge than for updating factual knowledge\. When factual knowledge transfer is the goal, forward\-KL supervision provides a more effective alternative\.

## References

- \[1\]R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. Bachem\(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21246–21263\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2610.09639#S3.SS3.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.09639#S4.SS2.p1.1),[§6\.3](https://arxiv.org/html/2610.09639#S6.SS3.p1.1)\.
- \[2\]M\. Balunovic, J\. Dekoninck, I\. Petrov, N\. Jovanović, and M\. Vechev\(2026\)Matharena: evaluating llms on uncontaminated math competitions\.Advances in Neural Information Processing Systems38\.Cited by:[Appendix D](https://arxiv.org/html/2610.09639#A4.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2610.09639#S5.I1.i2.p1.1)\.
- \[3\]F\. Behrens and L\. Zdeborová\(2026\)Dataset distillation for memorized data: soft labels can leak held\-out teacher knowledge\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 157366–157388\.Cited by:[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px3.p1.1)\.
- \[4\]A\. Cloud, M\. Le, J\. Chua, J\. Betley, A\. Sztyber\-Betley, S\. Mindermann, J\. Hilton, S\. Marks, and O\. Evans\(2026\)Language models transmit behavioural traits through hidden signals in data\.Nature652\(8110\),pp\. 615–621\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1)\.
- \[5\]W\. M\. Czarnecki, R\. Pascanu, S\. Osindero, S\. Jayakumar, G\. Swirszcz, and M\. Jaderberg\(2019\)Distilling policy distillation\.InThe 22nd international conference on artificial intelligence and statistics,pp\. 1331–1340\.Cited by:[§4\.2](https://arxiv.org/html/2610.09639#S4.SS2.p1.1)\.
- \[6\]Y\. Ding and R\. Zhang\(2026\)Does on\-policy distillation really distill? from noisy teacher to self\-improvement\.arXiv preprint arXiv:2608\.31046\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p4.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px2.p1.1)\.
- \[7\]W\. Du, S\. Toshniwal, B\. Kisacanin, S\. Mahdavi, I\. Moshkov, G\. Armstrong, S\. Ge, E\. Minasyan, F\. Chen, and I\. Gitman\(2026\)Nemotron\-math: efficient long\-context distillation of mathematical reasoning from multi\-mode supervision\.In3rd AI for Math Workshop: Toward Self\-Evolving Scientific Agents,External Links:[Link](https://openreview.net/forum?id=FK9LEhmjof)Cited by:[§5](https://arxiv.org/html/2610.09639#S5.SS0.SSS0.Px1.p2.1)\.
- \[8\]X\. Ge, Z\. Zhang, Y\. Huang, J\. Zhu, L\. Yuan, W\. Gu, W\. Wu, W\. Huang, X\. Zhang, B\. Han,et al\.\(2026\)Towards understanding on\-policy distillation through the lens of test\-time scaling\.arXiv preprint arXiv:2608\.11829\.Cited by:[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]Y\. Gu, L\. Dong, F\. Wei, and M\. Huang\(2024\)Minillm: knowledge distillation of large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 32694–32717\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2610.09639#S3.SS3.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.09639#S4.SS2.p3.1)\.
- \[10\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§B\.5](https://arxiv.org/html/2610.09639#A2.SS5.p2.1)\.
- \[12\]W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. Lee\(2026\)Entropy\-aware on\-policy distillation of language models\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=J5i09faOOf)Cited by:[§6\.3](https://arxiv.org/html/2610.09639#S6.SS3.p1.1)\.
- \[13\]S\. Jung, S\. Yoon, D\. Kim, and H\. Lee\(2025\)Todi: token\-wise distillation via fine\-grained divergence control\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 8089–8102\.Cited by:[§6\.3](https://arxiv.org/html/2610.09639#S6.SS3.p1.1)\.
- \[14\]J\. Kasai, K\. Sakaguchi, R\. Le Bras, A\. Asai, X\. Yu, D\. Radev, N\. A\. Smith, Y\. Choi, K\. Inui,et al\.\(2023\)Realtime qa: what’s the answer right now?\.Advances in neural information processing systems36,pp\. 49025–49043\.Cited by:[Appendix D](https://arxiv.org/html/2610.09639#A4.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.09639#S5.I1.i1.p1.1)\.
- \[15\]Y\. Kim and A\. M\. Rush\(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 1317–1327\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]B\. M\. Lake and M\. Baroni\(2023\)Human\-like systematic generalization through a meta\-learning neural network\.Nature623\(7985\),pp\. 115–121\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p2.1)\.
- \[17\]N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.\(2024\)Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[Appendix D](https://arxiv.org/html/2610.09639#A4.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.09639#S5.SS0.SSS0.Px1.p1.1)\.
- \[18\]J\. LI, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. C\. Huang, K\. Rasul, L\. Yu, A\. Jiang, Z\. Shen, Z\. Qin, B\. Dong, L\. Zhou, Y\. Fleureau, G\. Lample, and S\. Polu\(2024\)NuminaMath\.Numina\.Note:\[https://huggingface\.co/AI\-MO/NuminaMath\-CoT\]\(https://github\.com/project\-numina/aimo\-progress\-prize/blob/main/report/numina\_dataset\.pdf\)Cited by:[Appendix D](https://arxiv.org/html/2610.09639#A4.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2610.09639#S5.I1.i2.p1.1)\.
- \[19\]Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu,et al\.\(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p4.1)\.
- \[20\]K\. Lu and T\. M\. Lab\(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/on\-policy\-distillationExternal Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1),[§4\.2](https://arxiv.org/html/2610.09639#S4.SS2.p3.1)\.
- \[21\]J\. X\. Morris, C\. Sitawarin, C\. Guo, N\. Kokhlikyan, G\. E\. Suh, A\. M\. Rush, K\. Chaudhuri, and S\. Mahloujifar\(2026\)How much can language models memorize?\.InForty\-third International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p2.1),[§1](https://arxiv.org/html/2610.09639#S1.p5.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2610.09639#S3.SS1.SSS0.Px1.p1.1)\.
- \[22\]A\. Newell, H\. A\. Simon, and H\. A\. Simon\(1972\)Human problem solving\.Vol\.104,Prentice\-hall Englewood Cliffs, NJ\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.09639#S3.SS1.SSS0.Px2.p1.1)\.
- \[23\]U\. Ojha, Y\. Li, A\. Sundara Rajan, Y\. Liang, and Y\. J\. Lee\(2023\)What knowledge gets distilled in knowledge distillation?\.Advances in Neural Information Processing Systems36,pp\. 11037–11048\.Cited by:[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison,et al\.\(2025\)Olmo 3\.arXiv preprint arXiv:2512\.13961\.Cited by:[§B\.4](https://arxiv.org/html/2610.09639#A2.SS4.p1.1),[§3\.2](https://arxiv.org/html/2610.09639#S3.SS2.SSS0.Px1.p1.1)\.
- \[25\]T\. Pham, N\. P\. Nguyen, P\. Zunjare, W\. Chen, Y\. Tseng, and T\. Vu\(2026\)SealQA: raising the bar for reasoning in search\-augmented language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=zWb7ueH16c)Cited by:[Appendix D](https://arxiv.org/html/2610.09639#A4.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.09639#S5.I1.i1.p1.1)\.
- \[26\]I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal\(2026\)Self\-distillation enables continual learning\.InForty\-third International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px2.p1.1)\.
- \[27\]G\. Team, P\. Georgiev, V\. I\. Lei, R\. Burnell, L\. Bai, A\. Gulati, G\. Tanzer, D\. Vincent, Z\. Pan, S\. Wang,et al\.\(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1)\.
- \[28\]G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§B\.4](https://arxiv.org/html/2610.09639#A2.SS4.p1.1),[§3\.2](https://arxiv.org/html/2610.09639#S3.SS2.SSS0.Px1.p1.1)\.
- \[29\]Q\. Team\(2025\)Qwen3 technical report\.CoRRabs/2505\.09388\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2505.09388)Cited by:[§B\.4](https://arxiv.org/html/2610.09639#A2.SS4.p1.1),[§1](https://arxiv.org/html/2610.09639#S1.p1.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.09639#S3.SS2.SSS0.Px1.p1.1)\.
- \[30\]L\. Thede, K\. Roth, M\. Bethge, Z\. Akata, and T\. Hartvigsen\(2025\)WikiBigEdit: understanding the limits of lifelong knowledge editing in LLMs\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 59326–59354\.Cited by:[Appendix D](https://arxiv.org/html/2610.09639#A4.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.09639#S5.I1.i1.p1.1)\.
- \[31\]T\. Vu, M\. Iyyer, X\. Wang, N\. Constant, J\. Wei, J\. Wei, C\. Tar, Y\. Sung, D\. Zhou, Q\. Le,et al\.\(2024\)Freshllms: refreshing large language models with search engine augmentation\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 13697–13720\.Cited by:[Appendix D](https://arxiv.org/html/2610.09639#A4.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.09639#S5.I1.i1.p1.1)\.
- \[32\]R\. Wang, H\. Wang, Y\. Chen, B\. Xue, T\. Fang, W\. Yu, and K\. Wong\(2026\)Demystifying on\-policy distillation: roles, pathologies, and regulations\.arXiv preprint arXiv:2607\.13399\.Cited by:[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px2.p1.1)\.
- \[33\]L\. Yuan, W\. Chen, Y\. Zhang, G\. Cui, H\. Wang, Z\. You, N\. Ding, Z\. Liu, M\. Sun, and H\. Peng\(2026\)From f \(x\) and g \(x\) to f \(g \(x\)\): llms learn new skills in rl by composing old ones\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 147547–147574\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p2.1),[§2](https://arxiv.org/html/2610.09639#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2610.09639#S3.SS1.SSS0.Px2.p1.1)\.
- \[34\]H\. Zheng, L\. Shen, A\. Tang, Y\. Luo, H\. Hu, B\. Du, Y\. Wen, and D\. Tao\(2025\)Learning from models beyond fine\-tuning\.Nature Machine Intelligence7\(1\),pp\. 6–17\.Cited by:[§1](https://arxiv.org/html/2610.09639#S1.p1.1)\.

## Appendix

### Contents

## Appendix AExecution Examples

We provide complete input prompts and reference execution traces for a chain and a branching tree\. Both examples use the same eight four\-digit inputs and query each of the same seven initial facts, identified by\[OP\_1\]through\[OP\_7\], once\. The structures differ in how intermediate results are combined\. All values are computed from the experimental lookup tables\. Line breaks and indentation are added for readability\. Each completion consists of semicolon\-separated steps followed by the final answer within\[ANS\]and\[/ANS\]\.

#### Sequential Chain

Computation proceeds from left to right\. Each step after the first uses the previous step’s output as its left operand and the next leaf input as its right operand\.

> Input prompt \(\(\(\(\(\(1234 \[OP\_1\] 5678\) \[OP\_2\] 9012\) \[OP\_3\] 3456\) \[OP\_4\] 7890\) \[OP\_5\] 2468\) \[OP\_6\] 1357\) \[OP\_7\] 8024 = ? Reference completion 1234\[OP\_1\]5678=8495; 8495\[OP\_2\]9012=7688; 7688\[OP\_3\]3456=7701; 7701\[OP\_4\]7890=2436; 2436\[OP\_5\]2468=8325; 8325\[OP\_6\]1357=6416; 6416\[OP\_7\]8024=9196; \[ANS\] 9196 \[/ANS\]

#### Branching Tree

Following left\-to\-right postorder traversal, steps 1 to 3 evaluate the left subtree, steps 4 to 6 evaluate the right subtree, and step 7 combines their results\.

> Input prompt \(\(1234 \[OP\_1\] 5678\) \[OP\_3\] \(9012 \[OP\_2\] 3456\)\) \[OP\_7\] \(\(7890 \[OP\_4\] 2468\) \[OP\_6\] \(1357 \[OP\_5\] 8024\)\) = ? Reference completion 1234\[OP\_1\]5678=8495; 9012\[OP\_2\]3456=9508; 8495\[OP\_3\]9508=7122; 7890\[OP\_4\]2468=3507; 1357\[OP\_5\]8024=5086; 3507\[OP\_6\]5086=8064; 7122\[OP\_7\]8064=3761; \[ANS\] 3761 \[/ANS\]

At step 7, the left operand7122comes from step 3, while the right operand8064comes from step 6\. The model must retrieve the left subtree’s result after three intervening steps on the right subtree\. This balanced structure belongs to the unseen pool defined in Appendix[B\.3](https://arxiv.org/html/2610.09639#A2.SS3.SSS0.Px1)\.

## Appendix BAdditional Experimental Details

### B\.1Lookup Tables and Data Construction

Table entries are sampled uniformly from the ten digits\. Values are represented as four\-digit strings with leading zeros\. The tokenizer adds 25 fact identifiers, the identity marker\[OP\_ID\], and two answer delimiters\. The identity marker is excluded from the initial/held\-out fact partition\.

### B\.2Training Corpora and Budgets

Each student\-initialization or teacher\-training corpus has an estimated token budget of 10 million\. The estimate counts each special token and each remaining non\-whitespace character as one unit, summed over prompts and reference completions\.

TheS0S\_\{0\}corpus allocates 45% of this budget to single\-step queries about initial facts, 25% to chains of two, three, five, or seven steps, and 30% to groups of seven independent single\-step queries, totaling 166,365 records\. Each group is presented in one prompt; its queries use separate inputs and do not depend on one another’s results\. The completion lists the seven single\-step computations and encloses their comma\-separated answers within\[ANS\]and\[/ANS\]\.

Each trained teacher allocates 40% of its budget to the initialization tasks\. Within this portion, 30% covers single\-step queries, 40% chains, and 30% groups of seven independent queries, all over initial facts\. The remaining 60% differs by teacher\. ForTfT\_\{f\}, 30% of the total budget covers single\-step queries about held\-out facts and 30% covers chains querying held\-out facts\. ForTsT\_\{s\}, 60% covers trees over initial facts\. ForTf​sT\_\{fs\}, 20% covers single\-step queries about held\-out facts, 20% trees over initial facts, and 10% each chains and trees querying held\-out facts\. Single\-step queries about held\-out facts are split equally by budget between individual queries and groups of seven\. All teacher\-training chains and trees and all distillation data contain seven steps\.

Each distillation corpus contains 20,000 prompts, with queries excluded from student initialization and teacher training\. In teacher training and distillation, chains and trees querying held\-out facts contain one, three, or seven such steps, sampled with probabilities 0\.5, 0\.3, and 0\.2, respectively; any remaining steps query initial facts\.

### B\.3Tree Structures, Test Sets, and Metrics

#### Tree Structures Pools

Of the 429 ordered full binary\-tree structure with seven internal nodes, 64 form the shared pool, 64 the teacher\-only pool, and 128 the unseen pool; 173 are unused\. The balanced tree of height three, where height counts edges on the longest root\-to\-leaf path, belongs to the unseen pool\. The teacher\-only pool includes the left\-associative chain used during student initialization\. Thus, teacher\-only denotes exclusion from distillation prompts, not complete exclusion from the student’s earlier training\.

#### Test Set Definitions

The factual knowledge tests use held\-out facts\. The compositional skill tests use only initial facts and differ in their tree structure pool\. Each test contains 2,000 queries excluded from the training corpora\. All multi\-step tests use seven computational steps\.

#### Evaluation Metrics

*Greedy accuracy*is the percentage of queries whose greedily decoded answer exactly matches the reference four\-digit answer, including leading zeros\. Answers are extracted between\[ANS\]and\[/ANS\]; missing or unparseable answers count as incorrect\. This is the metric reported in Table[5](https://arxiv.org/html/2610.09639#A2.T5)\.

*Pass@32*is the percentage of queries for which at least one of 32 independently sampled completions has the correct final answer, with sampling temperature 1\.0\. It is distinct from greedy final\-answer accuracy and does not require the entire execution trace to be correct\.

### B\.4Student and Teacher Baselines

Table[5](https://arxiv.org/html/2610.09639#A2.T5)reports greedy final\-answer accuracy forS0S\_\{0\}and the specialized teachers across four models from three families: Qwen3\-0\.6B and Qwen3\-4B\[[29](https://arxiv.org/html/2610.09639#bib.bib11)\], Gemma\-2\-2B\[[28](https://arxiv.org/html/2610.09639#bib.bib18)\], and OLMo3\-7B\[[24](https://arxiv.org/html/2610.09639#bib.bib19)\]\.

For Qwen3\-4B,S0S\_\{0\}achieves 100% greedy final\-answer accuracy on seven\-step chains over initial facts, the chain task used during initialization, compared with 0\.45% onComp\-Shared\. Its final\-answer Pass@32 onComp\-Sharedis 1\.45%\.

Table 5:Greedy final\-answer accuracy \(%\) ofS0S\_\{0\}and the specialized teachers on factual knowledge and compositional skill tests\. Test definitions appear in Appendix[B\.3](https://arxiv.org/html/2610.09639#A2.SS3)\. All tests use exact match of the final four\-digit answer\. Green shading highlights high accuracy\.
### B\.5Optimization and Hyperparameters

All experiments use initialization and training seed 1\. Student initialization and teacher training use completion cross\-entropy with a maximum sequence length of 1,024 tokens, a learning rate of10−410^\{\-4\}, and three epochs\. OPD uses a learning rate of3×10−63\\times 10^\{\-6\}and trains for three epochs on 20,000 prompts per distillation condition\. Each visit to a prompt generates one student rollout at temperature 1\.0, capped at 256 new tokens\. The reverse KL is computed over the full vocabulary\.

All training stages employ AdamW with a cosine learning\-rate schedule, 3% linear warmup, zero weight decay, gradient norm clipping at 1\.0, and bfloat16 precision\. The global batch size is 32 for the Qwen3 and Gemma\-2 models and 16 for OLMo3\-7B\. Student initialization, teacher training, and OPD use LoRA adapters\[[11](https://arxiv.org/html/2610.09639#bib.bib20)\]on the attention and feed\-forward projection layers \(rank 16,α=32\\alpha=32, dropout 0\.05\)\. Teachers and distillation students initialize fromS0S\_\{0\}with its adapters merged into the base weights, then train fresh adapters\. Gradient masking restricts token\-embedding updates to newly added token rows\.

## Appendix CTraining Dynamics

Figure[5](https://arxiv.org/html/2610.09639#A3.F5)shows how rollout accuracy and the reverse\-KL objective evolve over the three training epochs\. UnderTsT\_\{s\}, rollout exact match rises from near zero to above 95% in all four models, and the reverse KL falls toward zero: once the student produces correct executions, its own trajectories coincide with the teacher’s supervision\. UnderTfT\_\{f\}, rollout exact match does not exceed about 30% in any model, and the reverse KL plateaus above zero instead of vanishing\. The residual gap concentrates at positions whose answer depends on a held\-out fact, where the teacher is confident about the correct digits while the student spreads probability across candidates it cannot distinguish\. The jointTf​sT\_\{fs\}run shows the same plateau\.

Figure 5:Training curves for on\-policy distillation in the synthetic experiments\. Columns correspond to distillation fromTfT\_\{f\}\(a, d\),TsT\_\{s\}\(b, e\), and the jointTf​sT\_\{fs\}\(c, f\)\. Top row: exact\-match accuracy of sampled student rollouts on the distillation prompts; bottom row: reverse KL in nats\. All runs train for three epochs over 20,000 prompts\.
## Appendix DReal\-World Benchmarks and Teacher Profiles

To evaluate whether the transfer asymmetry observed in synthetic tasks extends to natural language, we curate two evaluation suites covering post\-cutoff factual knowledge and competition\-level mathematical reasoning\.

#### Benchmark Construction

TheKnowledge\-2026benchmark evaluates factual recall using 809 question\-answer pairs that entered public knowledge in 2026, combining items from RealTimeQA\[[14](https://arxiv.org/html/2610.09639#bib.bib23)\], SealQA\[[25](https://arxiv.org/html/2610.09639#bib.bib24)\], FreshQA\[[31](https://arxiv.org/html/2610.09639#bib.bib25)\], and newly edited Wikipedia articles extracted via the WikiBigEdit pipeline\[[30](https://arxiv.org/html/2610.09639#bib.bib26)\]\. Because these items postdate the pretraining cutoff of Qwen3\-4B\-Base, the base student cannot rely on memorized pretraining associations\. TheMath\-2026benchmark evaluates multi\-step reasoning using 201 competition\-level math problems with verified step\-by\-step solutions, sampled from MathArena\[[2](https://arxiv.org/html/2610.09639#bib.bib27)\]and NuminaMath\-CoT\[[18](https://arxiv.org/html/2610.09639#bib.bib28)\]and held out from the curated 1,000\-problem pool described in Section[5](https://arxiv.org/html/2610.09639#S5)\.

#### Teacher Profiles

The initial studentS0S\_\{0\}is fine\-tuned from Qwen3\-4B\-Base using Tulu\-3\-SFT data\[[17](https://arxiv.org/html/2610.09639#bib.bib22)\]\. The specialized teachers shareS0S\_\{0\}’s initialization and training hyperparameters, but incorporate domain\-specific supervision:TKT\_\{K\}is fine\-tuned on the full set of 809 2026 factual QA pairs, andTRT\_\{R\}is fine\-tuned on verified competition\-math solutions\.

## Appendix EWeight Update Measurements

We evaluate parameter changes by analyzing the merged LoRA weight increments relative toS0S\_\{0\}\. For each adapted projectionmm, the increment isΔ​Wm=\(α/r\)​Bm​Am∈ℝdout×din\\Delta W\_\{m\}=\(\\alpha/r\)B\_\{m\}A\_\{m\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}, with rankr=16r=16and scaling factorα=32\\alpha=32\. Below we define the metrics reported in Table[3](https://arxiv.org/html/2610.09639#S6.T3)and Table[6](https://arxiv.org/html/2610.09639#A5.T6)\.

#### Update Norms

The global update norm measures the aggregate magnitude of parameter changes across all adapted linear projections, excluding embeddings:

‖Δ​W‖F=∑m‖Δ​Wm‖F2\.\\\|\\Delta W\\\|\_\{F\}=\\sqrt\{\\sum\_\{m\}\\\|\\Delta W\_\{m\}\\\|\_\{F\}^\{2\}\}\.\(3\)To inspect depth and architectural localization, we compute sub\-norms by restricting the sum to specific modules\. The layer norm‖Δ​Wℓ‖F\\\|\\Delta W\_\{\\ell\}\\\|\_\{F\}aggregates all projections in layerℓ\\ell, while the projection norm‖Δ​Wp‖F\\\|\\Delta W\_\{p\}\\\|\_\{F\}aggregates projection typep∈\{q,k,v,o,gate,up,down\}p\\in\\\{q,k,v,o,\\text\{gate\},\\text\{up\},\\text\{down\}\\\}across all layers\. For depth comparisons in Qwen3\-4B, layer norms are averaged within four consecutive depth quartiles of nine layers each \(Layers 0–8, 9–17, 18–26, and 27–35\)\.

#### Directional Similarity

To measure alignment between two training conditions within the same model family, we concatenate their projection increments into a single vector𝐯=vec⁡\(Δ​W\)\\mathbf\{v\}=\\mathrm\{vec\}\(\\Delta W\)\. The cosine similarity between conditions \(1\) and \(2\) is defined as

cos⁡\(𝐯\(1\),𝐯\(2\)\)=⟨𝐯\(1\),𝐯\(2\)⟩‖𝐯\(1\)‖2​‖𝐯\(2\)‖2\.\\cos\(\\mathbf\{v\}^\{\(1\)\},\\mathbf\{v\}^\{\(2\)\}\)=\\frac\{\\langle\\mathbf\{v\}^\{\(1\)\},\\mathbf\{v\}^\{\(2\)\}\\rangle\}\{\\\|\\mathbf\{v\}^\{\(1\)\}\\\|\_\{2\}\\\|\\mathbf\{v\}^\{\(2\)\}\\\|\_\{2\}\}\.\(4\)

#### Effective Rank

For each projection incrementΔ​Wm\\Delta W\_\{m\}with singular valuesσm,1≥⋯≥σm,r\>0\\sigma\_\{m,1\}\\geq\\dots\\geq\\sigma\_\{m,r\}\>0, we define the normalized spectral distributionpm,i=σm,i2/∑jσm,j2p\_\{m,i\}=\\sigma\_\{m,i\}^\{2\}/\\sum\_\{j\}\\sigma\_\{m,j\}^\{2\}\. The effective rank is the exponential Shannon entropy of this distribution,

reff\(ΔWm\)=exp\(−∑i=1rpm,ilogpm,i\),r\_\{\\mathrm\{eff\}\}\(\\Delta W\_\{m\}\)=\\exp\\left\(\-\\sum\_\{i=1\}^\{r\}p\_\{m,i\}\\log p\_\{m,i\}\\right\),\(5\)averaged across all adapted projections\. This metric reflects how uniformly update energy is distributed across singular directions, ranging from 1 for a rank\-1 update to 16 for a uniform spectrum\.

#### Embedding Drift

Embedding drift measures the Euclidean displacement of fact\-identifier tokens from theirS0S\_\{0\}initialization,‖𝐞k−𝐞k\(S0\)‖2\\\|\\mathbf\{e\}\_\{k\}\-\\mathbf\{e\}\_\{k\}^\{\(S\_\{0\}\)\}\\\|\_\{2\}\. We report the mean drift separately for held\-out tokens \(\[OP\_21\]through\[OP\_25\]\) and a representative sample of initial tokens \(\[OP\_1\]through\[OP\_5\]\)\.

Table 6:Complete weight updates relative toS0S\_\{0\}\. \(a\) Global update norm / mean effective rank\. \(b\) Cosine similarity between updates\. \(c\) Mean embedding drift for held\-out / initial fact identifiers\. \(d\) Mean layer norms by depth quartile\. \(e\) Norms by projection\. Ranges cover the four prefix–KL combinations; control ranges cover two prompt sets\.