Rethinking Reverse KL as Adaptive Entropy Distillation

# Rethinking Reverse KL as Adaptive Entropy Distillation
Source: [https://arxiv.org/html/2608.14685](https://arxiv.org/html/2608.14685)
Shizhen Li1, Zhiyu Shen1, Yuyin Lu1, Yunhe Pang1, Jielin Song1, Yanghui Rao1, Fu Lee Wang2 1School of Computer Science and Engineering, Sun Yat\-sen University, Guangzhou, China 2School of Science and Technology, Hong Kong Metropolitan University, Hong Kong SAR, China \{lishzh57, shenzhy23, luyy37, pangyh8, songjlin6\}@mail2\.sysu\.edu\.cn raoyangh@mail\.sysu\.edu\.cn pwang@hkmu\.edu\.hk

###### Abstract

Knowledge distillation \(KD\) is widely used to transfer the capabilities of large language models \(LLMs\) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation\. In particular, existing methods mainly combine FKL and RKL, overlooking that RKL itself provides a mechanism for adjusting the student’s imitation strength\. Motivated by this, we revisit on\-policy Reverse Kullback\-Leibler \(RKL\) distillation and decompose its objective into a teacher\-fitting term and a student\-entropy term, without introducing an explicit FKL branch\. We show theoretically that the token\-level optimal student distribution corresponds to a tempered variant of the teacher distribution, where the adaptive weight controls the trade\-off between mode\-seeking and uncertainty preservation\. Guided by this insight, we proposeAdaptive Entropy Distillation \(AED\), which uses the teacher’s entropy to dynamically calibrate token\-level imitation strength\. Experiments on instruction\-following and mathematical reasoning benchmarks demonstrate that AED achieves superior overall performance and generally improves teacher–student distributional and entropy alignment\.111Code:[https://github\.com/ShizhenL1/AED](https://github.com/ShizhenL1/AED)

Rethinking Reverse KL as Adaptive Entropy Distillation

Shizhen Li1, Zhiyu Shen1, Yuyin Lu1, Yunhe Pang1, Jielin Song1, Yanghui Rao1††thanks:Corresponding author, Fu Lee Wang21School of Computer Science and Engineering, Sun Yat\-sen University, Guangzhou, China2School of Science and Technology, Hong Kong Metropolitan University, Hong Kong SAR, China\{lishzh57, shenzhy23, luyy37, pangyh8, songjlin6\}@mail2\.sysu\.edu\.cnraoyangh@mail\.sysu\.edu\.cn pwang@hkmu\.edu\.hk

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.14685v1/x1.png)Figure 1:Comparison between fixed\-strength distillation and AED: fixed strength applies uniform imitation, while AED adapts to teacher uncertainty\.LLMs offer strong capabilities but incur high computational and deployment costs\. KD\(Hintonet al\.,[2015](https://arxiv.org/html/2608.14685#bib.bib1)\)addresses this issue by compressing large teacher models into lightweight students\. Conventional sequence\-level KD\(Kim and Rush,[2016](https://arxiv.org/html/2608.14685#bib.bib2)\)optimizes FKL divergence on teacher\-generated trajectories, but such off\-policy training suffers from exposure bias\(Bengioet al\.,[2015](https://arxiv.org/html/2608.14685#bib.bib3); Ranzatoet al\.,[2016](https://arxiv.org/html/2608.14685#bib.bib4)\)\. Recent work turns to on\-policy RKL distillation with student\-sampled trajectories\(Guet al\.,[2024](https://arxiv.org/html/2608.14685#bib.bib5)\), reducing the training\-inference mismatch\.

Nevertheless, fully exploiting the benefits of on\-policy distillation remains challenging due to the intrinsic trade\-off between the probability coverage of FKL and the probability concentration of RKL\(Guet al\.,[2024](https://arxiv.org/html/2608.14685#bib.bib5)\)\. Existing frameworks often address this through static interpolations\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.14685#bib.bib6)\)or variants like Skewed RKL\(Koet al\.,[2024](https://arxiv.org/html/2608.14685#bib.bib7)\)\. Yet, these global, fixed\-weighting strategies uniformly scale objectives across entire sequences, failing to account for varying token\-level teacher confidence\. To overcome this, recent methods such as AKL\(Wuet al\.,[2025](https://arxiv.org/html/2608.14685#bib.bib16)\)and ToDi\(Junget al\.,[2025](https://arxiv.org/html/2608.14685#bib.bib17)\)have introduced dynamic balancing strategies to avoid uniform scaling via token\-level mechanisms\.

While existing frameworks mainly focus on adaptively combining FKL and RKL, the internal structure of RKL itself already provides a mechanism for controlling the strength of the student’s imitation of the teacher\.

We therefore revisit on\-policy distillation from the perspective of distribution geometry\. We show that the RKL objective itself contains complementary forces for concentration and coverage, enabling an RKL\-only formulation by reweighting its two internal components\. Under this reweighted formulation, the analytical optimum corresponds to a tempered teacher distribution, providing a theoretical characterization of how the student’s imitation strength can be controlled\.

Building on these theoretical findings, we develop AED, a theory\-driven, uncertainty\-aware framework for on\-policy distillation, as illustrated in Figure[1](https://arxiv.org/html/2608.14685#S1.F1)\. AED is motivated by this theoretical optimum and adaptively adjusts the imitation strength at the token level\. To balance the two terms within the RKL, AED introduces an adaptive weighting mechanism\. We instantiate this mechanism by mapping the teacher’s entropy to the weight, encouraging the student internalizes the teacher’s distribution\. We evaluate AED on instruction distillation and mathematical reasoning benchmarks and further analyze its token\-level distributional behavior\.

Grounded in a geometric analysis of RKL, AED reweights its intrinsic components to yield an RKL\-only objective and uses teacher entropy to calibrate token\-level imitation strength\. Experiments demonstrate that this design improves task performance and teacher–student distributional alignment, highlighting the practical value of the theoretical characterization\.

## 2Related Work

### 2\.1Evolution of LLM KD

Traditional KD is a major model compression technique, typically following an off\-policy paradigm where the student learns from static teacher\-generated targets, such as soft logits, intermediate representations, or rationalesHintonet al\.\([2015](https://arxiv.org/html/2608.14685#bib.bib1)\); Sanhet al\.\([2019](https://arxiv.org/html/2608.14685#bib.bib8)\); Jiaoet al\.\([2020](https://arxiv.org/html/2608.14685#bib.bib9)\)\. For LLMs, it has been used to distill reasoning via chain\-of\-thought rationalesHsiehet al\.\([2023](https://arxiv.org/html/2608.14685#bib.bib10)\); Fuet al\.\([2023](https://arxiv.org/html/2608.14685#bib.bib11)\)and instruction\-following behaviors from synthetic datasetsJianget al\.\([2023](https://arxiv.org/html/2608.14685#bib.bib12)\)\. However, off\-policy KD suffers from exposure bias and cumulative distribution shiftBengioet al\.\([2015](https://arxiv.org/html/2608.14685#bib.bib3)\); Ranzatoet al\.\([2016](https://arxiv.org/html/2608.14685#bib.bib4)\), since the student is not trained on states induced by its own autoregressive errors\.

To address this, recent on\-policy distillation methods let the student sample from its own distribution while receiving teacher supervisionGuet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib5)\); Agarwalet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib6)\)\. MiniLLMGuet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib5)\)uses RKL divergence for token\-level corrective signals, while GKDAgarwalet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib6)\)generalizes on\-policy distillation with alternative divergence objectives\. However, these methods often treat RKL as a fixed objective; we further decompose and analyze RKL to better understand its distillation behavior\.

### 2\.2The FKL\-RKL Trade\-off in Generative Modeling

The choice of divergence measure strongly shapes the student’s behavior\. Prior work has explored alternative objectives for on\-policy distillation, including Skewed RKL in DistilLLMKoet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib7)\), JS divergence in GKDAgarwalet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib6)\), SKL and generalff\-divergencesWenet al\.\([2023](https://arxiv.org/html/2608.14685#bib.bib13)\)\. These objectives aim to balance the mode\-seeking behavior of RKL with broader distributional coverage\.

Beyond static divergence selection, recent work further explores adaptive balancing strategies\. Related adaptive weighting methods have been studied in computer vision\(Zheng and Yang,[2024](https://arxiv.org/html/2608.14685#bib.bib14); Amaraet al\.,[2022](https://arxiv.org/html/2608.14685#bib.bib15)\), while autoregressive sequence generation emphasizes token\-level dynamics for balancing different divergence behaviors\. AKLWuet al\.\([2025](https://arxiv.org/html/2608.14685#bib.bib16)\)adapts FKL and RKL based on their different fitting behaviors over the head and tail regions of the teacher distribution, whereas ToDiJunget al\.\([2025](https://arxiv.org/html/2608.14685#bib.bib17)\)coordinates them according to how their gradients increase or decrease token probabilities during on\-policy sampling\. More recently, EOPDJinet al\.\([2026](https://arxiv.org/html/2608.14685#bib.bib18)\)builds on RKL and introduces an additional FKL term for high\-entropy teacher tokens\. In contrast, AED does not interpolate FKL and RKL objectives or activate an additional FKL branch\. Instead, it exploits the distinct fitting roles of RKL’s two internal terms, yielding an adaptive target distribution and a simple RKL\-only on\-policy distillation objective\.

## 3Preliminaries and Analysis

This section first introduces the problem setup, notation, and KL\-based distillation objectives, and then examines the training behaviors induced by these objectives, which in turn motivate the adaptive method developed later\.

### 3\.1Problem Setup and Notation

LetPPdenote the teacher model andQθQ\_\{\\theta\}denote the student model parameterized byθ\\theta\. Given an input promptxxand an output sequencey=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\), the teacher and student define autoregressive sequence distributions as

p​\(y∣x\)\\displaystyle p\(y\\mid x\)=∏t=1Tp​\(yt∣x,y<t\),\\displaystyle=\\prod\_\{t=1\}^\{T\}p\(y\_\{t\}\\mid x,y\_\{<t\}\),\(1\)qθ​\(y∣x\)\\displaystyle q\_\{\\theta\}\(y\\mid x\)=∏t=1Tqθ​\(yt∣x,y<t\),\\displaystyle=\\prod\_\{t=1\}^\{T\}q\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\),wherey<t=\(y1,…,yt−1\)y\_\{<t\}=\(y\_\{1\},\\dots,y\_\{t\-1\}\)denotes the prefix preceding steptt\.

In on\-policy distillation, response are sampled from the student modely∼qθ\(⋅∣x\)y\\sim q\_\{\\theta\}\(\\cdot\\mid x\)\. For a given prefixy<ty\_\{<t\}, we denote the teacher and student conditional distributions over the entire vocabulary𝒱\\mathcal\{V\}at stepttasptp\_\{t\}andqtq\_\{t\}, respectively\. Specifically, for any tokenv∈𝒱v\\in\\mathcal\{V\}:

pt​\(v\)=p​\(v∣x,y<t\),qt​\(v\)=qθ​\(v∣x,y<t\)\.p\_\{t\}\(v\)=p\(v\\mid x,y\_\{<t\}\),\\quad q\_\{t\}\(v\)=q\_\{\\theta\}\(v\\mid x,y\_\{<t\}\)\.\(2\)Under this notation, sequence\-level objectives can be decomposed into token\-level terms along the sampled trajectory\.

### 3\.2Geometric Analysis of KL Distillation

The different behaviors of forward and reverse KL can be understood through the training objectives they induce\. To analyze the optimization dynamics in the on\-policy setting, we reformulate the sequence\-level objectives into token\-level expectations over student\-generated trajectories\. For RKL, minimizingDKL​\(qθ∥p\)D\_\{\\mathrm\{KL\}\}\(q\_\{\\theta\}\\parallel p\)is equivalent to minimizing the following on\-policy objective over responses sampled from the student model:

ℒRKL=𝔼y∼qθ\(⋅\|x\)​\[∑t=1T∑v∈𝒱qt​\(v\)​log⁡qt​\(v\)pt​\(v\)\]\.\\mathcal\{L\}\_\{\\mathrm\{RKL\}\}=\\mathbb\{E\}\_\{y\\sim q\_\{\\theta\}\(\\cdot\|x\)\}\\left\[\\sum\_\{t=1\}^\{T\}\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log\\frac\{q\_\{t\}\(v\)\}\{p\_\{t\}\(v\)\}\\right\]\.\(3\)This objective reveals two competing forces in training\. The log\-ratio naturally separates into−∑v∈𝒱qt​\(v\)​log⁡pt​\(v\)\-\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log p\_\{t\}\(v\)and∑v∈𝒱qt​\(v\)​log⁡qt​\(v\)\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log q\_\{t\}\(v\)\. The first term,−∑v∈𝒱qt​\(v\)​log⁡pt​\(v\)\-\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log p\_\{t\}\(v\), acts as a teacher\-derived cross\-entropy loss\. When minimized, it heavily penalizes the student for assigning non\-negligible probability mass to tokens that the teacher deems unlikely, thereby driving the student distribution toward the teacher’s high\-probability modes\. The second term,∑v∈𝒱qt​\(v\)​log⁡qt​\(v\)\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log q\_\{t\}\(v\), corresponds to the negative entropy of the student distribution, which inherently penalizes over\-concentration\.

As a consequence, RKL\-based training can be interpreted as balancing teacher\-driven mode seeking against entropy\-induced probability spreading on student\-sampled trajectories\.

Similarly, while standard FKL is off\-policy \(i\.e\., evaluated on teacher\-generated sequences\), recent on\-policy distillation methods often employ a token\-level FKL formulation to mitigate exposure bias\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.14685#bib.bib6)\)\. In this setting, prefixes are sampled from the student model, and the student is forced to match the teacher’s vocabulary distribution at each step via token\-level FKL\. The corresponding on\-policy FKL objective can be written as:

ℒOn\-FKL=𝔼y∼qθ\(⋅\|x\)​\[∑t=1T∑v∈𝒱pt​\(v\)​log⁡pt​\(v\)qt​\(v\)\]\.\\mathcal\{L\}\_\{\\text\{On\-FKL\}\}=\\mathbb\{E\}\_\{y\\sim q\_\{\\theta\}\(\\cdot\|x\)\}\\left\[\\sum\_\{t=1\}^\{T\}\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{t\}\(v\)\\log\\frac\{p\_\{t\}\(v\)\}\{q\_\{t\}\(v\)\}\\right\]\.\(4\)By separating the log\-ratio, the token\-level objective can be decomposed into two terms:−∑v∈𝒱pt​\(v\)​log⁡qt​\(v\)\-\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{t\}\(v\)\\log q\_\{t\}\(v\)and∑v∈𝒱pt​\(v\)​log⁡pt​\(v\)\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{t\}\(v\)\\log p\_\{t\}\(v\)\. In this decomposition, the first term,−∑v∈𝒱pt​\(v\)​log⁡qt​\(v\)\-\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{t\}\(v\)\\log q\_\{t\}\(v\)is the cross\-entropy term\. Under the teacher\-weighted objective, whenever the student assigns insufficient probability mass to teacher\-supported tokens, this term induces large corrective gradients and therefore forces the student to cover the support of the teacher distribution as broadly as possible\.

In contrast, the second term,∑vpt​\(v\)​log⁡pt​\(v\)\\sum\_\{v\}p\_\{t\}\(v\)\\log p\_\{t\}\(v\), equals−H​\(p\)\-H\(p\)and depends only on the teacher distribution, making it constant with respect to the student parameters\. It therefore contributes no gradient to student optimization\. As a result, FKL\-based training is driven entirely by the support\-covering pressure of the cross\-entropy term, which may lead to mean\-seeking behavior when the student family is less expressive than the teacher\.

RKL, however, has a different internal structure\. The decomposition above shows that it already contains two complementary terms: a teacher\-fitting term and a student\-entropy term\. In standard RKL, these two terms have equal relative weights\. This naturally raises the following question:what happens if we explicitly re\-weight them?Increasing the weight of the teacher\-fitting term sharpens the student distribution toward the teacher’s high\-probability regions, whereas increasing the weight of the student\-entropy term produces a smoother distribution with broader probability coverage\. This suggests that the trade\-off between probability concentration and coverage can be controlled directly through the intrinsic components of RKL, motivating the adaptive re\-weighting formulation introduced in the next section\.

![Refer to caption](https://arxiv.org/html/2608.14685v1/x2.png)Figure 2:Conceptual comparison between fixed\-weight distillation and AED\.

## 4Method

In this section, we present AED, a novel on\-policy distillation framework\. We begin by deriving the analytical optimum of our adaptive objective, establishing its formal equivalence to the RKL divergence to a tempered teacher distribution\. Guided by this theoretical insight, we introduce an entropy\-driven mechanism that dynamically modulates the imitation strength at the token level, enabling the student to accurately internalize the teacher’s distributional structure\.

### 4\.1Analytic Optimum and Equivalence to Tempered RKL

To provide a theoretical characterization of AED, we analyze the token\-level adaptive lossℒt\\mathcal\{L\}\_\{t\}at a fixed student\-generated prefix\. In this local analysis, the teacher distributionptp\_\{t\}and the adaptive weightλt\\lambda\_\{t\}are treated as fixed\. The objective is defined as

ℒt​\(qt\)=\\displaystyle\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)=\{\}∑v∈𝒱qt\(v\)\[−λtlogpt\(v\)\\displaystyle\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\Big\[\-\\lambda\_\{t\}\\log p\_\{t\}\(v\)\(5\)\+\(1−λt\)logqt\(v\)\]\.\\displaystyle\\quad\+\(1\-\\lambda\_\{t\}\)\\log q\_\{t\}\(v\)\\Big\]\.
#### Optimal student distribution\.

For a given teacher distributionptp\_\{t\}and0<λt<10<\\lambda\_\{t\}<1, minimizing Equation[5](https://arxiv.org/html/2608.14685#S4.E5)over the probability simplex yields

qt∗​\(v\)=\[pt​\(v\)\]αt∑u∈𝒱\[pt​\(u\)\]αt,αt=λt1−λt\.q\_\{t\}^\{\*\}\(v\)=\\frac\{\[p\_\{t\}\(v\)\]^\{\\alpha\_\{t\}\}\}\{\\sum\_\{u\\in\\mathcal\{V\}\}\[p\_\{t\}\(u\)\]^\{\\alpha\_\{t\}\}\},\\qquad\\alpha\_\{t\}=\\frac\{\\lambda\_\{t\}\}\{1\-\\lambda\_\{t\}\}\.\(6\)The boundary cases can be understood through their corresponding limits\. The derivation is provided in Appendix[A](https://arxiv.org/html/2608.14685#A1)\.

Equation[6](https://arxiv.org/html/2608.14685#S4.E6)shows that the optimizer is a tempered variant of the teacher distribution, withλt\\lambda\_\{t\}directly controlling the sharpness of the imitation target\. To further elucidate the relationship between our objective and classical divergence measures, we derive the following equivalence:

#### Equivalent tempered\-RKL form\.

To further clarify the relation between the adaptive objective and conventional divergence minimization, define the normalized tempered teacher distribution as

p~t\(αt\)​\(v\)=\[pt​\(v\)\]αtZt​\(αt\),Zt​\(αt\)=∑u∈𝒱\[pt​\(u\)\]αt\.\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\(v\)=\\frac\{\[p\_\{t\}\(v\)\]^\{\\alpha\_\{t\}\}\}\{Z\_\{t\}\(\\alpha\_\{t\}\)\},\\qquad Z\_\{t\}\(\\alpha\_\{t\}\)=\\sum\_\{u\\in\\mathcal\{V\}\}\[p\_\{t\}\(u\)\]^\{\\alpha\_\{t\}\}\.\(7\)The adaptive loss can then be rewritten as

ℒt​\(qt\)=\(1−λt\)​\[DKL​\(qt∥p~t\(αt\)\)−log⁡Zt​\(αt\)\]\.\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)=\(1\-\\lambda\_\{t\}\)\\left\[D\_\{\\mathrm\{KL\}\}\\left\(q\_\{t\}\\parallel\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\\right\)\-\\log Z\_\{t\}\(\\alpha\_\{t\}\)\\right\]\.\(8\)The corresponding derivation is provided in Appendix[A](https://arxiv.org/html/2608.14685#A1)\.

Since1−λt1\-\\lambda\_\{t\}is positive andlog⁡Zt​\(αt\)\\log Z\_\{t\}\(\\alpha\_\{t\}\)is independent ofqtq\_\{t\}, minimizing the adaptive loss with respect toqtq\_\{t\}at a fixed prefix is equivalent to minimizing the RKL divergence to the corresponding tempered teacher distribution\.

As AED averages over prefixes sampled fromqθq\_\{\\theta\}, it induces a prefix\-dependent conditional token\-level target throughout training\.

Equation[8](https://arxiv.org/html/2608.14685#S4.E8)demonstrates that AED implements a controllable mode\-seeking paradigm within the RKL framework\.In this context,αt\\alpha\_\{t\}serves as the inverse temperature: a highαt\\alpha\_\{t\}\(driven by a largeλt\\lambda\_\{t\}\) forces the student toward the teacher’s primary modes, whereas a lowαt\\alpha\_\{t\}encourages broader probability coverage\.

### 4\.2AED

Building upon the theoretical foundation established in equation[6](https://arxiv.org/html/2608.14685#S4.E6), equation[8](https://arxiv.org/html/2608.14685#S4.E8), and the detailed derivation in Appendix[A](https://arxiv.org/html/2608.14685#A1), we introduce AED, as illustrated in Figure[2](https://arxiv.org/html/2608.14685#S3.F2)\. The core principle of AED is to dynamically modulate the balance between the teacher\-fitting objective and the student\-entropy, thereby enabling the student to adapt its imitation strength according to the teacher’s local distributional characteristics\.

As demonstrated in Eq\. \([8](https://arxiv.org/html/2608.14685#S4.E8)\), the adaptive weightλt\\lambda\_\{t\}serves as the critical controller for the inverse temperatureαt\\alpha\_\{t\}, which governs the sharpness of the imitation target\. To operationalize this, we define the sequence\-level on\-policy distillation loss for a given inputxxas:

ℒAED​\(x\)=𝔼y∼qθ\(⋅\|x\)​\[∑t=1Tℒt​\(qt\)\],\\mathcal\{L\}\_\{\\text\{AED\}\}\(x\)=\\mathbb\{E\}\_\{y\\sim q\_\{\\theta\}\(\\cdot\|x\)\}\\left\[\\sum\_\{t=1\}^\{T\}\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)\\right\],\(9\)whereℒt​\(qt\)\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)denotes the token\-level adaptive loss defined in Eq\. \([5](https://arxiv.org/html/2608.14685#S4.E5)\)\.

We formalize this uncertainty\-aware calibration by mapping the teacher’s token\-level entropyHt​\(p\)H\_\{t\}\(p\)to an adaptive weightλt∈\[0,1\]\\lambda\_\{t\}\\in\[0,1\]via a generalized mapping functionf​\(⋅\)f\(\\cdot\):

λt=f​\(Ht​\(p\)\)\.\\lambda\_\{t\}=f\\big\(H\_\{t\}\(p\)\\big\)\.\(10\)Specifically, the mapping functionf​\(⋅\)f\(\\cdot\)must satisfy two fundamental constraints to ensure valid distillation dynamics: \(i\) the output must be bounded within the interval\[0,1\]\[0,1\]to serve as a proper weighting factor, and \(ii\) it must be monotonically decreasing with respect to the teacher’s entropyHt​\(p\)H\_\{t\}\(p\), ensuring that the adaptive weight systematically decays as the teacher’s uncertainty increases\.

To avoid introducing additional mapping hyperparameters, we instantiatef​\(⋅\)f\(\\cdot\)using the normalized divergence between the teacher distribution and the uniform distributionU​\(v\)=1/\|V\|U\(v\)=1/\|V\|\. Specifically,

DKL​\(pt∥U\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\parallel U\)=∑v∈Vpt​\(v\)​\(log⁡pt​\(v\)\+log⁡\|V\|\)\\displaystyle=\\sum\_\{v\\in V\}p\_\{t\}\(v\)\\left\(\\log p\_\{t\}\(v\)\+\\log\|V\|\\right\)\(11\)=log⁡\|V\|−Ht​\(p\)\.\\displaystyle=\\log\|V\|\-H\_\{t\}\(p\)\.We therefore set

λt=DKL​\(pt∥U\)log⁡\|V\|=1−Ht​\(p\)log⁡\|V\|\.\\lambda\_\{t\}=\\frac\{D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\parallel U\)\}\{\\log\|V\|\}=1\-\\frac\{H\_\{t\}\(p\)\}\{\\log\|V\|\}\.\(12\)
Through this entropy\-driven coupling, AED adaptively adjusts the imitation strength: when the teacher has low entropy,λt\\lambda\_\{t\}approaches 1 and the student receives stronger guidance from the teacher\. For high\-entropy tokens, a smallerλt\\lambda\_\{t\}reduces the imitation strength and increases the contribution of the student\-entropy term\.

Under limited student capacity and the geometry of the optimization landscape, the student may not always be able to directly reach the teacher distribution\. By adapting imitation strength to teacher entropy, AED provides a more suitable optimization trajectory that helps the student recover the teacher’s most informative distributional structure\.

Importantly, AED begins by decomposing RKL and adaptively balancing its two terms according to teacher entropy\. The tempered teacher distribution in Equation[6](https://arxiv.org/html/2608.14685#S4.E6)provides an equivalent characterization of the resulting target, rather than serving as an initial design assumption or a separately introduced temperature\-scaling objective\. In practice, AED directly optimizes Equation[5](https://arxiv.org/html/2608.14685#S4.E5)\.

MethodsDollySelfInstVicunaS\-NIUnNIAvgGPT2 1\.5B→\\rightarrowGPT2 120MTeacher27\.18±0\.1815\.07±0\.2216\.14±0\.4627\.46±0\.1631\.53±0\.1423\.48SFT18\.74±0\.209\.43±0\.5611\.92±0\.4516\.88±0\.3718\.54±0\.0615\.10RKL24\.94±0\.3110\.12±0\.4316\.29±0\.2917\.97±0\.1621\.15±0\.1018\.09AKL24\.92±0\.3110\.36±0\.2615\.44±0\.4918\.41±0\.1421\.07±0\.0718\.04ToDi24\.95±0\.1910\.38±0\.3415\.22±0\.6417\.12±0\.1319\.70±0\.1717\.47EOPD23\.66±0\.3412\.66±0\.6715\.74±0\.5023\.52±0\.3424\.21±0\.1219\.96AED25\.44±0\.4112\.61±0\.2516\.83±0\.2025\.33±0\.1327\.74±0\.0621\.59LLaMA2 7B→\\rightarrowTinyLLaMA 1\.1BTeacher29\.59±0\.2521\.18±0\.2820\.58±0\.3032\.71±0\.1632\.72±0\.0827\.36SFT22\.59±0\.3116\.39±0\.5816\.13±0\.3626\.65±0\.1428\.06±0\.1521\.96RKL25\.23±0\.2817\.94±0\.5917\.03±0\.3729\.04±0\.1730\.54±0\.1323\.96AKL25\.20±0\.1917\.31±0\.3116\.52±0\.6227\.53±0\.2129\.56±0\.1923\.22ToDi25\.31±0\.5017\.32±0\.2917\.36±0\.6026\.98±0\.2729\.44±0\.0823\.28EOPD22\.53±0\.2415\.16±0\.7615\.62±0\.3125\.30±0\.4027\.29±0\.1721\.18AED27\.28±0\.2318\.93±0\.3118\.52±0\.2030\.98±0\.2732\.79±0\.1325\.70Qwen3\-4B→\\rightarrowQwen3\-0\.6BTeacher31\.08±0\.5325\.82±0\.3223\.05±0\.7642\.51±0\.3039\.00±0\.0932\.29SFT26\.61±0\.2018\.74±0\.5918\.57±0\.5135\.77±0\.4235\.03±0\.2126\.94RKL28\.33±0\.3020\.10±0\.5219\.76±0\.6036\.94±0\.2736\.90±0\.0528\.41AKL28\.68±0\.2921\.16±0\.4420\.29±0\.2438\.29±0\.2837\.69±0\.1229\.22ToDi29\.09±0\.2020\.74±0\.4519\.75±0\.3636\.85±0\.0837\.90±0\.0728\.86EOPD25\.55±0\.1917\.05±0\.6522\.24±0\.1932\.59±0\.2934\.13±0\.1026\.31AED28\.99±0\.1820\.01±0\.3320\.96±0\.2438\.69±0\.1939\.10±0\.1129\.55Table 1:ROUGE\-L results on five instruction\-following benchmarks\. Results are reported as mean±\\pmstandard deviation across random seeds\.Boldandunderlinedvalues indicate the best and second\-best results within each model configuration, respectively, among the five distillation methods\.

## 5Experiments

### 5\.1Experimental Setup

#### Training Data and Models\.

FollowingGuet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib5)\), we train ondatabricks/dolly\-15k\. We consider three teacher\-student configurations: \(1\) GPT2: GPT\-2 1\.5B distilling to GPT\-2 120M\(Radfordet al\.,[2019](https://arxiv.org/html/2608.14685#bib.bib20)\); \(2\) LLaMA: LLaMA2\-7B\(Touvronet al\.,[2023](https://arxiv.org/html/2608.14685#bib.bib21)\)distilling to TinyLLaMA\-1\.1B\(Zhanget al\.,[2024](https://arxiv.org/html/2608.14685#bib.bib22)\); \(3\) Qwen: Qwen3\-4B\(Team,[2025](https://arxiv.org/html/2608.14685#bib.bib33)\)distilling to Qwen3\-0\.6B\(Team,[2025](https://arxiv.org/html/2608.14685#bib.bib33)\)\. Further implementation details and hyperparameters are provided in Appendix[B](https://arxiv.org/html/2608.14685#A2)\.

#### Evaluation Benchmarks and Metrics\.

Following the evaluation protocol ofGuet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib5)\), we evaluate AED on five instruction\-following benchmarks:Dolly,SelfInst\(Wanget al\.,[2023](https://arxiv.org/html/2608.14685#bib.bib23)\),Vicuna\(Zhenget al\.,[2023](https://arxiv.org/html/2608.14685#bib.bib24)\),S\-NI\(Wanget al\.,[2022](https://arxiv.org/html/2608.14685#bib.bib25)\), andUnNI\(Honovichet al\.,[2023](https://arxiv.org/html/2608.14685#bib.bib26)\)\. Evaluation details are provided in Appendix[B](https://arxiv.org/html/2608.14685#A2)\. We useROUGE\-L\(Lin,[2004](https://arxiv.org/html/2608.14685#bib.bib27)\)as the primary metric and further conduct LLM\-as\-a\-Judge pairwise preference evaluation to assess model alignment\.

#### Baselines\.

We compare AED with the standard RKL baseline and three recent adaptive distillation methods: Adaptive KL \(AKL\)\(Wuet al\.,[2025](https://arxiv.org/html/2608.14685#bib.bib16)\), Token\-level Distillation \(ToDi\)\(Junget al\.,[2025](https://arxiv.org/html/2608.14685#bib.bib17)\), and Entropy\-Aware On\-Policy Distillation \(EOPD\)\(Jinet al\.,[2026](https://arxiv.org/html/2608.14685#bib.bib18)\)\. We reproduce all baselines using their official implementations and align key training settings\.

![Refer to caption](https://arxiv.org/html/2608.14685v1/x3.png)Figure 3:Box plots of ROUGE\-L scores for AED and its ablation variants across five evaluation benchmarks\.

### 5\.2Main Results

#### Overall Performance\.

As shown in the Table[1](https://arxiv.org/html/2608.14685#S4.T1), the proposed AED consistently achieves favorable overall performance across different model architectures and parameter scales, securing the highest average scores\.

AED surpasses the teacher on several benchmarks, consistent with findings fromFurlanelloet al\.\([2018](https://arxiv.org/html/2608.14685#bib.bib28)\)andGuet al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib5)\)\. This outperformance likely stems from two factors: \(i\) the SFT\-trained teacher inherently suffers from exposure bias during generation\(Guet al\.,[2024](https://arxiv.org/html/2608.14685#bib.bib5)\); \(ii\) our adaptive weighting helps the student achieve sharper probability concentration on high\-confidence tokens, resulting in cleaner target distributions\.

![Refer to caption](https://arxiv.org/html/2608.14685v1/x4.png)Figure 4:Pairwise preference win\-tie\-lose rates comparing AED against RKL, AKL, ToDi and EOPD\.
#### LLM\-as\-a\-Judge Evaluation\.

Following the evaluation framework ofDuboiset al\.\([2024](https://arxiv.org/html/2608.14685#bib.bib29)\), we further conduct pairwise preference evaluation using GPT\-5\.4, accessed via API, as the LLM judge\. The evaluation is performed with the TinyLLaMA\-1\.1B student on the full S\-NI test set and the first 500 examples of UnNI\. As shown in Figure[4](https://arxiv.org/html/2608.14685#S5.F4), we compare AED against RKL, AKL, ToDi, and EOPD, and report the win, tie, and loss rates of AED\. AED achieves higher win rates than loss rates in all comparisons, indicating better response quality under the judge evaluation\.

### 5\.3Ablation Analysis

To investigate the specific contribution of our adaptive weighting mechanism, we compare the standard AED, implemented with GPT\-2 as the base language model, with two experimental variants: \(1\)RKL \(λ=0\.5\\lambda=0\.5\), which removes adaptivity by assigning an equal, static weight of 0\.5 to both constituent terms within the objective; \(2\)λ\\lambda\(FKL\+RKL\), which utilizes the adaptiveλ\\lambdato balance FKL and RKL components while retaining both objectives simultaneously\.

As illustrated in Figure[3](https://arxiv.org/html/2608.14685#S5.F3), the standard AED significantly outperforms all variants across the majority of benchmarks\. The superiority of AED over the fixedλ=0\.5\\lambda=0\.5baseline demonstrates that a static balance fails to capture the intricate token\-level trade\-offs between probability concentration and exploration\.

For theλ\\lambda\(FKL\+RKL\) variant, directly applying adaptive weighting to FKL and RKL lacks explicit theoretical motivation and is therefore more heuristic in nature\. In addition, the mean\-seeking nature of FKL encourages the student to cover the full support of the teacher’s distribution\. In capacity\-limited settings, the support\-covering pressure of FKL can trade off against concentration on high\-confidence teacher modes\.

### 5\.4Distributional and Entropy Alignment Analysis

MethodsRKL↓\\downarrowMAEG↓\\downarrowGPT\-2 1\.5B→\\rightarrowGPT\-2 120MRKL2\.95560\.4002AKL2\.72810\.4042ToDi2\.67280\.4104EOPD1\.38140\.4180AED1\.32120\.3251LLaMA2 7B→\\rightarrowTinyLLaMA 1\.1BRKL2\.60690\.4792AKL2\.56200\.4840ToDi2\.91300\.4613EOPD2\.53500\.6715AED2\.24280\.4243Qwen3\-4B→\\rightarrowQwen3\-0\.6BRKL2\.72570\.2549AKL2\.59110\.2606ToDi2\.57180\.2652EOPD2\.34960\.3161AED2\.35540\.2747Table 2:RKL and entropy alignment under three teacher–student configurations\.Boldandunderlinedvalues indicate the best and second\-best results within each configuration, respectively\.MethodMATH\-500AMC23AIME 2024Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8RKL14\.63%53\.40%9\.06%37\.50%0\.42%3\.33%AKL7\.23%34\.00%3\.44%20\.00%0\.83%6\.67%ToDi9\.75%43\.40%8\.13%35\.00%0\.42%3\.33%EOPD40\.65%66\.00%17\.50%35\.00%0\.83%6\.67%AED47\.80%71\.40%23\.75%50\.00%2\.08%6\.67%Table 3:Accuracy \(%\) results on three mathematical reasoning benchmarks\.Boldandunderlinedvalues indicate the best and second\-best results, respectively\.![Refer to caption](https://arxiv.org/html/2608.14685v1/x5.png)Figure 5:Pass@k results on the evaluated benchmarks\.To conduct a fine\-grained analysis, we consider three teacher–student model pairs\. For each pair, the teacher generates eight distinct responses for every prompt in the S\-NI dataset, and we evaluate the student’s predictive distribution against that of the teacher at each token position\. We assess distributional alignment using token\-level RKL and entropy alignment using the Mean Absolute Entropy Gap \(MAEG\), defined as the average absolute difference between the teacher and student predictive entropies\. Both metrics are averaged over all tokens across the eight responses, with lower values indicating closer teacher–student alignment\.

As shown in Table[2](https://arxiv.org/html/2608.14685#S5.T2), AED achieves the lowest RKL and MAEG values in the GPT\-2 and LLaMA2–TinyLLaMA configurations\. Under Qwen3, AED obtains competitive RKL and MAEG alignment with the best\-performing methods\. Overall, these results show that entropy\-guided adjustment improves the alignment of predictive distributions and uncertainty levels between the student and teacher\. Additional alignment results on mathematical reasoning datasets are provided in Appendix[C](https://arxiv.org/html/2608.14685#A3)\.

### 5\.5Mathematical Reasoning

To further evaluate the generalization ability of AED beyond instruction\-following tasks, we extend the comparison to mathematical reasoning\. We use Qwen3\-4B\-GRPO\(Liet al\.,[2026](https://arxiv.org/html/2608.14685#bib.bib34)\)as the teacher model and Qwen3\-0\.6B\(Team,[2025](https://arxiv.org/html/2608.14685#bib.bib33)\)as the student model\. The student models are trained on the MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.14685#bib.bib30)\)training set\. We evaluate the resulting student models on MATH\-500\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.14685#bib.bib30)\), AMC 2023\(Mathematical Association of America,[2023](https://arxiv.org/html/2608.14685#bib.bib32)\)and AIME 2024\(Mathematical Association of America,[2024](https://arxiv.org/html/2608.14685#bib.bib31)\)using Avg@8 and Pass@8\.

As shown in Table[3](https://arxiv.org/html/2608.14685#S5.T3), AED achieves the best overall performance across all three benchmarks\. Figure[5](https://arxiv.org/html/2608.14685#S5.F5)compares Pass@kkfork∈\{1,2,4,8\}k\\in\\\{1,2,4,8\\\}across three mathematical reasoning benchmarks\. AED achieves higher Pass@1 and consistently higher Pass@kkunder the same sampling budget, indicating better single\-sample accuracy and sampling efficiency\.

## 6Conclusion

In this work, we decompose the RKL objective and derive its equivalence to a tempered RKL framework\. Based on this analysis, we propose AED, which adaptively adjusts token\-level distillation intensity according to teacher uncertainty\. Experiments demonstrate that AED achieves strong performance in task evaluation, distributional alignment, and entropy alignment\.

## Limitations

Despite its advantages, certain limitations of our work warrant discussion\. First, AED is designed for open\-source LLMs and requires vocabulary consistency between the teacher and student to perform token\-level alignment\. Extending this to black\-box APIs or heterogeneous vocabularies remains an open direction\. Second, due to computational resource constraints, we did not scale experiments to ultra\-large models\. Crucially, our theoretical formulations are entirely scale\-agnostic, supporting seamless application to LLMs of any capacity when resources are available\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. Bachem \(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.14685#S3.SS2.p3.4)\.
- I\. Amara, N\. M\. Sepahvand, B\. H\. Meyer, W\. J\. Gross, and J\. J\. Clark \(2022\)BD\-KD: balancing the divergences for online knowledge distillation\.CoRRabs/2212\.12965\.External Links:[Link](https://doi.org/10.48550/arXiv.2212.12965),[Document](https://dx.doi.org/10.48550/ARXIV.2212.12965),2212\.12965Cited by:[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p2.1)\.
- S\. Bengio, O\. Vinyals, N\. Jaitly, and N\. Shazeer \(2015\)Scheduled sampling for sequence prediction with recurrent neural networks\.InAdvances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7\-12, 2015, Montreal, Quebec, Canada,C\. Cortes, N\. D\. Lawrence, D\. D\. Lee, M\. Sugiyama, and R\. Garnett \(Eds\.\),pp\. 1171–1179\.External Links:[Link](https://proceedings.neurips.cc/paper/2015/hash/e995f98d56967d946471af29d7bf99f1-Abstract.html)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto \(2024\)Length\-controlled AlpacaEval: A simple way to debias automatic evaluators\.CoRRabs/2404\.04475\.External Links:[Link](https://doi.org/10.48550/arXiv.2404.04475),[Document](https://dx.doi.org/10.48550/ARXIV.2404.04475),2404\.04475Cited by:[§5\.2](https://arxiv.org/html/2608.14685#S5.SS2.SSS0.Px2.p1.1)\.
- Y\. Fu, H\. Peng, L\. Ou, A\. Sabharwal, and T\. Khot \(2023\)Specializing smaller language models towards multi\-step reasoning\.InInternational Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research,pp\. 10421–10430\.External Links:[Link](https://proceedings.mlr.press/v202/fu23d.html)Cited by:[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- T\. Furlanello, Z\. C\. Lipton, M\. Tschannen, L\. Itti, and A\. Anandkumar \(2018\)Born\-again neural networks\.InProceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10\-15, 2018,J\. G\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research,pp\. 1602–1611\.External Links:[Link](http://proceedings.mlr.press/v80/furlanello18a.html)Cited by:[§5\.2](https://arxiv.org/html/2608.14685#S5.SS2.SSS0.Px1.p2.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p1.1),[§1](https://arxiv.org/html/2608.14685#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p2.1),[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2608.14685#S5.SS2.SSS0.Px1.p2.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual,J\. Vanschoren and S\. Yeung \(Eds\.\),External Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html)Cited by:[§5\.5](https://arxiv.org/html/2608.14685#S5.SS5.p1.1)\.
- G\. E\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.CoRRabs/1503\.02531\.External Links:[Link](http://arxiv.org/abs/1503.02531),1503\.02531Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- O\. Honovich, T\. Scialom, O\. Levy, and T\. Schick \(2023\)Unnatural instructions: tuning language models with \(almost\) no human labor\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 14409–14428\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.806),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.806)Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px2.p1.1)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Findings of ACL,pp\. 8003–8017\.External Links:[Link](https://doi.org/10.18653/v1/2023.findings-acl.507),[Document](https://dx.doi.org/10.18653/V1/2023.FINDINGS-ACL.507)Cited by:[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- Y\. Jiang, C\. Chan, M\. Chen, and W\. Wang \(2023\)Lion: adversarial distillation of proprietary large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6\-10, 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 3134–3154\.External Links:[Link](https://doi.org/10.18653/v1/2023.emnlp-main.189),[Document](https://dx.doi.org/10.18653/V1/2023.EMNLP-MAIN.189)Cited by:[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. Liu \(2020\)TinyBERT: distilling BERT for natural language understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16\-20 November 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Findings of ACL,pp\. 4163–4174\.External Links:[Link](https://doi.org/10.18653/v1/2020.findings-emnlp.372),[Document](https://dx.doi.org/10.18653/V1/2020.FINDINGS-EMNLP.372)Cited by:[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- W\. Jin, T\. Min, Y\. Yang, S\. R\. Kadhe, Y\. Zhou, D\. Wei, N\. Baracaldo, and K\. Lee \(2026\)Entropy\-aware on\-policy distillation of language models\.CoRRabs/2603\.07079\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.07079),[Document](https://dx.doi.org/10.48550/ARXIV.2603.07079),2603\.07079Cited by:[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p2.1),[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px3.p1.1)\.
- S\. Jung, S\. Yoon, D\. Kim, and H\. Lee \(2025\)ToDi: token\-wise distillation via fine\-grained divergence control\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 8078–8091\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.409),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.409)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p2.1),[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px3.p1.1)\.
- Y\. Kim and A\. M\. Rush \(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1\-4, 2016,J\. Su, X\. Carreras, and K\. Duh \(Eds\.\),pp\. 1317–1327\.External Links:[Link](https://doi.org/10.18653/v1/d16-1139),[Document](https://dx.doi.org/10.18653/V1/D16-1139)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p1.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: towards streamlined distillation for large language models\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 24872–24895\.External Links:[Link](https://proceedings.mlr.press/v235/ko24c.html)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p1.1)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.CoRRabs/2604\.13016\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.13016),[Document](https://dx.doi.org/10.48550/ARXIV.2604.13016),2604\.13016Cited by:[§5\.5](https://arxiv.org/html/2608.14685#S5.SS5.p1.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px2.p1.1)\.
- Mathematical Association of America \(2023\)2023 american mathematics competitions \(amc 10 and amc 12\)\.Note:MAA American Mathematics CompetitionsAccessed: 2026\-07\-29External Links:[Link](https://maa.org/student-programs/amc/)Cited by:[§5\.5](https://arxiv.org/html/2608.14685#S5.SS5.p1.1)\.
- Mathematical Association of America \(2024\)American invitational mathematics examination 2024\.Note:MAA American Mathematics CompetitionsAIME I and AIME IIExternal Links:[Link](https://maa.org/maa-invitational-competitions/)Cited by:[§5\.5](https://arxiv.org/html/2608.14685#S5.SS5.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.Technical reportOpenAI\.External Links:[Link](https://api.semanticscholar.org/CorpusID:160025533)Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px1.p1.1)\.
- M\. Ranzato, S\. Chopra, M\. Auli, and W\. Zaremba \(2016\)Sequence level training with recurrent neural networks\.In4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2\-4, 2016, Conference Track Proceedings,Y\. Bengio and Y\. LeCun \(Eds\.\),External Links:[Link](http://arxiv.org/abs/1511.06732)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf \(2019\)DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\.CoRRabs/1910\.01108\.External Links:[Link](http://arxiv.org/abs/1910.01108),1910\.01108Cited by:[§2\.1](https://arxiv.org/html/2608.14685#S2.SS1.p1.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.CoRRabs/2505\.09388\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.09388),[Document](https://dx.doi.org/10.48550/ARXIV.2505.09388),2505\.09388Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px1.p1.1),[§5\.5](https://arxiv.org/html/2608.14685#S5.SS5.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. Canton\-Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023\)Llama 2: open foundation and fine\-tuned chat models\.CoRRabs/2307\.09288\.External Links:[Link](https://doi.org/10.48550/arXiv.2307.09288),[Document](https://dx.doi.org/10.48550/ARXIV.2307.09288),2307\.09288Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 13484–13508\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.754),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.754)Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px2.p1.1)\.
- Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap, E\. Pathak, G\. Karamanolakis, H\. G\. Lai, I\. Purohit, I\. Mondal, J\. Anderson, K\. Kuznia, K\. Doshi, K\. K\. Pal, M\. Patel, M\. Moradshahi, M\. Parmar, M\. Purohit, N\. Varshney, P\. R\. Kaza, P\. Verma, R\. S\. Puri, R\. Karia, S\. Doshi, S\. K\. Sampat, S\. Mishra, S\. R\. A, S\. Patro, T\. Dixit, and X\. Shen \(2022\)Super\-NaturalInstructions: generalization via declarative instructions on 1600\+ NLP tasks\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\-11, 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),pp\. 5085–5109\.External Links:[Link](https://doi.org/10.18653/v1/2022.emnlp-main.340),[Document](https://dx.doi.org/10.18653/V1/2022.EMNLP-MAIN.340)Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px2.p1.1)\.
- Y\. Wen, Z\. Li, W\. Du, and L\. Mou \(2023\)F\-divergence minimization for sequence\-level knowledge distillation\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 10817–10834\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.605),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.605)Cited by:[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p1.1)\.
- T\. Wu, C\. Tao, J\. Wang, R\. Yang, Z\. Zhao, and N\. Wong \(2025\)Rethinking Kullback\-Leibler divergence in knowledge distillation for large language models\.InProceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19\-24, 2025,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),pp\. 5737–5755\.External Links:[Link](https://aclanthology.org/2025.coling-main.383/)Cited by:[§1](https://arxiv.org/html/2608.14685#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p2.1),[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px3.p1.1)\.
- P\. Zhang, G\. Zeng, T\. Wang, and W\. Lu \(2024\)TinyLlaMA: an open\-source small language model\.CoRRabs/2401\.02385\.External Links:[Link](https://doi.org/10.48550/arXiv.2401.02385),[Document](https://dx.doi.org/10.48550/ARXIV.2401.02385),2401\.02385Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px1.p1.1)\.
- K\. Zheng and E\. Yang \(2024\)Knowledge distillation based on transformed teacher matching\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=MJ3K7uDGGl)Cited by:[§2\.2](https://arxiv.org/html/2608.14685#S2.SS2.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets%5C_and%5C_Benchmarks.html)Cited by:[§5\.1](https://arxiv.org/html/2608.14685#S5.SS1.SSS0.Px2.p1.1)\.

## Appendix AAnalytic Optimum of the Token\-Level Adaptive Objective

We derive the analytic optimum of our token\-level adaptive loss and demonstrate its mathematical equivalence to minimizing the Reverse RKL divergence between the student distribution and a temperature\-scaled teacher distribution\.

### A\.1Lagrangian Optimization for the Adaptive Objective

Letpt​\(v\)=p​\(yt=v∣x,y<t\)p\_\{t\}\(v\)=p\(y\_\{t\}=v\\mid x,y\_\{<t\}\)andqt​\(v\)=qθ​\(yt=v∣x,y<t\)q\_\{t\}\(v\)=q\_\{\\theta\}\(y\_\{t\}=v\\mid x,y\_\{<t\}\)denote the teacher and student probabilities over the vocabulary𝒱\\mathcal\{V\}at steptt, respectively\. Given a fixed prefixy<ty\_\{<t\}, the token\-level expected loss to be minimized is defined as:

ℒt\(qt\)=∑v∈𝒱qt\(v\)\[−\\displaystyle\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)=\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\Big\[\-λt​log⁡pt​\(v\)\\displaystyle\\lambda\_\{t\}\\log p\_\{t\}\(v\)\(13\)\+\(1−λt\)logqt\(v\)\]\.\\displaystyle\+\(1\-\\lambda\_\{t\}\)\\log q\_\{t\}\(v\)\\Big\]\.
To find the optimal student distributionqt∗q\_\{t\}^\{\*\}, we minimizeℒt​\(qt\)\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)subject to the probability simplex constraint∑vqt​\(v\)=1\\sum\_\{v\}q\_\{t\}\(v\)=1\. We construct the Lagrangian:

ℒ​\(qt,η\)\\displaystyle\\mathcal\{L\}\(q\_\{t\},\\eta\)=−λt​∑v∈𝒱qt​\(v\)​log⁡pt​\(v\)\\displaystyle=\-\\lambda\_\{t\}\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log p\_\{t\}\(v\)\(14\)\+\(1−λt\)​∑v∈𝒱qt​\(v\)​log⁡qt​\(v\)\\displaystyle\\quad\+\(1\-\\lambda\_\{t\}\)\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log q\_\{t\}\(v\)\+η​\(∑v∈𝒱qt​\(v\)−1\)\.\\displaystyle\\quad\+\\eta\\left\(\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\-1\\right\)\.
Setting the partial derivative with respect toqt​\(v\)q\_\{t\}\(v\)to zero yields:

∂ℒ∂qt​\(v\)\\displaystyle\\frac\{\\partial\\mathcal\{L\}\}\{\\partial q\_\{t\}\(v\)\}=−λt​log⁡pt​\(v\)\\displaystyle=\-\\lambda\_\{t\}\\log p\_\{t\}\(v\)\(15\)\+\(1−λt\)​\(1\+log⁡qt​\(v\)\)\+η=0\.\\displaystyle\\quad\+\(1\-\\lambda\_\{t\}\)\\big\(1\+\\log q\_\{t\}\(v\)\\big\)\+\\eta=0\.
Solving forlog⁡qt​\(v\)\\log q\_\{t\}\(v\), we have:

log⁡qt​\(v\)=λt1−λt​log⁡pt​\(v\)\+C,\\log q\_\{t\}\(v\)=\\frac\{\\lambda\_\{t\}\}\{1\-\\lambda\_\{t\}\}\\log p\_\{t\}\(v\)\+C,\(16\)whereC=−1−η1−λtC=\-1\-\\frac\{\\eta\}\{1\-\\lambda\_\{t\}\}is a constant independent ofvv\. Thus, the analytic optimum explicitly takes the form of a scaled distribution:

qt∗​\(v\)∝pt​\(v\)αt,whereαt=λt1−λt\.q\_\{t\}^\{\*\}\(v\)\\propto p\_\{t\}\(v\)^\{\\alpha\_\{t\}\},\\quad\\text\{where\}\\quad\\alpha\_\{t\}=\\frac\{\\lambda\_\{t\}\}\{1\-\\lambda\_\{t\}\}\.\(17\)

### A\.2Mathematical Equivalence to Tempered RKL

This optimality condition reveals that the adaptive objective implicitly minimizes a divergence against a temperature\-scaled transformation of the teacher’s predictive distribution\. To explicitly establish the connection to the RKL framework, we define the normalized, temperature\-scaled \(tempered\) teacher distribution as:

p~t\(αt\)​\(v\)=pt​\(v\)αtZt​\(αt\),where​Zt​\(αt\)=∑u∈𝒱pt​\(u\)αt\.\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\(v\)=\\frac\{p\_\{t\}\(v\)^\{\\alpha\_\{t\}\}\}\{Z\_\{t\}\(\\alpha\_\{t\}\)\},\\,\\text\{where\}\\,Z\_\{t\}\(\\alpha\_\{t\}\)=\\sum\_\{u\\in\\mathcal\{V\}\}p\_\{t\}\(u\)^\{\\alpha\_\{t\}\}\.\(18\)We can now rewrite the original lossℒt​\(qt\)\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)usingαt\\alpha\_\{t\}:

ℒt​\(qt\)\\displaystyle\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)=\(1−λt\)∑v∈𝒱qt\(v\)\[−αtlogpt\(v\)\\displaystyle=\(1\-\\lambda\_\{t\}\)\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\Big\[\-\\alpha\_\{t\}\\log p\_\{t\}\(v\)\(19\)\+logqt\(v\)\]\.\\displaystyle\\quad\+\\log q\_\{t\}\(v\)\\Big\]\.
Taking the logarithm of the tempered teacher distribution givesαt​log⁡pt​\(v\)=log⁡p~t\(αt\)​\(v\)\+log⁡Zt​\(αt\)\\alpha\_\{t\}\\log p\_\{t\}\(v\)=\\log\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\(v\)\+\\log Z\_\{t\}\(\\alpha\_\{t\}\)\. Substituting this back into the loss yields:

ℒt​\(qt\)\\displaystyle\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)=\(1−λt\)∑v∈𝒱qt\(v\)\[logqt​\(v\)p~t\(αt\)​\(v\)\\displaystyle=\(1\-\\lambda\_\{t\}\)\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\Bigg\[\\log\\frac\{q\_\{t\}\(v\)\}\{\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\(v\)\}\(20\)−logZt\(αt\)\]\\displaystyle\\qquad\\qquad\-\\log Z\_\{t\}\(\\alpha\_\{t\}\)\\Bigg\]=\(1−λt\)​∑v∈𝒱qt​\(v\)​log⁡qt​\(v\)p~t\(αt\)​\(v\)\\displaystyle=\(1\-\\lambda\_\{t\}\)\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log\\frac\{q\_\{t\}\(v\)\}\{\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\(v\)\}−\(1−λt\)​log⁡Zt​\(αt\)​∑v∈𝒱qt​\(v\)\.\\displaystyle\\quad\-\(1\-\\lambda\_\{t\}\)\\log Z\_\{t\}\(\\alpha\_\{t\}\)\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\.
Since∑vqt​\(v\)=1\\sum\_\{v\}q\_\{t\}\(v\)=1, the first term inside the summation can be naturally expressed as the standard Kullback\-Leibler divergence:

∑v∈𝒱qt​\(v\)​log⁡qt​\(v\)p~t\(αt\)​\(v\)=DKL​\(qt∥p~t\(αt\)\)\.\\sum\_\{v\\in\\mathcal\{V\}\}q\_\{t\}\(v\)\\log\\frac\{q\_\{t\}\(v\)\}\{\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\(v\)\}=D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{t\}\\,\\\|\\,\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\\right\)\.\(21\)
Ultimately, the expected loss objective simplifies to:

ℒt​\(qt\)=\(1−λt\)​\[DKL​\(qt∥p~t\(αt\)\)−log⁡Zt​\(αt\)\]\.\\mathcal\{L\}\_\{t\}\(q\_\{t\}\)=\(1\-\\lambda\_\{t\}\)\\left\[D\_\{\\text\{KL\}\}\(q\_\{t\}\\parallel\\tilde\{p\}\_\{t\}^\{\(\\alpha\_\{t\}\)\}\)\-\\log Z\_\{t\}\(\\alpha\_\{t\}\)\\right\]\.\(22\)This derivation reveals that minimizing our token\-level adaptive loss is mathematically equivalent to minimizing the RKL divergence between the student distribution and the tempered teacher distribution\. When directly targetingptp\_\{t\}, standard RKL can exhibit stronger mode\-seeking behavior\. Our objective executes a controllable mode\-seeking paradigm\. By dynamically adjusting the inverse temperatureαt\\alpha\_\{t\}in response to teacher entropy, the objective safely smooths the target distribution in high\-uncertainty regions, while preserving confident mode\-seeking in low\-uncertainty regions\.

## Appendix BImplementation Details

### B\.1Training details

#### Instruction Following\.

We use thedatabricks/dolly\-15kdataset for training, which contains 11K training samples, 1K validation samples, and 500 test samples\. All training processes are conducted on a server equipped with two NVIDIA RTX A100 \(80GB\) GPUs\. The training cost is approximately 3 GPU hours for GPT\-2, 30 GPU hours for TinyLLaMA, and 21\.5 GPU hours for Qwen3\. Detailed training configurations for the three model architectures are summarized in Table[4](https://arxiv.org/html/2608.14685#A2.T4)\.

ParameterGPT\-2 \(120M\)TinyLLaMA \(1\.1B\)Qwen3 \(0\.6B\)LoRANoYesYesOptimizerAdamWAdamWAdamWLR2×10−52\\times 10^\{\-5\}5×10−65\\times 10^\{\-6\}5×10−65\\times 10^\{\-6\}LR schedulerCosineCosineCosineMini\-batch size321616Batch size323232Max response length256256256Temperature1\.01\.01\.0Top\-pp1\.01\.01\.0Epochs31010Table 4:Instruction\-following hyperparameters\.
#### Mathematical Reasoning\.

We use theverl\-team/lighteval\-MATH\-preprocesseddataset for training, which contains 7\.5K training samples and 5K test samples\. All training processes are conducted on a server equipped with two NVIDIA A100 \(80GB\) GPUs\. The training cost is approximately 3\.93 GPU hours for Qwen3\. Detailed training configuration is summarized in Table[5](https://arxiv.org/html/2608.14685#A2.T5)\.

ParameterQwen3 \(0\.6B\)LR5×10−65\\times 10^\{\-6\}LR schedulerCosineOptimizerAdamWBatch size128Mini\-batch size32Top\-pp1\.0Max response length1024Temperature1\.0Epochs3Table 5:Mathematical\-reasoning hyperparameters\.

### B\.2Evaluation details

#### Instruction Following\.

For generation\-based evaluation, we sample model responses with temperature set to1\.01\.0,top\-kset to0, andtop\-pset to1\.01\.0\. For the evaluation datasets, we use the full test sets of DollyEval, SelfInst, Vicuna, and S\-NI, while for UnNI we evaluate on the first 10,000 examples\. To account for randomness in generation, we conduct evaluations with five random seeds,\{10,20,30,40,50\}\\\{10,20,30,40,50\\\}, and report the mean ROUGE\-L score together with the standard deviation across these runs\. The final training checkpoint is used for evaluation\.

#### Mathematical Reasoning\.

For mathematical reasoning evaluation, we generate eight responses for each problem with a maximum response length of 2,048 tokens\. Sampling is performed with a temperature of1\.01\.0,top\-kof0, andtop\-pof0\.80\.8\. We report Avg@8, the average accuracy over the eight responses, and Pass@8, the proportion of problems for which at least one response is correct\. The final training checkpoint is used for evaluation\.

## Appendix CDistributional and Entropy Alignment in Mathematical Reasoning

MethodsRKL↓\\downarrowMAEG↓\\downarrowQwen3\-4B\-GRPO→\\rightarrowQwen3\-0\.6BRKL0\.32830\.1502AKL0\.35920\.1627ToDi0\.35900\.1622EOPD0\.25040\.0987AED0\.24730\.0870Table 6:Distributional and entropy alignment for mathematical reasoning distillation on AMC23\.We further evaluate token\-level RKL and entropy alignment on the AMC23 dataset using the MAEG\. Except for the dataset, the evaluation settings follow those in Section[5\.4](https://arxiv.org/html/2608.14685#S5.SS4)\. The results show that AED obtains the best alignment\.