Consistently Informative Soft-Label Temperature for Knowledge Distillation

# Consistently Informative Soft-Label Temperature for Knowledge Distillation
Source: [https://arxiv.org/html/2605.20357](https://arxiv.org/html/2605.20357)
Hoang\-Chau Luong1,†Nghia Van Vo2,†Kaiqi Zhao2Lingwei Chen1 1Rochester Institute of Technology, Rochester, NY, USA 2Oakland University, Rochester, MI, USA †These authors contributed equally cl6300@rit\.edu, \{nghiavo, kaiqizhao\}@oakland\.edu, lwcics@rit\.edu

###### Abstract

Knowledge distillation \(KD\) transfers knowledge from a high\-capacity teacher to a compact student by matching their predictive distributions, with temperature scalingτ\\tauserving as a central mechanism for smoothing teacher predictions and exposing informative “dark knowledge” beyond the hard label\. However, the standard fixed\-temperature design is inherently sample\-agnostic\. Since samples differ in logit scale and learning difficulty, a single global temperature produces teacher soft labels with highly inconsistent entropy: some predictions remain overly sharp and provide limited inter\-class information, whereas others become over\-smoothed and lose class\-discriminative information\. Moreover, sharing the same temperature between teacher and student further imposes rigid logit\-scale alignment despite their capacity mismatch\. To address these limitations, we proposeCIST\(ConsistentlyInformativeSoft\-labelTemperature\), which assigns separate sample\-wise adaptive temperatures to the teacher and student\. This design produces consistently informative teacher soft labels while relaxing rigid teacher–student logit\-scale matching\. It also reweights the distillation objective according to teacher confidence and student learning difficulty\. Theoretically, we show that teacher\-label entropy is largely governed by the ratio between the maximum teacher logit and the temperature, providing a principled basis for adaptive smoothing\. Empirically, CIST mitigates the inconsistency induced by fixed temperature, and experiments on both vision and language distillation tasks show consistent improvements over standard KD and strong baselines with negligible computational overhead\.

## 1Introduction

Deep neural networks \(DNNs\) have achieved state\-of\-the\-art performance across a wide range of tasks, largely driven by the continual scaling of model capacity\. However, this improvement often comes with substantial computational and memory costs, making large models difficult to deploy in resource\-constrained environments such as mobile and edge devices\. To address this challenge, model compression has been extensively studied\. Among existing compression techniques, knowledge distillation \(KD\)\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56)\)has emerged as a simple and effective approach, where a compact student model learns from a high\-capacity teacher by mimicking its predictive behavior\(Romeroet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib83); Cho and Hariharan,[2019](https://arxiv.org/html/2605.20357#bib.bib4); Gouet al\.,[2021](https://arxiv.org/html/2605.20357#bib.bib84)\)\.

A central mechanism in KD is temperatureτ\\tau, which divides teacher and student logits by a temperature hyperparameter before the softmax operation\. By increasingτ\\tau, the teacher distribution becomes less peaked and exposes inter\-class relationships, commonly known as “dark knowledge”\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56); Tanget al\.,[2020](https://arxiv.org/html/2605.20357#bib.bib24); Zhaoet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib60); Wei and Bai,[2024](https://arxiv.org/html/2605.20357#bib.bib42)\)\. In standard KD, however,τ\\tauis selected as a single global constant\. This choice implicitly assumes that all teacher predictions require the same amount of smoothing, even though their logit scales can vary substantially across samples\. We argue that this assumption is problematic because the entropy of the softened teacher distribution is highly sensitive to the ratio between the dominant logit and the temperature\. When the dominant logit is large relative toτ\\tau, the softened label remains sharply concentrated and provides little non\-target information\. Conversely, when the logit gaps are small relative toτ\\tau, the distribution can become excessively flat, weakening class\-discriminative signals\. Figure[1](https://arxiv.org/html/2605.20357#S1.F1)illustrates this issue: under the same fixed temperature, teacher soft\-label entropy varies widely across samples\.

![Refer to caption](https://arxiv.org/html/2605.20357v1/x1.png)Figure 1:Teacher soft label generation in KD\. With fixed temperature, some samples remain overconfident and uninformative \(low entropy\), while others are oversmoothed \(high entropy\)\. These inappropriately smoothed samples can reduce knowledge transfer\.This entropy inconsistency reveals that samples do not contribute uniformly to distillation: some provide informative soft supervision, while others provide unreliable training signals\. Standard KD assigns the same loss weight to all samples, although teacher predictions differ in confidence and the student may learn from different samples at different rates\. Low\-confidence teacher predictions can provide unreliable soft targets, while hard samples for the current student may lead to noisy or unstable gradients\. This motivates a curriculum\-based distillation strategy, where training emphasizes samples that provide reliable and informative supervision\(Bengioet al\.,[2009](https://arxiv.org/html/2605.20357#bib.bib85); Liet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib23)\)\. Although prior work has explored curriculum temperatures\(Liet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib23)\), logit standardization before softmax\(Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\), and entropy\-based loss weighting\(Suet al\.,[2025](https://arxiv.org/html/2605.20357#bib.bib43)\), they do not directly address the entropy inconsistency of teacher soft labels under fixed temperature\. In addition, standard KD commonly shares the same temperature between teacher and student, which can implicitly encourage rigid logit\-scale matching despite their capacity mismatch\(Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\)\.

To address these limitations, we proposeCIST\(ConsistentlyInformativeSoft\-labelTemperature\), an effective distillation framework that combines consistent informative soft labels with confidence\-aware curriculum regularization\. It first assigns sample\-wise adaptive temperatures to the teacher based on the dominant teacher logits, stabilizing teacher\-label entropy and producing consistently informative soft supervision\. It then applies separate adaptive temperatures to the teacher and student, relaxing the rigid logit\-scale matching imposed by shared\-temperature KD\. Finally, CIST reweights the distillation loss according to teacher confidence and student learning difficulty, emphasizing reliable and learnable soft targets while down\-weighting uncertain or poorly learned samples\. Theoretically, we show that teacher\-label entropy is largely governed by the ratio between the maximum teacher logit and the temperature, providing a principled basis for our adaptive design\. We validate CIST on both vision and language distillation tasks\. On CIFAR\-100 and ImageNet, CIST consistently improves standard KD across diverse teacher–student architectures\. On instruction\-following language distillation, CIST also outperforms other distillation baselines across multiple teacher–student pairs and evaluation benchmarks\. Overall, CIST achieves strong performance compared with competitive distillation baselines while introducing negligible computational overhead\.

In summary, our contributions are threefold:

- •We show a key limitation of fixed\-temperature KD: a single global temperature induces inconsistent teacher\-label entropy across samples, leading to uninformative soft labels\.
- •We proposeCIST, a logit\-based KD framework motivated by an entropy\-based analysis of teacher soft labels\. By combining sample\-wise temperature, independent teacher–student temperature, and confidence\-aware curriculum regularization, CIST produces informative soft targets, relaxes rigid logit\-scale alignment, and emphasizes reliable distillation signals\.
- •We validate CIST across vision and language distillation tasks, where it consistently outperforms strong baselines on CIFAR\-100, ImageNet, and instruction\-following language distillation\.

## 2Related Works

Knowledge distillationaims to transfer the dark knowledge from a high\-capacity teacher model to a lightweight student model\. By learning from the soft labels produced by the teacher, the student often achieves better generalization than when trained solely on hard labels\. Traditionally, KD trains the student by minimizing the Kullback–Leibler \(KL\) divergence between the teacher’s and student’s predicted probability distributions\. These probabilities are obtained by applying the softmax function to the models’ respective logits\. KD methods can broadly be categorized into three groups: logit\-based approaches\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56); Zhaoet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib60); Jinet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib25); Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8); Zheng and Yang,[2024](https://arxiv.org/html/2605.20357#bib.bib18)\), which directly match the output distributions; feature\-based methods\(Romeroet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib83); Zagoruyko and Komodakis,[2017](https://arxiv.org/html/2605.20357#bib.bib39); Parket al\.,[2019](https://arxiv.org/html/2605.20357#bib.bib31); Tianet al\.,[2020](https://arxiv.org/html/2605.20357#bib.bib33); Heoet al\.,[2019](https://arxiv.org/html/2605.20357#bib.bib27); Chenet al\.,[2021](https://arxiv.org/html/2605.20357#bib.bib40)\), which align intermediate representations, and relation\-based approaches\(Tung and Mori,[2019](https://arxiv.org/html/2605.20357#bib.bib34); Huanget al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib1); Liet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib35)\), which transfer structural relationships among samples or features\. Our proposed method falls under logit\-based category, as it extends the standard KD\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56)\)by applying an adaptive temperature mechanism for both teacher and student\.

Temperature scaling in KD\. Temperatureτ\\tauis a central mechanism in logit\-based KD\. By dividing logits by a temperatureτ\\taubefore softmax, KD smooths the teacher distribution and reveals non\-target class information that is absent from hard labels\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56); Tanget al\.,[2020](https://arxiv.org/html/2605.20357#bib.bib24); Liuet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib37); Zhaoet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib60)\)\. Most existing methods, however, use a fixed temperature for all samples and often share the same temperature between teacher and student\. This design is convenient but sample\-agnostic: it assumes that a single smoothing strength is suitable for all teacher predictions, despite large variations in logit scale, and sample difficulty\.

Recent studies have begun to revisit the role of temperature in KD, but they address different aspects of the problem\. CTKD\(Liet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib23)\)learns a curriculum\-based temperature to control distillation difficulty, but still uses a shared teacher–student temperature\.Chandrasegaranet al\.\([2022](https://arxiv.org/html/2605.20357#bib.bib36)\)study how smoothing interacts with teacher quality and label smoothing, showing that inappropriate smoothing can weaken distillation\. Logit Standardization\(Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\)normalizes logits to reduce sensitivity to temperature choice, but retains a fixed shared base temperature for stability\.Suet al\.\([2025](https://arxiv.org/html/2605.20357#bib.bib43)\)use entropy for sample\-wise loss reweighting but still rely on the single global temperature of standard KD, whereas our work uses entropy to regulate the informativeness of teacher soft labels\. In contrast, we address a distinct limitation of fixed teacher temperature: the induced inconsistency in soft\-label entropy across samples, which leads to uneven and sometimes uninformative distillation supervision\.

## 3Limitations of Fixed and Shared Temperatures

Preliminaries\. Temperature is used in KD to soften teacher and student output distributions, revealing inter\-class relationships\. LetC∈ℕC\\in\\mathbb\{N\}be the number of classes andτ\>0\\tau\>0be the temperature\. For theii\-th sample with teacher logits𝐯i∈ℝC\\mathbf\{v\}\_\{i\}\\in\\mathbb\{R\}^\{C\}and student logits𝐳i∈ℝC\\mathbf\{z\}\_\{i\}\\in\\mathbb\{R\}^\{C\}, their softened distributions are

𝐩iτ=softmax​\(𝐯iτ\),𝐪iτ=softmax​\(𝐳iτ\)\.\\mathbf\{p\}\_\{i\}^\{\\tau\}=\\text\{softmax\}\\left\(\\frac\{\\mathbf\{v\}\_\{i\}\}\{\\tau\}\\right\),\\quad\\mathbf\{q\}\_\{i\}^\{\\tau\}=\\text\{softmax\}\\left\(\\frac\{\\mathbf\{z\}\_\{i\}\}\{\\tau\}\\right\)\.\(1\)The softmax function is defined for classjj\-th assoftmax​\(vi,j\)=exp⁡\(vi,j\)/∑c=1Cexp⁡\(vi,c\)\\text\{softmax\}\(v\_\{i,j\}\)=\\exp\(v\_\{i,j\}\)/\\sum^\{C\}\_\{c=1\}\\exp\(v\_\{i,c\}\)\. To quantify the smoothness and informativeness of the teacher’s soft output, we compute the entropy of the softened distribution for training sampleiias follows:

H​\(𝐩iτ\)=−∑c=1Cpi,cτ​log⁡pi,cτ\.H\\left\(\\mathbf\{p\}^\{\\tau\}\_\{i\}\\right\)=\-\\sum\_\{c=1\}^\{C\}p^\{\\tau\}\_\{i,c\}\\log p^\{\\tau\}\_\{i,c\}\.\(2\)An effective temperature in KD should produce output distributions with moderate entropy, which is soft enough to reveal inter\-class relationships but not so uncertain that the information becomes noisy\. Prior work\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56); Tanget al\.,[2020](https://arxiv.org/html/2605.20357#bib.bib24); Chandrasegaranet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib36)\)supports this view, showing that well\-tuned temperatures help the teacher reveal meaningful inter\-class relationships and improve student performance\. This highlights the critical role of temperature selection in KD\.

![Refer to caption](https://arxiv.org/html/2605.20357v1/x2.png)\(a\)CIFAR\-100\.
![Refer to caption](https://arxiv.org/html/2605.20357v1/x3.png)\(b\)ImageNet\.
![Refer to caption](https://arxiv.org/html/2605.20357v1/x4.png)\(c\)Dolly\.

Figure 2:Entropy distribution of teacher soft labels under standard KD and CIST across architectures and datasets\. Soft labels from fixed\-temperature KD show high entropy variance and many outliers, indicating ineffective smoothing\. In contrast, CIST achieves consistent smoothing\.Entropy inconsistency under fixed temperature\. In real\-world scenarios, the magnitudes of teacher logits vary substantially due to factors such as sample complexity, teacher confidence, and model inductive biases\. As a result, applying a single fixed temperature can produce softened teacher distributions with highly inconsistent entropy\.

We illustrate this phenomenon in Figure[2](https://arxiv.org/html/2605.20357#S3.F2), which shows the entropy distribution of teacher soft labels from CIFAR\-100\(Krizhevskyet al\.,[2009](https://arxiv.org/html/2605.20357#bib.bib9)\), ImageNet\(Russakovskyet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib10)\), and Dolly\(Conoveret al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib73)\), using various teacher architectures from vision and language models\. The results show substantial variation in entropy across samples: some have low entropy \(near or even below 2\.0 for CIFAR\-100\), while others reach up to 4\.5 for CIFAR\-100 and 6\.8 for ImageNet, which are close to the entropy of a uniform distribution over 100 classes \(log⁡\(100\)≈4\.6\\log\(100\)\\approx 4\.6\) and 1000 classes \(log⁡\(1000\)≈6\.9\\log\(1000\)\\approx 6\.9\), respectively\. This inconsistency suggests that some soft labels lack meaningful inter\-class information, while others are excessively uncertain and resemble random noise\. Such variation highlights the limitation of fixed\-temperature KD in providing consistently informative supervision, which impairs the effectiveness of distillation\.

Table 1:Test accuracy \(%\) on CIFAR\-100\. Results are averaged over three trials\.Impact of entropy outliers\. To investigate the effect of low\-entropy outliers on student performance, we apply different treatments to these samples and examine whether they benefit distillation\. Specifically, we define low\-entropy outliers as the bottom 5% of training samples according to the entropy of the teacher soft targets\. These samples correspond to highly confident teacher predictions that provide limited inter\-class information\. We introduce two KD variants:

- •KD\-EntOut\-CE: for entropy outliers, we replace the distillation loss with standard cross\-entropy, while keeping distillation unchanged for all other samples\.
- •KD\-EntOut\-HT: we retain distillation for all samples, but use a higher temperature for entropy outliers, increasingτ\\taufrom44to55to further soften their predictions\.

As shown in Table[1](https://arxiv.org/html/2605.20357#S3.T1), both KD\-EntOut\-CE and KD\-EntOut\-HT consistently improve over standard KD\. For example, KD\-EntOut\-CE improves VGG13→\\rightarrowVGG8 from72\.98%72\.98\\%to73\.75%73\.75\\%and RN32x4→\\rightarrowRN8x4 from73\.33%73\.33\\%to74\.20%74\.20\\%, while KD\-EntOut\-HT achieves the best result on WRN40\-2→\\rightarrowWRN16\-2\. These results suggest that inappropriate smoothing of highly confident teacher predictions can degrade student performance\. Moreover, these results indicate that entropy outliers benefit from sample\-specific treatment, which can improve student performance\. This motivates an effective temperature\-scaling strategy for producing consistently informative soft targets\.

Shared temperature enforces rigid logit matching\. Following the prior analysis\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56)\), we show that using a shared temperature between the teacher and student leads to a rigid logit matching behavior\. The distillation loss is computed as the Kullback–Leibler divergence between the softened output probabilities of the student and teacher, denoted byqcq\_\{c\}andpcp\_\{c\}, respectively\. These are obtained by applying softmax with temperaturesτs\\tau^\{s\}\(student\) andτt\\tau^\{t\}\(teacher\) to their logits\. The gradient of the KD loss with respect to the student logitzcz\_\{c\}is:

∂LKD∂zc=1τs​\(qc−pc\)=1τs​\(exp⁡\(zc/τs\)∑jexp⁡\(zj/τs\)−exp⁡\(vc/τt\)∑jexp⁡\(vc/τt\)\)\.\\displaystyle\\frac\{\\partial L\_\{\\text\{KD\}\}\}\{\\partial z\_\{c\}\}=\\frac\{1\}\{\\tau^\{s\}\}\(q\_\{c\}\-p\_\{c\}\)=\\frac\{1\}\{\\tau^\{s\}\}\\left\(\\frac\{\\exp\(z\_\{c\}/\\tau^\{s\}\)\}\{\\sum\_\{j\}\\exp\(z\_\{j\}/\\tau^\{s\}\)\}\-\\frac\{\\exp\(v\_\{c\}/\\tau^\{t\}\)\}\{\\sum\_\{j\}\\exp\(v\_\{c\}/\\tau^\{t\}\)\}\\right\)\.\(3\)In the high\-temperature regime \(i\.e\., when the logits are small relative to the temperature\), we apply a first\-order Taylor expansion to the exponential and assume zero\-mean logits per sample\(∑jzj=∑jvj=0\)\(\\sum\_\{j\}z\_\{j\}=\\sum\_\{j\}v\_\{j\}=0\)\. Under these assumptions, the gradient of the distillation loss simplifies to:

∂LKD∂zc≈1τs​\(1\+zc/τsC\+∑jzj/τs−1\+vc/τtC\+∑jvj/τt\)=1C​τs​τt​\(zc​τtτs−vc\)\.\\displaystyle\\frac\{\\partial L\_\{\\text\{KD\}\}\}\{\\partial z\_\{c\}\}\\approx\\frac\{1\}\{\\tau^\{s\}\}\\left\(\\frac\{1\+z\_\{c\}/\\tau^\{s\}\}\{C\+\\sum\_\{j\}z\_\{j\}/\\tau^\{s\}\}\-\\frac\{1\+v\_\{c\}/\\tau^\{t\}\}\{C\+\\sum\_\{j\}v\_\{j\}/\\tau^\{t\}\}\\right\)=\\frac\{1\}\{C\\tau^\{s\}\\tau^\{t\}\}\\left\(z\_\{c\}\\frac\{\\tau^\{t\}\}\{\\tau^\{s\}\}\-v\_\{c\}\\right\)\.\(4\)When the teacher and student share the same temperature,τt=τs=τ\\tau^\{t\}=\\tau^\{s\}=\\tau, this reduces to1/\(C​τ2\)∗\(zc−vc\)1/\(C\\tau^\{2\}\)\*\(z\_\{c\}\-v\_\{c\}\)which drives the student toward exact teacher\-logit matching\. This rigid constraint can be suboptimal, particularly when the teacher and student differ in representational capacity or logit scale\(Liet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib35); Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\)\.

## 4CIST \(Consistently Informative Soft\-label Temperature\)

We propose CIST, which consists of three components: \(i\) sample\-wise temperature scaling to stabilize soft\-label entropy, \(ii\) independent teacher–student temperatures to relax rigid logit\-scale matching, and \(iii\) confidence\-aware loss rescaling to induce a curriculum over training samples\. The pseudo\-code for CIST is presented in Algorithm[1](https://arxiv.org/html/2605.20357#alg1)\.

###### Proposition 1

Let𝐯i,𝐯j∈ℝC\\mathbf\{v\}\_\{i\},\\mathbf\{v\}\_\{j\}\\in\\mathbb\{R\}^\{C\}be the logits for two samples and letm=arg⁡maxc⁡vi,cm=\\arg\\max\_\{c\}v\_\{i,c\}andm′=arg⁡maxc⁡vj,cm^\{\\prime\}=\\arg\\max\_\{c\}v\_\{j,c\}\. Their softened softmax outputs, computed with temperaturesτi\\tau\_\{i\}andτj\\tau\_\{j\}, are denoted by𝐩iτi\\mathbf\{p\}\_\{i\}^\{\\tau\_\{i\}\}and𝐩jτj\\mathbf\{p\}\_\{j\}^\{\\tau\_\{j\}\}, respectively\. If both distributions are dominated by their maximum logit, the entropy difference\|H​\(𝐩iτi\)−H​\(𝐩jτj\)\|\\left\|H\\left\(\\mathbf\{p\}\_\{i\}^\{\\tau\_\{i\}\}\\right\)\-H\\left\(\\mathbf\{p\}\_\{j\}^\{\\tau\_\{j\}\}\\right\)\\right\|is minimized when the ratio of the dominant logit to temperature is equal across samples, i\.e\.,vi,mτi=vj,m′τj=ρ\\frac\{v\_\{i,m\}\}\{\\tau\_\{i\}\}=\\frac\{v\_\{j,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}=\\rhofor some constantρ\>0\\rho\>0\.

The proof of Proposition[1](https://arxiv.org/html/2605.20357#ThmTheorem1)is provided in Appendix[B](https://arxiv.org/html/2605.20357#A2), which shows that, for a strong well\-trained teacher, teacher\-label entropy is mainly controlled by the ratio between the dominant teacher logit and the temperature\. Thus, a fixed global temperature cannot ensure consistent entropy: high\-confidence samples require stronger smoothing, while lower\-confidence samples require weaker smoothing\. This motivates sample\-wise temperature adaptation to match the sample\-specific logit scale\.

Sample\-wise adaptive temperature\. Following Proposition[1](https://arxiv.org/html/2605.20357#ThmTheorem1), CIST chooses the teacher temperature by normalizing the dominant logit to a constantρ\\rho:

τit=vi,mρ\.\\tau^\{t\}\_\{i\}=\\frac\{v\_\{i,m\}\}\{\\rho\}\.\(5\)This formulation assigns the larger temperatures to highly confident teacher predictions, preventing them from remaining overly sharp, while assigning smaller temperatures to less confident predictions, preventing excessive smoothing\. As a result, CIST reduces entropy variation across samples and produces teacher soft labels with more consistent informativeness\.

To empirically evaluate entropy stabilization, we compare the entropy distributions of teacher soft labels produced by fixed\-temperature KD and CIST across different teacher architectures and datasets\. As shown in Figure[2](https://arxiv.org/html/2605.20357#S3.F2), fixed\-temperature KD exhibits high entropy variance, with many samples producing either overly sharp or overly smooth targets\. This pattern is consistent across both vision and language datasets\. Note that the entropy ranges differ across datasets due to the different numbers of classes\. In contrast, CIST substantially reduces entropy variance across all three datasets and produces more consistent soft\-label entropy, with values concentrated around5\.755\.75–6\.56\.5on ImageNet\.

Independent teacher and student temperature\. Standard KD typically uses the same temperature for the teacher and student, implicitly forcing them to align under an exact logit scale despite their capacity mismatch\. CIST addresses this issue by assigning separate sample\-wise temperatures to the teacher and student\. Letvi,m=maxc⁡vi,cv\_\{i,m\}=\\max\_\{c\}v\_\{i,c\}denote the dominant teacher logit, and letzi,m′=maxc⁡\|zi,c\|z\_\{i,m^\{\\prime\}\}=\\max\_\{c\}\|z\_\{i,c\}\|denote the maximum student logit magnitude for sampleii\. We define the teacher and student temperatures asτit=vi,mρ,τis=zi,m′ρ,\\tau\_\{i\}^\{t\}=\\frac\{v\_\{i,m\}\}\{\\rho\},\\ \\tau\_\{i\}^\{s\}=\\frac\{z\_\{i,m^\{\\prime\}\}\}\{\\rho\},respectively\. Using separate temperatures allows the teacher and student predictions to be smoothed according to their respective logit scales\. Substituting these temperatures into the high\-temperature approximation in Eq\. \([4](https://arxiv.org/html/2605.20357#S3.E4)\) gives

∂LKD∂zc≈1C​τis​τit​\(zi,c​vi,mzi,m′−vi,c\)\.\\frac\{\\partial L\_\{\\mathrm\{KD\}\}\}\{\\partial z\_\{c\}\}\\approx\\frac\{1\}\{C\\tau\_\{i\}^\{s\}\\tau\_\{i\}^\{t\}\}\\left\(z\_\{i,c\}\\frac\{v\_\{i,m\}\}\{z\_\{i,m^\{\\prime\}\}\}\-v\_\{i,c\}\\right\)\.\(6\)CIST introduces the relative scale factorvi,m/zi,m′v\_\{i,m\}/z\_\{i,m^\{\\prime\}\}, allowing the student to match the teacher’s logit structure without being forced to reproduce the teacher’s absolute logit magnitude\. This relaxes rigid logit\-scale matching and encourages the student to preserve inter\-class relations\.

Confidence\-aware curriculum regularization\. Adaptive temperatures change both the shape of the softened distributions and the scale of the KD gradients\. As shown in Eq\. \([6](https://arxiv.org/html/2605.20357#S4.E6)\), the gradient magnitude is inversely proportional toτis​τit\\tau\_\{i\}^\{s\}\\tau\_\{i\}^\{t\}\. Therefore, to maintain a stable balance between the KL distillation loss and the hard\-label cross\-entropy loss, we follow the standard temperature\-scaled KD principle\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56)\)and rescale the KL term for each sampleii:

ℒKL=τit​τis⋅KL​\(𝐩iτit∥𝐪iτis\),\\mathcal\{L\}\_\{\\text\{KL\}\}=\\tau\_\{i\}^\{t\}\\tau\_\{i\}^\{s\}\\cdot\\mathrm\{KL\}\\left\(\\mathbf\{p\}\_\{i\}^\{\\tau\_\{i\}^\{t\}\}\\parallel\\mathbf\{q\}\_\{i\}^\{\\tau\_\{i\}^\{s\}\}\\right\),\(7\)where𝐩iτt\\mathbf\{p\}\_\{i\}^\{\\tau^\{t\}\}and𝐪iτs\\mathbf\{q\}\_\{i\}^\{\\tau^\{s\}\}are the temperature\-scaled teacher and student output probabilities\. Importantly, this rescaling factor also acts as a confidence\-aware curriculum weight\. Sinceτit\\tau\_\{i\}^\{t\}andτis\\tau\_\{i\}^\{s\}are determined by the teacher and student logit magnitudes, their product assigns larger weights to samples where the teacher provides confident soft targets and the student produces confident responses\. Conversely, samples with low teacher confidence or weak student responses receive smaller weights, reducing the influence of unreliable or difficult supervision\. Thus, CIST induces a sample\-wise curriculum: it prioritizes distillation signals that are both reliable from the teacher and learnable for the student\.

## 5Experiments

### 5\.1Implementation details

Vision task\. We evaluate on CIFAR\-100\(Krizhevskyet al\.,[2009](https://arxiv.org/html/2605.20357#bib.bib9)\)and ImageNet\(Russakovskyet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib10)\)\. Following standard KD protocols\(Tianet al\.,[2020](https://arxiv.org/html/2605.20357#bib.bib33); Zhaoet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib60)\), we consider diverse teacher–student architectures, including VGG\(Simonyan and Zisserman,[2014](https://arxiv.org/html/2605.20357#bib.bib16)\), ResNet \(RN\)\(Heet al\.,[2016](https://arxiv.org/html/2605.20357#bib.bib14)\), WideResNet \(WRN\)\(Zagoruyko and Komodakis,[2016](https://arxiv.org/html/2605.20357#bib.bib15)\), MobileNet \(MN\)\(Howard,[2017](https://arxiv.org/html/2605.20357#bib.bib11); Sandleret al\.,[2018](https://arxiv.org/html/2605.20357#bib.bib12)\), and ShuffleNet \(SHN\)\(Zhanget al\.,[2018](https://arxiv.org/html/2605.20357#bib.bib13)\)\. We compare against logit\-based KD methods, including KD\(Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56)\), DKD\(Zhaoet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib60)\), CTKD\(Liet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib23)\), MLKD\(Jinet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib25)\), LS\(Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\), and EA\(Suet al\.,[2025](https://arxiv.org/html/2605.20357#bib.bib43)\), as well as feature\-based methods including FitNet\(Romeroet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib83)\), RKD\(Parket al\.,[2019](https://arxiv.org/html/2605.20357#bib.bib31)\), CRD\(Penget al\.,[2019](https://arxiv.org/html/2605.20357#bib.bib32)\), OFD\(Heoet al\.,[2019](https://arxiv.org/html/2605.20357#bib.bib27)\), and ReviewKD\(Chenet al\.,[2021](https://arxiv.org/html/2605.20357#bib.bib40)\)\.

Language task\. We use the instruction–response data fromdatabricks\-dolly\-15k\(Conoveret al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib73)\)for training\. The dataset contains 14k training samples, 500 validation samples, and 500 test samples\. We evaluate distillation from GPT\-2 XL\(Radfordet al\.,[2019](https://arxiv.org/html/2605.20357#bib.bib61)\)and OPT\-6\.7B\(Zhanget al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib66)\)teachers to GPT\-2 Base and OPT\-1\.3B students\. Evaluation is conducted on four benchmarks consisting of Dolly Eval\(Conoveret al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib73)\), Vicuna Eval\(Chianget al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib75)\), Super\-Natural Instructions\(Wanget al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib76)\), and Unnatural Instructions\(Honovichet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib77)\)\. We compare CIST with common KL\-based baselines in language model distillation, including FKL, RKL\(Guet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib55)\), Sym\-KL, JS\(Agarwalet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib54)\), SRKL\(Koet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib64)\), AKL\(Wuet al\.,[2025](https://arxiv.org/html/2605.20357#bib.bib48)\), andα\\alpha\-β\\betadivergence \(AB\)\(Wanget al\.,[2025](https://arxiv.org/html/2605.20357#bib.bib65)\)\.

Additional implementation details for both tasks, including training resources and hyperparameter settings, are provided in the Appendix[A](https://arxiv.org/html/2605.20357#A1)\.

### 5\.2Results on Vision Task

Table 2:Top\-1 test accuracy \(%\) on CIFAR\-100\. Distillation methods are grouped by category, and student\-teacher pairs are grouped by architecture type\. The best and second\-best results are marked inboldandunderlined, respectively\.ΔKD\\Delta\_\{\\text\{KD\}\}andΔMLKD\\Delta\_\{\\text\{MLKD\}\}report the improvement of KD\+Ours over KD and MLKD\+Ours over MLKD, respectively\. The results are averaged over three trials\.CIFAR\-100\. Table[2](https://arxiv.org/html/2605.20357#S5.T2)reports Top\-1 accuracy on CIFAR\-100 across homogeneous \(same architecture family\) and heterogeneous \(different architecture family\) teacher–student pairs\. Overall, MLKD\+Ours consistently achieves the best performance, surpassing the second\-best method by0\.29%0\.29\\%–1\.63%1\.63\\%\. These gains are significant in KD, where improvements above0\.15%0\.15\\%are considered non\-trivial in recent work\(Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\)and our smallest gain is nearly twice this threshold\.

In the*homogeneous setting*, KD\+Ours substantially improves standard KD, with gains of\+3\.63%\+3\.63\\%on RN32x4→\\rightarrowRN8x4 and\+1\.67%\+1\.67\\%on WRN40\-2→\\rightarrowWRN16\-2\. It also outperforms strong logit\-based baselines such as DKD, while remaining competitive with feature\-based methods such as ReviewKD\. When combined with MLKD, our method further improves performance by0\.30%0\.30\\%–1\.63%1\.63\\%over MLKD and achieves the best results across all homogeneous pairs, including a gain exceeding5%5\\%over standard KD on RN32x4→\\rightarrowRN8x4\.

In the*heterogeneous setting*, where architectural mismatch makes distillation more challenging, KD\+Ours outperforms standard KD and other logit\-based baselines in most cases\. Although feature\-based methods such as OFD and ReviewKD benefit from explicit intermediate\-representation alignment, MLKD\+Ours consistently achieves the best overall performance\. It surpasses the second\-best method by up to1\.18%1\.18\\%on WRN40\-2→\\rightarrowResNet8x4 and outperforms feature\-based baselines by margins ranging from0\.5%0\.5\\%to over3%3\\%\.

Table 3:Top\-1 and top\-5 accuracy on the ImageNet validation set\. The best and second best results are inboldandunderlined\.ImageNet\. Table[3](https://arxiv.org/html/2605.20357#S5.T3)compares CIST \(KD\+Ours\) with representative feature\-based and logit\-based distillation methods on ImageNet in terms of Top\-1 and Top\-5 accuracy\. For a fair efficiency comparison, we exclude MLKD since it incurs substantially higher training cost than standard KD, whereas CIST preserves the training cost of logit\-based KD\. Overall, KD\+Ours achieves the best performance among all compared methods across both logit\- and feature\-based categories\. In particular, KD\+Ours consistently surpasses the strongest logit\-based baselines, including DKD and KD\+LS, by clear margins: \+0\.27% on ResNet34\-ResNet18 and \+0\.57% on ResNet50\-MobileNetV1\. Compared to standard KD, CIST yields up to \+2\.25% Top\-1 improvement on ResNet50\-MobileNetV1, demonstrating its effectiveness under challenging teacher–student capacity gaps\. Moreover, KD\+Ours also outperforms strong feature\-based methods such as OFD and ReviewKD, which require additional trainable modules and introduce non\-trivial training overhead\.

![Refer to caption](https://arxiv.org/html/2605.20357v1/x5.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.20357v1/x6.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.20357v1/x7.png)\(c\)

Figure 3:\(a\) Effect ofρ\\rhoon the entropy distribution of teacher soft targets from ResNet32x4\. \(b\) The impact of variousλKL\\lambda\_\{\\text\{KL\}\}andρ\\rhovalues on student test accuracy \(%\) in CIST\. \(c\) Training time per batch and test accuracy \(%\) of different distillation methods on the ResNet32x4\-ResNet8x4 pair\.
### 5\.3Results on Language Task

Table 4:ROUGE\-L for GPT\-2 and OPT teacher–student pairs, with mean and standard deviation computed over five random seeds\. The best results areboldfacedand the second\-best areunderlined\.StudentMethod / TeacherDollyVicunaSuper\-NIUnNIAvg\.\(↑\\uparrow\)GPT\-20\.1B\\cellcolorteachergray Teacher: GPT\-2 1\.5B\\cellcolorteachergray27\.00±\\pm0\.19\\cellcolorteachergray16\.31±\\pm0\.32\\cellcolorteachergray26\.46±\\pm0\.41\\cellcolorteachergray31\.10±\\pm0\.06\\cellcolorteachergray25\.22FKL23\.80±\\pm0\.5514\.53±\\pm0\.3116\.27±\\pm0\.2419\.03±\\pm0\.0918\.41RKL24\.67±\\pm0\.1315\.82±\\pm0\.4321\.03±\\pm0\.2123\.94±\\pm0\.1221\.37Sym\-KL24\.38±\\pm0\.1415\.72±\\pm0\.7519\.92±\\pm0\.2522\.21±\\pm0\.0320\.56JS24\.08±\\pm0\.2615\.47±\\pm0\.3719\.66±\\pm0\.2322\.17±\\pm0\.2120\.34SRKL24\.48±\\pm0\.3714\.96±\\pm0\.3423\.25±\\pm0\.3624\.01±\\pm0\.0521\.68AKL21\.83±\\pm0\.1913\.70±\\pm0\.2115\.40±\\pm0\.2018\.06±\\pm0\.1317\.25AB24\.32±\\pm0\.2915\.82±\\pm0\.5823\.08±\\pm0\.1924\.32±\\pm0\.1421\.88\\cellcoloroursblueCIST \(Ours\)\\cellcoloroursblue25\.42±\\pm0\.10\\cellcoloroursblue16\.10±\\pm0\.07\\cellcoloroursblue21\.81±\\pm0\.23\\cellcoloroursblue25\.94±\\pm0\.13\\cellcoloroursblue22\.32OPT1\.3B\\cellcolorteachergray Teacher: OPT 6\.7B\\cellcolorteachergray27\.52±\\pm0\.29\\cellcolorteachergray17\.64±\\pm0\.27\\cellcolorteachergray30\.41±\\pm0\.46\\cellcolorteachergray31\.39±\\pm0\.20\\cellcolorteachergray26\.74FKL26\.07±\\pm0\.6516\.71±\\pm0\.3322\.11±\\pm0\.3826\.82±\\pm0\.1022\.93RKL26\.58±\\pm0\.1117\.54±\\pm0\.1322\.64±\\pm0\.1626\.29±\\pm0\.1023\.26Sym\-KL25\.73±\\pm0\.4016\.95±\\pm0\.3723\.40±\\pm0\.2225\.44±\\pm0\.0722\.88JS26\.39±\\pm0\.7117\.15±\\pm0\.3924\.02±\\pm0\.3126\.59±\\pm0\.1123\.54SRKL26\.04±\\pm0\.3617\.34±\\pm0\.2522\.96±\\pm0\.2224\.71±\\pm0\.2222\.76AKL26\.20±\\pm0\.2717\.26±\\pm0\.6622\.12±\\pm0\.1825\.89±\\pm0\.2022\.87AB26\.86±\\pm0\.1717\.59±\\pm0\.4124\.39±\\pm0\.1126\.62±\\pm0\.0923\.87\\cellcoloroursblueCIST \(Ours\)\\cellcoloroursblue27\.74±\\pm0\.26\\cellcoloroursblue17\.69±\\pm0\.23\\cellcoloroursblue25\.19±\\pm0\.25\\cellcoloroursblue28\.41±\\pm0\.12\\cellcoloroursblue24\.76

GPT\-2\. Table[4](https://arxiv.org/html/2605.20357#S5.T4)shows that CIST consistently delivers strong performance across instruction\-following benchmarks, ranking first on three out of four tasks and achieving the best average score\. On UnNI, CIST yields the largest gain, outperforming RKL by2\.002\.00and the second\-best method by1\.621\.62\. On Dolly and Vicuna, CIST further improves over the second\-best method by0\.750\.75and0\.280\.28, respectively\. Although CIST ranks third on Super\-NI, slightly behind AB and SRKL, its overall performance highlights its robustness across diverse evaluation tasks\.

OPT\. For larger teacher–student model pairs, CIST consistently outperforms state\-of\-the\-art distillation objectives, ranking first on all four benchmarks and achieving the best average score\. On Dolly, CIST exceeds the second\-best method, AB, by0\.880\.88points and the strong RKL baseline by1\.161\.16points\. On Vicuna, although the improvement over the second\-best method is smaller \(0\.100\.10points\), CIST obtains a lower standard deviation, indicating more stable gains\. The largest improvement is observed on UnNI, where CIST outperforms AB by1\.791\.79points\.

### 5\.4Ablation Study

Impact ofρ\\rhoon entropy and student performance\. As shown in Figure[3\(a\)](https://arxiv.org/html/2605.20357#S5.F3.sf1), increasingρ\\rholeads to lower adaptive temperaturesτi\\tau\_\{i\}, producing sharper teacher outputs with reduced entropy\. In contrast, smallerρ\\rhovalues yield largerτi\\tau\_\{i\}and higher\-entropy soft targets\. For example, the average entropy decreases from 4\.5 \(nearly uniform distribution\) to 2\.6 asρ\\rhoincreases from 2\.0 to 5\.0\. These findings confirm that the hyperparameterρ\\rhoprovides effective control over the entropy of the teacher’s soft labels and maintains consistent entropy behavior across diverse samples, even whenρ\\rhovaries widely\.

To evaluate the sensitivity of CIST to the hyperparameterρ\\rho, we conduct experiments on the ResNet32x4→\\rightarrowResNet8x4 pair withρ∈\{3\.0,4\.0,5\.0\}\\rho\\in\\\{3\.0,4\.0,5\.0\\\}and KL loss weightλKL∈\{1,2,4,6,8,10,12\}\\lambda\_\{\\mathrm\{KL\}\}\\in\\\{1,2,4,6,8,10,12\\\}\. As shown in Figure[3\(b\)](https://arxiv.org/html/2605.20357#S5.F3.sf2), CIST consistently outperforms vanilla KD across all configurations, demonstrating its robustness to the choice ofρ\\rho\. Performance remains stable forλKL\\lambda\_\{\\mathrm\{KL\}\}in the range of44to1010, while overly large distillation weights can degrade accuracy\. The best result is achieved withρ=3\.0\\rho=3\.0andλKL=8\\lambda\_\{\\mathrm\{KL\}\}=8, reaching76\.96%76\.96\\%accuracy\.

t\-SNE visualization\. We present t\-SNE visualizations using ResNet32x4 as the teacher and ResNet8x4 as the student on CIFAR\-100\. The results demonstrate that the representations learned with CIST are more class\-separable compared to those from standard KD, indicating that CIST enhances the discriminability of learned features\.

![Refer to caption](https://arxiv.org/html/2605.20357v1/figure/tsne/best_ckpt_kd.jpeg)\(a\)KD
![Refer to caption](https://arxiv.org/html/2605.20357v1/figure/tsne/resnet32x4_resnet8x4_ckpt_best.jpeg)\(b\)CIST
![Refer to caption](https://arxiv.org/html/2605.20357v1/x8.png)\(c\)Grad\-CAM

Figure 4:\(a\) t\-SNE visualization of KD\. \(b\) t\-SNE visualization of CIST\. \(c\) Grad\-CAM visualizations for the teacher, student without KD, student with KD, and student with CIST on high\- and low\-entropy samples\. The number on the left indicates the entropy value\.Training efficiency\. We evaluate training efficiency by comparing the performance–cost trade\-offs of different KD methods\. As shown in Figure[3\(c\)](https://arxiv.org/html/2605.20357#S5.F3.sf3), CIST achieves the best balance between accuracy and training time\. While feature\-based methods such as ReviewKD, CRD, and OFD incur additional overhead from auxiliary modules, CIST preserves the simplicity and efficiency of vanilla KD\. All training times are measured on an NVIDIA RTX 3090 GPU\.

Distillation with ViT models and larger teachers\. We further evaluate CIST on Vision Transformer \(ViT\) backbones and larger teachers, increasing architectural diversity and teacher–student capacity gaps \(see Appendix[C](https://arxiv.org/html/2605.20357#A3)\)\. CIST consistently improves vanilla KD by a clear margin, showing that its effectiveness extends beyond CNN\-based models and remains strong with larger teachers\.

Grad\-CAM\. We select two high\-entropy and two low\-entropy samples based on the soft labels produced by a pretrained ResNet34 teacher, drawn from class 0 in the ImageNet dataset\. Grad\-CAM attribution maps are visualized for the teacher, a student trained without KD, a student trained with standard KD, and a student trained with CIST\. Warm colors \(e\.g\., red and yellow\) indicate discriminative regions, while cool colors \(e\.g\., blue and purple\) highlight less informative areas\. As shown in Figure[4\(c\)](https://arxiv.org/html/2605.20357#S5.F4.sf3), two key observations emerge: \(1\) the student trained with standard KD produces attribution maps that resemble those of the non\-KD student rather than the teacher, suggesting that fixed\-temperature KD fails to transfer spatially meaningful supervision; and \(2\) the student trained with CIST better focuses on the target object \(e\.g\., the fish\-tench\), demonstrating that CIST provides more informative soft labels and enables more effective knowledge transfer\.

Table 5:Importance of CIST components\.Effect of CIST components\. Table[5](https://arxiv.org/html/2605.20357#S5.T5)reports a component\-wise ablation of CIST\. Confidence\-aware reweighting alone improves standard KD to74\.41%74\.41\\%, showing that sample\-wise weighting provides useful curriculum regularization\. CIST contributes a larger gain: removing the reweighting term while keeping adaptive temperatures reaches76\.49%76\.49\\%, improving KD by\+3\.16%\+3\.16\\%\. This confirms that stabilizing teacher\-label entropy is the primary driver of CIST\. The full CIST objective achieves the best accuracy\.

## 6Conclusion

In this work, we analyze temperature scaling in knowledge distillation and show that the standard fixed\-temperature design leads to inconsistent teacher\-label entropy and rigid teacher–student logit\-scale alignment\. We propose CIST, a sample\-wise adaptive temperature\-scaling framework that assigns separate temperatures to the teacher and student and reweights distillation according to teacher confidence and student learning difficulty\. Guided by our entropy analysis, CIST produces consistently informative soft labels, relaxes restrictive logit\-scale matching, and emphasizes reliable samples\. Experiments on both vision and language tasks show that CIST consistently outperforms standard KD and strong distillation baselines while adding negligible computational overhead\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.InThe twelfth international conference on learning representations,Cited by:[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston \(2009\)Curriculum learning\.InProceedings of the 26th Annual International Conference on Machine Learning,ICML ’09,New York, NY, USA,pp\. 41–48\.External Links:ISBN 9781605585161,[Link](https://doi.org/10.1145/1553374.1553380),[Document](https://dx.doi.org/10.1145/1553374.1553380)Cited by:[§1](https://arxiv.org/html/2605.20357#S1.p3.1)\.
- A\. Chan, H\. Silva, S\. Lim, T\. Kozuno, A\. R\. Mahmood, and M\. White \(2022\)Greedification operators for policy optimization: investigating forward and reverse kl divergences\.Journal of Machine Learning Research23\(253\),pp\. 1–79\.Cited by:[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p3.5)\.
- K\. Chandrasegaran, N\. Tran, Y\. Zhao, and N\. Cheung \(2022\)Revisiting label smoothing and knowledge distillation compatibility: what was missing?\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p3.1),[§3](https://arxiv.org/html/2605.20357#S3.p1.9)\.
- P\. Chen, S\. Liu, H\. Zhao, and J\. Jia \(2021\)Distilling knowledge via knowledge review\.InCVPR,Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez, I\. Stoica, and E\. P\. Xing \(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality\.External Links:[Link](https://lmsys.org/blog/2023-03-30-vicuna/)Cited by:[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- J\. H\. Cho and B\. Hariharan \(2019\)On the efficacy of knowledge distillation\.InProceedings of the IEEE/CVF international conference on computer vision,Cited by:[Appendix C](https://arxiv.org/html/2605.20357#A3.p1.1),[§1](https://arxiv.org/html/2605.20357#S1.p1.1)\.
- M\. Conover, M\. Hayes, A\. Mathur, J\. Xie, J\. Wan, S\. Shah, A\. Ghodsi, P\. Wendell, M\. Zaharia, and R\. Xin \(2023\)Cited by:[§3](https://arxiv.org/html/2605.20357#S3.p3.2),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- J\. Gou, B\. Yu, S\. J\. Maybank, and D\. Tao \(2021\)Knowledge distillation: a survey\.International journal of computer vision129\(6\),pp\. 1789–1819\.Cited by:[§1](https://arxiv.org/html/2605.20357#S1.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)Minillm: knowledge distillation of large language models\.InInternational Conference on Learning Representations,Cited by:[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p2.6),[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p3.5),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- B\. Heo, J\. Kim, S\. Yun, H\. Park, N\. Kwak, and J\. Y\. Choi \(2019\)A comprehensive overhaul of feature distillation\.InProceedings of the IEEE/CVF international conference on computer vision,Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- B\. Heo, S\. Yun, D\. Han, S\. Chun, J\. Choe, and S\. J\. Oh \(2021\)Rethinking spatial dimensions of vision transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),Cited by:[Appendix C](https://arxiv.org/html/2605.20357#A3.p2.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[Appendix A](https://arxiv.org/html/2605.20357#A1.p1.12),[§1](https://arxiv.org/html/2605.20357#S1.p1.1),[§1](https://arxiv.org/html/2605.20357#S1.p2.5),[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§2](https://arxiv.org/html/2605.20357#S2.p2.2),[§3](https://arxiv.org/html/2605.20357#S3.p1.9),[§3](https://arxiv.org/html/2605.20357#S3.p5.5),[§4](https://arxiv.org/html/2605.20357#S4.p6.2),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- O\. Honovich, T\. Scialom, O\. Levy, and T\. Schick \(2023\)Unnatural instructions: tuning language models with \(almost\) no human labor\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 14409–14428\.Cited by:[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- A\. G\. Howard \(2017\)Mobilenets: efficient convolutional neural networks for mobile vision applications\.arXiv preprint arXiv:1704\.04861\.Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- T\. Huang, S\. You, F\. Wang, C\. Qian, and C\. Xu \(2022\)Knowledge distillation from a stronger teacher\.Advances in Neural Information Processing Systems\.Cited by:[Appendix C](https://arxiv.org/html/2605.20357#A3.p1.1),[§2](https://arxiv.org/html/2605.20357#S2.p1.1)\.
- Y\. Jin, J\. Wang, and D\. Lin \(2023\)Multi\-level logit distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p1.1),[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p6.1),[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- G\. Kim, D\. Jang, and E\. Yang \(2024\)Promptkd: distilling student\-friendly knowledge for generative language models via prompt tuning\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 6266–6282\.Cited by:[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p3.5)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: towards streamlined distillation for large language models\.InForty\-first International Conference on Machine Learning,Cited by:[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p3.5),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- A\. Krizhevsky, G\. Hinton,et al\.\(2009\)Learning multiple layers of features from tiny images\.Cited by:[§3](https://arxiv.org/html/2605.20357#S3.p3.2),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- G\. Li, X\. Li, Y\. Wang, S\. Zhang, Y\. Wu, and D\. Liang \(2022\)Knowledge distillation for object detection via rank mimicking and prediction\-guided feature imitation\.InProceedings of the AAAI conference on artificial intelligence,Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§3](https://arxiv.org/html/2605.20357#S3.p5.8)\.
- Z\. Li, X\. Li, L\. Yang, B\. Zhao, R\. Song, L\. Luo, J\. Li, and J\. Yang \(2023\)Curriculum temperature for knowledge distillation\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p7.1),[§1](https://arxiv.org/html/2605.20357#S1.p3.1),[§2](https://arxiv.org/html/2605.20357#S2.p3.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- J\. Liu, B\. Liu, H\. Li, and Y\. Liu \(2022\)Meta knowledge distillation\.arXiv preprint arXiv:2202\.07940\.Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p2.2)\.
- H\. Luong, D\. B\. Tran, and L\. Chen \(2026\)Diversity\-aware reverse kullback\-leibler divergence for large language model distillation\.arXiv preprint arXiv:2604\.00223\.Cited by:[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p3.5)\.
- W\. Park, D\. Kim, Y\. Lu, and M\. Cho \(2019\)Relational knowledge distillation\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- B\. Peng, X\. Jin, J\. Liu, D\. Li, Y\. Wu, Y\. Liu, S\. Zhou, and Z\. Zhang \(2019\)Correlation congruence for knowledge distillation\.InProceedings of the IEEE/CVF international conference on computer vision,Cited by:[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- A\. Romero, N\. Ballas, S\. E\. Kahou, A\. Chassang, C\. Gatta, and Y\. Bengio \(2015\)FitNets: hints for thin deep nets\.InIn Proceedings of ICLR,Cited by:[§1](https://arxiv.org/html/2605.20357#S1.p1.1),[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- O\. Russakovsky, J\. Deng, H\. Su, J\. Krause, S\. Satheesh, S\. Ma, Z\. Huang, A\. Karpathy, A\. Khosla, M\. Bernstein,et al\.\(2015\)Imagenet large scale visual recognition challenge\.International journal of computer vision\.Cited by:[§3](https://arxiv.org/html/2605.20357#S3.p3.2),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- M\. Sandler, A\. Howard, M\. Zhu, A\. Zhmoginov, and L\. Chen \(2018\)Mobilenetv2: inverted residuals and linear bottlenecks\.InProceedings of the IEEE conference on computer vision and pattern recognition,Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- K\. Simonyan and A\. Zisserman \(2014\)Very deep convolutional networks for large\-scale image recognition\.arXiv preprint arXiv:1409\.1556\.Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- C\. Su, C\. Tseng, B\. Pu, L\. Zhao, J\. Yang, Z\. Chen, and S\. Lee \(2025\)EA\-kd: entropy\-based adaptive knowledge distillation\.InProceedings of the IEEE/CVF International Conference on Computer Vision,Cited by:[§1](https://arxiv.org/html/2605.20357#S1.p3.1),[§2](https://arxiv.org/html/2605.20357#S2.p3.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- S\. Sun, W\. Ren, J\. Li, R\. Wang, and X\. Cao \(2024\)Logit standardization in knowledge distillation\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p1.1),[Appendix C](https://arxiv.org/html/2605.20357#A3.p2.1),[§1](https://arxiv.org/html/2605.20357#S1.p3.1),[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§2](https://arxiv.org/html/2605.20357#S2.p3.1),[§3](https://arxiv.org/html/2605.20357#S3.p5.8),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2605.20357#S5.SS2.p1.3)\.
- J\. Tang, R\. Shivanna, Z\. Zhao, D\. Lin, A\. Singh, E\. H\. Chi, and S\. Jain \(2020\)Understanding and improving knowledge distillation\.arXiv preprint arXiv:2002\.03532\.Cited by:[§1](https://arxiv.org/html/2605.20357#S1.p2.5),[§2](https://arxiv.org/html/2605.20357#S2.p2.2),[§3](https://arxiv.org/html/2605.20357#S3.p1.9)\.
- Y\. Tian, D\. Krishnan, and P\. Isola \(2020\)Contrastive representation distillation\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- H\. Touvron, M\. Cord, M\. Douze, F\. Massa, A\. Sablayrolles, and H\. Jegou \(2021\)Training data\-efficient image transformers and distillation through attention\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research\.Cited by:[Appendix C](https://arxiv.org/html/2605.20357#A3.p2.1)\.
- F\. Tung and G\. Mori \(2019\)Similarity\-preserving knowledge distillation\.InProceedings of the IEEE/CVF international conference on computer vision,Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1)\.
- G\. Wang, Z\. Yang, Z\. Wang, S\. Wang, Q\. Xu, and Q\. Huang \(2025\)ABKD: pursuing a proper allocation of the probability mass in knowledge distillation via $\\alpha$\-$\\beta$\-divergence\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 65167–65212\.Cited by:[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p3.5),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap,et al\.\(2022\)Super\-naturalinstructions: generalization via declarative instructions on 1600\+ nlp tasks\.InProceedings of the 2022 conference on empirical methods in natural language processing,pp\. 5085–5109\.Cited by:[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- Y\. Wei and Y\. Bai \(2024\)Dynamic temperature knowledge distillation\.arXiv preprint arXiv:2404\.12711\.Cited by:[§1](https://arxiv.org/html/2605.20357#S1.p2.5)\.
- T\. Wu, C\. Tao, J\. Wang, R\. Yang, Z\. Zhao, and N\. Wong \(2025\)Rethinking kullback\-leibler divergence in knowledge distillation for large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 5737–5755\.Cited by:[§A\.2](https://arxiv.org/html/2605.20357#A1.SS2.p3.5),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- L\. Yuan, Y\. Chen, T\. Wang, W\. Yu, Y\. Shi, Z\. Jiang, F\. E\.H\. Tay, J\. Feng, and S\. Yan \(2021\)Tokens\-to\-token vit: training vision transformers from scratch on imagenet\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),Cited by:[Appendix C](https://arxiv.org/html/2605.20357#A3.p2.1)\.
- S\. Zagoruyko and N\. Komodakis \(2016\)Wide residual networks\.arXiv preprint arXiv:1605\.07146\.Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- S\. Zagoruyko and N\. Komodakis \(2017\)Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer\.ICLR\.Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1)\.
- S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.\(2022\)Opt: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p2.2)\.
- X\. Zhang, X\. Zhou, M\. Lin, and J\. Sun \(2018\)Shufflenet: an extremely efficient convolutional neural network for mobile devices\.InProceedings of the IEEE conference on computer vision and pattern recognition,Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- B\. Zhao, Q\. Cui, R\. Song, Y\. Qiu, and J\. Liang \(2022\)Decoupled knowledge distillation\.InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition,pp\. 11953–11962\.Cited by:[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p1.1),[§A\.1](https://arxiv.org/html/2605.20357#A1.SS1.p3.10),[§1](https://arxiv.org/html/2605.20357#S1.p2.5),[§2](https://arxiv.org/html/2605.20357#S2.p1.1),[§2](https://arxiv.org/html/2605.20357#S2.p2.2),[§5\.1](https://arxiv.org/html/2605.20357#S5.SS1.p1.1)\.
- K\. Zheng and E\. Yang \(2024\)Knowledge distillation based on transformed teacher matching\.InThe Twelfth International Conference on Learning Representations \(ICLR 2024\),External Links:[Link](https://openreview.net/pdf?id=MJ3K7uDGGl)Cited by:[§2](https://arxiv.org/html/2605.20357#S2.p1.1)\.

## Appendix AImplementation Details

Implementation\. CIST is computationally efficient and easy to implement\. It requires computing the maximum logitvi,maxv\_\{i,\\max\}per sample and dividing it byρ\\rhoto obtain the temperatureτi\\tau\_\{i\}\. To preserve the shift\-invariance property of softmax, each logit vector is centered by subtracting its mean: for a logit vector𝐱\\mathbf\{x\}, we use𝐱−𝐱¯\\mathbf\{x\}\-\\bar\{\\mathbf\{x\}\}, where𝐱¯\\bar\{\\mathbf\{x\}\}is the mean of𝐱\\mathbf\{x\}\. Following the temperature\-scaling principle ofHintonet al\.\[[2015](https://arxiv.org/html/2605.20357#bib.bib56)\], whereτ\>1\\tau\>1smooths the output distribution andτ=1\\tau=1recovers the original prediction without additional smoothing, we define the adaptive temperature asτi=max⁡\(vi,maxρ,1\)\.\\tau\_\{i\}=\\max\(\\frac\{v\_\{i,\\max\}\}\{\\rho\},1\)\.For reproducibility, we provide the pseudo\-code in Algorithm[1](https://arxiv.org/html/2605.20357#alg1)\. Note thatλCE\\lambda\_\{\\mathrm\{CE\}\}andλKL\\lambda\_\{\\mathrm\{KL\}\}in Algorithm[1](https://arxiv.org/html/2605.20357#alg1)are standard hyperparameters in traditional KD\[Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56)\], rather than additional hyperparameters introduced by our method\.

How to setρ\\rho\. The hyperparameterρ\\rhocontrols the target entropy of the softened predictions\. To selectρ\\rho, we estimate the practical entropy range for a given dataset and teacher model using a small calibration subset, e\.g\., 512 training samples\. The maximum entropy islog⁡K\\log K, whereKKis the number of classes; for ImageNet\-1k,log⁡\(1000\)≈6\.9\\log\(1000\)\\approx 6\.9, corresponding to a uniform distribution\. The lower end can be approximated by the entropy of the original teacher predictions before temperature scaling\. We then sweep candidate values ofρ\\rhoand choose an intermediate region, as shown in Figure[3\(a\)](https://arxiv.org/html/2605.20357#S5.F3.sf1)\. Smallρ\\rhovalues induce larger temperatures, pushing softened distributions toward high\-entropy, nearly uniform predictions\. Largeρ\\rhovalues yield temperatures close to11, producing sharp predictions with limited dark knowledge\. This process is performed once before training, so its computational overhead is negligible\.

CIST hyperparameter setting\. CIST replaces the fixed temperature hyperparameter with an entropy\-control parameterρ\\rho, which determines the target level of soft\-label smoothing\. We tuneρ\\rhoand the KD loss weightλKL\\lambda\_\{\\mathrm\{KL\}\}by grid search, as shown in Figure[3\(b\)](https://arxiv.org/html/2605.20357#S5.F3.sf2)\. Based on this analysis, we setρ=3\\rho=3andλKL=8\\lambda\_\{\\mathrm\{KL\}\}=8as the default configuration, which provides strong and stable performance across settings\.

Training resources\. All vision experiments were conducted on an Ubuntu Linux machine equipped with an NVIDIA RTX 3090 GPU with 24GB memory\. Language experiments were conducted on a GPU cluster with four NVIDIA A100 GPUs, each with 40GB memory\.

### A\.1Vision Task

We adopt standard training protocols widely used in prior distillation studies\[Zhaoet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib60), Jinet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib25), Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\]\. We use SGD with momentum and employ the original loss weights of each baseline method\. Reported results are averaged over three independent runs\.

Evaluation results\. We report Top\-1 test accuracy \(%\) for CIFAR\-100 and Top\-1 test accuracy \(%\) and Top\-5 test accuracy \(%\) for ImageNet, computed as the classification accuracy on the test set\.

CIFAR\-100\. All CIFAR\-100 experiments are trained for240240epochs with batch size6464, except MLKD\[Jinet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib25)\], which follows its original480480\-epoch schedule\. We use SGD with momentum 0\.9 and weight decay5×10−45\\times 10^\{\-4\}\. The initial learning rate is set to 0\.01 for lightweight students \(MobileNet\[Howard,[2017](https://arxiv.org/html/2605.20357#bib.bib11), Sandleret al\.,[2018](https://arxiv.org/html/2605.20357#bib.bib12)\], ShuffleNet\[Zhanget al\.,[2018](https://arxiv.org/html/2605.20357#bib.bib13)\]\) and 0\.05 for other architectures \(ResNet\[Heet al\.,[2016](https://arxiv.org/html/2605.20357#bib.bib14)\], WRN\[Zagoruyko and Komodakis,[2016](https://arxiv.org/html/2605.20357#bib.bib15)\], VGG\[Simonyan and Zisserman,[2014](https://arxiv.org/html/2605.20357#bib.bib16)\]\)\. The learning rate is decayed by a factor of0\.10\.1at epochs150150,180180, and210210\. The cross\-entropy \(CE\) loss weight is kept identical to the original baselines \(0\.10\.1for KD\[Hintonet al\.,[2015](https://arxiv.org/html/2605.20357#bib.bib56)\]and CTKD\[Liet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib23)\];1\.01\.0for DKD\[Zhaoet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib60)\]and MLKD\[Jinet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib25)\]\)\.

ImageNet\. For ImageNet, we train for 100 epochs using SGD with batch size512512, momentum0\.90\.9, and weight decay10−410^\{\-4\}\. The initial learning rate is0\.20\.2and is divided by1010at epochs3030,6060, and9090\. We keep the baseline CE loss weight unchanged \(0\.10\.1for KD and CTKD;0\.50\.5for the remaining baselines\)\. For CIST, we do not have the temperature hyperparameter and instead use entropy controlρ=3\\rho=3with KD loss weightλ=8\\lambda=8by default\.

Setting for Table 1\. For fair comparison, KD\-EntOut\-CE and KD\-EntOut\-HT use the same hyperparameters as standard KD such as CE loss weight0\.10\.1and KL loss weight0\.90\.9\.

Combining CIST with other distillation objectives\. We evaluate CIST on top of MLKD\[Jinet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib25)\]\. MLKD performs logit alignment at multiple granularities \(instance, batch and class level\) via temperature\-scaled softmax at each level\. We therefore integrate CIST by replacing the original temperature scaling in MLKD with our entropy\-guided, sample\-wise temperature scaling CIST\.

We do not apply CIST to CTKD\[Liet al\.,[2023](https://arxiv.org/html/2605.20357#bib.bib23)\]as CTKD optimizes a learnable global temperature parameter, while CIST eliminates shared temperature scaling and assigns entropy\-guided sample\-wise temperatures\. Hence, the two formulations are not directly compatible without modifying CTKD\.

### A\.2Language Task

Evaluation results\. We focus on the off\-policy distillation setting, where training is performed using fixed teacher\-generated responses\. Therefore, we do not compare with methods that require on\-policy sampling or additional external datasets\. All methods are trained under the same setup, using the same dataset, identical hyperparameters, a shared teacher checkpoint, and the same student initialization to ensure a fair comparison\. For evaluation, checkpoints are saved after each epoch, and we report results from the checkpoint with the best validation ROUGE\-L\. ROUGE\-L is averaged over five random seeds\{10,20,30,40,50\}\\\{10,20,30,40,50\\\}, and the decoding temperature is set to11by default\.

Hyperparameter setting\. FollowingGuet al\.\[[2024](https://arxiv.org/html/2605.20357#bib.bib55)\], we use AdamW with a weight decay of0\.010\.01\. The learning rate is set to5×10−45\\times 10^\{\-4\}for the GPT\-2 0\.1B student and5×10−55\\times 10^\{\-5\}for OPT\-1\.3B\. We use a batch size of3232and train each student model for2020epochs\. The maximum input length is fixed to512512tokens for all model families\.

Motivated by prior studies showing that RKL often outperforms FKL in language model distillation\[Chanet al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib50), Guet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib55), Kimet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib53), Wuet al\.,[2025](https://arxiv.org/html/2605.20357#bib.bib48), Luonget al\.,[2026](https://arxiv.org/html/2605.20357#bib.bib86)\], we apply CIST to RKL in our language experiments\. For baseline\-specific hyperparameters, we follow the hyperparameter settings from the original works and use the implementation provided in DistilLM\[Koet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib64)\]: Sym\-KL which is0\.5​FKL\+0\.5​RKL0\.5\\,\\mathrm\{FKL\}\+0\.5\\,\\mathrm\{RKL\}, SRKL use smoothing coefficientα=0\.1\\alpha=0\.1\[Koet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib64)\], whileα\\alpha\-β\\betadivergence uses\(α,β\)=\(0\.2,0\.7\)\(\\alpha,\\beta\)=\(0\.2,0\.7\)\[Wanget al\.,[2025](https://arxiv.org/html/2605.20357#bib.bib65)\]\.

Algorithm 1Consistently Informative Soft\-label Temperature \(CIST\)1:Train set

𝒟=\{\(𝐱n,yn\)\}n=1N\\mathcal\{D\}=\\\{\(\\mathbf\{x\}\_\{n\},y\_\{n\}\)\\\}\_\{n=1\}^\{N\}, teacher model

fTf\_\{T\}, student model

fSf\_\{S\}, number of epochs

EE, entropy control constant

ρ\\rho, CE loss weight

λCE\\lambda\_\{\\rm CE\}, KL loss weight

λKL\\lambda\_\{\\rm KL\}\.

2:Trained student model

3:for

epoch=1,2,…,E\\text\{epoch\}=1,2,\\dots,Edo

4:for all

\(𝐱n,yn\)\(\\mathbf\{x\}\_\{n\},y\_\{n\}\)in

𝒟\\mathcal\{D\}do

5:

𝐯n=fT​\(𝐱n\),𝐳n=fS​\(𝐱n\)\\mathbf\{v\}\_\{n\}=f\_\{T\}\(\\mathbf\{x\}\_\{n\}\),\\ \\ \\mathbf\{z\}\_\{n\}=f\_\{S\}\(\\mathbf\{x\}\_\{n\}\)
6:

𝐯^n=𝐯n−𝐯¯n,𝐳^n=𝐳n−𝐳¯n\\hat\{\\mathbf\{v\}\}\_\{n\}=\\mathbf\{v\}\_\{n\}\-\\bar\{\\mathbf\{v\}\}\_\{n\},\\ \\ \\hat\{\\mathbf\{z\}\}\_\{n\}=\\mathbf\{z\}\_\{n\}\-\\bar\{\\mathbf\{z\}\}\_\{n\}
7:

vmax=max⁡\(𝐯^n\),zmax=max⁡\(𝐳^n\),v\_\{\\max\}=\\max\(\\hat\{\\mathbf\{v\}\}\_\{n\}\),\\ \\ z\_\{\\max\}=\\max\(\\hat\{\\mathbf\{z\}\}\_\{n\}\),
8:

τnt←max⁡\(vmax/ρ,1\),τns←max⁡\(zmax/ρ,1\)\\tau^\{t\}\_\{n\}\\leftarrow\\max\(v\_\{\\max\}/\\rho,1\),\\ \\ \\tau^\{s\}\_\{n\}\\leftarrow\\max\(z\_\{\\max\}/\\rho,1\)
9:

𝐩n=softmax​\(𝐯^n/τnt\),𝐪n=softmax​\(𝐳^n/τns\)\\mathbf\{p\}\_\{n\}=\\mathrm\{softmax\}\(\\hat\{\\mathbf\{v\}\}\_\{n\}/\\tau^\{t\}\_\{n\}\),\\ \\ \\mathbf\{q\}\_\{n\}=\\mathrm\{softmax\}\(\\hat\{\\mathbf\{z\}\}\_\{n\}/\\tau^\{s\}\_\{n\}\)
10:Update student by minimizing

λCE​ℒCE​\(yn,𝐪n\)\+λKL​τnt​τns​ℒKL​\(𝐩n∥𝐪n\)\\lambda\_\{\\text\{CE\}\}\\mathcal\{L\}\_\{\\text\{CE\}\}\(y\_\{n\},\\mathbf\{q\}\_\{n\}\)\+\\lambda\_\{\\text\{KL\}\}\\tau^\{t\}\_\{n\}\\tau^\{s\}\_\{n\}\\mathcal\{L\}\_\{\\rm KL\}\(\\mathbf\{p\}\_\{n\}\\\|\\mathbf\{q\}\_\{n\}\)
11:endfor

12:endfor

## Appendix BProof of Proposition[1](https://arxiv.org/html/2605.20357#ThmTheorem1)

Proof\. Substituting𝐩iτi\\mathbf\{p\}^\{\\tau\_\{i\}\}\_\{i\}as the softmax output for a given sampleiiwith temperatureτi\\tau\_\{i\}into entropy Eq\. \([2](https://arxiv.org/html/2605.20357#S3.E2)\) and using the property that∑c=1Cpi,cτi=1\\sum^\{C\}\_\{c=1\}p\_\{i,c\}^\{\\tau\_\{i\}\}=1, we can rewrite it as follows

H​\(𝐩iτi\)\\displaystyle H\\left\(\\mathbf\{p\}^\{\\tau\_\{i\}\}\_\{i\}\\right\)=log​∑c=1Cexp⁡\(vi,cτi\)−∑c=1Cpi,cτi​vi,cτi\.\\displaystyle=\\log\\sum^\{C\}\_\{c=1\}\\exp\\left\(\\frac\{v\_\{i,c\}\}\{\\tau\_\{i\}\}\\right\)\-\\sum\_\{c=1\}^\{C\}p^\{\\tau\_\{i\}\}\_\{i,c\}\\frac\{v\_\{i,c\}\}\{\\tau\_\{i\}\}\.\(8\)Considering two samplesiiandjj, the magnitude of entropy difference between their soft labels can be expressed as:

\|H​\(𝐩iτi\)−H​\(𝐩jτj\)\|\\displaystyle\\left\|H\(\\mathbf\{p\}^\{\\tau\_\{i\}\}\_\{i\}\)\-H\(\\mathbf\{p\}^\{\\tau\_\{j\}\}\_\{j\}\)\\right\|=\|log⁡∑c=1Cexp⁡\(vi,cτi\)∑c=1Cexp⁡\(vj,cτj\)\+∑c=1C\(pj,cτi​vj,cτj−pi,cτj​vi,cτi\)\|\.\\displaystyle=\\left\|\\log\\frac\{\\small\{\\sum^\{C\}\_\{c=1\}\}\\exp\(\\frac\{v\_\{i,c\}\}\{\\tau\_\{i\}\}\)\}\{\\small\{\\sum^\{C\}\_\{c=1\}\}\\exp\(\\frac\{v\_\{j,c\}\}\{\\tau\_\{j\}\}\)\}\+\\small\{\\sum\_\{c=1\}^\{C\}\}\\big\(p^\{\\tau\_\{i\}\}\_\{j,c\}\\frac\{v\_\{j,c\}\}\{\\tau\_\{j\}\}\-p^\{\\tau\_\{j\}\}\_\{i,c\}\\frac\{v\_\{i,c\}\}\{\\tau\_\{i\}\}\\big\)\\right\|\.\(9\)Letm=arg⁡maxc⁡zi,cm=\\arg\\max\_\{c\}z\_\{i,c\}andm′=arg⁡maxc⁡zj,cm^\{\\prime\}=\\arg\\max\_\{c\}z\_\{j,c\}be the indices of the maximum logits for samplesiiandjj, respectively\. The first term, denoted asAA, simplifies to:

A\\displaystyle A=log⁡exp⁡\(vi,mτi\)​\(1\+∑c≠mCexp⁡\(vi,c−vi,mτi\)\)exp⁡\(vj,m′τj\)​\(1\+∑c≠m′Cexp⁡\(vj,c−vj,m′τj\)\)\\displaystyle=\\log\\frac\{\\exp\(\\frac\{v\_\{i,m\}\}\{\\tau\_\{i\}\}\)\\left\(1\+\\sum^\{C\}\_\{c\\not=m\}\\exp\(\\frac\{v\_\{i,c\}\-v\_\{i,m\}\}\{\\tau\_\{i\}\}\)\\right\)\}\{\\exp\(\\frac\{v\_\{j,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}\)\\left\(1\+\\sum^\{C\}\_\{c\\not=m^\{\\prime\}\}\\exp\(\\frac\{v\_\{j,c\}\-v\_\{j,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}\)\\right\)\}=log⁡exp⁡\(vi,mτi\)exp⁡\(vj,m′τj\)\+log⁡1\+∑c≠mCexp⁡\(vi,c−vi,mτi\)1\+∑c≠m′Cexp⁡\(vj,c−vj,m′τj\)\.\\displaystyle=\\log\\frac\{\\exp\(\\frac\{v\_\{i,m\}\}\{\\tau\_\{i\}\}\)\}\{\\exp\(\\frac\{v\_\{j,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}\)\}\+\\log\\frac\{1\+\\sum^\{C\}\_\{c\\not=m\}\\exp\(\\frac\{v\_\{i,c\}\-v\_\{i,m\}\}\{\\tau\_\{i\}\}\)\}\{1\+\\sum^\{C\}\_\{c\\not=m^\{\\prime\}\}\\exp\(\\frac\{v\_\{j,c\}\-v\_\{j,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}\)\}\.Since the teacher is well\-trained, the maximum logit dominates for both samplesiiandjj, i\.e\.,vi,m≫vi,c,∀c≠mv\_\{i,m\}\\gg v\_\{i,c\},\\forall c\\not=m, or equivalently,vi,c−vi,mv\_\{i,c\}\-v\_\{i,m\}becomes very negative\. The same applies to samplejj\. Therefore, the second term above is negligible, as the exponential of a large negative value approaches zero\. Then, termAAis approximated by the difference between the two dominant logits:

A≈log⁡exp⁡\(vi,mτi\)exp⁡\(vj,m′τj\)=vi,mτi−vj,m′τj\.\\displaystyle A\\approx\\log\\frac\{\\exp\(\\frac\{v\_\{i,m\}\}\{\\tau\_\{i\}\}\)\}\{\\exp\(\\frac\{v\_\{j,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}\)\}=\\frac\{v\_\{i,m\}\}\{\\tau\_\{i\}\}\-\\frac\{v\_\{j,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}\.\(10\)
We now analyze the second term in Eq\. \([9](https://arxiv.org/html/2605.20357#A2.E9)\), denoted asBB\. Letπi,πj\\pi\_\{i\},\\pi\_\{j\}be permutations that sort the logits of samplesiiandjjin decreasing order, such that index11corresponds to the maximum logit and indexCCto the minimum\. Then:

B\\displaystyle B=∑c=1C\(pj,πj​\(c\)τ​vj,πj​\(c\)τj−pi,πi​\(c\)τ​vi,πi​\(c\)τi\)\.\\displaystyle=\\sum\_\{c=1\}^\{C\}\\left\(p^\{\\tau\}\_\{j,\\pi\_\{j\}\(c\)\}\\frac\{v\_\{j,\\pi\_\{j\}\(c\)\}\}\{\\tau\_\{j\}\}\-p^\{\\tau\}\_\{i,\\pi\_\{i\}\(c\)\}\\frac\{v\_\{i,\\pi\_\{i\}\(c\)\}\}\{\\tau\_\{i\}\}\\right\)\.\(11\)The value ofBBis primarily influenced by the difference between the maximum logits,vi,πi​\(1\)=vi,mv\_\{i,\\pi\_\{i\}\(1\)\}=v\_\{i,m\}andvj,πj​\(1\)=vj,m′v\_\{j,\\pi\_\{j\}\(1\)\}=v\_\{j,m^\{\\prime\}\}, as well as their corresponding softened probabilities,pi,mτp^\{\\tau\}\_\{i,m\}andpj,m′τp^\{\\tau\}\_\{j,m^\{\\prime\}\}, which tend to be highly peaked in well\-trained models\. To reduce both termsAAandBB, we propose to adaptively adjust the temperaturesτi\\tau\_\{i\}andτj\\tau\_\{j\}so that the ratio between each sample’s maximum logit and its temperature is normalized to a predefined constantρ\\rho:

vi,mτi=vi,m′τj=ρ\.\\displaystyle\\frac\{v\_\{i,m\}\}\{\\tau\_\{i\}\}=\\frac\{v\_\{i,m^\{\\prime\}\}\}\{\\tau\_\{j\}\}=\\rho\.\(12\)Under this scaling, the termA≈0A\\approx 0as the dominant logits are aligned\. The value ofBBis also reduced, since its leading difference components become aligned across the two samples\. The remaining variation arises from differences in the smaller logits, which are not significant\. As a result, the overall entropy gap\|H​\(𝐩iτi\)−H​\(𝐩jτj\)\|\|H\(\\mathbf\{p\}^\{\\tau\_\{i\}\}\_\{i\}\)\-H\(\\mathbf\{p\}^\{\\tau\_\{j\}\}\_\{j\}\)\|is minimized, which completes the proof\.

## Appendix CMore Experiments

Table 6:Top\-1 test accuracy \(%\) of student WRN\-16\-2 distilled from various teachers on CIFAR\-100\. Best results are inbold\.Distillation with large teacher\. Prior works\[Cho and Hariharan,[2019](https://arxiv.org/html/2605.20357#bib.bib4), Huanget al\.,[2022](https://arxiv.org/html/2605.20357#bib.bib1)\]show that larger teachers do not always lead to better distillation performance due to capacity gaps with lightweight students\. As discussed in our analysis, CIST addresses this issue by relaxing strict logit matching and promoting relative scale alignment\. As shown in Table[6](https://arxiv.org/html/2605.20357#A3.T6), CIST significantly improves student performance over traditional KD and consistently achieves the best or second\-best accuracy across all six teacher models\.

Table 7:Top\-1 accuracy \(%\) of KD methods on CIFAR\-100\. The teacher is ResNet56\.ViT experiments\. To assess the generality of CIST beyond CNN backbones, we evaluate it on Transformer\-based architectures via knowledge distillation on four representative Vision Transformer \(ViT\) variants: DeiT\-Ti\[Touvronet al\.,[2021](https://arxiv.org/html/2605.20357#bib.bib44)\], T2T\-ViT\-7\[Yuanet al\.,[2021](https://arxiv.org/html/2605.20357#bib.bib45)\], and PiT\-Ti\[Heoet al\.,[2021](https://arxiv.org/html/2605.20357#bib.bib46)\]\. We follow the experimental protocol of prior work\[Sunet al\.,[2024](https://arxiv.org/html/2605.20357#bib.bib8)\]and conduct all experiments on CIFAR\-100, using the same training pipeline and hyperparameter settings to ensure fair comparison\. As reported in Table[7](https://arxiv.org/html/2605.20357#A3.T7), CIST \(KD\+Ours\) consistently improves over vanilla KD across all ViT backbones, indicating that CIST is effective on Transformer architectures and not limited to CNN\-based distillation\.

## Appendix DLimitations and Discussions

CIST is designed for logit\-based distillation, where KD matches softened output distributions\. Thus, it naturally applies to classification tasks and can be extended to classification heads in detection or segmentation\. However, applying CIST to regression heads or other non\-logit outputs is non\-trivial, since temperature scaling and soft\-label entropy are not directly defined in these settings\.

CIST also depends on teacher\-logit quality\. When the teacher is weak, poorly calibrated, or unreliable on many samples, its logits may provide poor confidence signals, limiting the benefit of adaptive temperature scaling\. This is a potential failure case, although it falls outside the standard KD assumption that the teacher is stronger than the student\.

Due to computational constraints, our language experiments use teacher models up to 7B parameters, and our vision experiments focus on CIFAR\-100 and ImageNet\. Evaluating CIST on larger models and broader vision\-language tasks remains future work\. Although on\-policy sampling, stronger student initialization, auxiliary supervision, and task\-specific enhancements may further improve performance, we exclude them to ensure a direct and fair comparison between distillation objectives under the same controlled setting\.