Temperature-Adaptive Transformed Teacher Matching

# Temperature-AdaptiveTransformed Teacher Matching
Source: [https://arxiv.org/html/2608.29099](https://arxiv.org/html/2608.29099)
Temperature\-Adaptive TTM

## Temperature\-Adaptive Transformed Teacher Matching

Yoshikazu Hayashi\\orcidlink0009\-0000\-6251\-3950Hiroaki AizawaYoshikazu Hayashi

###### Abstract

Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood\. Transformed Teacher Matching \(TTM\) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit Rényi entropy regularization on the student\. However, TTM still relies on a fixed temperature and does not specify how the teacher\-side temperature should be adapted for individual samples\. In this paper, we introduce a sample\-wise inverse\-temperature update for TTM by locally minimizing the Kullback\-Leibler divergence between the temperature\-scaled teacher distribution and the student’s prediction\. We derive closed\-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting\. This yields an efficient curvature\-aware update that requires one softmax evaluation and a constant number of class\-wise weighted sums\. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature\-adaptive distillation baselines\.

††email:hiroaki\-aizawa@hiroshima\-u\.ac\.jp††email:hayashi@cv\.info\.gifu\-u\.ac\.jp††affiliation:Graduate School of Advanced Science and Engineering
Hiroshima University, Japan
††affiliation:Intelligent Production Technology Research & Development Center for Aerospace
Gifu University, Japan
## 1Introduction

Knowledge distillation \(KD\) trains a student model by transferring knowledge from a teacher model, improving accuracy without increasing inference cost\[[12](https://arxiv.org/html/2608.29099#bib.bib1)\]\. This paradigm is particularly valuable when training resources or high\-quality labeled data are limited, since it can deliver strong performance with a smaller model budget\. In practice, however, the effectiveness of KD depends on several design choices that can be sensitive to the teacher, student, and dataset, and these choices often require careful tuning\.

![Refer to caption](https://arxiv.org/html/2608.29099v1/overview.png)Figure 1:Overview of the proposed Temperature\-Adaptive Transformed Teacher Matching\. Given an input sample, the teacher produces a predictive distribution, which is calibrated by a power transform with an adaptive inverse temperatureγ\\gamma\. The inverse temperatureγ\\gammais updated to obtain sample\-adaptive soft targets for distillation\. The student is trained using cross\-entropy and a distillation loss that matches the student prediction to the adapted teacher distribution\.A core component of standard KD is temperature scaling, which smooths predictive distributions to expose class\-to\-class relations, often referred to as dark knowledge\. In the common formulation\[[12](https://arxiv.org/html/2608.29099#bib.bib1)\], temperature scaling is applied to both teacher and student outputs, and the student is trained to match the resulting softened distributions\. This smoothing assigns meaningful probability mass and gradients to non\-maximum classes, making it easier for the student to learn the teacher’s relative class structure\. Despite its empirical success, the role of temperature scaling in KD has been less clear, and the design choice of whether to apply temperature to the teacher, the student, or both has historically been ambiguous\.

Transformed Teacher Matching \(TTM\)\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\]resolves this ambiguity in the distillation objective by using temperature scaling only on the teacher distribution and matching the transformed teacher distribution to the student’s temperature\-free prediction\. Beyond this objective\-level clarification, TTM provides a principled interpretation of temperature in distillation\. By reformulating the teacher\-side temperature transformation via power transforms, it shows that the resulting objective combines distribution matching with an implicit Rényi entropy regularization on the student\.

Temperature scaling is also widely used beyond distillation, including calibration\[[10](https://arxiv.org/html/2608.29099#bib.bib18)\], out\-of\-distribution detection\[[18](https://arxiv.org/html/2608.29099#bib.bib19)\], semi\- and self\-supervised learning\[[6](https://arxiv.org/html/2608.29099#bib.bib20),[5](https://arxiv.org/html/2608.29099#bib.bib21),[26](https://arxiv.org/html/2608.29099#bib.bib22),[4](https://arxiv.org/html/2608.29099#bib.bib23)\], and learning with label noise\[[16](https://arxiv.org/html/2608.29099#bib.bib24)\]\. In these settings, the temperature can substantially affect optimization behavior and final performance\. In practice, however, selecting an effective temperature often depends on the particular teacher, student, and dataset, and it is commonly chosen through empirical tuning\. While TTM clarifies the objective by applying temperature scaling only on the teacher side, it still relies on a fixed temperature and does not specify how the teacher\-side temperature should vary across samples\.

In this paper, we address this limitation by introducing a per\-sample temperature adaptation method for TTM\. Our approach optimizes the teacher\-side inverse temperature for each sample by locally minimizing the Kullback\-Leibler divergence between the transformed teacher distribution and the student’s prediction \(Fig\.[1](https://arxiv.org/html/2608.29099#S1.F1)\)\. We derive closed\-form first and second derivatives with respect to the inverse temperature, and show that they can be written as compact variance\-covariance expressions of centered teacher and student logits under the transformed teacher weighting\. These derivatives lead to an efficient curvature\-aware update that adds negligible overhead and can be directly integrated into TTM and WTTM\. As a result, the proposed method preserves the TTM framework while adapting the teacher target, and consequently the implicit Rényi entropy regularization, at the sample level\. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature\-adaptive distillation baselines\.

## 2Related Work

Knowledge distillation \(KD\) transfers knowledge from a teacher model to a student model by matching their predictive distributions, improving accuracy without increasing inference cost\[[12](https://arxiv.org/html/2608.29099#bib.bib1)\]\. Prior work has explored diverse transfer signals, including feature\-based distillation\[[23](https://arxiv.org/html/2608.29099#bib.bib4),[34](https://arxiv.org/html/2608.29099#bib.bib5),[1](https://arxiv.org/html/2608.29099#bib.bib6)\], relation and similarity matching methods\[[21](https://arxiv.org/html/2608.29099#bib.bib9),[20](https://arxiv.org/html/2608.29099#bib.bib8),[29](https://arxiv.org/html/2608.29099#bib.bib7)\], and contrastive objectives for representation distillation\[[27](https://arxiv.org/html/2608.29099#bib.bib10)\]\. On the logits side, a large line of work improves distillation by refining target distributions or loss formulations\[[35](https://arxiv.org/html/2608.29099#bib.bib12),[13](https://arxiv.org/html/2608.29099#bib.bib11)\], while self\-distillation provides a complementary direction\[[9](https://arxiv.org/html/2608.29099#bib.bib13)\]\.

Temperature scaling is central to KD and is also widely used in related settings such as calibration\[[10](https://arxiv.org/html/2608.29099#bib.bib18)\], out\-of\-distribution detection\[[18](https://arxiv.org/html/2608.29099#bib.bib19)\], and semi\- or self\-supervised learning\[[6](https://arxiv.org/html/2608.29099#bib.bib20),[5](https://arxiv.org/html/2608.29099#bib.bib21),[26](https://arxiv.org/html/2608.29099#bib.bib22),[4](https://arxiv.org/html/2608.29099#bib.bib23)\]\. Recent KD methods adapt temperature by introducing curriculum temperature schedules\[[17](https://arxiv.org/html/2608.29099#bib.bib16)\], logit standardization\[[25](https://arxiv.org/html/2608.29099#bib.bib14)\], entropy\-based adaptation\[[24](https://arxiv.org/html/2608.29099#bib.bib15)\], and precomputed sample\-wise temperatures based on pretrained\-teacher difficulty\[[32](https://arxiv.org/html/2608.29099#bib.bib17)\]\. In contrast to temperature adaptation, Transformed Teacher Matching \(TTM\) applies temperature scaling only to the teacher distribution and shows that the resulting formulation can be viewed as standard distillation with an implicit Rényi entropy regularization on the student\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\]\. WTTM further extends TTM with sample\-wise weighting to emphasize informative teacher signals\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\]\. Our work complements TTM and WTTM by adapting the teacher\-side inverse temperature at the sample level using a curvature\-aware update derived from closed\-form first and second derivatives of the teacher\-student mismatch, enabling temperature adaptation within the TTM framework\.

## 3Preliminaries

### 3\.1Setting and Notation

We considerKK\-class classification under a knowledge distillation setting\. Given an input\-label pair\(𝐱,y\)\(\\mathbf\{x\},y\)withy∈\{1,…,K\}y\\in\\\{1,\\dots,K\\\}, we train a student modelfs​\(⋅,θs\)f\_\{s\}\(\\cdot;\\theta\_\{s\}\)using both the ground\-truth label and the teacher’s predictive distribution produced by a teacher modelft​\(⋅,θt\)f\_\{t\}\(\\cdot;\\theta\_\{t\}\)\. The teacher parametersθt\\theta\_\{t\}are fixed during distillation, while the student parametersθs\\theta\_\{s\}are optimized\. Let the teacher and student models produce logits𝐳t=ft​\(𝐱,θt\)∈ℝK\\mathbf\{z\}\_\{t\}=f\_\{t\}\(\\mathbf\{x\};\\theta\_\{t\}\)\\in\\mathbb\{R\}^\{K\}and𝐳s=fs​\(𝐱,θs\)∈ℝK\\mathbf\{z\}\_\{s\}=f\_\{s\}\(\\mathbf\{x\};\\theta\_\{s\}\)\\in\\mathbb\{R\}^\{K\}\. The corresponding temperature\-free predictive distributions are

p=softmax⁡\(𝐳t\),q=softmax⁡\(𝐳s\),p=\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{t\}\),\\qquad q=\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{s\}\),\(1\)where\(softmax⁡\(𝐳\)\)i=exp⁡\(zi\)/∑j=1Kexp⁡\(zj\)\\bigl\(\\mathrm\{softmax\}\(\\mathbf\{z\}\)\\bigr\)\_\{i\}=\\exp\(z\_\{i\}\)\\big/\\sum\_\{j=1\}^\{K\}\\exp\(z\_\{j\}\)\. When the dependence on model parameters is important, we writepθt=softmax⁡\(ft​\(𝐱,θt\)\)p\_\{\\theta\_\{t\}\}=\\mathrm\{softmax\}\(f\_\{t\}\(\\mathbf\{x\};\\theta\_\{t\}\)\)andqθs=softmax⁡\(fs​\(𝐱,θs\)\)q\_\{\\theta\_\{s\}\}=\\mathrm\{softmax\}\(f\_\{s\}\(\\mathbf\{x\};\\theta\_\{s\}\)\); otherwise, we useppandqqfor notational simplicity\.

For distillation, we introduce a temperature parameterT\>0T\>0to control the sharpness \(or smoothness\) of the predictive distributions\. Specifically, we define the temperature\-scaled teacher and student distributions as

pT=softmax⁡\(𝐳t/T\),qT=softmax⁡\(𝐳s/T\)\.p\_\{T\}=\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{t\}/T\),\\qquad q\_\{T\}=\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{s\}/T\)\.\(2\)We useℒCE​\(⋅,⋅\)\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(\\cdot,\\cdot\)for the cross\-entropy loss andDKL\(⋅∥⋅\)D\_\{\\mathrm\{KL\}\}\(\\cdot\\\|\\cdot\)for the Kullback\-Leibler divergence\.

### 3\.2Knowledge Distillation and Transformed Teacher Matching

#### Knowledge distillation \(KD\)\[[12](https://arxiv.org/html/2608.29099#bib.bib1)\]\.

KD trains the student by combining supervised learning with distribution matching between temperature\-scaled teacher and student predictions\. A common formulation is

ℒKD=\(1−λ\)ℒCE\(y,q\)\+λT2DKL\(pT∥qT\),\\mathcal\{L\}\_\{\\mathrm\{KD\}\}=\(1\-\\lambda\)\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(y,q\)\+\\lambda T^\{2\}\\,D\_\{\\mathrm\{KL\}\}\(p\_\{T\}\\\|q\_\{T\}\),\(3\)whereλ∈\[0,1\]\\lambda\\in\[0,1\]balances the supervised and distillation terms\.

#### Transformed teacher matching \(TTM\)\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\]\.

To clarify the role of temperature on the teacher side, TTM interprets temperature scaling as a*power transform*\. Given the temperature\-free teacher predictionpp, we define the power\-transformed distribution

pi\(γ\)=piγ∑j=1Kpjγ,γ\>0\.p^\{\(\\gamma\)\}\_\{i\}=\\frac\{p\_\{i\}^\{\\gamma\}\}\{\\sum\_\{j=1\}^\{K\}p\_\{j\}^\{\\gamma\}\},\\qquad\\gamma\>0\.\(4\)By settingγ=1/T\\gamma=1/T, we obtain the equivalencepT=p\(γ\)p\_\{T\}=p^\{\(\\gamma\)\}and we computep\(γ\)p^\{\(\\gamma\)\}directly from logits as

p\(γ\)=softmax⁡\(γ​𝐳t\)\.p^\{\(\\gamma\)\}=\\mathrm\{softmax\}\(\\gamma\\,\\mathbf\{z\}\_\{t\}\)\.\(5\)
Building on the power\-transform view, TTM makes the connection to the standard KD objective explicit\. In particular, with the parameter correspondenceγ=1/T\\gamma=1/Tandβ=λ1−λ​T\\beta=\\frac\{\\lambda\}\{1\-\\lambda\}T, the TTM loss can be rewritten as

ℒTTM=ℒCE\(y,q\)\+βDKL\(p\(γ\)∥q\)=11−λ\[ℒKD−λT\(T−1\)ℋ1/T\(q\)\],\\mathcal\{L\}\_\{\\mathrm\{TTM\}\}=\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(y,q\)\+\\beta\\,D\_\{\\mathrm\{KL\}\}\(p^\{\(\\gamma\)\}\\\|q\)=\\frac\{1\}\{1\-\\lambda\}\\Bigl\[\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\-\\lambda T\(T\-1\)\\,\\mathcal\{H\}\_\{1/T\}\(q\)\\Bigr\],\(6\)whereℋα​\(q\)\\mathcal\{H\}\_\{\\alpha\}\(q\)denotes the Rényi entropy of orderα\\alpha\[[22](https://arxiv.org/html/2608.29099#bib.bib3)\]\. This identity highlights that TTM is not only a distribution\-matching objective but also introduces an implicit Rényi entropy regularization on the student prediction\.

#### Weighted TTM \(WTTM\)\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\]\.

WTTM further extends TTM by assigning a sample\-wise weight to the distillation term based on the sharpness of the teacher target\. Concretely, WTTM multiplies the distillation coefficientβ\\betain Eq\. \([6](https://arxiv.org/html/2608.29099#S3.E6)\) by a weightUγ​\(p\)U\_\{\\gamma\}\(p\)computed for each sample:

ℒWTTM=ℒCE\(y,q\)\+βUγ\(p\)DKL\(pT∥q\),Uγ\(p\)=∑j=1Kpjγ,\\mathcal\{L\}\_\{\\mathrm\{WTTM\}\}=\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(y,q\)\+\\beta U\_\{\\gamma\}\(p\)D\_\{\\mathrm\{KL\}\}\(p\_\{T\}\\\|q\),\\quad U\_\{\\gamma\}\(p\)=\\sum\_\{j=1\}^\{K\}p\_\{j\}^\{\\gamma\},\(7\)whereUγ​\(p\)U\_\{\\gamma\}\(p\)is a power\-sum of the teacher distribution that increases when the teacher target is smoother, thereby upweighting samples for which dark knowledge is more informative, while keeping the teacher\-side temperature unchanged\.

## 4Methodology

### 4\.1Overview: Per\-Sample Temperature Adaptation for TTM

We address the remaining question in TTM:*how to adapt the teacher temperature*\. Our goal is to enable sample\-adaptive temperature scaling within the TTM framework\. To this end, we optimize the inverse temperatureγ=1/T\\gamma=1/Tfor each input under fixed logits\. Specifically, we form the teacher distributionp\(γ\)=softmax⁡\(γ​𝐳t\)p^\{\(\\gamma\)\}=\\mathrm\{softmax\}\(\\gamma\\,\\mathbf\{z\}\_\{t\}\)and the student predictionq=softmax⁡\(𝐳s\)q=\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{s\}\), and minimize the per\-sample TTM discrepancyDKL\(p\(γ\)∥q\)D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\.

#### Local sensitivity and curvature of the KL discrepancy\.

To understand how the teacher temperature affects TTM,*i\.e*, to quantify how the teacher\-side inverse temperatureγ\\gammaaffects TTM at the sample level, we analyze the local behavior ofDKL\(p\(γ\)∥q\)D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)under small perturbationsΔ​γ\\Delta\\gamma\. This discrepancy is the teacher\-student matching term in TTM, and its value can vary substantially across samples as the student prediction changes during training\. Therefore, its local variation provides a natural signal for how the teacher\-side transformation should be adjusted for each sample\. Concretely, we approximate the change in the discrepancy by a second\-order Taylor expansion around the currentγ\\gamma:

DKL\(p\(γ\+Δ​γ\)∥q\)≈DKL\(p\(γ\)∥q\)⏟constant\+dd​γDKL\(p\(γ\)∥q\)Δγ⏟local sensitivity\+12d2d​γ2DKL\(p\(γ\)∥q\)\(Δγ\)2⏟local curvature,D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\+\\Delta\\gamma\)\}\\\|q\\bigr\)\\approx\\underbrace\{D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\}\_\{\\text\{constant\}\}\+\\underbrace\{\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\\Delta\\gamma\}\_\{\\text\{local sensitivity\}\}\+\\underbrace\{\\frac\{1\}\{2\}\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\(\\Delta\\gamma\)^\{2\}\}\_\{\\text\{local curvature\}\},\(8\)which characterizes both the local sensitivity \(first derivative\) and curvature \(second derivative\) with respect toγ\\gamma\. This quadratic model provides the local surrogate used below, and its derivatives are derived in Sec\.[4\.2](https://arxiv.org/html/2608.29099#S4.SS2)\.

#### Minimizing the local quadratic approximation\.

The local quadratic model above turns the sensitivity analysis into a practical update rule for the teacher\-side inverse temperature\. Standard TTM optimizes the student parameters for a fixed teacher\-side temperature\. In contrast, our method treats the teacher\-side inverse temperature as a sample\-wise auxiliary variable and refines it for the current student\. Conceptually, this gives a double\-minimization view\[[31](https://arxiv.org/html/2608.29099#bib.bib36),[33](https://arxiv.org/html/2608.29099#bib.bib35),[30](https://arxiv.org/html/2608.29099#bib.bib37)\]of the TTM objective:

minθsminγ\>0\[ℒCE\(y,qθs\)\+βDKL\(p\(γ\)∥qθs\)\]\.\\min\_\{\\theta\_\{s\}\}\\min\_\{\\gamma\>0\}\\left\[\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(y,q\_\{\\theta\_\{s\}\}\)\+\\beta D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\_\{\\theta\_\{s\}\}\\bigr\)\\right\]\.\(9\)This expression is used only to motivate the update direction; we do not solve the inner problem to convergence\. In practice, after warm\-up, we keep the current teacher and student logits fixed and updateγ\\gammaperiodically everyIγI\_\{\\gamma\}epochs using one local step based on the quadratic approximation in Eq\. \([8](https://arxiv.org/html/2608.29099#S4.E8)\)\. Dropping the constant term that does not affect the update, we minimize the quadratic surrogate

Δγ⋆≜argminΔ​γ∈ℝ\[dd​γDKL\(p\(γ\)∥q\)Δγ\+12d2d​γ2DKL\(p\(γ\)∥q\)\(Δγ\)2\]\.\\Delta\\gamma^\{\\star\}\\triangleq\\arg\\min\_\{\\Delta\\gamma\\in\\mathbb\{R\}\}\\;\\left\[\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\\,\\Delta\\gamma\+\\frac\{1\}\{2\}\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\\,\(\\Delta\\gamma\)^\{2\}\\right\]\.\(10\)

#### Curvature\-aware update\.

Because the surrogate in Eq\. \([10](https://arxiv.org/html/2608.29099#S4.E10)\) is quadratic inΔ​γ\\Delta\\gamma, its minimizer has a closed form:

Δ​γ⋆=−dd​γDKL\(p\(γ\)∥q\)d2d​γ2DKL\(p\(γ\)∥q\)\.\\Delta\\gamma^\{\\star\}=\-\\frac\{\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\}\{\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\}\.\(11\)We update the inverse temperature using a step sizeη\>0\\eta\>0, add a small constant to the denominator for numerical stability, and clip the result to a valid range:

γ←clip⁡\(γ−η​dd​γDKL\(p\(γ\)∥q\)d2d​γ2DKL\(p\(γ\)∥q\)\+δ,γmin,γmax\),\\gamma\\leftarrow\\mathrm\{clip\}\\\!\\left\(\\gamma\-\\eta\\,\\frac\{\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\}\{\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\+\\delta\},\\gamma\_\{\\min\},\\gamma\_\{\\max\}\\right\),\(12\)whereδ\>0\\delta\>0is a small constant for numerical stability,clip⁡\(x,a,b\)=min⁡\(max⁡\(x,a\),b\)\\mathrm\{clip\}\(x,a,b\)=\\min\(\\max\(x,a\),b\), andγmin,γmax\\gamma\_\{\\min\},\\gamma\_\{\\max\}are the lower and upper bounds that restrictγ\\gammato a valid range\. In Sec\.[4\.2](https://arxiv.org/html/2608.29099#S4.SS2), we show that these derivatives can be written in compact variance\-covariance forms, enabling an efficient per\-sample update\.

#### Temperature\-adaptive extensions of TTM and WTTM\.

The update in Eq\. \([12](https://arxiv.org/html/2608.29099#S4.E12)\) can be directly incorporated into TTM \(Eq\. \([6](https://arxiv.org/html/2608.29099#S3.E6)\)\) and WTTM \(Eq\. \([7](https://arxiv.org/html/2608.29099#S3.E7)\)\) by replacing the fixed inverse temperature with a sample\-wise value updated during training\. We denote the resulting methods as*Temperature\-Adaptive TTM \(TA\-TTM\)*and*Temperature\-Adaptive WTTM \(TA\-WTTM\)*, respectively\. In both cases, we keep the original loss definitions and introduce only the additionalγ\\gammaupdate computed withstop\-gradientlogits\. We maintainγ\\gammaas a per\-sample state throughout training and update it for observed samples during scheduled update epochs\. For TA\-WTTM, one could alternatively updateγ\\gammaby minimizing the full weighted WTTM term, including the sample weightUγ​\(p\)U\_\{\\gamma\}\(p\)\. However, this couples temperature adaptation with sample weighting, and our ablation in Sec\.[5\.4](https://arxiv.org/html/2608.29099#S5.SS4.SSS0.Px3)shows that this full\-objective variant underperforms the decoupled update in Eq\. \([12](https://arxiv.org/html/2608.29099#S4.E12)\)\. We therefore use the decoupled update as TA\-WTTM in the rest of the paper\. As a result, TA\-TTM adapts the teacher\-side temperature within the TTM objective, and TA\-WTTM augments WTTM with the same temperature adaptation while retaining its original sample weighting scheme\.

### 4\.2Closed\-Form Derivatives

The curvature\-aware update in Sec\.[4\.1](https://arxiv.org/html/2608.29099#S4.SS1)requires the first and second derivatives of the per\-sample discrepancyDKL\(p\(γ\)∥q\)D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)with respect to the teacher\-side inverse temperatureγ\\gamma\. Notably, both derivatives can be written using low\-order moments of teacher and student logits under the transformed teacher weighting\.

We recallp\(γ\)=softmax⁡\(γ​𝐳t\)p^\{\(\\gamma\)\}=\\mathrm\{softmax\}\(\\gamma\\mathbf\{z\}\_\{t\}\)andq=softmax⁡\(𝐳s\)q=\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{s\}\), and we differentiate with respect toγ\\gammawith stop\-gradient logits\. Letmt≜𝔼p\(γ\)​\[zt\]m\_\{t\}\\triangleq\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]andms≜𝔼p\(γ\)​\[zs\]m\_\{s\}\\triangleq\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\], where𝔼p\(γ\)​\[f\]≜∑i=1Kpi\(γ\)​fi\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\]\\triangleq\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}f\_\{i\}denotes the class\-wise expectation\. We write centered logits asz^t≜zt−mt\\hat\{z\}\_\{t\}\\triangleq z\_\{t\}\-m\_\{t\}andz^s≜zs−ms\\hat\{z\}\_\{s\}\\triangleq z\_\{s\}\-m\_\{s\}\(element\-wise\)\. With these definitions, the first derivative can be written in a compact variance\-covariance form underp\(γ\)p^\{\(\\gamma\)\}, while the second derivative yields a curvature expression that also remains a simple class\-wise weighted sum\. Concretely,

dd​γDKL\(p\(γ\)∥q\)\\displaystyle\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=γ​Varp\(γ\)​\(zt\)⏟teacher dispersion−Covp\(γ\)​\(zs,zt\)⏟teacher\-studentalignment,\\displaystyle=\\underbrace\{\\gamma\\,\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)\}\_\{\\text\{\\shortstack\{teacher dispersion\}\}\}\-\\underbrace\{\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},z\_\{t\}\)\}\_\{\\text\{\\shortstack\{teacher\-student\\\\ alignment\}\}\},\(13\)d2d​γ2DKL\(p\(γ\)∥q\)\\displaystyle\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=Varp\(γ\)​\(zt\)⏟base curvature−Covp\(γ\)​\(zs,z^t2\)⏟student\-teacherdeviation correlation\+γ​𝔼p\(γ\)​\[z^t3\]⏟teacher skewness,\\displaystyle=\\underbrace\{\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)\}\_\{\\text\{\\shortstack\{base curvature\}\}\}\-\\underbrace\{\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},\\hat\{z\}\_\{t\}^\{\\,2\}\)\}\_\{\\text\{\\shortstack\{student\-teacher\\\\ deviation correlation\}\}\}\+\\underbrace\{\\gamma\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,3\}\]\}\_\{\\text\{\\shortstack\{teacher skewness\}\}\},whereVarp\(γ\)​\(⋅\)\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(\\cdot\)andCovp\(γ\)​\(⋅,⋅\)\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(\\cdot,\\cdot\)are taken with respect to the weightingp\(γ\)p^\{\(\\gamma\)\}\. We provide the full derivations in the supplementary material\.

#### Interpretation of first derivative\.

Eq\. \([13](https://arxiv.org/html/2608.29099#S4.E13)\) shows that temperature adaptation is governed by simple statistics of teacher and student logits under the transformed teacher weightingp\(γ\)p^\{\(\\gamma\)\}\. In the first derivative, the update direction is determined by the balance between teacher dispersion and teacher\-student alignment\. The termγ​Varp\(γ\)​\(zt\)\\gamma\\,\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)grows when the transformed teacher distribution emphasizes classes with widely spread teacher logits, meaning that changingγ\\gammastrongly affects the smoothness ofp\(γ\)p^\{\(\\gamma\)\}\. In contrast,Covp\(γ\)​\(zs,zt\)\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},z\_\{t\}\)increases when the student logits align with the teacher logits on those emphasized classes, which counteracts the push to changeγ\\gamma\.

#### Interpretation of second derivative\.

The second derivative quantifies the local curvature that determines how aggressive the update should be\. The base curvatureVarp\(γ\)​\(zt\)\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)provides the dominant scale, while the student\-teacher deviation correlation termCovp\(γ\)​\(zs,z^t2\)\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},\\hat\{z\}\_\{t\}^\{\\,2\}\)adjusts this curvature depending on how the student weights classes where the teacher deviates strongly from its weighted mean\. Finally, the teacher skewness termγ​𝔼p\(γ\)​\[z^t3\]\\gamma\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,3\}\]captures asymmetry of teacher logits underp\(γ\)p^\{\(\\gamma\)\}\.

Algorithm 1Training with per\-sample inverse\-temperature adaptation0:Training data

\{\(𝐱,y\)\}\\\{\(\\mathbf\{x\},y\)\\\}, teacher

ftf\_\{t\}, student

fsf\_\{s\}, initial per\-sample inverse temperatures

\{γ\}\\\{\\gamma\\\}, total epochs

EE, warm\-up

EwarmE\_\{\\mathrm\{warm\}\}, epoch update interval

IγI\_\{\\gamma\}
1:forepoch

=1,…,E=1,\\dots,Edo

2:foreach mini\-batch

ℬ\\mathcal\{B\}do

3:Compute teacher logits

𝐳t=ft​\(𝐱\)\\mathbf\{z\}\_\{t\}=f\_\{t\}\(\\mathbf\{x\}\)and student logits

𝐳s=fs​\(𝐱\)\\mathbf\{z\}\_\{s\}=f\_\{s\}\(\\mathbf\{x\}\)
4:Compute

p\(γ\)←softmax⁡\(γ​𝐳t\)p^\{\(\\gamma\)\}\\leftarrow\\mathrm\{softmax\}\(\\gamma\\,\\mathbf\{z\}\_\{t\}\)and

q←softmax⁡\(𝐳s\)q\\leftarrow\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{s\}\)
5:ifepoch

\>Ewarm\>E\_\{\\mathrm\{warm\}\}and

\(epoch−Ewarm−1\)modIγ=0\(\\mathrm\{epoch\}\-E\_\{\\mathrm\{warm\}\}\-1\)\\bmod I\_\{\\gamma\}=0then

6:

mt←∑ipi\(γ\)​zt,im\_\{t\}\\leftarrow\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}z\_\{t,i\},

ms←∑ipi\(γ\)​zs,im\_\{s\}\\leftarrow\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}z\_\{s,i\}
7:

𝐳^t←𝐳t−mt\\hat\{\\mathbf\{z\}\}\_\{t\}\\leftarrow\\mathbf\{z\}\_\{t\}\-m\_\{t\},

𝐳^s←𝐳s−ms\\hat\{\\mathbf\{z\}\}\_\{s\}\\leftarrow\\mathbf\{z\}\_\{s\}\-m\_\{s\}
8:Compute derivatives using Eq\. \([13](https://arxiv.org/html/2608.29099#S4.E13)\) with detached logits

9:

γ←clip​\(γ−η​dd​γDKL\(p\(γ\)∥q\)d2d​γ2DKL\(p\(γ\)∥q\)\+δ,γmin,γmax\)\\gamma\\leftarrow\\texttt\{clip\}\\\!\\left\(\\gamma\-\\eta\\,\\frac\{\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\}\{\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)\+\\delta\},\\gamma\_\{\\min\},\\gamma\_\{\\max\}\\right\)
10:endif

11:Compute the TTM/WTTM distillation loss and update student parameters

12:endfor

13:endfor

#### Connection to implicit Rényi entropy regularization\.

This sample\-wise adaptation also has an interpretation through the implicit Rényi entropy regularization in TTM \(Eq\. \([6](https://arxiv.org/html/2608.29099#S3.E6)\)\)\. Since the teacher\-side temperature determines the transformed teacher distribution, it also controls how strongly the student is implicitly encouraged to adjust its predictive entropy under the TTM objective\. Therefore, updatingγ\\gammaper sample not only changes the teacher target but also modulates the strength of the implicit Rényi entropy regularization across samples, allowing the regularization effect to adapt to sample difficulty and teacher\-student mismatch\.

#### Curriculum learning perspective\.

Our per\-sample temperature adaptation can also be interpreted as an implicit curriculum\. The inverse temperatureγ\\gammacontrols the sharpness of the transformed teacher distribution, which affects how difficult the distillation target is for the student to match\. By updatingγ\\gammato reduce the teacher\-student discrepancy, the method can assign softer targets to samples with large mismatch and sharper targets to samples where the student is more aligned with the teacher\. In this sense, the teacher signal is adjusted at the sample level according to the current learning state, resembling curriculum or self\-paced learning\[[3](https://arxiv.org/html/2608.29099#bib.bib27),[14](https://arxiv.org/html/2608.29099#bib.bib28),[36](https://arxiv.org/html/2608.29099#bib.bib29),[2](https://arxiv.org/html/2608.29099#bib.bib30)\]\. Through Eq\. \([6](https://arxiv.org/html/2608.29099#S3.E6)\), this view is consistent with sample\-wise modulation of the implicit Rényi entropy regularization, since changingγ\\gammasimultaneously changes the strength of the regularization effect induced by TTM\.

### 4\.3Algorithm and Implementation

Algorithm[1](https://arxiv.org/html/2608.29099#alg1)summarizes the per\-sample inverse\-temperature update\. Each scheduled update requires one softmax overKKclasses and a constant number ofO⁡\(K\)O\(K\)weighted sums, yielding negligible overhead\. We initializeγ\\gammafrom the temperature used in TTM, keep it fixed during a short warm\-up, and then update it everyIγI\_\{\\gamma\}epochs, reusing the stored values between updates\. For a training set ofNNsamples, the per\-sample state consists ofNN32\-bit floating\-point \(FP32\) scalars, requiring4​N4Nbytes \(0\.2 MB for CIFAR100 and 5\.1 MB for ImageNet1k\)\. The state is indexed by dataset sample IDs, making it compatible with shuffling and standard data augmentation\. The intervalIγI\_\{\\gamma\}trades update frequency against noise and computation\. For stability, we addδ\\deltato the curvature denominator and clipγ\\gammato\[γmin,γmax\]\[\\gamma\_\{\\min\},\\gamma\_\{\\max\}\]; these safeguards handle small or negative estimated curvature and keep the stored inverse temperatures within the valid range\.

Table 1:Performance comparison on CIFAR100 in the same\-architecture distillation setting\.†\\daggerindicates results reproduced by us using the official implementation under the protocol described in the paper\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\]\.Δ\\DeltaandΔ†\\Delta^\{\\dagger\}denote accuracy differences from TTM/WTTM and TTM†/WTTM†, respectively; positive values are highlighted in red and negative values in blue\.Table 2:Performance comparison on CIFAR100 in the cross\-architecture distillation setting\.

## 5Evaluation

### 5\.1Evaluation Setting

#### Benchmarks and comparison methods\.

We evaluate our method on CIFAR100\[[15](https://arxiv.org/html/2608.29099#bib.bib26)\]and ImageNet1k\[[7](https://arxiv.org/html/2608.29099#bib.bib25)\], which are standard image classification benchmarks for distillation\. CIFAR100 contains 100 classes of32×3232\\times 32images, and ImageNet1k contains 1,000 classes of high\-resolution natural images\. We use widely adopted teacher\-student architectures and report top\-1 accuracy on the CIFAR100 test split and the ImageNet1k validation split\. We compare against TTM\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\], WTTM\[[37](https://arxiv.org/html/2608.29099#bib.bib2)\], strong KD baselines FitNet\[[23](https://arxiv.org/html/2608.29099#bib.bib4)\], AT\[[34](https://arxiv.org/html/2608.29099#bib.bib5)\], VID\[[1](https://arxiv.org/html/2608.29099#bib.bib6)\], RKD\[[20](https://arxiv.org/html/2608.29099#bib.bib8)\], PKT\[[21](https://arxiv.org/html/2608.29099#bib.bib9)\], CRD\[[27](https://arxiv.org/html/2608.29099#bib.bib10)\], DIST\[[13](https://arxiv.org/html/2608.29099#bib.bib11)\], and DKD\[[35](https://arxiv.org/html/2608.29099#bib.bib12)\], temperature\-adaptive KD methods LS\[[25](https://arxiv.org/html/2608.29099#bib.bib14)\]and EA\-KD\[[24](https://arxiv.org/html/2608.29099#bib.bib15)\], and the curriculum\-based KD method CTKD\[[17](https://arxiv.org/html/2608.29099#bib.bib16)\]\.

#### Training and evaluation protocol\.

We follow the training and evaluation protocols of prior distillation work\[[27](https://arxiv.org/html/2608.29099#bib.bib10)\]to ensure consistent comparisons\. Since our method extends TTM and WTTM with temperature adaptation, we reproduce TTM and WTTM using the official implementation111https://github\.com/zkxufo/TTMunder the protocol described in the original paper\. For our method, all hyperparameters follow the corresponding TTM or WTTM configuration, except for the step size introduced for temperature adaptation\. For the reproduced baselines and proposed methods in the main CIFAR100 experiments, we report the mean and standard deviation over five runs with different random seeds\. Full details are provided in the supplementary material\.

### 5\.2Comparison Results

#### CIFAR100 results\.

Tables[1](https://arxiv.org/html/2608.29099#S4.T1)and[2](https://arxiv.org/html/2608.29099#S4.T2)summarize results on CIFAR100 under same\-architecture and cross\-architecture distillation, respectively\. TA\-TTM is generally competitive with or better than the corresponding TTM baseline, while TA\-WTTM consistently strengthens WTTM\. The trend holds in both architecture settings, indicating that per\-sample temperature adaptation is complementary to teacher\-side temperature matching and remains useful when the teacher\-student gap becomes larger\. TTM and WTTM already use carefully tunedβ\\betaandγ\\gammaconfigurations, so the improvements obtained by changing only the teacher\-side temperature adaptation should be interpreted as modest refinements over strong baselines\.

#### ImageNet1k results\.

Table[3](https://arxiv.org/html/2608.29099#S5.T3)reports results on ImageNet1k for the ResNet\-34 teacher and ResNet\-18 student pair\. TA\-TTM improves over the reproduced TTM baseline, and TA\-WTTM further improves over WTTM while remaining competitive with the temperature\-adaptive KD baseline EA\-KD\. These results indicate that the proposed sample\-wise temperature adaptation remains effective at ImageNet1k scale and that its benefit is preserved when combined with the WTTM sample\-weighting scheme\.

Table 3:Performance comparison on ImageNet1k\. TTM and WTTM results are from our reproduced experiments\. These results are obtained from a single run\.Table 4:Top\-1 accuracy \(%\) on CIFAR100 same\-architecture pairs\. TTM/WTTM variants omit cross\-entropy; KD with cross\-entropy is included as a reference\. Results are reported over three seeds\.
#### Distillation without cross\-entropy\.

Table[4](https://arxiv.org/html/2608.29099#S5.T4)evaluates whether the proposed temperature adaptation remains effective when the supervised cross\-entropy term is removed from the TTM\-style objectives on representative pairs, which reflects recent practical settings where ground\-truth labels are not available and learning must rely on teacher\-provided supervision\. Both TA\-TTM and TA\-WTTM retain their advantage over the corresponding non\-adaptive baselines\. These results indicate that the gains are not solely due to interaction with the supervised cross\-entropy term; the adaptive teacher\-side transform also strengthens the distillation signal itself\.

Table 5:Performance comparison on CIFAR100 in the distillation setting from transformer\-based teachers to CNN\-based students\.
#### Distillation from transformer\-based teachers\.

Table[5](https://arxiv.org/html/2608.29099#S5.T5)evaluates distillation performance from transformer\-based teachers \(ViT\-S\[[8](https://arxiv.org/html/2608.29099#bib.bib31)\], Swin\-T\[[19](https://arxiv.org/html/2608.29099#bib.bib32)\], and Mixer\-B/16\[[28](https://arxiv.org/html/2608.29099#bib.bib33)\]\) to a CNN\-based student \(ResNet18\[[11](https://arxiv.org/html/2608.29099#bib.bib34)\]\)\. TTM and WTTM are strong baselines in this setting, and our adaptive variants further improve them\. This suggests that per\-sample temperature adaptation is also useful when transferring knowledge across model families, where the teacher\-student gap is pronounced\.

### 5\.3Analysis of Temperature Dynamics

\(a\) Dynamics

\(b\) Distribution

![Refer to caption](https://arxiv.org/html/2608.29099v1/x2.png)

\(c\) Difficulty correlation

Figure 2:Temperature diagnostics for TA\-TTM on the representative same\-architecture pair ResNet110→\\rightarrowResNet20\. The panels show the evolution of the mean and standard deviation of the inverse temperatureγ\\gamma, the stage\-wise distribution ofγ\\gamma, and the correlation between per\-sample cross\-entropy loss and the optimizedγ\\gamma, respectively\.\(a\) Dynamics

\(b\) Distribution

![Refer to caption](https://arxiv.org/html/2608.29099v1/x4.png)

\(c\) Difficulty correlation

Figure 3:Temperature diagnostics for TA\-WTTM on ResNet110→\\rightarrowResNet20, using the same layout as Fig\.[2](https://arxiv.org/html/2608.29099#S5.F2)\.#### Adaptation behavior of the average temperature\.

We analyze the training dynamics of the temperature\. Figs\.[2](https://arxiv.org/html/2608.29099#S5.F2)\(a\) and[3](https://arxiv.org/html/2608.29099#S5.F3)\(a\) show that the mean inverse temperatureγ\\gammaevolves in a structured manner rather than converging to a constant\. For both TA\-TTM and TA\-WTTM,γ\\gammadecreases from its initialization and then stabilizes after mid training, indicating that the effective temperatureT=1/γT=1/\\gammaincreases and the teacher targets become softer over time\. The red dashed curves plot the mixing coefficientλ\\lambdaimplied by the learnedγ\\gammaunder a fixedβ\\beta, and their evolution suggests that the balance between the distillation term and the supervised cross\-entropy term changes adaptively as a consequence of temperature adaptation\. In addition, the standard deviation ofγ\\gammatends to increase, suggesting that sample\-wise temperatures become more diverse as training progresses\. Corresponding cross\-architecture diagnostics are provided in the supplementary material\.

#### Stage\-wise temperature distribution\.

Figs\.[2](https://arxiv.org/html/2608.29099#S5.F2)\(b\) and[3](https://arxiv.org/html/2608.29099#S5.F3)\(b\) visualize how the inverse temperatureγ\\gammais distributed over training samples at three stages of training\. At epoch 100, the distributions are sharply peaked for both methods, indicating that most samples share a similar temperature early in training\. By epoch 200, the histograms become noticeably broader and their modes shift to smallerγ\\gammavalues, which corresponds to using a larger effective temperatureT=1/γT=1/\\gammaand thus softer teacher targets on average\. This widening suggests that the proposed update progressively differentiates samples and assigns a wider range of temperatures rather than converging to a single global value\. From epoch 200 to epoch 240, the distributions change more mildly, implying that the sample\-wise temperatures stabilize in the late stage\.

#### Correlation with sample difficulty\.

Figs\.[2](https://arxiv.org/html/2608.29099#S5.F2)\(c\) and[3](https://arxiv.org/html/2608.29099#S5.F3)\(c\) examine how the optimized inverse temperatureγ\\gammarelates to sample difficulty, measured by the per\-sample cross\-entropy loss, which is widely used as a difficulty proxy in curriculum and self\-paced learning\[[14](https://arxiv.org/html/2608.29099#bib.bib28)\]\. Across both methods and training stages, we observe a clear negative correlation: samples with larger cross\-entropy loss tend to be assigned smallerγ\\gamma\. Since smallerγ\\gammacorresponds to a larger effective temperatureT=1/γT=1/\\gamma, the proposed adaptation softens the teacher target for harder samples while keeping it sharper for easier ones\. This relationship remains pronounced throughout training, as reflected by the consistent downward trends in the point clouds\. Overall, these results support a curriculum\-like behavior in which the supervision is adjusted according to the current sample difficulty and teacher\-student mismatch\.

### 5\.4Ablation Results

Table 6:Temperature\-design ablations on CIFAR100 for WRN\-40\-2→\\rightarrowWRN\-16\-2\. \(a\) compares inverse\-temperature granularity using a commonη=0\.001\\eta=0\.001and reports top\-1 best accuracy\. \(b\) fixes sample\-wiseγ\\gammaand compares optimization rules using method\-specific settings, reporting five\-run top\-1 best accuracy \(mean±\\pmstd\.\)\.\(a\) Temperature granularity

\(b\) Sample\-wise temperature optimization

Table 7:Ablation study of the TA\-WTTM temperature update on CIFAR100\. We compare TA\-WTTM, which uses the decoupled update, with the full\-objective update on representative same\- and different\-architecture pairs\.Δ\\Deltadenotes TA\-WTTM minus TA\-WTTM\-full\.#### Temperature granularity\.

Table[6](https://arxiv.org/html/2608.29099#S5.T6)\(a\) compares global, class\-wise, and sample\-wise inverse temperatures on CIFAR100 for WRN\-40\-2→\\rightarrowWRN\-16\-2 using a commonη=0\.001\\eta=0\.001\. For both methods, the sample\-wise variant achieves the highest top\-1 best accuracy, outperforming the global and class\-wise alternatives\. This representative ablation supports adapting the teacher transform at the sample level, where teacher\-student mismatch can vary even within the same class\.

#### Temperature optimization rule\.

Table[6](https://arxiv.org/html/2608.29099#S5.T6)\(b\) fixes sample\-wiseγ\\gammaand compares trainable, first\-order, and Newton updates over five runs using method\-specific settings\. The Newton update performs best for both TA\-TTM and TA\-WTTM, improving over directly trainingγ\\gammaand updating it with only the first derivative\. On this representative pair, the result supports using the closed\-form curvature rather than treating the inverse temperature as a generic trainable state\.

#### Full\-objective temperature update\.

Table[7](https://arxiv.org/html/2608.29099#S5.T7)compares two ways of updating the sample\-wise inverse temperature in TA\-WTTM on representative pairs\. The decoupled variant, used as our default, updatesγ\\gammaby minimizing the teacher\-student discrepancyDKL\(p\(γ\)∥q\)D\_\{\\mathrm\{KL\}\}\(p^\{\(\\gamma\)\}\\\|q\), while retaining the WTTM sample weight only in the training loss\. The full\-objective variant instead updatesγ\\gammausing the weighted WTTM objective, thereby coupling the temperature update with the sample\-weighting termUγ​\(p\)U\_\{\\gamma\}\(p\)\. TA\-WTTM outperforms the full\-objective variant in both same\- and cross\-architecture settings, indicating that separating temperature adaptation from sample weighting provides a more stable update forγ\\gamma\.

\(a\) Update rateη\\eta

\(b\) Warm\-up length

\(c\) Update interval

Figure 4:Hyperparameter sensitivity of the proposed temperature adaptation\. We evaluate the update rateη\\eta, warm\-up length, and update interval for the sample\-wise inverse temperature update\.
#### Sensitivity to hyperparameters\.

Fig\.[4](https://arxiv.org/html/2608.29099#S5.F4)summarizes the sensitivity to the update rateη\\eta, warm\-up length, and update interval\. The results are relatively stable across warm\-up lengths and update intervals, indicating that the method does not rely on a specific start epoch or very frequent temperature updates\. The update rate has a clearer effect: overly conservative updates can under\-adapt the temperature, while a moderate rate gives better accuracy\. Overall, the proposed update is robust to scheduling choices, withη\\etabeing the main hyperparameter to tune\.

## 6Conclusion

We addressed how to adapt the teacher temperature in Transformed Teacher Matching \(TTM\) and proposed a per\-sample update of the teacher\-side inverse temperature within the TTM framework\. Using closed\-form first and second derivatives, expressed through variance\-covariance statistics of centered teacher and student logits under the transformed teacher weighting, we derived an efficient curvature\-aware update with negligible overhead\. Experiments on standard image classification distillation benchmarks show that TA\-TTM and TA\-WTTM generally improve the reproduced TTM and WTTM baselines across multiple CIFAR100 settings, and also outperform the compared temperature\-adaptive baseline in the ImageNet1k setting\. Our analysis further reveals structured temperature dynamics and a negative correlation betweenγ\\gammaand per\-sample difficulty, suggesting softer teacher targets for harder samples and a curriculum\-like modulation of the implicit Rényi entropy regularization in TTM\.

## References

- \[1\]S\. Ahn, S\. X\. Hu, A\. Damianou, N\. D\. Lawrence, and Z\. Dai\(2019\)Variational information distillation for knowledge transfer\.InCVPR,pp\. 9163–9171\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[2\]H\. Aizawa and Y\. Hayashi\(2025\)Flatness\-aware curriculum learning via adversarial difficulty\.InBMVC,pp\. 1–12\.Cited by:[§4\.2](https://arxiv.org/html/2608.29099#S4.SS2.SSS0.Px4.p1.1)\.
- \[3\]Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston\(2009\)Curriculum learning\.InICML,pp\. 41–48\.Cited by:[§4\.2](https://arxiv.org/html/2608.29099#S4.SS2.SSS0.Px4.p1.1)\.
- \[4\]D\. Berthelot, N\. Carlini, I\. Goodfellow, N\. Papernot, A\. Oliver, and C\. A\. Raffel\(2019\)Mixmatch: a holistic approach to semi\-supervised learning\.NeurIPS32\.Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p4.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1)\.
- \[5\]M\. Caron, H\. Touvron, I\. Misra, H\. Jégou, J\. Mairal, P\. Bojanowski, and A\. Joulin\(2021\)Emerging properties in self\-supervised vision transformers\.InICCV,pp\. 9650–9660\.Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p4.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1)\.
- \[6\]T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton\(2020\)A simple framework for contrastive learning of visual representations\.InICML,pp\. 1597–1607\.Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p4.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1)\.
- \[7\]J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei\(2009\)Imagenet: a large\-scale hierarchical image database\.InCVPR,pp\. 248–255\.Cited by:[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[8\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly,et al\.\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InICLR,Cited by:[§5\.2](https://arxiv.org/html/2608.29099#S5.SS2.SSS0.Px4.p1.1)\.
- \[9\]T\. Furlanello, Z\. Lipton, M\. Tschannen, L\. Itti, and A\. Anandkumar\(2018\)Born again neural networks\.InICML,pp\. 1607–1616\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1)\.
- \[10\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InICML,pp\. 1321–1330\.Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p4.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1)\.
- \[11\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InCVPR,pp\. 770–778\.Cited by:[§5\.2](https://arxiv.org/html/2608.29099#S5.SS2.SSS0.Px4.p1.1)\.
- \[12\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p1.1),[§1](https://arxiv.org/html/2608.29099#S1.p2.1),[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§3\.2](https://arxiv.org/html/2608.29099#S3.SS2.SSS0.Px1)\.
- \[13\]T\. Huang, S\. You, F\. Wang, C\. Qian, and C\. Xu\(2022\)Knowledge distillation from a stronger teacher\.NeurIPS35,pp\. 33716–33727\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[14\]L\. Jiang, D\. Meng, Q\. Zhao, S\. Shan, and A\. Hauptmann\(2015\)Self\-paced curriculum learning\.InAAAI,Vol\.29\.Cited by:[§4\.2](https://arxiv.org/html/2608.29099#S4.SS2.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2608.29099#S5.SS3.SSS0.Px3.p1.1)\.
- \[15\]A\. Krizhevsky G\. Hintonet al\.\(2009\)Learning multiple layers of features from tiny images\.Cited by:[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[16\]J\. Li, R\. Socher, and S\. C\. Hoi\(2020\)DivideMix: learning with noisy labels as semi\-supervised learning\.InICLR,Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p4.1)\.
- \[17\]Z\. Li, X\. Li, L\. Yang, B\. Zhao, R\. Song, L\. Luo, J\. Li, and J\. Yang\(2023\)Curriculum temperature for knowledge distillation\.InAAAI,Vol\.37,pp\. 1504–1512\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[18\]S\. Liang, Y\. Li, and R\. Srikant\(2018\)Enhancing the reliability of out\-of\-distribution image detection in neural networks\.InICLR,Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p4.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1)\.
- \[19\]Z\. Liu, Y\. Lin, Y\. Cao, H\. Hu, Y\. Wei, Z\. Zhang, S\. Lin, and B\. Guo\(2021\)Swin transformer: hierarchical vision transformer using shifted windows\.InICCV,pp\. 10012–10022\.Cited by:[§5\.2](https://arxiv.org/html/2608.29099#S5.SS2.SSS0.Px4.p1.1)\.
- \[20\]W\. Park, D\. Kim, Y\. Lu, and M\. Cho\(2019\)Relational knowledge distillation\.InCVPR,pp\. 3967–3976\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[21\]N\. Passalis and A\. Tefas\(2018\)Learning deep representations with probabilistic knowledge transfer\.InECCV,pp\. 268–284\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[22\]A\. Rényi\(1961\)On measures of entropy and information\.InProceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics,Vol\.4,pp\. 547–562\.Cited by:[§3\.2](https://arxiv.org/html/2608.29099#S3.SS2.SSS0.Px2.p2.2)\.
- \[23\]A\. Romero, N\. Ballas, S\. E\. Kahou, A\. Chassang, C\. Gatta, and Y\. Bengio\(2015\)FitNets: hints for thin deep nets\.InICLR,Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[24\]C\. Su, C\. Tseng, B\. Pu, L\. Zhao, J\. Yang, Z\. Chen, and S\. Lee\(2025\)EA\-kd: entropy\-based adaptive knowledge distillation\.InICCV,pp\. 731–740\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[25\]S\. Sun, W\. Ren, J\. Li, R\. Wang, and X\. Cao\(2024\)Logit standardization in knowledge distillation\.InCVPR,pp\. 15731–15740\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[26\]A\. Tarvainen and H\. Valpola\(2017\)Mean teachers are better role models: weight\-averaged consistency targets improve semi\-supervised deep learning results\.NeurIPS30\.Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p4.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1)\.
- \[27\]Y\. Tian, D\. Krishnan, and P\. Isola\(2019\)Contrastive representation distillation\.InICLR,Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px2.p1.1)\.
- \[28\]I\. O\. Tolstikhin, N\. Houlsby, A\. Kolesnikov, L\. Beyer, X\. Zhai, T\. Unterthiner, J\. Yung, A\. Steiner, D\. Keysers, J\. Uszkoreit,et al\.\(2021\)Mlp\-mixer: an all\-mlp architecture for vision\.NeurIPS34,pp\. 24261–24272\.Cited by:[§5\.2](https://arxiv.org/html/2608.29099#S5.SS2.SSS0.Px4.p1.1)\.
- \[29\]F\. Tung and G\. Mori\(2019\)Similarity\-preserving knowledge distillation\.InICCV,pp\. 1365–1374\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1)\.
- \[30\]E\. Yang, S\. M\. Hamidi, L\. Ye, R\. Tan, and B\. Yang\(2025\)Conditional mutual information constrained deep learning for classification\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§4\.1](https://arxiv.org/html/2608.29099#S4.SS1.SSS0.Px2.p1.1)\.
- \[31\]E\. Yang and L\. Ye\(2024\)Markov knowledge distillation: make nasty teachers trained by self\-undermining knowledge distillation fully distillable\.InECCV,pp\. 154–171\.Cited by:[§4\.1](https://arxiv.org/html/2608.29099#S4.SS1.SSS0.Px2.p1.1)\.
- \[32\]S\. Yang, X\. Yang, J\. Ren, L\. Xu, J\. Yang, Z\. Huang, Z\. Gong, and W\. Wang\(2025\)Adaptive temperature distillation method for mining hard samples’ knowledge\.Neurocomputing636,pp\. 129745\.Cited by:[Appendix B](https://arxiv.org/html/2608.29099#A2.p1.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1)\.
- \[33\]L\. Ye, S\. M\. Hamidi, and E\. YANG\(2025\)Towards undistillable models by minimizing conditional mutual information\.TMLR\.Cited by:[§4\.1](https://arxiv.org/html/2608.29099#S4.SS1.SSS0.Px2.p1.1)\.
- \[34\]S\. Zagoruyko and N\. Komodakis\(2017\)Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer\.InICLR,Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[35\]B\. Zhao, Q\. Cui, R\. Song, Y\. Qiu, and J\. Liang\(2022\)Decoupled knowledge distillation\.InCVPR,pp\. 11953–11962\.Cited by:[§2](https://arxiv.org/html/2608.29099#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.
- \[36\]D\. Zhao, Y\. S\. Koh, G\. Dobbie, H\. Hu, and P\. Fournier\-Viger\(2024\)Symmetric self\-paced learning for domain generalization\.InAAAI,Vol\.38,pp\. 16961–16969\.Cited by:[§4\.2](https://arxiv.org/html/2608.29099#S4.SS2.SSS0.Px4.p1.1)\.
- \[37\]K\. Zheng and E\. YANG\(2024\)Knowledge distillation based on transformed teacher matching\.InICLR,Cited by:[§1](https://arxiv.org/html/2608.29099#S1.p3.1),[§2](https://arxiv.org/html/2608.29099#S2.p2.1),[§3\.2](https://arxiv.org/html/2608.29099#S3.SS2.SSS0.Px2),[§3\.2](https://arxiv.org/html/2608.29099#S3.SS2.SSS0.Px3),[Table 1](https://arxiv.org/html/2608.29099#S4.T1),[§5\.1](https://arxiv.org/html/2608.29099#S5.SS1.SSS0.Px1.p1.1)\.

## Appendix ADerivations for Per\-Sample Temperature Adaptation

### A\.1Sample\-wise optimization of the teacher temperature

We interpret the teacher temperatureTTin TTM viaγ=1/T\\gamma=1/Tand optimizeγ\\gammaper sample\. Let the teacher\-side transformed distribution and the student prediction be

p\(γ\)=softmax⁡\(γ​𝐳t\),q=softmax⁡\(𝐳s\),γ\>0\.p^\{\(\\gamma\)\}=\\mathrm\{softmax\}\(\\gamma\\,\\mathbf\{z\}\_\{t\}\),\\qquad q=\\mathrm\{softmax\}\(\\mathbf\{z\}\_\{s\}\),\\qquad\\gamma\>0\.\(14\)For a fixed pair\(𝐳t,𝐳s\)\(\\mathbf\{z\}\_\{t\},\\mathbf\{z\}\_\{s\}\), we minimize the TTM discrepancy

DKL\(p\(γ\)∥q\)=∑i=1Kpi\(γ\)logpi\(γ\)qi\.D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}\\log\\frac\{p^\{\(\\gamma\)\}\_\{i\}\}\{q\_\{i\}\}\.\(15\)
#### Remark \(stop\-gradient\)\.

In the following derivations,𝐳t\\mathbf\{z\}\_\{t\}and𝐳s\\mathbf\{z\}\_\{s\}\(henceqq\) are treated as constants when differentiating with respect toγ\\gamma\. In implementation, this corresponds to computing theγ\\gamma\-update with stop\-gradient on logits\.

### A\.2Log\-sum\-exp identities and an expectation derivative lemma

We start from thelog\-sum\-expform of the softmax:

pi\(γ\)=exp⁡\(γ​zt,i\)∑j=1Kexp⁡\(γ​zt,j\)=exp\(γzt,i−log∑j=1Kexp\(γzt,j\)\)p^\{\(\\gamma\)\}\_\{i\}=\\frac\{\\exp\(\\gamma z\_\{t,i\}\)\}\{\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\}=\\exp\\left\(\\gamma z\_\{t,i\}\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\\right\)\(16\)and

logpi\(γ\)=γzt,i−log∑j=1Kexp\(γzt,j\)\.\\log p^\{\(\\gamma\)\}\_\{i\}=\\gamma z\_\{t,i\}\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\.\(17\)Differentiatinglog⁡pi\(γ\)\\log p^\{\(\\gamma\)\}\_\{i\}with respect toγ\\gammayields

∂∂γ​log⁡pi\(γ\)\\displaystyle\\frac\{\\partial\}\{\\partial\\gamma\}\\log p^\{\(\\gamma\)\}\_\{i\}=zt,i−∑j=1Kzt,j​exp⁡\(γ​zt,j\)∑j=1Kexp⁡\(γ​zt,j\)=zt,i−∑j=1Kpj\(γ\)​zt,j\.\\displaystyle=z\_\{t,i\}\-\\frac\{\\sum\_\{j=1\}^\{K\}z\_\{t,j\}\\exp\(\\gamma z\_\{t,j\}\)\}\{\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\}=z\_\{t,i\}\-\\sum\_\{j=1\}^\{K\}p^\{\(\\gamma\)\}\_\{j\}z\_\{t,j\}\.\(18\)Define thep\(γ\)p^\{\(\\gamma\)\}\-weighted mean of teacher logits

mt≜∑j=1Kpj\(γ\)​zt,j=𝔼p\(γ\)​\[zt\]\.m\_\{t\}\\triangleq\\sum\_\{j=1\}^\{K\}p^\{\(\\gamma\)\}\_\{j\}z\_\{t,j\}=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\.\(19\)Then Eq\. \([18](https://arxiv.org/html/2608.29099#A1.E18)\) becomes

∂∂γ​log⁡pi\(γ\)=zt,i−mt\.\\frac\{\\partial\}\{\\partial\\gamma\}\\log p^\{\(\\gamma\)\}\_\{i\}=z\_\{t,i\}\-m\_\{t\}\.\(20\)
If the teacher logit are normalized as centered teacher logitz^t,i≜zt,i−mt\\hat\{z\}\_\{t,i\}\\triangleq z\_\{t,i\}\-m\_\{t\}, then Eq\. \([18](https://arxiv.org/html/2608.29099#A1.E18)\) is

∂∂γ​log⁡pi\(γ\)=z^t,i\.\\frac\{\\partial\}\{\\partial\\gamma\}\\log p^\{\(\\gamma\)\}\_\{i\}=\\hat\{z\}\_\{t,i\}\.\(21\)
For the derivative∂∂γ​pi\(γ\)\\frac\{\\partial\}\{\\partial\\gamma\}p^\{\(\\gamma\)\}\_\{i\}, we obtain it by the chain rule andlog\-sum\-expform:

∂∂γ​pi\(γ\)\\displaystyle\\frac\{\\partial\}\{\\partial\\gamma\}p^\{\(\\gamma\)\}\_\{i\}=∂∂γexp\(γzt,i−log∑j=1Kexp\(γzt,j\)\)\\displaystyle=\\frac\{\\partial\}\{\\partial\\gamma\}\\exp\\left\(\\gamma z\_\{t,i\}\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\\right\)\(22\)=p\(γ\)i⋅∂∂γ\(γzt,i−log∑j=1Kexp\(γzt,j\)\)\\displaystyle=p^\{\(\\gamma\)\}\_\{i\}\\cdot\\frac\{\\partial\}\{\\partial\\gamma\}\\left\(\\gamma z\_\{t,i\}\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\\right\)=pi\(γ\)⋅∂∂γ​log⁡pi\(γ\)\\displaystyle=p^\{\(\\gamma\)\}\_\{i\}\\cdot\\frac\{\\partial\}\{\\partial\\gamma\}\\log p^\{\(\\gamma\)\}\_\{i\}=pi\(γ\)​z^t,i\.\\displaystyle=p^\{\(\\gamma\)\}\_\{i\}\\,\\hat\{z\}\_\{t,i\}\.
###### Lemma 1\.

For any vectorf=\(fi\)i=1Kf=\(f\_\{i\}\)\_\{i=1\}^\{K\}independent ofγ\\gamma, define the class\-wise expectation𝔼p\(γ\)​\[f\]≜∑i=1Kpi\(γ\)​fi\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\]\\triangleq\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}f\_\{i\}\. Then

dd​γ​𝔼p\(γ\)​\[f\]=𝔼p\(γ\)​\[f​z^t\]=Covp\(γ\)​\(f,zt\),\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\]=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\\,\\hat\{z\}\_\{t\}\]=\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(f,z\_\{t\}\),\(23\)where products are element\-wise andCovp\(γ\)​\(f,zt\)\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(f,z\_\{t\}\)is the covariance under the weightingp\(γ\)p^\{\(\\gamma\)\}\.

###### Proof\.

Using Eq\. \([22](https://arxiv.org/html/2608.29099#A1.E22)\),

dd​γ​𝔼p\(γ\)​\[f\]\\displaystyle\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\]=∑id​pi\(γ\)d​γ​fi\\displaystyle=\\sum\_\{i\}\\frac\{dp^\{\(\\gamma\)\}\_\{i\}\}\{d\\gamma\}\\,f\_\{i\}\(24\)=∑ipi\(γ\)​z^t,i​fi\\displaystyle=\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}\\hat\{z\}\_\{t,i\}f\_\{i\}=𝔼p\(γ\)​\[f​z^t\]\.\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\\,\\hat\{z\}\_\{t\}\]\.Then we have

dd​γ​𝔼p\(γ\)​\[f\]\\displaystyle\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\]=∑ipi\(γ\)​z^t,i​fi\\displaystyle=\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}\\hat\{z\}\_\{t,i\}f\_\{i\}\(25\)=∑ipi\(γ\)​\(zt,i−mt\)​fi\\displaystyle=\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}\(z\_\{t,i\}\-m\_\{t\}\)f\_\{i\}=∑ipi\(γ\)​zt,i​fi−∑ipi\(γ\)​mt​fi\\displaystyle=\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}z\_\{t,i\}f\_\{i\}\-\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}m\_\{t\}f\_\{i\}=∑ipi\(γ\)​zt,i​fi−mt​∑ipi\(γ\)​fi\\displaystyle=\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}z\_\{t,i\}f\_\{i\}\-m\_\{t\}\\sum\_\{i\}p^\{\(\\gamma\)\}\_\{i\}f\_\{i\}=𝔼p\(γ\)​\[f​zt\]−𝔼p\(γ\)​\[zt\]​𝔼p\(γ\)​\[f\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\\,z\_\{t\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[f\]=Covp\(γ\)​\(f,zt\)\.\\displaystyle=\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(f,z\_\{t\}\)\.∎

###### Corollary 1\.

Takingf=ztf=z\_\{t\}in Eq\. \([23](https://arxiv.org/html/2608.29099#A1.E23)\) gives

dd​γ​𝔼p\(γ\)​\[zt\]=𝔼p\(γ\)​\[z^t2\]=Varp\(γ\)​\(zt\)\.\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]=\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)\.\(26\)

### A\.3Expanding the KL discrepancy

From Eq\. \([15](https://arxiv.org/html/2608.29099#A1.E15)\),

DKL\(p\(γ\)∥q\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=∑i=1Kpi\(γ\)​\(log⁡pi\(γ\)−log⁡qi\)\\displaystyle=\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}\\bigl\(\\log p^\{\(\\gamma\)\}\_\{i\}\-\\log q\_\{i\}\\bigr\)\(27\)=∑i=1Kp\(γ\)i\(γzt,i−log∑j=1Kexp\(γzt,j\)−logqi\)\\displaystyle=\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}\\left\(\\gamma z\_\{t,i\}\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\-\\log q\_\{i\}\\right\)=γ​∑i=1Kpi\(γ\)​zt,i−∑i=1Kpi\(γ\)​log​∑j=1Kexp⁡\(γ​zt,j\)−∑i=1Kpi\(γ\)​log​qi\\displaystyle=\\gamma\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}z\_\{t,i\}\-\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\-\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}\\log q\_\{i\}=γ∑i=1Kp\(γ\)izt,i−log∑j=1Kexp\(γzt,j\)∑i=1Kp\(γ\)i−∑i=1Kp\(γ\)ilogqi\\displaystyle=\\gamma\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}z\_\{t,i\}\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}\-\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}\\log q\_\{i\}=γ𝔼p\(γ\)\[zt\]−log∑j=1Kexp\(γzt,j\)−𝔼p\(γ\)\[logq\],\\displaystyle=\\gamma\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\log q\],where we use∑i=1Kpi\(γ\)=1\\sum\_\{i=1\}^\{K\}p^\{\(\\gamma\)\}\_\{i\}=1\.

### A\.4First derivative

Differentiate Eq\. \([27](https://arxiv.org/html/2608.29099#A1.E27)\) with respect toγ\\gamma:

dd​γDKL\(p\(γ\)∥q\)\\displaystyle\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=dd​γ\{γ𝔼p\(γ\)\[zt\]−log∑j=1Kexp\(γzt,j\)−𝔼p\(γ\)\[logq\]\}\\displaystyle=\\frac\{d\}\{d\\gamma\}\\left\\\{\\gamma\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\-\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\log q\]\\right\\\}\(28\)=dd​γ​γ​𝔼p\(γ\)​\[zt\]−dd​γ​log​∑j=1Kexp⁡\(γ​zt,j\)−dd​γ​𝔼p\(γ\)​\[log⁡q\]\\displaystyle=\\frac\{d\}\{d\\gamma\}\\gamma\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\-\\frac\{d\}\{d\\gamma\}\\log\\sum\_\{j=1\}^\{K\}\\exp\(\\gamma z\_\{t,j\}\)\-\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\log q\]=𝔼p\(γ\)​\[zt\]\+γ​dd​γ​𝔼p\(γ\)​\[zt\]−𝔼p\(γ\)​\[zt\]−dd​γ​𝔼p\(γ\)​\[log⁡q\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\+\\gamma\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\-\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\log q\]=γ​dd​γ​𝔼p\(γ\)​\[zt\]−dd​γ​𝔼p\(γ\)​\[log⁡q\]\\displaystyle=\\gamma\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\-\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\log q\]By the lemma and corollary, we obtain

dd​γDKL\(p\(γ\)∥q\)\\displaystyle\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=γ​𝔼p\(γ\)​\[z^t2\]−𝔼p\(γ\)​\[z^t​log⁡q\]\\displaystyle=\\gamma\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}\\,\\log q\]\(29\)=γ​Varp\(γ\)​\(zt\)−Covp\(γ\)​\(log⁡q,zt\)\.\\displaystyle=\\gamma\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)\-\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(\\log q,z\_\{t\}\)\.
Next, note that student log\-probabilities satisfylogqi=zs,i−log∑jezs,j\\log q\_\{i\}=z\_\{s,i\}\-\\log\\sum\_\{j\}e^\{z\_\{s,j\}\}\. Since the second term is class\-independent constantcc, it does not affect covariance withztz\_\{t\}, hence

Covp\(γ\)​\(log⁡q,zt\)\\displaystyle\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(\\log q,z\_\{t\}\)=𝔼p\(γ\)​\[\(log⁡q\)​zt\]−𝔼p\(γ\)​\[log⁡q\]​𝔼p\(γ\)​\[zt\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(\\log q\)\\,z\_\{t\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\log q\]\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\(30\)=𝔼p\(γ\)​\[\(zs−c\)​zt\]−𝔼p\(γ\)​\[zs−c\]​𝔼p\(γ\)​\[zt\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{s\}\-c\)\\,z\_\{t\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\-c\]\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]=\(𝔼p\(γ\)​\[zs​zt\]−c​𝔼p\(γ\)​\[zt\]\)−\(𝔼p\(γ\)​\[zs\]−c\)​𝔼p\(γ\)​\[zt\]\\displaystyle=\\Bigl\(\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\]\-c\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\\Bigr\)\-\\Bigl\(\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\]\-c\\Bigr\)\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]=𝔼p\(γ\)​\[zs​zt\]−𝔼p\(γ\)​\[zs\]​𝔼p\(γ\)​\[zt\]=Covp\(γ\)​\(zs,zt\)\.\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\]\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]=\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},z\_\{t\}\)\.
Therefore, we obtain

dd​γDKL\(p\(γ\)∥q\)=γVarp\(γ\)\(zt\)−Covp\(γ\)\(zs,zt\)\.\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=\\gamma\\,\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)\-\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},z\_\{t\}\)\.\(31\)

### A\.5Second derivative

Before differentiating Eq\. \([31](https://arxiv.org/html/2608.29099#A1.E31)\), we rearrange it using the centered logitsz^t,z^s\\hat\{z\}\_\{t\},\\hat\{z\}\_\{s\}\. In addition tomtm\_\{t\}andz^t\\hat\{z\}\_\{t\}defined above, letms≜𝔼p\(γ\)​\[zs\]m\_\{s\}\\triangleq\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\]andz^s≜zs−ms\\hat\{z\}\_\{s\}\\triangleq z\_\{s\}\-m\_\{s\}element\-wise\. By the definitions of variance and covariance,

Varp\(γ\)​\(zt\)\\displaystyle\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)=𝔼p\(γ\)​\[\(zt−mt\)2\]=𝔼p\(γ\)​\[z^t2\],\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{t\}\-m\_\{t\}\)^\{2\}\]=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\],\(32\)Covp\(γ\)​\(zs,zt\)\\displaystyle\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},z\_\{t\}\)=𝔼p\(γ\)​\[\(zs−ms\)​\(zt−mt\)\]=𝔼p\(γ\)​\[z^s​z^t\]\.\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{s\}\-m\_\{s\}\)\(z\_\{t\}\-m\_\{t\}\)\]=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\,\\hat\{z\}\_\{t\}\]\.Substituting these results into first derivative yeilds the following expression:

dd​γDKL\(p\(γ\)∥q\)=γ𝔼p\(γ\)\[z^t2\]−𝔼p\(γ\)\[z^sz^t\]\.\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=\\gamma\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\,\\hat\{z\}\_\{t\}\]\.\(33\)
For calculating the second derivative, let

a⁡\(γ\)≜𝔼p\(γ\)​\[z^t2\],b⁡\(γ\)≜𝔼p\(γ\)​\[z^s​z^t\]\.a\(\\gamma\)\\triangleq\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\],\\qquad b\(\\gamma\)\\triangleq\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\,\\hat\{z\}\_\{t\}\]\.\(34\)Thendd​γDKL\(p\(γ\)∥q\)=γa\(γ\)−b\(γ\)\\frac\{d\}\{d\\gamma\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=\\gamma a\(\\gamma\)\-b\(\\gamma\)and

d2d​γ2DKL\(p\(γ\)∥q\)=a\(γ\)\+γa′\(γ\)−b′\(γ\)\.\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=a\(\\gamma\)\+\\gamma a^\{\\prime\}\(\\gamma\)\-b^\{\\prime\}\(\\gamma\)\.\(35\)Althoughz^t,z^s\\hat\{z\}\_\{t\},\\hat\{z\}\_\{s\}depend onγ\\gammathroughmt,msm\_\{t\},m\_\{s\}, the derivatives can be computed by expanding into these terms\.

First, we expanda⁡\(γ\)a\(\\gamma\):

a⁡\(γ\)=𝔼p\(γ\)​\[\(zt−mt\)2\]\\displaystyle a\(\\gamma\)=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{t\}\-m\_\{t\}\)^\{2\}\]=𝔼p\(γ\)​\[zt2\]−2​mt​𝔼p\(γ\)​\[zt\]\+mt2\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}^\{2\}\]\-2m\_\{t\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\+m\_\{t\}^\{2\}\(36\)=𝔼p\(γ\)​\[zt2\]−mt2,\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}^\{2\}\]\-m\_\{t\}^\{2\},since𝔼p\(γ\)​\[zt\]=mt\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]=m\_\{t\}\. Differentiating this equation gives

a′​\(γ\)=dd​γ​𝔼p\(γ\)​\[zt2\]−2​mt​mt′\.a^\{\\prime\}\(\\gamma\)=\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}^\{2\}\]\-2m\_\{t\}\\,m\_\{t\}^\{\\prime\}\.\(37\)Becausezt2z\_\{t\}^\{2\}isγ\\gamma\-independent, we can apply the lemma

dd​γ​𝔼p\(γ\)​\[zt2\]=𝔼p\(γ\)​\[zt2​z^t\]\.\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}^\{2\}\]=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}^\{2\}\\,\\hat\{z\}\_\{t\}\]\.\(38\)Similarly, applying the lemma withf=ztf=z\_\{t\}yields

mt′\\displaystyle m\_\{t\}^\{\\prime\}=dd​γ​𝔼p\(γ\)​\[zt\]\\displaystyle=\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\(39\)=𝔼p\(γ\)​\[zt​z^t\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\\,\\hat\{z\}\_\{t\}\]=𝔼p\(γ\)​\[\(zt−mt\)​zt\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{t\}\-m\_\{t\}\)z\_\{t\}\]=𝔼p\(γ\)​\[z^t​\(z^t\+mt\)\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\\\!\\left\[\\hat\{z\}\_\{t\}\(\\hat\{z\}\_\{t\}\+m\_\{t\}\)\\right\]=𝔼p\(γ\)​\[z^t2\]\+mt​𝔼p\(γ\)​\[z^t\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\+m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}\]=𝔼p\(γ\)\[z^t2\]\(∵𝔼p\(γ\)\[z^t\]=0\)\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\\qquad\(\\because~\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}\]=0\)=a⁡\(γ\)\.\\displaystyle=a\(\\gamma\)\.Substituting these results intoa′​\(γ\)a^\{\\prime\}\(\\gamma\)gives

a′​\(γ\)\\displaystyle a^\{\\prime\}\(\\gamma\)=𝔼p\(γ\)​\[zt2​z^t\]−2​mt​𝔼p\(γ\)​\[z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}^\{2\}\\,\\hat\{z\}\_\{t\}\]\-2m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\(40\)=𝔼p\(γ\)​\[\(z^t\+mt\)2​z^t\]−2​mt​𝔼p\(γ\)​\[z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\\big\[\(\\hat\{z\}\_\{t\}\+m\_\{t\}\)^\{2\}\\,\\hat\{z\}\_\{t\}\\big\]\-2m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]=𝔼p\(γ\)​\[\(z^t2\+2​mt​z^t\+mt2\)​z^t\]−2​mt​𝔼p\(γ\)​\[z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\\big\[\(\\hat\{z\}\_\{t\}^\{2\}\+2m\_\{t\}\\hat\{z\}\_\{t\}\+m\_\{t\}^\{2\}\)\\hat\{z\}\_\{t\}\\big\]\-2m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]=𝔼p\(γ\)​\[z^t3\]\+2​mt​𝔼p\(γ\)​\[z^t2\]\+mt2​𝔼p\(γ\)​\[z^t\]−2​mt​𝔼p\(γ\)​\[z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,3\}\]\+2m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\+m\_\{t\}^\{2\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}\]\-2m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]=𝔼p\(γ\)​\[z^t3\],\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,3\}\],where we used𝔼p\(γ\)​\[z^t\]=𝔼p\(γ\)​\[zt−mt\]=0\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}\]=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\-m\_\{t\}\]=0\.

Next, we derive the derivative ofb⁡\(γ\)b\(\\gamma\)\. we expand

b⁡\(γ\)\\displaystyle b\(\\gamma\)=𝔼p\(γ\)​\[\(zs−ms\)​\(zt−mt\)\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{s\}\-m\_\{s\}\)\(z\_\{t\}\-m\_\{t\}\)\]\(41\)=𝔼p\(γ\)​\[zs​zt\]−ms​𝔼p\(γ\)​\[zt\]−mt​𝔼p\(γ\)​\[zs\]\+ms​mt\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\]\-m\_\{s\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]\-m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\]\+m\_\{s\}m\_\{t\}=𝔼p\(γ\)​\[zs​zt\]−ms​mt,\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\]\-m\_\{s\}m\_\{t\},since𝔼p\(γ\)​\[zt\]=mt\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{t\}\]=m\_\{t\}and𝔼p\(γ\)​\[zs\]=ms\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\]=m\_\{s\}\. Differentiating this equation yields

b′​\(γ\)=dd​γ​𝔼p\(γ\)​\[zs​zt\]−ms′​mt−ms​mt′\.b^\{\\prime\}\(\\gamma\)=\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\]\-m\_\{s\}^\{\\prime\}m\_\{t\}\-m\_\{s\}m\_\{t\}^\{\\prime\}\.\(42\)Becausezs​ztz\_\{s\}z\_\{t\}isγ\\gamma\-independent \(logits are treated as fixed w\.r\.t\.γ\\gamma\), we apply the lemma withf=zs​ztf=z\_\{s\}z\_\{t\}:

dd​γ​𝔼p\(γ\)​\[zs​zt\]=𝔼p\(γ\)​\[zs​zt​z^t\]\.\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\]=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\\,\\hat\{z\}\_\{t\}\]\.\(43\)Moreover, applying the lemma withf=zsf=z\_\{s\}gives

ms′\\displaystyle m\_\{s\}^\{\\prime\}=dd​γ​𝔼p\(γ\)​\[zs\]\\displaystyle=\\frac\{d\}\{d\\gamma\}\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\]\(44\)=𝔼p\(γ\)​\[zs​z^t\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\\,\\hat\{z\}\_\{t\}\]=𝔼p\(γ\)​\[\(\(zs−ms\)\+ms\)​z^t\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\\\!\\left\[\\bigl\(\(z\_\{s\}\-m\_\{s\}\)\+m\_\{s\}\\bigr\)\\hat\{z\}\_\{t\}\\right\]=𝔼p\(γ\)​\[\(zs−ms\)​z^t\]\+ms​𝔼p\(γ\)​\[z^t\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{s\}\-m\_\{s\}\)\\hat\{z\}\_\{t\}\]\+m\_\{s\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}\]=𝔼p\(γ\)\[\(zs−ms\)z^t\]\(∵𝔼p\(γ\)\[z^t\]=0\)\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(z\_\{s\}\-m\_\{s\}\)\\hat\{z\}\_\{t\}\]\\qquad\(\\because~\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}\]=0\)=𝔼p\(γ\)​\[z^s​z^t\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\,\\hat\{z\}\_\{t\}\]=b⁡\(γ\)\.\\displaystyle=b\(\\gamma\)\.and we already havemt′=a⁡\(γ\)m\_\{t\}^\{\\prime\}=a\(\\gamma\)\. Substituting these results intob′​\(γ\)b^\{\\prime\}\(\\gamma\), we obtain

b′​\(γ\)\\displaystyle b^\{\\prime\}\(\\gamma\)=𝔼p\(γ\)​\[zs​zt​z^t\]−b⁡\(γ\)​mt−ms​a​\(γ\)\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}z\_\{t\}\\,\\hat\{z\}\_\{t\}\]\-b\(\\gamma\)m\_\{t\}\-m\_\{s\}a\(\\gamma\)\(45\)=𝔼p\(γ\)​\[zs​\(z^t\+mt\)​z^t\]−mt​𝔼p\(γ\)​\[z^s​z^t\]−ms​𝔼p\(γ\)​\[z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\\big\[z\_\{s\}\(\\hat\{z\}\_\{t\}\+m\_\{t\}\)\\hat\{z\}\_\{t\}\\big\]\-m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\hat\{z\}\_\{t\}\]\-m\_\{s\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]=𝔼p\(γ\)​\[zs​z^t2\]\+mt​𝔼p\(γ\)​\[zs​z^t\]−mt​𝔼p\(γ\)​\[z^s​z^t\]−ms​𝔼p\(γ\)​\[z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\\hat\{z\}\_\{t\}^\{\\,2\}\]\+m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\\hat\{z\}\_\{t\}\]\-m\_\{t\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\hat\{z\}\_\{t\}\]\-m\_\{s\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]=𝔼p\(γ\)​\[zs​z^t2\]−ms​a​\(γ\),\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\\hat\{z\}\_\{t\}^\{\\,2\}\]\-m\_\{s\}\\,a\(\\gamma\),sincems′=b⁡\(γ\)m\_\{s\}^\{\\prime\}=b\(\\gamma\)\. Finally, writezs=z^s\+msz\_\{s\}=\\hat\{z\}\_\{s\}\+m\_\{s\}:

𝔼p\(γ\)​\[zs​z^t2\]−ms​a​\(γ\)\\displaystyle\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[z\_\{s\}\\hat\{z\}\_\{t\}^\{\\,2\}\]\-m\_\{s\}\\,a\(\\gamma\)=𝔼p\(γ\)​\[\(z^s\+ms\)​z^t2\]−ms​𝔼p\(γ\)​\[z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\(\\hat\{z\}\_\{s\}\+m\_\{s\}\)\\hat\{z\}\_\{t\}^\{\\,2\}\]\-m\_\{s\}\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\(46\)=𝔼p\(γ\)​\[z^s​z^t2\]\.\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\,\\hat\{z\}\_\{t\}^\{\\,2\}\]\.
Finally, we obtain

d2d​γ2DKL\(p\(γ\)∥q\)\\displaystyle\\frac\{d^\{2\}\}\{d\\gamma^\{2\}\}D\_\{\\mathrm\{KL\}\}\\\!\\bigl\(p^\{\(\\gamma\)\}\\\|q\\bigr\)=𝔼p\(γ\)​\[z^t2\]\+γ​𝔼p\(γ\)​\[z^t3\]−𝔼p\(γ\)​\[z^s​z^t2\]\\displaystyle=\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,2\}\]\+\\gamma\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,3\}\]\-\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{s\}\\,\\hat\{z\}\_\{t\}^\{\\,2\}\]\(47\)=Varp\(γ\)​\(zt\)−Covp\(γ\)​\(zs,z^t2\)\+γ​𝔼p\(γ\)​\[z^t3\]\.\\displaystyle=\\mathrm\{Var\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{t\}\)\-\\mathrm\{Cov\}\_\{p^\{\(\\gamma\)\}\}\(z\_\{s\},\\hat\{z\}\_\{t\}^\{\\,2\}\)\+\\gamma\\,\\mathbb\{E\}\_\{p^\{\(\\gamma\)\}\}\[\\hat\{z\}\_\{t\}^\{\\,3\}\]\.

## Appendix BComparison with Adaptive Temperature Distillation

Adaptive Temperature Distillation \(ATD\)\[[32](https://arxiv.org/html/2608.29099#bib.bib17)\]precomputes sample\-wise temperatures from the difficulty estimated by a pretrained teacher\. For a fair comparison of temperature\-adaptation strategies, we use the CNN\-RIS\-AT setting from ATD on CIFAR100 with WRN\-40\-2→\\rightarrowWRN\-16\-2 and apply only its temperature regulation strategy, without Mixup\.

Table 8:Comparison with ATD on CIFAR100 for WRN\-40\-2→\\rightarrowWRN\-16\-2\. Results are Best Acc@1\.TA\-TTM and TA\-WTTM outperform ATD by 0\.59 and 0\.74 percentage points, respectively, while also improving over TTM and WTTM\. The methods differ in when and how the sample\-wise temperatures are determined: ATD computes them from pretrained\-teacher difficulty before distillation, whereas our method maintains and repeatedly updates each sample’sγ\\gammaduring training from the current teacher–student mismatch within the TTM objective\.

## Appendix CAdditional Diagnostics of Temperature Adaptation

### C\.1Cross\-architecture temperature diagnostics

Figure[5](https://arxiv.org/html/2608.29099#A3.F5)complements the same\-architecture diagnostics in the main paper with the cross\-architecture pair WRN\-40\-2→\\rightarrowShuffleNetV1\. TA\-TTM and TA\-WTTM show the same qualitative pattern: the inverse\-temperature distributions broaden during training, and larger per\-sample cross\-entropy losses tend to be associated with smallerγ\\gamma\.

\(a\) TA\-TTM dynamics

\(b\) TA\-TTM distribution

![Refer to caption](https://arxiv.org/html/2608.29099v1/x6.png)

\(c\) TA\-TTM correlation

\(d\) TA\-WTTM dynamics

\(e\) TA\-WTTM distribution

![Refer to caption](https://arxiv.org/html/2608.29099v1/x8.png)

\(f\) TA\-WTTM correlation

Figure 5:Cross\-architecture temperature diagnostics for WRN\-40\-2→\\rightarrowShuffleNetV1\. The first and second rows show TA\-TTM and TA\-WTTM, respectively; the columns show temperature dynamics, stage\-wise inverse\-temperature distributions, and correlations with per\-sample cross\-entropy loss\.
### C\.2Curvature and clipping

We examine the curvature used by the temperature update for WRN\-40\-2→\\rightarrowWRN\-16\-2\. Table[9](https://arxiv.org/html/2608.29099#A3.T9)reports statistics of the estimated curvaturehupdateh\_\{\\mathrm\{update\}\}reconstructed over all 50,000 training samples at three update epochs\. Negative curvature occurs primarily at epoch 100, but only for 0\.164% of samples for either method\. It disappears for TA\-TTM at epochs 200 and 240, and falls to 0\.004% and then 0% for TA\-WTTM\. We observe no samples with\|hupdate\|<10−8\|h\_\{\\mathrm\{update\}\}\|<10^\{\-8\}, the threshold associated with the denominator guard, and only 0\.002% with\|hupdate\|<10−3\|h\_\{\\mathrm\{update\}\}\|<10^\{\-3\}in one setting\. Consistent with these curvature statistics, clipping is almost never activated\. Thus, damping and clipping act mainly as numerical safeguards in this experiment rather than routinely determining the temperature update\.

Table 9:Curvature diagnostics for WRN\-40\-2→\\rightarrowWRN\-16\-2, evaluated over all 50,000 training samples\. The last three columns report the percentage of samples satisfying each condition\.
### C\.3Correlation between inverse temperature and sample difficulty

We quantify the relationship visualized in the main\-paper diagnostic figures using the same protocol of 1,000 samples per epoch\. Table[10](https://arxiv.org/html/2608.29099#A3.T10)reports both Pearson and Spearman correlations between the learned inverse temperatureγ\\gammaand the per\-sample cross\-entropy loss for WRN\-40\-2→\\rightarrowWRN\-16\-2\. All coefficients are negative at epochs 100, 200, and 240 for both methods, with Pearson correlations ranging from−0\.713\-0\.713to−0\.645\-0\.645and Spearman correlations from−0\.744\-0\.744to−0\.681\-0\.681\. Across all 13 settings used for the paper figures, the mean Pearson correlation is−0\.611\-0\.611for TA\-TTM and−0\.595\-0\.595for TA\-WTTM\. These quantitative results corroborate the negative trend shown in the main\-paper diagnostic figures and summarized in its conclusion: samples with higher cross\-entropy loss tend to receive smallerγ\\gamma, corresponding to softer teacher targets\.

Table 10:Correlation between the learned inverse temperatureγ\\gammaand per\-sample cross\-entropy loss for WRN\-40\-2→\\rightarrowWRN\-16\-2, using 1,000 samples per epoch\.

## Appendix DHyperparameters

For CIFAR100, Table[11](https://arxiv.org/html/2608.29099#A4.T11)summarizes the hyperparameters of our proposed method\. We use the original TTM and WTTM configurations for the initial inverse temperatureγinit\\gamma\_\{\\mathrm\{init\}\}andβ\\beta, and grid\-search the additional step sizeη\\etaover\{0\.001,0\.002,…,0\.005\}\\\{0\.001,0\.002,\\ldots,0\.005\\\}\. For ImageNet1k, we use the TTM configuration and grid\-searchη\\etaover the same range\.

Table 11:Hyperparameters used for each teacher–student pair\.