Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

# Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation
Source: [https://arxiv.org/html/2609.13199](https://arxiv.org/html/2609.13199)
## Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge DistillationDOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN:978\-1\-4503\-XXXX\-X/2018/06CCS:Computing methodologies Machine learning Learning paradigms Supervised learning

Dawen JiangAffiliation:Wuhan University of Technology,Wuhan,Hubei,Chinaemail:[davin@whut\.edu\.cn](mailto:[email protected])OrcID:[0009\-0006\-5898\-4403](https://orcid.org/0009-0006-5898-4403)Zhishu ShenNote:Zhishu Shen is the corresponding author\.Affiliation:Wuhan University of Technology,Wuhan,Hubei,Chinaemail:[z\_shen@ieee\.org](mailto:[email protected])OrcID:[0000\-0002\-3123\-4390](https://orcid.org/0000-0002-3123-4390),Zeyu LiuAffiliation:Wuhan University of Technology,Wuhan,Hubei,Chinaemail:[357932@whut\.edu\.cn](mailto:[email protected])OrcID:[0009\-0009\-5083\-3856](https://orcid.org/0009-0009-5083-3856)andTiehua ZhangAffiliation:Tongji University,Shanghai,Chinaemail:[tiehuaz@tongji\.edu\.cn](mailto:[email protected])OrcID:[0000\-0002\-7195\-4472](https://orcid.org/0000-0002-7195-4472)

2018© , 2018;

###### Abstract\.

Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher models\. However, a substantial size gap between teacher and student models often impedes effective knowledge transfer\. Most existing approaches adopt a static, one\-way teacher\-to\-student distillation paradigm, which overlooks the dynamic nature of student learning and fails to provide targeted guidance on hard samples\. In this paper, we propose adaptive reciprocal knowledge distillation \(AR\-KD\), a novel method that improves knowledge transfer by simplifying the teacher’s output distribution\. Specifically, AR\-KD performs reciprocal adaptation on the teacher by matching its class correlation matrix to the student’s relational representation, which reshapes the teacher’s prediction structure to better suit the student’s capacity\. This relational alignment mitigates the collapse of inter\-class dark knowledge caused by overconfident teachers, enabling the student to learn from richer and more compatible supervisory signals\. We evaluate AR\-KD on CIFAR\-100 and ImageNet\-1k classification datasets, where it outperforms state\-of\-the\-art knowledge distillation baselines\. Specifically, AR\-KD improves student performance across homogeneous and heterogeneous setups: up to 7\.13% accuracy gain for students, 1\.42%–4\.15% higher than vanilla KD on average, and further improvements when integrated with other advanced methods\. Our code is available at https://anonymous\.4open\.science/r/ARKD/\.

###### Keywords:

Data mining, Knowledge distillation, Reciprocal learning

## 1\.Introduction

Deep neural networks \(DNNs\)\([9](https://arxiv.org/html/2609.13199#bib.bib10);[20](https://arxiv.org/html/2609.13199#bib.bib11);[30](https://arxiv.org/html/2609.13199#bib.bib16)\)have achieved remarkable success in various computer vision tasks\. Models such as ResNet and Vision Transformers have significantly pushed the boundaries of accuracy by leveraging large\-scale architectures and massive data\. However, the impressive performance of these models comes at the cost of enormous computational and memory demands\. To tackle this challenge, a variety of model compression methods have been proposed\. Knowledge distillation \(KD\)\([12](https://arxiv.org/html/2609.13199#bib.bib1)\)has emerged as one of the most effective and generalizable strategies\. KD aims to transfer the knowledge from a large, well\-trained teacher model to a smaller student model by minimizing the Kullback\-Leibler \(KL\) divergence between their outputs\. To alleviate distillation degradation caused by large teacher–student capacity gaps, Peiet al\.proposed self\-boosting feature distillation \(SFD\)\([28](https://arxiv.org/html/2609.13199#bib.bib36)\), which enhances student representations through feature integration and parameter self\-distillation without additional overhead\. Similarly, Kimet al\.introduced integrating matched features \(IMF\) using attentive logit\([18](https://arxiv.org/html/2609.13199#bib.bib37)\), leveraging intermediate feature distillers and an ENA layer to effectively integrate matched features, achieving improved performance across diverse tasks\.

Numerous KD methods have been developed to improve the performance of compact student models by leveraging the guidance of pre\-trained teachers\. These methods generally fall into two categories: logits\-based distillation\([12](https://arxiv.org/html/2609.13199#bib.bib1);[26](https://arxiv.org/html/2609.13199#bib.bib3);[31](https://arxiv.org/html/2609.13199#bib.bib4);[42](https://arxiv.org/html/2609.13199#bib.bib2);[22](https://arxiv.org/html/2609.13199#bib.bib8);[6](https://arxiv.org/html/2609.13199#bib.bib7);[37](https://arxiv.org/html/2609.13199#bib.bib9);[17](https://arxiv.org/html/2609.13199#bib.bib5);[24](https://arxiv.org/html/2609.13199#bib.bib18);[43](https://arxiv.org/html/2609.13199#bib.bib39);[32](https://arxiv.org/html/2609.13199#bib.bib6)\), which transfers softened output probabilities from teacher to student, and feature\-based distillation\([40](https://arxiv.org/html/2609.13199#bib.bib13);[36](https://arxiv.org/html/2609.13199#bib.bib15);[29](https://arxiv.org/html/2609.13199#bib.bib14);[10](https://arxiv.org/html/2609.13199#bib.bib19);[27](https://arxiv.org/html/2609.13199#bib.bib20);[4](https://arxiv.org/html/2609.13199#bib.bib17);[8](https://arxiv.org/html/2609.13199#bib.bib22);[2](https://arxiv.org/html/2609.13199#bib.bib21)\), which aligns intermediate representations between models\. These methods have consistently demonstrated superior performance compared to training student models from scratch\. However, a substantial architectural gap often hinders effective knowledge transfer, as the student struggles to mimic the teacher’s overconfident and complex output distributions\.

To bridge this gap, recent works have proposed approaches such as softening the teacher outputs, introducing assistant teacher models, and employing multi\-teacher distillation\. However, most existing methods adopt a static, one\-way teacher\-to\-student distillation paradigm, which overlooks the dynamic nature of the student’s learning process\. Moreover, they focus more on enabling the student to capture the teacher’s correctly predicted classes, while ignoring the correlation between categories\. This issue often prevents the student from correctly capturing category correlations\. In large\-scale classification tasks such as e\-commerce product categorization, document topic annotation, and medical code prediction, the category correlation information is essential\.

![Refer to caption](https://arxiv.org/html/2609.13199v1/pic/Figure1.png)Figure 1\.The core motivation of our AR\-KD lies in addressing the capacity gap in traditional KD\. According to\([42](https://arxiv.org/html/2609.13199#bib.bib2)\), the high confidence of teacher outputs results in low weights for non\-target class knowledge \(NCKD\), making it difficult for the student to effectively learn from the dark knowledge\. In contrast, AR\-KD introduces a reciprocal distillation strategy that adaptively simplifies the teacher outputs based on feedback from the student, reducing overconfidence and softening the knowledge, making knowledge transfer more effective and student\-friendly\.In this paper, we propose adaptive reciprocal knowledge distillation \(AR\-KD\), a novel logits\-based distillation framework that enhances the compatibility between teacher and student through reciprocal model adaptation\. As illustrated in Figure[1](https://arxiv.org/html/2609.13199#S1.F1), instead of following a conventional one\-way teacher\-to\-student paradigm, AR\-KD introduces a bidirectional training scheme in which the student also provides feedback to guide the teacher, forming a closed\-loop knowledge refinement process\. Specifically, AR\-KD adopts a recursive distillation pipeline that integrates a reciprocal teacher adaptation mechanism into standard KD\. In the first stage, the student is trained with the original teacher to obtain an initial representation of the task\. In the second stage, the teacher is further adapted by aligning its category correlation structure with that of the student\. This is achieved by matching the class correlation matrices derived from their softened prediction distributions, which allows the teacher to adjust its relational knowledge to better fit the student’s capacity\. Through this reciprocal adaptation, the teacher learns to generate more compatible and transferable outputs for the student\. In the final stage, the adapted teacher is used to distill knowledge back to the student, enabling the student to learn from a teacher whose category correlations are explicitly aligned with its own representation\. This bidirectional relationally\-aware design allows the student to indirectly influence the teacher and leads to a more adaptive and effective distillation process\. Moreover, since AR\-KD operates at the level of class correlation structure, it can be seamlessly integrated with existing KD methods as a plug\-and\-play enhancement module\.

The main contributions of this paper are summarized as follows:

- •We propose a novel logits\-based distillation framework AR\-KD that transforms knowledge transfer into a closed\-loop process\. Rather than treating the teacher as a static supervisor, our approach explicitly allows the student to adapt the teacher, thereby discovering and preserving essential category correlation knowledge that would otherwise be obscured by teacher overconfidence\.
- •We design a relational structure alignment strategy that matches the category correlation matrices derived from the softened prediction distributions of both teacher and student models\. This mechanism enables the teacher to dynamically adjust its relational knowledge, generating highly compatible and transferable outputs tailored to the student’s representation capability\.
- •We validate the effectiveness of our method through extensive experiments conducted on CIFAR\-100 and ImageNet\-1k against state\-of\-the\-art baselines under both homogeneous and heterogeneous teacher–student configurations\.

## 2\.Related Work

KD aims to transfer knowledge from a large, complex teacher model to a lightweight student model, improving the student’s performance with fewer parameters and reduced computational cost\([21](https://arxiv.org/html/2609.13199#bib.bib42);[35](https://arxiv.org/html/2609.13199#bib.bib43);[16](https://arxiv.org/html/2609.13199#bib.bib41)\)\. Since its introduction, numerous studies have been conducted based on this concept\. Current distillation methods can be broadly divided into two categories: logit\-based distillation\([12](https://arxiv.org/html/2609.13199#bib.bib1);[26](https://arxiv.org/html/2609.13199#bib.bib3);[31](https://arxiv.org/html/2609.13199#bib.bib4);[42](https://arxiv.org/html/2609.13199#bib.bib2);[22](https://arxiv.org/html/2609.13199#bib.bib8);[6](https://arxiv.org/html/2609.13199#bib.bib7);[37](https://arxiv.org/html/2609.13199#bib.bib9);[17](https://arxiv.org/html/2609.13199#bib.bib5);[24](https://arxiv.org/html/2609.13199#bib.bib18);[32](https://arxiv.org/html/2609.13199#bib.bib6)\)and intermediate feature\-based distillation\([40](https://arxiv.org/html/2609.13199#bib.bib13);[36](https://arxiv.org/html/2609.13199#bib.bib15);[29](https://arxiv.org/html/2609.13199#bib.bib14);[10](https://arxiv.org/html/2609.13199#bib.bib19);[27](https://arxiv.org/html/2609.13199#bib.bib20);[4](https://arxiv.org/html/2609.13199#bib.bib17);[8](https://arxiv.org/html/2609.13199#bib.bib22);[2](https://arxiv.org/html/2609.13199#bib.bib21)\)\. Sunet al\.\([33](https://arxiv.org/html/2609.13199#bib.bib31)\)proposed Exp\-KD, which adopts class activation map \(CAM\) as the knowledge carrier to unify label\-dependent and structure\-related knowledge, thereby improving interpretability\. Experiments on CIFAR\-100 and ImageNet\-1K demonstrate its superior performance\. Gouet al\.\([7](https://arxiv.org/html/2609.13199#bib.bib32)\)proposed a hierarchical multi\-attention transfer framework \(HMAT\) to transfer hierarchical multi\-attention knowledge in feature\-based distillation, achieving state\-of\-the\-art performance on multiple vision tasks\. Most KD methods rely on a pre\-trained, complex teacher to supervise a simpler student\. However, when the capacity gap is excessively large, the student may struggle to absorb the distilled knowledge, limiting distillation effectiveness\.

To address this issue, several knowledge distillation methods have been proposed\. For example, teacher assistant knowledge distillation \(TAKD\)\([26](https://arxiv.org/html/2609.13199#bib.bib3)\)introduces an intermediate teacher assistant to reduce the capacity gap, but may propagate teacher errors multiple times, causing error accumulation\. To alleviate this problem, densely guided knowledge distillation \(DGKD\)\([31](https://arxiv.org/html/2609.13199#bib.bib4)\)employs tightly guided supervision to reduce compounded errors\. Decoupled knowledge distillation \(DKD\)\([42](https://arxiv.org/html/2609.13199#bib.bib2)\)tackles performance degradation by decoupling the distillation loss and introducing hyperparameters to better handle large teacher–student gaps\. Similarly, Yanget al\.\([38](https://arxiv.org/html/2609.13199#bib.bib30)\)proposed a unified KD and self\-KD framework with NKD and USKD, compatible with both CNN and ViT architectures and achieving state\-of\-the\-art performance with minimal overhead\.

Beyond single\-teacher settings, several extensions further improve distillation\. For example, multi\-level logit knowledge distillation \(MLKD\)\([17](https://arxiv.org/html/2609.13199#bib.bib5)\)aligns predictions at multiple levels, while multi\-teacher KD with reinforcement learning \(MTKD\-RL\)\([37](https://arxiv.org/html/2609.13199#bib.bib9)\)dynamically weights multiple teachers to overcome suboptimal fixed weighting\. Liet al\.\([23](https://arxiv.org/html/2609.13199#bib.bib27)\)proposed DTSKD, a dual\-teacher self\-KD framework leveraging lightweight branches and MFM to mitigate semantic gaps with high efficiency\. Beyond image classification, KD has also been extended to time series analysis\. For instance, Boileauet al\.\([1](https://arxiv.org/html/2609.13199#bib.bib29)\)distilled reasoning knowledge from foundation models into lightweight language models to enhance temporal interpretability, while Dehigahawattageet al\.\([11](https://arxiv.org/html/2609.13199#bib.bib28)\)proposed TSD to extract temporal saliency and further improve fidelity and interpretability\.

In contrast to the aforementioned methods, AR\-KD introduces a feedback mechanism that adaptively refines the teacher model based on feedback from the student\. This dynamic adjustment enables the teacher to generate knowledge better aligned with the student’s learning capacity, effectively narrowing the teacher\-student capacity gap\.

In this section, we first introduce the motivation and preliminaries underlying our proposed AR\-KD, including the limitations of conventional knowledge distillation under large teacher–student capacity gaps\. We then describe our three\-stage framework, consisting of initial learning, reciprocal learning, and final learning, and formulate the corresponding optimization objectives\. Finally, we provide an analysis of why reciprocal learning can alleviate category correlation collapse and improve the compatibility of the transferred knowledge with the student model\.

![Refer to caption](https://arxiv.org/html/2609.13199v1/pic/Figure2_newnewnew.png)Figure 2\.AR\-KD performs reciprocal distillation in three stages: \(left\) traditional KD, \(middle\) category correlations based teacher adaptation guided by a frozen student, and \(right\) re\-distillation with the adapted teacher\. This bidirectional process generates simplified and compatible knowledge tailored to the student’s learning capacity\.### 3\.1\.Motivation and Notations

We start from the classical KD method proposed in\([12](https://arxiv.org/html/2609.13199#bib.bib1)\)\. To illustrate the procedure of KD, we consider aCC\-way classification task and denote the logit output of a single input asz=\[z1,z2,…,zC\]∈ℝ1×Cz=\[z\_\{1\},z\_\{2\},\.\.\.,z\_\{C\}\]\\in\\mathbb\{R\}^\{1\\times C\}, then the probability of theii\-th class can be calculated by

\(1\)pi=ezi/τ∑j=1Cezj/τ,p\_\{i\}=\\frac\{e^\{z\_\{i\}/\\tau\}\}\{\\sum\_\{j=1\}^\{C\}e^\{z\_\{j\}/\\tau\}\},whereziz\_\{i\}represents the logit of theii\-th class andτ\\tauis the temperature scaling hyperparameter\.

According to previous work, the loss function of classical KD can be reformulated by

\(2\)LK​D\\displaystyle L\_\{KD\}=KL\(pt​e​a\|\|ps​t​u\)\\displaystyle=\\text\{KL\}\(p^\{tea\}\|\|p^\{stu\}\)=ptt​e​a​log⁡\(ptt​e​apts​t​u\)\+∑i=1,i≠tCpit​e​a​log⁡\(pit​e​apis​t​u\)\\displaystyle=p\_\{t\}^\{tea\}\\log\\left\(\\frac\{p\_\{t\}^\{tea\}\}\{p\_\{t\}^\{stu\}\}\\right\)\+\\sum\_\{i=1,i\\neq t\}^\{C\}p\_\{i\}^\{tea\}\\log\\left\(\\frac\{p\_\{i\}^\{tea\}\}\{p\_\{i\}^\{stu\}\}\\right\)=KL\(𝐛t​e​a∥𝐛s​t​u\)\+\(1−ptt​e​a\)KL\(p^t​e​a∥p^s​t​u\),\\displaystyle=\\text\{KL\}\(\\mathbf\{b\}^\{tea\}\\\|\\mathbf\{b\}^\{stu\}\)\+\(1\-p\_\{t\}^\{tea\}\)\\text\{KL\}\(\\hat\{p\}^\{tea\}\\\|\\hat\{p\}^\{stu\}\),
wheret​e​a\{tea\}ands​t​u\{stu\}denote the teacher and student models, respectively\.ps​t​up^\{\{stu\}\}andpt​e​ap^\{\{tea\}\}are the predicted distributions of the student model and the teacher model\. We define the information content of the teacher model by its entropy as

\(3\)H\(𝒯\)=−∑i=1Cpit​e​alogpit​e​a\.H\(\\mathcal\{T\}\)=\-\\sum\_\{i=1\}^\{C\}p^\{tea\}\_\{i\}\\log p^\{tea\}\_\{i\}\.KD can be viewed as an information transfer process from the teacher model to the student model, quantified by the mutual information

\(4\)I⁡\(𝒯,𝒮\)=H⁡\(𝒯\)−H⁡\(𝒯∣𝒮\)\.I\(\\mathcal\{T\};\\mathcal\{S\}\)=H\(\\mathcal\{T\}\)\-H\(\\mathcal\{T\}\\mid\\mathcal\{S\}\)\.As can be observed, whenmaxi⁡pit​e​a→1\\max\_\{i\}p^\{tea\}\_\{i\}\\rightarrow 1, the coefficient associated with non\-target class relational terms in \([2](https://arxiv.org/html/2609.13199#S3.E2)\) tends to zero andH⁡\(𝒯\)→0H\(\\mathcal\{T\}\)\\rightarrow 0, indicating a substantial reduction in the information content encapsulated within the teacher model’s outputs; consequently, the achievable mutual informationI⁡\(𝒯,𝒮\)I\(\\mathcal\{T\};\\mathcal\{S\}\)is inherently limited, which renders it extremely challenging for the student model to approximate the performance of the teacher model\.

### 3\.2\.Overview of AR\-KD

#### 3\.2\.1\.Problem Formulation

Figure[2](https://arxiv.org/html/2609.13199#S3.F2)illustrates an overview of our proposed AR\-KD, which includes three stages: initial learning, reciprocal learning, and final learning\. The details of these stages are summarized below: First, we explicitly model the dark knowledge, namely the category correlation inherent in the classification task\. We state that model predictions can depict the relationships between classes\. In a single prediction, if a class is similar to the ground truth, its corresponding prediction value will be relatively higher than those of other classes, indicating that similar classes exhibit high co\-activation in the predictive distribution\. Such class correlations can be modeled through predictions on the data as

\(5\)G=p⊤​p,Ga​b=∑k=1Bpk,a⋅pk,b,G=p^\{\\top\}p,\\,G\_\{ab\}=\\sum^\{B\}\_\{k=1\}\{p\_\{k,a\}\\cdot p\_\{k,b\}\},whereGGis aC×CC\\times Cmatrix,BBis the batch size,ppindicates the predictions of the model, andGa​bG\_\{ab\}represents the probability that the inputs in this batch are classified into theaa\-th category and thebb\-th category simultaneously, which quantifies the relationship between the two classes\. However, when the teacher model exhibits excessively high prediction confidence on some samples, it leads to an extremely sharp output distribution, i\.e\., the vast majority of the probability mass concentrates on the target class, while the probabilities of non\-target classes approach zero\. Under such circumstances, the values of the off\-diagonal elements in the class correlation matrix computed by \([5](https://arxiv.org/html/2609.13199#S3.E5)\) become extremely small, due to the minimal degree of simultaneous activation across different classes\. This causes the class correlation matrix to degenerate into an approximately diagonal matrix, thereby losing the rich semantic relational information among classes\.

#### 3\.2\.2\.Stage 1: Initial Learning

In the initial stage, we employ a traditional KD approach to train a preliminary student model, which acts as a baseline representation to be refined in subsequent stages\. The overall loss function for this stage is formulated as

\(6\)LI​n​i​t\\displaystyle L\_\{Init\}=\(1−α\)⋅LCE\+α⋅LK​D\\displaystyle=\(1\-\\alpha\)\\cdot L\_\{\\text\{CE\}\}\+\\alpha\\cdot L\_\{KD\}=\(α−1\)⋅∑i=1Cyi​log⁡pis​t​u\+α⋅∑i=1Cpit​e​a​log⁡\(pit​e​apis​t​u\),\\displaystyle=\(\\alpha\-1\)\\cdot\\sum\_\{i=1\}^\{C\}y\_\{i\}\\log p\_\{i\}^\{\{stu\}\}\+\\alpha\\cdot\\sum\_\{i=1\}^\{C\}p\_\{i\}^\{tea\}\\log\\left\(\\frac\{p\_\{i\}^\{tea\}\}\{p\_\{i\}^\{stu\}\}\\right\),whereyiy\_\{i\}denotes the ground\-truth probability of theii\-th class,pis​t​up\_\{i\}^\{stu\}is the predicted probability of theii\-th class by the student model,pit​e​ap\_\{i\}^\{tea\}is the predicted probability of theii\-th class by the teacher model, andα\\alphais the weighting coefficient that balances the two loss terms\.

#### 3\.2\.3\.Stage 2: Reciprocal Learning

To alleviate the issues discussed in the Problem Formulation section, we propose Reciprocal Learning to enhance the information content of the class correlation matrix by adaptively smoothing the predictive distribution of the teacher model\. Specifically, we construct a feedback loss function that adapts and optimizes the teacher model under the premise of a fixed student model\. This approach reduces the excessive output confidence of the teacher model and enhances the information entropy of category relationships in its outputs, thereby providing the student model with more abundant knowledge\.

\(7\)LRL=\(1−α\)⋅LCE⏟Preservation\+α⋅1C​‖Gtea−Gstu‖F2⏟Adaptation​,L\_\{\\text\{RL\}\}=\\underbrace\{\(1\-\\alpha\)\\cdot L\_\{\\text\{CE\}\}\}\_\{\\text\{Preservation\}\}\+\\underbrace\{\\alpha\\cdot\\frac\{1\}\{C\}\\left\\\|G^\{\\text\{tea\}\}\-G^\{\\text\{stu\}\}\\right\\\|\_\{F\}^\{2\}\}\_\{\\mathclap\{\\text\{Adaptation\}\}\}\\text\{,\}whereGt​e​aG^\{\{tea\}\}andGs​t​uG^\{\{stu\}\}are the category correlation matrix computed from the teacher’s and student’s predictions, respectively\.

#### 3\.2\.4\.Stage 3: Final Learning

In this stage, the prediction distribution of the new teacher model has been smoothed, and the class correlation matrix contains abundant inter\-class knowledge, which better matches the current learning capacity of the student model\. Meanwhile, we align the student model with both the KL divergence of the new teacher model and the class correlation matrix of the teacher model, thereby helping the student model absorb knowledge more efficiently\. The final distillation loss function is defined as

\(8\)LF​i​n​a​l\\displaystyle L\_\{Final\}=LK​D\+1C​‖Gt​e​a−Gs​t​u‖F2,\\displaystyle=L\_\{KD\}\+\\frac\{1\}\{C\}\\left\\\|G^\{\{tea\}\}\-G^\{\{stu\}\}\\right\\\|\_\{F\}^\{2\},whereLK​DL\_\{KD\}denotes the KL divergence\. The overall loss function in Stage 3 is defined as

\(9\)LA​R​K​D=\(1−α\)⋅LCE\+α⋅LF​i​n​a​l,\\displaystyle L\_\{ARKD\}=\(1\-\\alpha\)\\cdot L\_\{\\text\{CE\}\}\+\\alpha\\cdot L\_\{Final\},whereα\\alphais the weighting coefficient that balances the two loss terms\.

## 4\.Why Does Reciprocal Learning Work?

In this section, we provide an intuitive explanation of why the proposed reciprocal learning mechanism can improve knowledge transfer\. Rather than presenting rigorous theoretical guarantees, we focus on identifying key properties of AR\-KD from information\-theoretic and optimization perspectives\. These properties help us understand when and why adapting the teacher via category correlation structure alignment is beneficial\.

### 4\.1\.Preliminaries and Assumptions

Consider aCC\-way classification task\. Letpt​e​a,ps​t​u∈ℝ1×Cp^\{tea\},p^\{stu\}\\in\\mathbb\{R\}^\{1\\times C\}denote the probability distributions of the teacher and student models for a given input, respectively\. Following standard KD, we view distillation as a process of transferring information from teacher model𝒯\\mathcal\{T\}to student model𝒮\\mathcal\{S\}\. We adopt the following assumptions:

Teacher Overconfidence\.A well\-trained large teacher often produces sharply peaked distributions, i\.e\.,

\(10\)maxi⁡pit​e​a→1,‖pt​e​a‖22→1,\\max\_\{i\}p^\{tea\}\_\{i\}\\to 1,\\quad\\left\\\|p^\{tea\}\\right\\\|\_\{2\}^\{2\}\\to 1,which implies low entropyH⁡\(𝒯\)H\(\\mathcal\{T\}\)\.

Student as Noisy Approximation\.The student prediction can be modeled as a noisy transformation of the teacher signal\([15](https://arxiv.org/html/2609.13199#bib.bib40)\)

\(11\)ps​t​u=f⁡\(pt​e​a\)\+ε,p^\{stu\}=f\(p^\{tea\}\)\+\\varepsilon,whereε\\varepsilonis a zero\-mean noise term reflecting capacity and optimization limitations\.

Under these assumptions, the effectiveness of distillation depends on how much information about𝒯\\mathcal\{T\}is preserved in𝒮\\mathcal\{S\}, quantified by the mutual information

\(12\)I⁡\(𝒯,𝒮\)=H⁡\(𝒯\)−H⁡\(𝒯\|𝒮\)\.I\(\\mathcal\{T\};\\mathcal\{S\}\)=H\(\\mathcal\{T\}\)\-H\(\\mathcal\{T\}\|\\mathcal\{S\}\)\.

### 4\.2\.Category Correlation Collapse Under Overconfident Teacher

According to Section 3\.2, for convenience of derivation, we analyze a single input of a batch, so we have the class correlation matrix for an instance as

\(13\)G=p⊤​p,Ga​b=pa​pb\.G=p^\{\\top\}p,\\quad G\_\{ab\}=p\_\{a\}p\_\{b\}\.
Relational Degeneracy\.If the teacher distribution is sharply peaked, then the off\-diagonal mass ofGt​e​aG^\{tea\}collapses

\(14\)∑a≠bGa​bt​e​a=1−‖pt​e​a‖22→0\.\\sum\_\{a\\neq b\}G^\{tea\}\_\{ab\}=1\-\\left\\\|p^\{tea\}\\right\\\|\_\{2\}^\{2\}\\to 0\.
Proof\.Since∑ipi=1\\sum\_\{i\}p\_\{i\}=1,

\(15\)∑a≠bpa​pb=\(∑ipi\)2−∑ipi2=1−‖p‖22\.\\sum\_\{a\\neq b\}p\_\{a\}p\_\{b\}=\\left\(\\sum\_\{i\}p\_\{i\}\\right\)^\{2\}\-\\sum\_\{i\}p\_\{i\}^\{2\}=1\-\\left\\\|p\\right\\\|\_\{2\}^\{2\}\.
Whenmaxi⁡pi→1\\max\_\{i\}p\_\{i\}\\to 1, we have‖p‖22→1\\left\\\|p\\right\\\|\_\{2\}^\{2\}\\to 1, and thus the off\-diagonal sum vanishes\.

This implies that highly confident teachers encode little category correlation structure\. The class correlation matrix degenerates into an approximately diagonal form, removing informative dark knowledge about class similarities\.

### 4\.3\.Reciprocal Learning as Entropy and Compatibility Regularization

In Stage 2, AR\-KD adapts the teacher by minimizing

\(16\)LR​L=\(1−α\)⋅LCE\+α⋅‖Gt​e​a−Gs​t​u‖F2,L\_\{RL\}=\(1\-\\alpha\)\\cdot L\_\{\\text\{CE\}\}\+\\alpha\\cdot\\left\\\|G^\{tea\}\-G^\{stu\}\\right\\\|\_\{F\}^\{2\},withps​t​up^\{stu\}fixed\.

Expanding the relational term gives

\(17\)‖Gt​e​a−Gs​t​u‖F2=‖pt​e​a‖44\+‖ps​t​u‖44−2​\(pt​e​a⊤​ps​t​u\)2\.\\left\\\|G^\{tea\}\-G^\{stu\}\\right\\\|\_\{F\}^\{2\}=\\left\\\|p^\{tea\}\\right\\\|\_\{4\}^\{4\}\+\\left\\\|p^\{stu\}\\right\\\|\_\{4\}^\{4\}\-2\(p^\{tea\\top\}p^\{stu\}\)^\{2\}\.
Overconfidence Suppression\.Minimizing ‖Gt​e​a−Gs​t​u‖F2\\left\\\|G^\{tea\}\-G^\{stu\}\\right\\\|\_\{F\}^\{2\}with respect topt​e​ap^\{tea\}penalizes large‖pt​e​a‖2\\left\\\|p^\{tea\}\\right\\\|\_\{2\}and thus discourages overly concentrated teacher distributions\.

Proof\.Since‖p‖44≥1C​‖p‖24\\left\\\|p\\right\\\|\_\{4\}^\{4\}\\geq\\frac\{1\}\{C\}\\left\\\|p\\right\\\|\_\{2\}^\{4\}and‖p‖22→1\\left\\\|p\\right\\\|\_\{2\}^\{2\}\\to 1for peaked distributions, the objective increases rapidly aspt​e​ap^\{tea\}becomes concentrated\. Therefore, the optimization favors smoother distributions with higher entropy\. The adapted teacher𝒯′\\mathcal\{T\}^\{\\prime\}obtained after reciprocal learning satisfies

\(18\)H⁡\(𝒯′\)\>H⁡\(𝒯\)\.H\(\\mathcal\{T\}^\{\\prime\}\)\>H\(\\mathcal\{T\}\)\.
![Refer to caption](https://arxiv.org/html/2609.13199v1/pic/Figure7.png)Figure 3\.Class correlation matrix heatmaps for a single sample\.Thus, reciprocal learning explicitly increases the entropy of teacher outputs while preserving task correctness via the CE term\.

As shown in Figure[3](https://arxiv.org/html/2609.13199#S4.F3), the color intensity represents the strength of category correlation\. The off\-diagonal regions of the adapted teacher𝒯′\\mathcal\{T\}^\{\\prime\}are no longer pure white, indicating that𝒯′\\mathcal\{T\}^\{\\prime\}retains richer category correlation knowledge and thus contains more information compared to the original teacher𝒯\\mathcal\{T\}\.

### 4\.4\.Implications for Mutual Information and Student Learning

Mutual Information Enlargement\.Under Assumption 2, increasingH⁡\(𝒯\)H\(\\mathcal\{T\}\)while maintaining boundedH⁡\(T\|S\)H\(T\|S\)leads to larger mutual informationI⁡\(𝒯,𝒮\)I\(\\mathcal\{T\};\\mathcal\{S\}\)

\(19\)I⁡\(𝒯′,𝒮\)=H⁡\(𝒯′\)−H⁡\(𝒯′\|𝒮\)≥I⁡\(𝒯,𝒮\)\.I\(\\mathcal\{T\}^\{\\prime\};\\mathcal\{S\}\)=H\(\\mathcal\{T\}^\{\\prime\}\)\-H\(\\mathcal\{T\}^\{\\prime\}\|\\mathcal\{S\}\)\\geq I\(\\mathcal\{T\};\\mathcal\{S\}\)\.
Proof\.SinceH⁡\(𝒯′\|𝒮\)H\(\\mathcal\{T\}^\{\\prime\}\|\\mathcal\{S\}\)is dominated by the student noise termε\\varepsilon, and reciprocal learning does not increaseε\\varepsilon, raisingH⁡\(𝒯′\)H\(\\mathcal\{T\}^\{\\prime\}\)increasesI⁡\(𝒯′,𝒮\)I\(\\mathcal\{T\}^\{\\prime\};\\mathcal\{S\}\)directly\.

## 5\.Experiments

In this section, we conduct extensive experiments to evaluate the effectiveness of AR\-KD\. We first introduce the datasets and implementation settings, followed by comparisons with state\-of\-the\-art knowledge distillation methods\. We then conduct ablation studies and additional analysis to examine the contributions of different components, the effectiveness of reciprocal teacher refinement, and the compatibility of AR\-KD with feature\-based distillation methods\. Finally, we visualize the learned representations to provide further insights into the effectiveness of the proposed method\.

### 5\.1\.Datasets and Settings

We evaluate AR\-KD on two widely used image classification benchmarks: CIFAR\-100\([19](https://arxiv.org/html/2609.13199#bib.bib23)\)and ImageNet\-1k\([5](https://arxiv.org/html/2609.13199#bib.bib33)\)\. CIFAR\-100 consists of 32×32 RGB images from 100 object categories, with 50,000 images for training and 10,000 images for testing, and is commonly adopted for benchmarking methods on small\-scale image classification tasks\. ImageNet\-1k is a large\-scale image classification dataset comprising approximately 1\.2 million training images and 50,000 validation images across 1,000 object categories\. Following standard practice, all ImageNet images are resized to224×224224\\times 224during training, enabling evaluation under a high\-resolution and large\-category setting\.

### 5\.2\.Implementation Details

For CIFAR\-100, we evaluate AR\-KD under two settings: \(1\) Homogeneous architecture, where the teacher and student share the same architecture type \(e\.g\., VGG13→\\toVGG8\), and \(2\) Heterogeneous architecture, where the two differ structurally \(e\.g\., VGG13→\\toMobileNetV2\)\. Our experiments cover a range of classical models, including ResNet, WRN, VGG, ShuffleNet\-V1/V2, and MobileNetV2\([9](https://arxiv.org/html/2609.13199#bib.bib10);[39](https://arxiv.org/html/2609.13199#bib.bib25);[30](https://arxiv.org/html/2609.13199#bib.bib16);[41](https://arxiv.org/html/2609.13199#bib.bib26);[14](https://arxiv.org/html/2609.13199#bib.bib24)\)\.

For ImageNet\-1k, we consider two representative teacher–student configurations: Res34→\\toRes18 for the homogeneous architecture setting and Res50→\\toMobileNetV2 for the heterogeneous setting\. All experiments are conducted under the same hardware environment and follow the standard ImageNet\-1k training protocol\. Performance is reported in terms of Top\-1 and Top\-5 accuracy, in accordance with the conventional evaluation protocol for image classification\. We use the SGD optimizer for 240 training epochs, with an initial learning rate of 0\.01 for CIFAR\-100 and 0\.1 for ImageNet\. Specifically, for the second stage, we adopt an initial learning rate of 0\.001 and perform fine\-tuning training for 40 epochs\. The distillation temperatureτ\\tauis set to 4\. All training is conducted on A100 GPUs\. Each experiment is run five times with five different random seeds to eliminate randomness, and we report the averaged results for reliability\. T\-tests are conducted on all experimental results, and all performance improvements achieved by our AR\-KD framework are statistically significant withpp\-value < 0\.05\.

Table 1\.Comparison of Top\-1 accuracy \(%\) with different methods on CIFAR\-100\. Note: R32x4, R8x4, R56, R50, R20, W40\-2, W40\-1, W16\-2, MV2, SV1, and SV2 stand for ResNet32x4, ResNet8x4, ResNet56, ResNet50, ResNet20, WRN\-40\-2, WRN\-40\-1, WRN\-16\-2, MobileNetV2, ShuffleNetV1, and ShuffleNetV2\.Homogeneous architectureHeterogeneous architectureMethodTeacherR32x4R56W40\-2W40\-2VGG13R32x4W40\-2VGG13R50R32x4Avg\.79\.4272\.3475\.6175\.6174\.6479\.4275\.6174\.6479\.3479\.42StudentR8x4R20W40\-1W16\-2VGG8SV1SV1MV2MV2SV272\.5069\.0671\.9873\.2670\.3670\.5070\.5064\.6064\.6071\.80FeatureFitNet \(ICLR15’\)73\.5069\.2172\.2473\.5871\.0273\.5973\.7364\.1463\.1673\.5470\.77CRD \(ICLR20’\)75\.5171\.1674\.1475\.4873\.9475\.1176\.0569\.7369\.1175\.6573\.59WCoRD \(CVPR21’\)75\.9571\.5674\.7375\.8874\.5575\.4076\.3269\.4770\.4575\.9674\.03ReviewKD \(CVPR22’\)75\.6371\.8975\.0976\.1274\.8477\.4577\.1470\.3769\.8677\.7874\.62NORM \(ICLR23’\)76\.4971\.3574\.8275\.6573\.9577\.4277\.0668\.9470\.5678\.0774\.43LogitKD \(NIPS14’\)73\.3370\.6673\.5474\.9272\.9874\.0774\.8367\.3767\.3574\.4572\.35TAKD \(AAAI20’\)73\.8170\.8373\.7875\.1273\.2374\.5375\.3467\.9168\.0274\.8272\.74DKD \(CVPR22’\)76\.3271\.9774\.8176\.2474\.6876\.4576\.7069\.7170\.3577\.0774\.43CTKD \(AAAI23’\)73\.7071\.1973\.9375\.4573\.5274\.4875\.7868\.4668\.4775\.3173\.03LSKD \(CVPR24’\)76\.6271\.4374\.3776\.1174\.36\-\-68\.6169\.0275\.56\-RCKA \(IJCAI24’\)76\.11\-75\.3476\.5174\.9776\.9777\.2170\.12\-\-\-TeKAP \(ICLR25’\)74\.7971\.3273\.8075\.2174\.0074\.9276\.7567\.3969\.0075\.4373\.26RLD \(ICCV25’\)76\.6472\.0074\.88\-74\.93\-\-69\.9770\.7677\.56\-AR\-KD \(Ours\)77\.2972\.0875\.5176\.7875\.2277\.6377\.5270\.1270\.0278\.6075\.08Gap\-Base4\.793\.023\.533\.524\.867\.137\.025\.525\.46\.85\.16Gap\-AR\-KD2\.130\.260\.10\-1\.17\-0\.581\.79\-1\.914\.529\.320\.821\.53Δ\\Delta\(to KD\)\+3\.96\+1\.42\+1\.97\+1\.86\+2\.24\+3\.56\+2\.69\+2\.75\+2\.67\+4\.15\+2\.72

Table 2\.Comparisons of Top\-1 and Top\-5 accuracy \(%\) on ImageNet\-1k\. The original accuracies of the teacher and student model are also reported\.
### 5\.3\.Experimental Result

#### 5\.3\.1\.Comparison with State\-of\-the\-art Baselines

We validate our proposed AR\-KD against the following state\-of\-the\-art baselines including feature\-based distillation methods \(FitNet\([29](https://arxiv.org/html/2609.13199#bib.bib14)\), AT\([40](https://arxiv.org/html/2609.13199#bib.bib13)\), CRD\([36](https://arxiv.org/html/2609.13199#bib.bib15)\), WCoRD\([3](https://arxiv.org/html/2609.13199#bib.bib34)\), ReviewKD\([4](https://arxiv.org/html/2609.13199#bib.bib17)\), NORM\([25](https://arxiv.org/html/2609.13199#bib.bib12)\), CAT\-KD\([8](https://arxiv.org/html/2609.13199#bib.bib22)\)\), and logit\-based distillation methods \(KD\([12](https://arxiv.org/html/2609.13199#bib.bib1)\), TAKD\([26](https://arxiv.org/html/2609.13199#bib.bib3)\), DKD\([42](https://arxiv.org/html/2609.13199#bib.bib2)\), CTKD\([24](https://arxiv.org/html/2609.13199#bib.bib18)\), MLKD\([17](https://arxiv.org/html/2609.13199#bib.bib5)\), LSKD\([32](https://arxiv.org/html/2609.13199#bib.bib6)\), RCKA\([43](https://arxiv.org/html/2609.13199#bib.bib39)\), TeKAP\([13](https://arxiv.org/html/2609.13199#bib.bib35)\), and RLD\([34](https://arxiv.org/html/2609.13199#bib.bib38)\)\)\.

Tables[1](https://arxiv.org/html/2609.13199#S5.T1)and[2](https://arxiv.org/html/2609.13199#S5.T2)summarize the performance of AR\-KD under both homogeneous and heterogeneous teacher–student settings on CIFAR\-100 and ImageNet\-1k, respectively\. As shown in Table[1](https://arxiv.org/html/2609.13199#S5.T1), AR\-KD improves student performance across both homogeneous and heterogeneous settings on CIFAR\-100\. Compared to baseline students, AR\-KD yields up to 7\.13% absolute gains and outperforms vanilla KD by 1\.42%–4\.15%\. AR\-KD achieves comparable or superior performance to feature\-based methods while only utilizing output logits, demonstrating the effectiveness and generalizability of our adaptive reciprocal distillation framework\.

For ImageNet\-1k, in the homogeneous ResNet34→\\toResNet18 setup, AR\-KD outperforms MLKD, a strong logit\-based method, by 0\.23% in Top\-1 accuracy\. Meanwhile, in the heterogeneous ResNet50→\\toMobileNetV2 configuration, AR\-KD achieves a Top\-1 accuracy that is only 0\.06% lower than MLKD, the best\-performing logit\-based method, while matching the highest reported Top\-5 accuracy\. These results demonstrate that AR\-KD remains competitive across different architectural settings on ImageNet\-1k, benefiting from a lightweight and architecture\-agnostic distillation framework\.

Table 3\.Ablation study result\. The experiments are conducted on CIFAR\-100, with ResNet32x4 as the teacher model, ResNet8x4 as the student model, and Top\-1 accuracy as the evaluation metric\.Table 4\.Performance comparison of AR\-KD \(KL vs\. Ours\) on CIFAR\-100\.While AR\-KD delivers strong and often leading performance on both CIFAR\-100 and ImageNet\-1k, its relative improvements are more pronounced in homogeneous teacher–student settings than in heterogeneous ones\. This trend can be attributed to the fact that AR\-KD operates without exploiting intermediate representations, which are known to be particularly beneficial when transferring knowledge across architectures with large structural discrepancies\. Nevertheless, across both small\-scale \(CIFAR\-100\) and large\-scale \(ImageNet\-1k\) benchmarks, AR\-KD achieves performance that is comparable to or competitive with representative feature\-based methods, while maintaining a simpler and more flexible formulation\. These results highlight the robustness and general applicability of AR\-KD across diverse datasets and distillation scenarios\.

#### 5\.3\.2\.Comparison with Teacher Model

We further examine the performance gap between the student and teacher models in our proposed AR\-KD method\. As shown in Table[1](https://arxiv.org/html/2609.13199#S5.T1), we calculate the accuracy difference between the student and teacher models \(denoted as Gap\-ARKD\), where a negative gap indicates that the student model outperforms the teacher model\. It can be observed that, with our carefully designed approach, the performance of the student model \(AR\-KD\) is highly aligned with that of the teacher model, with an average accuracy gap of only 1\.53\. More notably, the student model even slightly surpasses the teacher in some experimental scenarios\.

We observe that if the accuracy gap between the teacher model and the student model is not extremely large without distillation, our method can boost the student model’s accuracy to a level very close to that of the teacher model\. However, if the student model itself has extremely low accuracy and a substantial gap from the teacher model, such as ResNet50→\\toMobileNetV2, the student model still exhibits a noticeable accuracy gap from the teacher model even after optimization with our method\. This phenomenon stems from the student model’s limited capacity; once this capacity bound is reached, further improvement becomes exceedingly challenging\.

#### 5\.3\.3\.Ablation Study

To investigate the contributions of each stage in our method, i\.e\., initial learning, reciprocal learning, and final learning, we conduct four groups of controlled experiments\. As shown in Table[3](https://arxiv.org/html/2609.13199#S5.T3), the experimental results are analyzed as follows: In the first group, where only the first stage is adopted for training, the performance of our method is comparable to that of the standard KD method\. In the second group, we directly perform the third\-stage training without fine\-tuning the teacher model\. Compared with the first group, this configuration achieves a significant performance improvement, indicating that the category correlation matrix encodes rich inter\-class semantic relationships—information that is overlooked when only KL divergence alignment is employed\. In the third group, we conduct the third stage by aligning the updated teacher model and student model solely via KL divergence, without incorporating the category correlation matrix as a supplement\. This setup also outperforms the first group remarkably, demonstrating that the teacher model fine\-tuning in the second stage plays a pivotal role\. In the fourth group, where all three stages are integrated, our method achieves superior performance over all other experimental groups\. This confirms the importance of all three stages to the overall performance gain\.

#### 5\.3\.4\.Comparison with KL\-Based Teacher Refinement

In the early stage of our framework’s design, we attempted to fine\-tune and constrain the teacher model by aligning the KL divergence between the teacher and student models in the second stage\. The experimental results are presented in Table[4](https://arxiv.org/html/2609.13199#S5.T4)\. The results indicate that the fine\-tuning method based on the category correlation matrix achieves superior performance\. We speculate that this KL divergence\-based alignment approach struggles to capture the deep category correlation information; it may even lead to the teacher model being misguided by the student model\. This further demonstrates the superiority of our proposed method\.

Table 5\.Training time for each batch\.
#### 5\.3\.5\.Combining with Feature Knowledge Distillation Methods

AR\-KD is a logit\-based distillation method whose core objective is to obtain a teacher model that is better suited to the student model\. Therefore, we can integrate AR\-KD with feature\-based distillation approaches\. As shown in Table[6](https://arxiv.org/html/2609.13199#S5.T6), we combine ReviewKD\([2](https://arxiv.org/html/2609.13199#bib.bib21)\)with AR\-KD to conduct our experiments\. The results verify the satisfactory scalability of AR\-KD, and its combination with other distillation strategies can achieve higher performance\.

#### 5\.3\.6\.Time cost

AR\-KD requires extra category correlation matrix computation in training\. We evaluate computational overhead by comparing training time for each batch of representative KD methods on CIFAR\-100 under the same hardware configuration, with homogeneous architectures for the teacher and student models\. As shown in Table[5](https://arxiv.org/html/2609.13199#S5.T5), despite the extra matrix operation, AR\-KD skips intermediate layers computations and achieves lower per\-batch training latency than feature\-based distillation methods including CRD and ReviewKD\.

### 5\.4\.Visualizations

Figure[4](https://arxiv.org/html/2609.13199#S5.F4)illustrates the t\-SNE visualization of features learned by vanilla KD and AR\-KD on CIFAR\-100\. It demonstrates that representations of our method are more separable than vanilla KD, demonstrating that AR\-KD improves the discriminability of deep features\.

![Refer to caption](https://arxiv.org/html/2609.13199v1/pic/Figure4.png)Figure 4\.Visualization of vanilla KD \(left\) and AR\-KD \(right\)\.Table 6\.Combine with feature KD method on CIFAR\-100\.

## 6\.Rethinking AR\-KD

### 6\.1\.Concern About the Scalability of AR\-KD

A potential concern of AR\-KD is the additional computational overhead introduced by the reciprocal teacher adaptation stage\. We clarify that AR\-KD does not retrain the teacher model from scratch\. Instead, the second stage only performs lightweight adaptation on a pretrained teacher for a relatively small number of epochs, making the additional optimization cost substantially lower than the original teacher training process\.

Moreover, AR\-KD operates purely at the logit level and does not require intermediate feature extraction, feature projection modules, or auxiliary alignment networks commonly used in feature\-based distillation methods\. Therefore, the overall training pipeline remains simple and computationally efficient\.

An important practical advantage of AR\-KD is that the adapted teacher can be reused across multiple student models once reciprocal learning is completed\. In practice, we observe that the refined teacher consistently provides more compatible supervision signals for different lightweight students without requiring repeated adaptation\. Consequently, the reciprocal learning stage introduces a one\-time optimization cost, while its benefits can be amortized throughout the entire distillation pipeline\.

From this perspective, AR\-KD achieves a favorable trade\-off between computational overhead and distillation effectiveness, particularly under large teacher–student capacity gaps where conventional KD methods often suffer from limited transferability\.

![Refer to caption](https://arxiv.org/html/2609.13199v1/pic/Figure8_new.png)Figure 5\.A toy case where two teacher models,𝒯1\\mathcal\{T\}\_\{1\}and𝒯2\\mathcal\{T\}\_\{2\}, adapted in two methods\.𝒯1\\mathcal\{T\}\_\{1\}merely scales the overall distribution, leaving the interrelationships among non\-target classes largely hidden\. The original probabilities are 0\.07, 0\.07, 0\.06, and after temperature scaling they remain nearly identical at 0\.208, 0\.208, 0\.201, preserving the same structure\. In contrast,𝒯2\\mathcal\{T\}\_\{2\}with AR\-KD explicitly restructures the category correlation: frog is actively suppressed to 0\.15, creating a clear distinction from cat and bird\. This enables the dark knowledge to be effectively transferred\.
### 6\.2\.Why Simple Temperature Smoothing Is Not Enough

A natural question arises from our formulation: if overconfident teacher predictions hinder knowledge transfer, why not simply increase the distillation temperatureτ\\tauto soften the teacher outputs?

Although temperature scaling can flatten the prediction distribution globally, we argue that it is fundamentally different from the reciprocal adaptation mechanism in AR\-KD\. Specifically, temperature scaling only rescales the logits uniformly, which changes the sharpness of the probability distribution while preserving the original relational structure among classes\. Consequently, the relative similarity relationships encoded by the teacher remain largely unchanged\.

In contrast, AR\-KD does not merely smooth the output distribution\. Instead, it explicitly adapts the category correlation structure of the teacher according to the student representation through correlation matrix alignment\. This process modifies how semantic relationships are organized in the teacher outputs, enabling the adapted teacher to generate knowledge that is structurally compatible with the student capacity\.

From the perspective of class correlations, temperature scaling mainly enlarges small probabilities uniformly, whereas AR\-KD reshapes the relational dependency among categories\. Therefore, the proposed reciprocal learning mechanism can recover richer and more transferable dark knowledge beyond simple entropy smoothing\.

Figure[5](https://arxiv.org/html/2609.13199#S6.F5)illustrates a toy case comparing two teacher models,𝒯1\\mathcal\{T\}\_\{1\}and𝒯2\\mathcal\{T\}\_\{2\}, adapted with different methods\.𝒯1\\mathcal\{T\}\_\{1\}merely scales the overall logit distribution via temperature scaling, leaving the interrelationships among non\-target classes largely hidden\. For instance, the original probabilities for three non\-target classes are 0\.07, 0\.07, 0\.06; after scaling they become nearly identical \(0\.208, 0\.208, 0\.201\), preserving the same flat structure\. In contrast,𝒯2\\mathcal\{T\}\_\{2\}with AR\-KD explicitly restructures the category correlation: the probability of frog is actively suppressed to 0\.15, creating a clear distinction from cat and bird\. This restructuring makes the dark knowledge more discernible and thus more effectively transferable to a student model\.

### 6\.3\.Rethinking the Capacity Gap in Distillation

Most existing KD studies typically treat the teacher–student capacity gap primarily as a performance discrepancy\. However, our findings reveal that this gap fundamentally reflects a representation compatibility issue\. As the capacity gap increases, the relational structures encoded by the teacher model may exceed the optimization and representation capabilities of the student model, making direct imitation progressively less effective\.

From this perspective, the core challenge of KD is not merely “transferring stronger knowledge", but rather how to adapt complex relational structures into a form that is inherently learnable for the lightweight student models\. This indicates that future KD paradigms should shift their focus from strictly enforcing teacher\-student alignment to ensuring representation compatibility\. KD achieves its full potential only when the teacher’s knowledge is projected into a structural format that the student can successfully assimilate\. Ultimately, in the context of extreme capacity gaps, representation compatibility supersedes raw capacity, and targeted adaptation proves more critical than rigid imitation\.

## 7\.Conclusion

We propose adaptive reciprocal knowledge distillation \(AR\-KD\), a novel distillation framework that narrows the teacher–student capacity gap by allowing the student to adapt the teacher through reciprocal learning\. By aligning category correlation structures via the category correlation matrix, AR\-KD produces smoother and more compatible supervision signals, enabling the student to absorb richer dark knowledge\. Extensive experiments on CIFAR\-100 and ImageNet\-1k demonstrate consistent improvements over state\-of\-the\-art KD baselines, validating the effectiveness and generality of our reciprocal, relationally\-aware distillation paradigm\.

## GenAI Usage Disclosure

During the preparation of this work, the authors used AI tools only to improve writing and fix grammar in existing content\. After using AI tools, the authors carefully reviewed and edited all changed contents\. No AI tools were used for generating research ideas, model configuration choices, or experimental results\. The authors take full responsibility for the paper\.

## References

- Boileauet al\.\(2025\)M\. Boileau, P\. Helluy, J\. Pawlus, and S\. VyetrenkoTowards interpretable time series foundation models\.InProceedings of the International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p3.1)\.
- Chenet al\.\(2022\)D\. Chen, J\. Mei, H\. Zhang, C\. Wang, Y\. Feng, and C\. ChenKnowledge distillation with the reused teacher classifier\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition \(CVPR\),pp\. 11933–11942\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.5](https://arxiv.org/html/2609.13199#S5.SS3.SSS5.p1.1)\.
- Chenet al\.\(2021a\)L\. Chen, Z\. Gan, D\. Wang, J\. Liu, R\. Henao, and L\. CarinWasserstein contrastive representation distillation\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 16296–16305\.Cited by:[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Chenet al\.\(2021b\)P\. Chen, S\. Liu, H\. Zhao, and J\. JiaDistilling knowledge via knowledge review\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 5008–5017\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Denget al\.\(2009\)J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-FeiImageNet: a large\-scale hierarchical image database\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 248–255\.Cited by:[§5\.1](https://arxiv.org/html/2609.13199#S5.SS1.p1.1)\.
- Fukudaet al\.\(2017\)T\. Fukuda, M\. Suzuki, G\. Kurata, S\. Thomas, J\. Cui, and B\. RamabhadranEfficient knowledge distillation from an ensemble of teachers\.InProceedings of the International Speech Communication Association \(INTERSPEECH\),pp\. 3697–3701\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Gouet al\.\(2024\)J\. Gou, L\. Sun, B\. Yu, S\. Wan, and D\. TaoHierarchical multi\-attention transfer for knowledge distillation\.ACM Transactions on Multimedia Computing, Communications and Applications20\(2\),pp\. 51:1–20\.Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Guoet al\.\(2023\)Z\. Guo, H\. Yan, H\. Li, and X\. LinClass attention transfer based knowledge distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 11868–11877\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Heet al\.\(2016\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 770–778\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p1.1),[§5\.2](https://arxiv.org/html/2609.13199#S5.SS2.p1.1)\.
- Heoet al\.\(2019\)B\. Heo, J\. Kim, S\. Yun, H\. Park, N\. Kwak, and J\. Y\. ChoiA comprehensive overhaul of feature distillation\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 1921–1930\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Hewa Dehigahawattageet al\.\(2025\)N\. U\. Hewa Dehigahawattage, K\. Nandakishor, and M\. PalaniswamiLearning to reason: temporal saliency distillation for interpretable knowledge transfer\.InProceedings of the European Conference on Artificial Intelligence \(ECAI\),pp\. 2866–2873\.Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p3.1)\.
- Hinton \(2014\)G\. HintonDistilling the knowledge in a neural network\.InProceedings of the Annual Conference on Neural Information Processing Systems \(NIPS\),Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p1.1),[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§3\.1](https://arxiv.org/html/2609.13199#S3.SS1.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Hossainet al\.\(2025\)M\. I\. Hossain, S\. Akhter, C\. S\. Hong, and E\. HuhSingle teacher, multiple perspectives: teacher knowledge augmentation for enhanced knowledge distillation\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Howardet al\.\(2017\)A\. G\. Howard, M\. Zhu, B\. Chen, D\. Kalenichenko, W\. Wang, T\. Weyand, M\. Andreetto, and H\. AdamMobileNets: efficient convolutional neural networks for mobile vision applications\.arXiv preprint arXiv:1704\.04861\.Cited by:[§5\.2](https://arxiv.org/html/2609.13199#S5.SS2.p1.1)\.
- Huanget al\.\(2023\)T\. Huang, Y\. Zhang, M\. Zheng, S\. You, F\. Wang, C\. Qian, and C\. XuKnowledge diffusion for distillation\.Advances in Neural Information Processing Systems36,pp\. 65299–65316\.Cited by:[§4\.1](https://arxiv.org/html/2609.13199#S4.SS1.p3.1)\.
- Jianget al\.\(2025\)D\. Jiang, Z\. Shen, Q\. Zheng, T\. Zhang, W\. Xiang, and J\. JinFarm\-LightSeek: an edge\-centric multimodal agricultural IoT data analytics framework with lightweight LLMs\.IEEE Internet of Things Magazine8\(5\),pp\. 72–79\.Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Jinet al\.\(2023\)Y\. Jin, J\. Wang, and D\. LinMulti\-level logit distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 24276–24285\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§2](https://arxiv.org/html/2609.13199#S2.p3.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Kimet al\.\(2023\)J\. Kim, H\. Lee, and S\. S\. WooIMF: integrating matched features using attentive logit in knowledge distillation\.InProceedings of the International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 974–981\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p1.1)\.
- Krizhevskyet al\.\(2009\)A\. Krizhevsky G\. Hintonet al\.Learning multiple layers of features from tiny images\.Cited by:[§5\.1](https://arxiv.org/html/2609.13199#S5.SS1.p1.1)\.
- Krizhevskyet al\.\(2012\)A\. Krizhevsky, I\. Sutskever, and G\. E\. HintonImageNet classification with deep convolutional neural networks\.InProceedings of Advances in Neural Information Processing Systems \(NIPS\),Vol\.25\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p1.1)\.
- Liet al\.\(2025\)S\. Li, T\. Su, X\. Zhang, and Z\. WangContinual learning with knowledge distillation: a survey\.IEEE Transactions on Neural Networks and Learning Systems36\(6\),pp\. 9798–9818\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2024.3476068)Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Liet al\.\(2021\)X\. Li, S\. Li, B\. Omar, F\. Wu, and X\. LiReskd: residual\-guided knowledge distillation\.IEEE Transactions on Image Processing30,pp\. 4735–4746\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Liet al\.\(2024\)Z\. Li, X\. Li, L\. Yang, R\. Song, J\. Yang, and Z\. PanDual teachers for self\-knowledge distillation\.Pattern Recognition151,pp\. 110422\.Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p3.1)\.
- Liet al\.\(2023\)Z\. Li, X\. Li, L\. Yang, B\. Zhao, R\. Song, L\. Luo, J\. Li, and J\. YangCurriculum temperature for knowledge distillation\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),pp\. 1504–1512\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Liuet al\.\(2023\)X\. Liu, L\. Li, C\. Li, and A\. YaoNorm: knowledge distillation via n\-to\-one representation matching\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Mirzadehet al\.\(2020\)S\. I\. Mirzadeh, M\. Farajtabar, A\. Li, N\. Levine, A\. Matsukawa, and H\. GhasemzadehImproved knowledge distillation via teacher assistant\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),Vol\.34,pp\. 5191–5198\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§2](https://arxiv.org/html/2609.13199#S2.p2.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Parket al\.\(2019\)W\. Park, D\. Kim, Y\. Lu, and M\. ChoRelational knowledge distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 3967–3976\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Peiet al\.\(2021\)Y\. Pei, Y\. Qu, and J\. ZhangSelf\-boosting for feature distillation\.InProceedings of the International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 945–951\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p1.1)\.
- Romeroet al\.\(2015\)A\. Romero, N\. Ballas, S\. E\. Kahou, A\. Chassang, C\. Gatta, and Y\. BengioFitNets: hints for thin deep nets\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Simonyan and Zisserman \(2014\)K\. Simonyan and A\. ZissermanVery deep convolutional networks for large\-scale image recognition\.arXiv preprint arXiv:1409\.1556\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p1.1),[§5\.2](https://arxiv.org/html/2609.13199#S5.SS2.p1.1)\.
- Sonet al\.\(2021\)W\. Son, J\. Na, J\. Choi, and W\. HwangDensely guided knowledge distillation using multiple teacher assistants\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 9395–9404\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§2](https://arxiv.org/html/2609.13199#S2.p2.1)\.
- Sunet al\.\(2024\)S\. Sun, W\. Ren, J\. Li, R\. Wang, and X\. CaoLogit standardization in knowledge distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 15731–15740\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Sunet al\.\(2025a\)T\. Sun, H\. Chen, G\. Hu, and C\. ZhaoExplainability\-based knowledge distillation\.Pattern Recognition159,pp\. 111095\.External Links:ISSN 0031\-3203Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Sunet al\.\(2025b\)W\. Sun, D\. Chen, S\. Lyu, G\. Chen, C\. Chen, and C\. WangKnowledge distillation with refined logits\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 1110–1119\.Cited by:[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Tianet al\.\(2025\)Y\. Tian, S\. Pei, X\. Zhang, C\. Zhang, and N\. V\. ChawlaKnowledge distillation on graphs: a survey\.ACM Computing Surveys57\(8\)\.Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p1.1)\.
- Tianet al\.\(2020\)Y\. Tian, D\. Krishnan, and P\. IsolaContrastive representation distillation\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Yanget al\.\(2025\)C\. Yang, X\. Yu, H\. Yang, Z\. An, C\. Yu, L\. Huang, and Y\. XuMulti\-teacher knowledge distillation with reinforcement learning for visual recognition\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),Vol\.39,pp\. 9148–9156\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§2](https://arxiv.org/html/2609.13199#S2.p3.1)\.
- Yanget al\.\(2023\)Z\. Yang, A\. Zeng, Z\. Li, T\. Zhang, C\. Yuan, and Y\. LiFrom knowledge distillation to self\-knowledge distillation: a unified approach with normalized loss and customized soft labels\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 17185–17194\.Cited by:[§2](https://arxiv.org/html/2609.13199#S2.p2.1)\.
- Zagoruyko and Komodakis \(2016\)S\. Zagoruyko and N\. KomodakisWide residual networks\.arXiv preprint arXiv:1605\.07146\.Cited by:[§5\.2](https://arxiv.org/html/2609.13199#S5.SS2.p1.1)\.
- Zagoruyko and Komodakis \(2017\)S\. Zagoruyko and N\. KomodakisPaying more attention to attention: improving the performance of convolutional neural networks via attention transfer\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Zhanget al\.\(2018\)X\. Zhang, X\. Zhou, M\. Lin, and J\. SunShuffleNet: an extremely efficient convolutional neural network for mobile devices\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6848–6856\.Cited by:[§5\.2](https://arxiv.org/html/2609.13199#S5.SS2.p1.1)\.
- Zhaoet al\.\(2022\)B\. Zhao, Q\. Cui, R\. Song, Y\. Qiu, and J\. LiangDecoupled knowledge distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 11953–11962\.Cited by:[Figure 1](https://arxiv.org/html/2609.13199#S1.F1),[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§2](https://arxiv.org/html/2609.13199#S2.p1.1),[§2](https://arxiv.org/html/2609.13199#S2.p2.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.
- Zhouet al\.\(2024\)Z\. Zhou, Y\. Shen, S\. Shao, L\. Gong, and S\. LinRethinking centered kernel alignment in knowledge distillation\.InProceedings of the International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 5680–5688\.Cited by:[§1](https://arxiv.org/html/2609.13199#S1.p2.1),[§5\.3\.1](https://arxiv.org/html/2609.13199#S5.SS3.SSS1.p1.1)\.