A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

# A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
Source: [https://arxiv.org/html/2607.08776](https://arxiv.org/html/2607.08776)
###### Abstract

Despite the success of knowledge distillation \(KD\) in Large Language Models \(LLMs\), the underlying mechanism behind its efficacy remains unclear\. In this paper, we propose a unified approach to explore the common mechanism of various KD methods using interactions\. Specifically, we decompose the output score of the LLM into the sum of numerous interactions\. Each interaction represents a nonlinear relationship involving a set of input variables \(e\.g\., words\)\. Based on the decomposed interactions, we discover that the common mechanism underlying various KD methods is the sparsification of interactions,i\.e\., student models retain fewer interactions for inference while suppressing other interactions to zero effects\. Furthermore, we discover that the performance variance across different KD methods arises from their capabilities in handling complex interactions\. A KD method typically yields better performance if it enables the student model to achieve higher sparsity of complex interactions\. Motivated by these insights, we propose a plug\-and\-play loss function called Complex Interaction Penalty \(CIP\) to explicitly enforce the sparsity of complex interactions during the distillation process\. Extensive experiments demonstrate that integrating CIP consistently improves the performance of diverse KD methods on both in\-domain and out\-of\-distribution benchmarks\.

## 1Introduction

Although LLMs excel in text generation tasks, their practical deployment is constrained by substantial computing expense and high\-quality training data\. Therefore, KD\(Hintonet al\.,[2015](https://arxiv.org/html/2607.08776#bib.bib1)\)serves as an essential technique to compress LLMs into compact student models while retaining their capabilities\. Despite the empirical success of various KD methods\(Kim and Rush,[2016](https://arxiv.org/html/2607.08776#bib.bib2); Linet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib3); Agarwalet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib4); Guet al\.,[2024a](https://arxiv.org/html/2607.08776#bib.bib5); Koet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib6),[2025](https://arxiv.org/html/2607.08776#bib.bib7)\), the underlying mechanism ofwhyKD works remains unclear\. Current research primarily focuses on improving the performance of KD, yet lacks interpretability of the distillation process: is there a unified mechanism that explains why knowledge distillation consistently enhances model performance?

![Refer to caption](https://arxiv.org/html/2607.08776v1/x1.png)Figure 1:Comparing the interaction patterns encoded by the student model before and after distillation\. The distillation processdrives the sparsification of interactionsby suppressing most of the interactions to near\-zero effects, which is visualized by the fading color of the bars\. Meanwhile, the student modelimproves the alignment of interactions with the teacher modelby learning more salient interactions from the teacher model \(highlighted by red boxes\)\.In this paper, we aim to explore the common mechanism behind the efficacy of various KD methods, thereby offering better guidance for the distillation process\. Recent research\(Chenet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib46); Zhouet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib51); Renet al\.,[2025a](https://arxiv.org/html/2607.08776#bib.bib53),[b](https://arxiv.org/html/2607.08776#bib.bib54)\)has leveraged game\-theoretic interactions to explain the internal logic of LLMs\. Inspired by these studies, we employ interactions to provide a unified interpretation of the learning process across various KD methods\.

Specifically, letxxdenote an input sequence withnninput variables \(e\.g\., words\), indexed by the setN=\{1,…,n\}N=\\\{1,\\dots,n\\\}\. An interaction represents a nonlinear relationship associated with a specific combination of input variables\. Considering the sentencex=“I am a green hand means that I am a”,x=\\textit\{\`\`I\\ am\\ a\\ green\\ hand\\ means\\ that\\ I\\ am\\ a",\}the words“green”and“hand”have distinct semantic meanings; their co\-occurrence in the setS=\{green,hand\}S=\\\{\\textit\{green\},\\textit\{hand\}\\\}triggers a special interaction effectISI\_\{S\}, which pushes the LLM’s inference towards the ground truth answer“beginner\.”Li and Zhang \([2023](https://arxiv.org/html/2607.08776#bib.bib42)\)rigorously proved that the scalar outputv​\(x\)v\(x\)of an LLM is always equivalent to the output of an interaction\-based logical modelϕ​\(x\)=∑S⊆NIS\\phi\(x\)=\\sum\\nolimits\_\{S\\subseteq N\}I\_\{S\}\. That is,v​\(x\)=ϕ​\(x\)=∑S⊆NISv\(x\)=\\phi\(x\)=\\sum\\nolimits\_\{S\\subseteq N\}I\_\{S\}\. Thus, the inference logic of an LLM can be equivalently explained by a set of interactions\.

Based on interactions, we conduct a comprehensive analysis of various KD methods and uncover a unified mechanism similar toOccam’s Razor\(Blumeret al\.,[1987](https://arxiv.org/html/2607.08776#bib.bib38)\):the essence of distillation lies in the sparsification of interactions\. This means that after distillation, the student model retains much fewer interactions with high effects \(which we termsalient interactions\) and suppresses more interactions to near\-zero effects than the original model\. Crucially, we find that during this sparsification process, the student model does not retain random interaction patterns but rather retains salient interactions encoded by the teacher model\. As shown in Figure[1](https://arxiv.org/html/2607.08776#S1.F1), the student model after distillation retain more salient interactions learned from the teacher model than the model before distillation, while setting many other interactions to nearly zero effects\.

Furthermore, we investigate the underlying reasons of the above mechanism from the perspective of interaction complexity\. Specifically, the complexity of an interaction is defined by the number of input variables involved\. A simple interaction indicates the relationship between few input variables, while a complex interaction represents the relationship between many input variables\. By decomposing the overall sparsity of interactions into the sparsity of simple interactions and the sparsity of complex interactions, we identify thatthe sparsification of complex interactions is the primary driver of the overall sparsification of interactions\. In other words, a high proportion of complex interactions are suppressed, but a relatively small part of simple interactions are suppressed after KD\. In addition, for salient interactions learned from the teacher model, a higher proportion of simple interactions are retained than complex interactions\. We attribute this to the findings inZhouet al\.\([2024](https://arxiv.org/html/2607.08776#bib.bib51)\): simple interactions typically encode generalizable knowledge essential for maintaining the model’s general capabilities\. In contrast, complex interactions are more likely to be considered as non\-generalizable noises\.

Based on the above insights, we explore why the performances of different KD methods vary\. Empirically, we find that a KD method achieves better performance if it induces higher sparsity in complex interactions \(i\.e\., suppressing more complex interactions to nearly zero effects\) and when it retains more salient complex interactions encoded by the teacher model\. Successful KD methods act as a selective filter: they effectively suppress non\-salient complex interactions which are mainly considered as noise by the teacher model while faithfully preserving salient complex interactions learned from the teacher model\.

Based on the common mechanism we identified, we propose a plug\-and\-play loss term calledComplex Interaction Penalty \(CIP\)\. This penalty term explicitly enforces the sparsity of complex interactions during the training process\. By integrating CIP into current KD methods, we demonstrate that it enhances the performance of various KD methods on both in\-domain and out\-of\-distribution benchmarks\.

## 2Related Work

Knowledge distillation\.KD\(Hintonet al\.,[2015](https://arxiv.org/html/2607.08776#bib.bib1)\)and SeqKD\(Kim and Rush,[2016](https://arxiv.org/html/2607.08776#bib.bib2)\)establish the foundation for model compression\. To address the distribution mismatch, methods like ImitKD\(Linet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib3)\), GKD\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib4)\), and MiniLLM\(Guet al\.,[2024a](https://arxiv.org/html/2607.08776#bib.bib5)\)utilize student\-generated outputs for optimization\. Recent advances focus on efficiency and performance bounds: DistiLLM\(Koet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib6)\)improves efficiency via Skew KL, DistiLLM\-2\(Koet al\.,[2025](https://arxiv.org/html/2607.08776#bib.bib7)\)integrates contrastive learning, and Speculative KD\(Xuet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib8)\)bridges the teacher\-student gap through interleaved sampling strategies \(More KD methods are introduced in Appendix[A](https://arxiv.org/html/2607.08776#A1)\)\. Regarding explanation for the mechanism of KD, existing studies fall into theoretical verification and representation analysis\. Theoretical verifications\(Phuong and Lampert,[2019](https://arxiv.org/html/2607.08776#bib.bib33); Allen\-Zhu and Li,[2020](https://arxiv.org/html/2607.08776#bib.bib34)\)prove that KD accelerates convergence and transfers implicit “dark knowledge” inaccessible to single model\. Representation analyses\(Chenget al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib35); Ojhaet al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib37); Wuet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib36)\)observe that student models inherit conceptual features and causal mechanisms from teacher models\. However, a unified framework is still missing to explicitly explainthe common mechanism of why various KD methods can improve model performance\.

Using game\-theoretic interactions to explain LLMs\.Conventional explanation methods\(Tenneyet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib39); Zhanget al\.,[2020b](https://arxiv.org/html/2607.08776#bib.bib40)\)suffer from a lack of mathematical faithfulness, implying their outputs may not accurately represent the internal logic encoded by the LLM\. To bridge this gap,Renet al\.\([2023a](https://arxiv.org/html/2607.08776#bib.bib43)\)proposed to use interactions between input variables to explain LLMs and provided a series of theoretical guarantees for the method’s validity\. Furthermore,Li and Zhang \([2023](https://arxiv.org/html/2607.08776#bib.bib42)\)provided empirical evidence that interactions indeed represent meaningful concepts encoded by the LLM\.Liuet al\.\([2023](https://arxiv.org/html/2607.08776#bib.bib57)\)discovered that simple interactions are more likely to be learned by LLMs than complex interactions\. At the application level, the interaction framework has demonstrated its efficacy across a wide range of complicated tasks, such as adversarial transferability\(Denget al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib48)\), model generalization\(Chenet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib46); Zhouet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib51)\), model training process\(Renet al\.,[2025a](https://arxiv.org/html/2607.08776#bib.bib53),[b](https://arxiv.org/html/2607.08776#bib.bib54)\)and overfitting\(Zhouet al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib49); Renet al\.,[2023b](https://arxiv.org/html/2607.08776#bib.bib50)\)\. In this paper, we use the interaction framework to analyze the process of knowledge distillation\.

## 3Preliminaries: Interactions

Given an LLMvvand an input sentencexxwithnnwords111We take words instead of tokens as input variables because different LLMs may divide the same word into different tokens\. For example, GPT\-2\-0\.1B tokenizes the word “Elements” into two tokens “Element” and “s”, while LLaMA\-7B treats it as one token\.indexed byN=\{1,2,…,n\}N=\\\{1,2,\\ldots,n\\\}, letv​\(x\)∈ℝv\(x\)\\in\\mathbb\{R\}denote the scalar output of the LLM\. In practice, the scalar outputv​\(x\)v\(x\)is defined asv​\(x\)=log​p¯1−p¯∈ℝv\(x\)=\\text\{log\}\\frac\{\\bar\{p\}\}\{1\-\\bar\{p\}\}\\in\\mathbb\{R\}, wherep¯\\bar\{p\}denotes the probability of generating the ground\-truth sequence in the text generation task\. Specifically, given the ground truth sequencey∗=\(y1∗,…,yL∗\)y^\{\*\}=\(y^\{\*\}\_\{1\},\\ldots,y^\{\*\}\_\{L\}\), we definep¯\\bar\{p\}as the geometric mean of the probabilities of each ground truth token iny∗y^\{\*\},i\.e\.,p¯=\(∏l=1Lp​\(yl\*\|x,y<l\*\)\)1/L\\bar\{p\}=\\left\(\\prod\_\{l=1\}^\{L\}p\(y\_\{l\}^\{\\text\{\*\}\}\|x,y\_\{<l\}^\{\\text\{\*\}\}\)\\right\)^\{1/L\}\. Please see Appendix[C](https://arxiv.org/html/2607.08776#A3)for why we choose this metric\. Recent studies\(Li and Zhang,[2023](https://arxiv.org/html/2607.08776#bib.bib42); Renet al\.,[2023a](https://arxiv.org/html/2607.08776#bib.bib43)\)proved Theorem[3\.1](https://arxiv.org/html/2607.08776#S3.Thmtheorem1), which shows that the output scorev​\(x\)v\(x\)on any randomly masked222We employ standard masking technique, such as replacing the target token with a specific\[MASK\]token\. Please see Appendix[B](https://arxiv.org/html/2607.08776#A2)for an introduction to masking strategies\.inputxTx\_\{T\}can be accurately calculated by the logical modelϕ​\(⋅\)\\phi\(\\cdot\):

ϕ​\(xT\)≜ϕ​\(x∅\)\+∑S⊆N,S≠∅𝟙​\(S∣xT\)⋅IS,\\small\\phi\(x\_\{T\}\)\\triangleq\\phi\(x\_\{\\emptyset\}\)\+\\sum\\nolimits\_\{S\\subseteq N,S\\neq\\emptyset\}\\mathds\{1\}\(\{S\\mid x\_\{T\}\}\)\\cdot I\_\{S\},\(1\)wherexTx\_\{T\}denotes the input when we mask all the words inN∖TN\\setminus Tand keep all the words inTTunchanged\. The term𝟙​\(S∣xT\)∈\{0,1\}\\mathds\{1\}\(\{S\\mid x\_\{T\}\}\)\\in\\\{0,1\\\}serves as an indicator function for anAND interaction pattern, which represents anAND relationshipbetween input variables inSS\. The scalar weightISI\_\{S\}represents theinteraction effect, quantifying the effect of an AND relationship\. An AND relationship is activated only if every input variable within the setSSis present \(unmasked\) in the inputxTx\_\{T\}\. Taking the sentencex=“I am a green hand means that I am a”x=\\textit\{\`\`I\\ am\\ a\\ green\\ hand\\ means\\ that\\ I\\ am\\ a"\}as example, the co\-occurrence of the input variables in the setS=\{green,hand\}S=\\\{\\textit\{green\},\\textit\{hand\}\\\}pushes the surrogate logical model’s inference towards generating the ground truth answer“beginner”by adding a numerical effectISI\_\{S\}to it\. If an AND interactionSSis activated,i\.e\.,𝟙​\(S∣xT\)=1\\mathds\{1\}\(\{S\\mid x\_\{T\}\}\)=1, the interaction effectISI\_\{S\}is added to the output of the logical model\. Otherwise, if any word inSSis masked and the AND interaction is not activated,i\.e\.,𝟙​\(S∣xT\)=0\\mathds\{1\}\(\{S\\mid x\_\{T\}\}\)=0, the interaction effectISI\_\{S\}is not added to the output of the logical model\.

###### Theorem 3\.1\(Universal matching property, proven in Appendix[D\.1](https://arxiv.org/html/2607.08776#A4.SS1)\)\.

Consider an inputxxwithnninput variables\. Let\{xT∣T⊆N\}\\\{x\_\{T\}\\mid T\\subseteq N\\\}denote the set of all2n2^\{n\}different masked inputs\. For every masked inputxTx\_\{T\}, when the scalar weightISI\_\{S\}in the logical modelϕ​\(⋅\)\\phi\(\\cdot\)are set toIS=∑S′⊆S\(−1\)\|S\|−\|S′\|⋅v​\(xS′\),ϕ​\(x∅\)=v​\(x∅\)I\_\{S\}=\\sum\\nolimits\_\{S^\{\\prime\}\\subseteq S\}\(\-1\)^\{\|S\|\-\|S^\{\\prime\}\|\}\\cdot v\(x\_\{S^\{\\prime\}\}\),\\phi\(x\_\{\\emptyset\}\)=v\(x\_\{\\emptyset\}\), the output of the logical modelϕ​\(⋅\)\\phi\(\\cdot\)can always match the LLM’s output scorev​\(⋅\)v\(\\cdot\)\.

∀T⊆N,v​\(xT\)=ϕ​\(xT\)\\small\\forall\\ T\\subseteq N,\\ \\ v\(x\_\{T\}\)=\\phi\(x\_\{T\}\)\(2\)

Equation \([1](https://arxiv.org/html/2607.08776#S3.E1)\) and Theorem[3\.1](https://arxiv.org/html/2607.08776#S3.Thmtheorem1)establish a rigorous theoretical foundation, allowing us to treat interactions as the detailed inference patterns that constitute the model’s internal logic\.

## 4Explaining KD via Interactions

### 4\.1Extracting Interactions in LLMs

In this paper, we use interactions as an analytical tool to investigate how interaction patterns of the student model change before and after distillation, thereby analyzing the underlying mechanism of the distillation process\.

Experiment setup\. We conduct experiments on six widely used KD methods, including KD\(Hintonet al\.,[2015](https://arxiv.org/html/2607.08776#bib.bib1)\), SeqKD\(Kim and Rush,[2016](https://arxiv.org/html/2607.08776#bib.bib2)\), ImitKD\(Linet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib3)\), MiniLLM\(Guet al\.,[2024a](https://arxiv.org/html/2607.08776#bib.bib5)\), GKD\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib4)\), and DISTILLM\(Koet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib6)\)\. For comparison, we employ the original pre\-trained student model without any further training as theBasemodel and the student model trained via standard supervised fine\-tuning as theSFTmodel\. We analyze three LLM families with the following teacher\-student configurations: \(1\) GPT\-2 family\(Radfordet al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib64)\), we use GPT\-2\-1\.5B as the teacher model and select GPT\-2\-0\.1B, 0\.3B, and 0\.7B as student models; \(2\) OPT family\(Zhanget al\.,[2022](https://arxiv.org/html/2607.08776#bib.bib65)\), we use OPT\-13B as the teacher model and select OPT\-1\.3B, 2\.7B, and 6\.7B as student models; and \(3\) LLaMA family\(Touvronet al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib66)\), we use LLaMA\-13B as the teacher model and select LLaMA\-7B as the student model\. All student models are distilled on thedatabricks\-dolly\-15kdataset\(Conoveret al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib68)\)\. We followKoet al\.\([2024](https://arxiv.org/html/2607.08776#bib.bib6)\)to split the dataset into training, validation and test set\. We evaluate model performance using the ROUGE\-L metric\(Lin,[2004](https://arxiv.org/html/2607.08776#bib.bib67)\)and compute interactions on the test set\. For computational feasibility, we take words in the“Instruction”segment of the input as input variables\. Meanwhile, the system prompt is treated as fixed, unmasked background context\. Please see Appendix[E](https://arxiv.org/html/2607.08776#A5)for the details of implementation\.

Figure[2](https://arxiv.org/html/2607.08776#S4.F2)plots the probability density of absolute values of the normalized interaction effects for a representative input samplexx\. Specifically, the normalized interaction effect is calculated asI~S=ISMax\\widetilde\{I\}\_\{S\}=\\frac\{I\_\{S\}\}\{\\textit\{Max\}\}, whereMaxis the maximum absolute values of all2n−12^\{n\}\-1interaction effects ofxx\. Results show that the non\-distilled model exhibits a relatively flat and uniform distribution of interactions\. In contrast, the distributions of distilled models exhibit an obvious peak near zero, indicating that the majority of interactions are suppressed to nearly zero effects\. It demonstrates that the distribution of interactions turns more sparse after the distillation process\. Notably, we observe that SFT models do not exhibit similar level of sparsification as distilled models\. We hypothesize thatsparsification of interactions is a unique mechanism of KD, distinguishing it from standard training methods\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x2.png)Figure 2:Comparison of the probability density between distilled and non\-distilled models\. Non\-distilled models exhibit a flat distribution, while distilled models exhibit sharp peaks near zero\.
### 4\.2Interaction\-Based Metrics for Analyzing KD

Inspired by the phenomenon, in this subsection, we propose three interaction\-based metrics to analyze how a KD method change the interactions encoded by the student model\.

Interaction sparsity\.To evaluate the uniformity of the interaction distribution encoded by LLMs and thereby quantify the sparsity of interactions, we introduce theGini coefficientandShannon entropy\.

Gini coefficient\. Letℐ=\{\|I~S\|:S⊆N,S≠∅\}\\mathcal\{I\}=\\\{\|\\widetilde\{I\}\_\{S\}\|:S\\subseteq N,S\\neq\\emptyset\\\}denote the set of absolute value of all normalized interaction effects extracted from the inputxx\. LetM=\|ℐ\|=2n−1M=\|\\mathcal\{I\}\|=2^\{n\}\-1denote the number of all the interactions\. Let the values inℐ\\mathcal\{I\}be sorted in ascending order asu1≤u2≤⋯≤uMu\_\{1\}\\leq u\_\{2\}\\leq\\dots\\leq u\_\{M\}\. For the inputxx, the Gini coefficientG​\(x\)G\(x\)is defined as:

G​\(x\)=2​∑i=1Mi​uiM​∑i=1Mui−M\+1M\.\\small G\(x\)=\\frac\{2\\sum\_\{i=1\}^\{M\}iu\_\{i\}\}\{M\\sum\_\{i=1\}^\{M\}u\_\{i\}\}\-\\frac\{M\+1\}\{M\}\.\(3\)
Shannon entropy\. The Shannon entropyH​\(x\)H\(x\)\(abbreviated as entropy\) is defined asH​\(x\)=−∑SpS​log⁡pSH\(x\)=\-\\sum\_\{S\}p\_\{S\}\\log p\_\{S\}, wherepS=\|I~S\|/∑S′\|I~S′\|p\_\{S\}=\|\\widetilde\{I\}\_\{S\}\|/\\sum\_\{S^\{\\prime\}\}\|\\widetilde\{I\}\_\{S^\{\\prime\}\}\|is the normalized probability\. A highG​\(x\)G\(x\)or low entropyH​\(x\)H\(x\)indicates a sharp interaction distribution, reflecting high sparsity of interactions\.

Interaction alignment\.To evaluate whether the student model inherits the critical interaction patterns of the teacher model during distillation, we employ the overlap rate of interactions to measure the alignment of the salient interactions between the student and teacher model\. We define a threshold ratiok∈\(0,1\]k\\in\(0,1\]\. LetΩteacher\(k\)\\Omega\_\{\\text\{teacher\}\}^\{\(k\)\}andΩstudent\(k\)\\Omega\_\{\\text\{student\}\}^\{\(k\)\}represent the sets containing the indices of the top\-⌊k×M⌋\\lfloor k\\times M\\rfloorinteractions with the largest absolute effects in the teacher and student models, respectively\. For the inputxx, the student\-teacher overlap rateO​v​e​r​l​a​p​@​k​\(x\)Overlap@k\(x\)is defined as:

O​v​e​r​l​a​p​@​k​\(x\)=\|Ωteacher\(k\)∩Ωstudent\(k\)\|⌊k×M⌋\.\\small Overlap@k\(x\)=\\frac\{\\left\|\\Omega\_\{\\text\{teacher\}\}^\{\(k\)\}\\cap\\Omega\_\{\\text\{student\}\}^\{\(k\)\}\\right\|\}\{\\lfloor k\\times M\\rfloor\}\.\(4\)A high overlap rate indicates that the student model has effectively captured the teacher model’s salient interactions, implying a strong logic alignment\.

### 4\.3Exploring the Common Mechanism of KD Methods

Based on the proposed metrics, we discover the following two common mechanisms shared by different KD methods\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x3.png)Figure 3:The distribution of the Gini coefficient and entropy across all samples\. The results demonstrate that, compared to non\-distilled models \(Base models and SFT models\), distilled models consistently exhibit higher Gini coefficients and lower entropy\. It indicates that the interactions encoded by the LLM become more sparse after distillation\.Table 1:Comparing the student\-teacher overlap rate of salient interactions before and after distillation\.𝔼x​\[O​v​e​r​l​a​p​@​k​\(x\)\]\\mathbb\{E\}\_\{x\}\\left\[Overlap@k\(x\)\\right\]represents the average overlap rate across all samples\.Distillation enhances interaction sparsity\.We employ the Gini coefficientG​\(x\)G\(x\)and Shannon entropyH​\(x\)H\(x\)to quantify interaction sparsity across the whole dataset\. Figure[3](https://arxiv.org/html/2607.08776#S4.F3)illustrates the distributions of these metrics across all samples\. Results333Results on more model family and model size in Appendix[G\.2](https://arxiv.org/html/2607.08776#A7.SS2)exhibit the same conclusion\.show that distilled student models demonstrate an obvious distributional shift towards higher Gini values and lower entropy compared to corresponding base models and SFT models\. It proves that the distillation process enhances the sparsity of interactions encoded by the student model\.

Distillation enhances interaction alignment\.The above phenomenon is counter\-intuitive\. Although KD suppresses most of the interactions to nearly zero effects and makes the student model rely on fewer salient interactions for inference, it conversely improves the performance of the student model\. To explain such a counter\-intuitive phenomenon, we further analyze the relationship between the salient interactions retained by the distilled student model and those encoded by the teacher model by the metricO​v​e​r​l​a​p​@​k​\(x\)Overlap@k\(x\)\.

Table[1](https://arxiv.org/html/2607.08776#S4.T1)shows that distilled student models exhibit a higher overlap rate with the teacher model in terms of salient interactions than that of the non\-distilled student models[3](https://arxiv.org/html/2607.08776#footnote3)\. This indicates that KD drives student models to learn more salient interactions encoded by the teacher model, which accounts for the enhanced performance of the distilled model\.

In conclusion, the sparsification of interactions demonstrates that the distillation process effectively compresses interaction patterns encoded by the student model\. Furthermore, the alignment of salient interactions between the student model and the teacher model confirms that the compression is achieved by selectivelylearning the teacher model’s salient interactionswhilediscarding other interactions\.

### 4\.4Exploring the Underlying Reasons behind the Common Mechanism

The mechanism of KD can be summarized as“discarding the dross and selecting the essential\.”To explore the underlying reason of the mechanism, we further identify which specific types of interactions are primarily treated as“dross”to be discarded, and which specific types of interactions are treated as“essential”to be selected\.

To answer the above question, we analyze interactions with different complexities\. The complexity of an interactionSSis defined as the number of input variables involved,i\.e\.,\|S\|\|S\|\. Based on this, we partition interactions into two types: simple interactions, which represent relationships involving few input variables, and complex interactions, which represent relationships involving many input variables\. In this way, the overall sparsity of all interactions can be decomposed into the sparsity of simple interactions and the sparsity of complex interactions\. FollowingLiet al\.\([2025](https://arxiv.org/html/2607.08776#bib.bib72)\), we partition interactions into two groups:Ωsimple\\Omega^\{\\text\{simple\}\}andΩcomplex\\Omega^\{\\text\{complex\}\}\. Given an input sentencexxwithnnwords indexed byN=\{1,…,n\}N=\\\{1,\\dots,n\\\}, we defineΩsimple=\{S⊆N∣1≤\|S\|<β\},Ωcomplex=\{S⊆N∣β≤\|S\|≤n\}\\Omega^\{\\text\{simple\}\}=\\\{S\\subseteq N\\mid 1\\leq\|S\|<\\beta\\\},\\Omega^\{\\text\{complex\}\}=\\\{S\\subseteq N\\mid\\beta\\leq\|S\|\\leq n\\\}, where we setβ=⌈n/3⌉\\beta=\\lceil n/3\\rceil\. To verify the robustness of the choice ofβ\\beta, we also conduct experiments onβ=⌈n/5⌉\\beta=\\lceil n/5\\rceilandβ=⌈n/2⌉\\beta=\\lceil n/2\\rceil\. Results in Appendix[G\.3](https://arxiv.org/html/2607.08776#A7.SS3)show that our findings are consistent when choosing differentβ\\beta\. Accordingly, we denote the metrics in Section[4\.2](https://arxiv.org/html/2607.08776#S4.SS2)for interaction typeτ∈\{simple,complex\}\\tau\\in\\\{\\text\{simple\},\\text\{complex\}\\\}and model typeℱ∈\{base,distilled\}\\mathcal\{F\}\\in\\\{\\text\{base\},\\text\{distilled\}\\\}asGℱτ​\(x\)G^\{\\tau\}\_\{\\mathcal\{F\}\}\(x\),Hℱτ​\(x\)H^\{\\tau\}\_\{\\mathcal\{F\}\}\(x\), andOverlap​@​kℱτ​\(x\)\\text\{Overlap\}@k^\{\\tau\}\_\{\\mathcal\{F\}\}\(x\), respectively\.Gℱτ​\(x\)G^\{\\tau\}\_\{\\mathcal\{F\}\}\(x\)is computed by sorting all interaction effects in ascending order in the subsetΩτ\\Omega^\{\\tau\}\.Hℱτ​\(x\)H^\{\\tau\}\_\{\\mathcal\{F\}\}\(x\)is derived from a probability distributionre\-normalizedwithin the subsetΩτ\\Omega^\{\\tau\}, defined aspS=\|I~S\|/∑S′∈Ωτ\|I~S′\|p\_\{S\}=\|\\widetilde\{I\}\_\{S\}\|/\\sum\_\{S^\{\\prime\}\\in\\Omega^\{\\tau\}\}\|\\widetilde\{I\}\_\{S^\{\\prime\}\}\|\.Overlap​@​kℱτ​\(x\)\\text\{Overlap\}@k^\{\\tau\}\_\{\\mathcal\{F\}\}\(x\)is defined as the overlap rate of top\-kkinteractions rankedexclusivelywithin the subsetΩτ\\Omega^\{\\tau\}\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x4.png)Figure 4:Comparison of the changes in Gini coefficients between distilled and base models for simple versus complex interactions\.![Refer to caption](https://arxiv.org/html/2607.08776v1/x5.png)Figure 5:Comparison of student\-teacher overlap rate between simple interactions and complex interactions after distillation\. Results show that the student\-teacher overlap rate of simple interactions is higher compared to complex interactions after distillation\.![Refer to caption](https://arxiv.org/html/2607.08776v1/x6.png)Figure 6:Correlation between the sparsity of complex interactions and model performance\. Results show that increased sparsity \(higher Gini coefficients and lower entropy\) of complex interactions is positively associated with higher ROUGE\-L scores\. However, there is no distinct correlation between the sparsity of simple interactions and model performance, as shown in[Figure16](https://arxiv.org/html/2607.08776#A7.F16)\.First, we aim to explore whether the sparsification of interactions is driven mainly by the sparsification of simple interactions or complex interactions\. Figure[4](https://arxiv.org/html/2607.08776#S4.F4)plots the change of Gini coefficients between distilled and base models for both simple and complex interactions, which is defined asΔ​Gτ​\(x\)=Gdistilledτ​\(x\)−Gbaseτ​\(x\)\\Delta G^\{\\tau\}\(x\)=G^\{\\tau\}\_\{\\text\{distilled\}\}\(x\)\-G^\{\\tau\}\_\{\\text\{base\}\}\(x\)\. Results[3](https://arxiv.org/html/2607.08776#footnote3)show that the sparsity of complex interactions exhibits an obvious increase after distillation\. In contrast, the sparsity of simple interactions exhibits lower increases or even decrease in some cases\. This indicates thatthe sparsification of complex interactions contributes more to the overall sparsification of interactions after distillation\. In other words, the interactions discarded by the LLM during the distillation process are more likely to be complex interactions\.

Second, we investigate whether the salient interactions learned from the teacher model are mainly simple or complex interactions\. Specifically, we compare the overlap rate of simple interactions against that of complex interactions between the teacher and student models\. Figure[5](https://arxiv.org/html/2607.08776#S4.F5)shows that the overlap rate of salient simple interactions consistently surpasses that of complex interactions[3](https://arxiv.org/html/2607.08776#footnote3)\. This indicates thatthe student model learn more simple salient interactions than complex salient interactions from the teacher model\.

Thus, we can conclude thatcomplex interactions are primarily treated as“dross”to be discarded and simple interactions are primarily treated as“essential”to be selected during the distillation process\. We attribute this phenomenon to the findings ofZhouet al\.\([2024](https://arxiv.org/html/2607.08776#bib.bib51)\), which suggest that simple interactions typically encode fundamental, generalizable knowledge, while most of the complex interactions are considered as non\-generalizable noise\. Thus, the underlying reason of the common mechanism is thatthe distillation process makes the student model retain more simple interactions to stabilize the student model’s general capabilities, while discarding more complex interactions to filter out non\-generalizable noise\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x7.png)Figure 7:Correlation between the student\-teacher overlap rate of complex interactions and model performance\. Results show that increased student\-teacher overlap rate of complex interactions is positively associated with higher ROUGE\-L scores\.
### 4\.5Explaining the Performance Variance across Different KD methods

We further explorewhy the performances of different KD methods differfrom the perspective of interactions\. Specifically, we discover the correlation between the sparsity of complex interactions and the model performance of student models across different KD methods\. As shown in Figure[6](https://arxiv.org/html/2607.08776#S4.F6),the sparsity of complex interactions is positively correlated with model performance[3](https://arxiv.org/html/2607.08776#footnote3)\. Student models that exhibit higher Gini coefficients and lower entropy \(indicating higher sparsity\) in their complex interactions generally achieve superior ROUGE\-L scores\. Furthermore, we analyze the relationship between the student\-teacher overlap rate of complex interactions and model performance\. As shown in Figure[7](https://arxiv.org/html/2607.08776#S4.F7), the results[3](https://arxiv.org/html/2607.08776#footnote3)reveala positive correlation between model performance and the overlap rate of complex interactions\. This indicates that student models with higher performance learn more complex interactions from the teacher model compared to student models with lower performance\. Counter\-intuitively, the overlap rate of simple interactions shows no distinct correlation with model performance, as shown in[Figure18](https://arxiv.org/html/2607.08776#A7.F18)in Appendix[G\.2](https://arxiv.org/html/2607.08776#A7.SS2)\. Across all the results, the Pearson correlation coefficient between simple interactions and model performance is 0\.34 \(±\\pm0\.12\), whereas for complex interactions, it is 0\.81 \(±\\pm0\.11\)\. This implies that while retaining more simple interactions is a common reason for general performance improvement in all KD methods, it does not account for the performance gap between different KD methods\. Instead, the retainment of complex interactions serves as the decisive factor\.

In conclusion, the performance variance across KD methods is related to their abilities in handling complex interactions\. In general, superior performance is achieved when student models exhibit higher sparsity and higher alignment \(overlap rate\) with the teacher model in complex interactions\.

## 5Guiding KD via Interactions

In[Section4\.5](https://arxiv.org/html/2607.08776#S4.SS5), we find that the sparsity of complex interactions is positively correlated with model performance\. The more complex interactions are suppressed to nearly zero effects, the better the model performance becomes\. Based on this insight, we introduce a plug\-and\-play loss function calledComplexInteractionPenalty \(CIP\), which is designed to suppress complex interactions encoded by the student model during distillation\. We define the loss functionℒCIP\\mathcal\{L\}\_\{\\text\{CIP\}\}as follows\.

ℒCIP=𝔼x​\[𝔼S∈Ωcomplex​\[\|I​\(S\)\|\]\]\\small\\mathcal\{L\}\_\{\\text\{CIP\}\}=\\mathbb\{E\}\_\{x\}\\left\[\\mathbb\{E\}\_\{S\\in\\Omega^\{\\text\{complex\}\}\}\[\|I\(S\)\|\]\\right\]\(5\)
![Refer to caption](https://arxiv.org/html/2607.08776v1/x8.png)Figure 8:The comparison of ROUGE\-L and GPT\-5 scores between different KD methods without and with the CIP loss function across various model families and model sizes\. Results show that adding the CIP loss improves the model performance on almost all the KD methods and datasets\. Please see[Table5](https://arxiv.org/html/2607.08776#A7.T5)in Appendix[G](https://arxiv.org/html/2607.08776#A7)for results of all models with the CIP loss\.According to[Equation5](https://arxiv.org/html/2607.08776#S5.E5), directly computing all complex interactions is NP\-hard\. To alleviate computational complexity while preserving efficacy, we adopt a sampling method to approximateℒCIP\\mathcal\{L\}\_\{\\text\{CIP\}\}by followingZhanget al\.\([2020a](https://arxiv.org/html/2607.08776#bib.bib73)\)andLiet al\.\([2025](https://arxiv.org/html/2607.08776#bib.bib72)\)\. Specifically, given an input sequencexxwithnnwords indexed byN=\{1,…,n\}N=\\\{1,\\dots,n\\\}, we randomly partition the setNNintommdisjoint subsetsS1,S2,…,SmS\_\{1\},S\_\{2\},\\dots,S\_\{m\}, such that⋃i=1mSi=N\\bigcup\_\{i=1\}^\{m\}S\_\{i\}=NandSi∩Sj=∅S\_\{i\}\\cap S\_\{j\}=\\emptyset\. Based on this, we define the approximated loss termℒCIP′\\mathcal\{L\}^\{\\prime\}\_\{\\text\{CIP\}\}as follows\.

ℒCIP′=𝔼x​\[𝔼K⊆\{1,…,m\},K≠∅​\[\|I​\(⋃i∈KSi\)\|\]\]\\small\\mathcal\{L\}\_\{\\text\{CIP\}\}^\{\\prime\}=\\mathbb\{E\}\_\{x\}\\left\[\\mathbb\{E\}\_\{K\\subseteq\\\{1,\\dots,m\\\},K\\neq\\emptyset\}\\left\[\\left\|I\\left\(\\bigcup\_\{i\\in K\}S\_\{i\}\\right\)\\right\|\\right\]\\right\]\(6\)To ensure that any sampled interaction⋃i∈KSi\\bigcup\_\{i\\in K\}S\_\{i\}is a complex interaction, that is,\|⋃i∈KSi\|≥⌈n/3⌉\|\\bigcup\_\{i\\in K\}S\_\{i\}\|\\geq\\lceil n/3\\rceil,mmmust meet the requirementm≤n⌈n/3⌉≤3m\\leq\\frac\{n\}\{\\lceil n/3\\rceil\}\\leq 3\. The detailed proof is in Appendix[D\.2](https://arxiv.org/html/2607.08776#A4.SS2)\. Therefore, the approximation method effectively reduces the complexity from2n2^\{n\}to2m2^\{m\}, making the computational cost negligible compared to other steps in the distillation\. See Appendix[G\.1](https://arxiv.org/html/2607.08776#A7.SS1)for the comparison of training time with and without the CIP loss\.

The total loss function for KD is defined as the weighted sum of the original distillation lossℒKD\\mathcal\{L\}\_\{\\text\{KD\}\}and the CIP loss:

ℒ=ℒKD\+λ​ℒCIP′,\\small\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{KD\}\}\+\\lambda\\mathcal\{L\}\_\{\\text\{CIP\}\}^\{\\prime\},\(7\)whereλ\\lambdais the weight of the CIP loss \(λ\>0\\lambda\>0\)\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x9.png)Figure 9:Exploring the effects of the weightλ\\lambdafor on the sparsity of complex interactions \(a and b\), the performance of the student model \(c\), and the student\-teacher overlap rate of complex interactions \(d\)\. Here the base model is GPT\-2\-0\.3B\.We integrate the proposed CIP loss into six widely used KD methods and train on three LLM families, same as the experimental setup in[Section4\.2](https://arxiv.org/html/2607.08776#S4.SS2)\. The LLMs are distilled on thedatabricks\-dolly\-15kdataset and evaluated across four diverse instruction\-following benchmarks, including DollyEval, SelfInst\(Wanget al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib69)\), Super\-Natural Instructions\(Wanget al\.,[2022](https://arxiv.org/html/2607.08776#bib.bib70)\), and Vicuna evaluation\(Chianget al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib71)\), to comprehensively assess their capabilities\. We adopt two metrics for evaluation: ROUGE\-L\(Lin,[2004](https://arxiv.org/html/2607.08776#bib.bib67)\)and GPT\-5 feedback\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib78)\)\. See Appendix[F](https://arxiv.org/html/2607.08776#A6)for details on the experimental setup\.

[Figure8](https://arxiv.org/html/2607.08776#S5.F8)presents a detailed performance comparison across varying models and KD baselines, both with and without the CIP loss function \(Here the weightλ\\lambdais set as 0\.001 for all the models\)\. Results show thatadding the CIP loss yields improvements in ROUGE\-L and GPT\-5 scores in most of the settings, regardless of the specific model or KD method\. Crucially, these improvements demonstrate strong generalization capabilities, extending beyond the in\-domain Dolly dataset to out\-of\-distribution benchmarks like Self\-Instruct, SNI and Vicuna\. In addition,[Table6](https://arxiv.org/html/2607.08776#A7.T6)in Appendix[G](https://arxiv.org/html/2607.08776#A7)shows that student models trained with CIP exhibit higher Gini coefficients and lower entropy compared to their counterparts without CIP\. This confirms thatthe CIP loss can truly increase the sparsity of complex interactions, thus effectively improving the model performance\.

Impact of the hyperparameterλ\\lambda\.As shown in[Figure9](https://arxiv.org/html/2607.08776#S5.F9)\(a\) and \(b\), the sparsity of complex interactions increases withλ\\lambda\. In contrast, the ROUGE\-L score \([Figure9](https://arxiv.org/html/2607.08776#S5.F9)\(c\)\) and the overlap rate of complex interactions \([Figure9](https://arxiv.org/html/2607.08776#S5.F9)\(d\)\) initially improve withλ\\lambda, but subsequently degrade asλ\\lambdacontinues to increase\. We attribute this to the trade\-off between the two loss terms in Equation \([7](https://arxiv.org/html/2607.08776#S5.E7)\), whereℒ′CIP\\mathcal\{L^\{\\prime\}\}\_\{\\text\{CIP\}\}is designed to suppress the effects of complex interactions, whileℒKD\\mathcal\{L\}\_\{\\text\{KD\}\}compels the student model to align the output consistency with the teacher model\. Whenλ\\lambdais relatively small, the impact ofℒ′CIP\\mathcal\{L^\{\\prime\}\}\_\{\\text\{CIP\}\}is mild\. At this stage,ℒKD\\mathcal\{L\}\_\{\\text\{KD\}\}guides the student model to prioritize the suppression of task\-irrelevant complex interactions while retaining salient interactions learned from the teacher model to maintain output consistency\. Consequently, the overlap rate of complex interactions increases, driving an improvement in model performance\. However, asλ\\lambdacontinues to increase, the strong impact ofℒ′CIP\\mathcal\{L^\{\\prime\}\}\_\{\\text\{CIP\}\}leads to a decline in the effects of salient complex interactions learned from the teacher model\. Consequently, although the sparsity of complex interactions increases, the overlap rate drops significantly, leading to the degradation in model performance\. Therefore, there exists an optimal range for the weightλ\\lambda\. By choosing a validλ\\lambda, we can strike a balance between the sparsification of interactions and the retainment of salient interactions learned from the teacher model\.

## 6Conclusion

In this paper, we present a comprehensive study on the mechanism of KD via interactions\. We discover the common mechanism of KD is retaining salient simple interactions of the teacher model while suppressing other interactions\. We further demonstrate that the performance gap among various KD methods is positively correlated with the sparsity and alignment of complex interactions\. Guided by these observations, we introduce a plug\-and\-play loss function named CIP that explicitly suppress complex interactions\. Experiments confirm that CIP generally improves various KD baselines on both in\-domain and out\-of\-distribution benchmarks\. Our study offers a new perspective on the “black box” of KD and inspires future research to utilize interactions to optimize the distillation process of LLMs\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. Bachem \(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InThe Twelfth International Conference on Learning Representations,Cited by:[5th item](https://arxiv.org/html/2607.08776#A5.I1.i5.p1.1),[§1](https://arxiv.org/html/2607.08776#S1.p1.1),[§2](https://arxiv.org/html/2607.08776#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- Z\. Allen\-Zhu and Y\. Li \(2020\)Towards understanding ensemble, knowledge distillation and self\-distillation in deep learning\.arXiv preprint arXiv:2012\.09816\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p1.1)\.
- M\. Ancona, C\. Oztireli, and M\. Gross \(2019\)Explaining deep neural networks with a polynomial time algorithm for shapley value approximation\.InInternational conference on machine learning,pp\. 272–281\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- S\. Bird, E\. Klein, and E\. Loper \(2009\)Natural language processing with python: analyzing text with the natural language toolkit\.” O’Reilly Media, Inc\.”\.Cited by:[Appendix E](https://arxiv.org/html/2607.08776#A5.p1.4)\.
- A\. Blumer, A\. Ehrenfeucht, D\. Haussler, and M\. K\. Warmuth \(1987\)Occam’s razor\.Information processing letters24\(6\),pp\. 377–380\.Cited by:[§1](https://arxiv.org/html/2607.08776#S1.p4.1)\.
- L\. Chen, S\. Lou, B\. Huang, and Q\. Zhang \(2024\)Defining and extracting generalizable interaction primitives from dnns\.arXiv preprint arXiv:2401\.16318\.Cited by:[§1](https://arxiv.org/html/2607.08776#S1.p2.1),[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- X\. Cheng, Z\. Rao, Y\. Chen, and Q\. Zhang \(2020\)Explaining knowledge distillation by quantifying the knowledge\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 12925–12935\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p1.1)\.
- W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez,et al\.\(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality\.See https://vicuna\. lmsys\. org \(accessed 14 April 2023\)2\(3\),pp\. 6\.Cited by:[3rd item](https://arxiv.org/html/2607.08776#A6.I1.i3.p1.1),[§5](https://arxiv.org/html/2607.08776#S5.p4.1)\.
- J\. H\. Cho and B\. Hariharan \(2019\)On the efficacy of knowledge distillation\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 4794–4802\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- M\. Conover, M\. Hayes, A\. Mathur, J\. Xie, J\. Wan, S\. Shah, A\. Ghodsi, P\. Wendell, M\. Zaharia, and R\. Xin \(2023\)Free dolly: introducing the world’s first truly open instructiontuned llm\.Cited by:[1st item](https://arxiv.org/html/2607.08776#A6.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- I\. Covert, S\. Lundberg, and S\. Lee \(2020\)Feature removal is a unifying principle for model explanation methods\.arXiv preprint arXiv:2011\.03623\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- P\. Dabkowski and Y\. Gal \(2017\)Real time image saliency for black box classifiers\.Advances in neural information processing systems30\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- H\. Deng, N\. Zou, M\. Du, W\. Chen, G\. Feng, Z\. Yang, Z\. Li, and Q\. Zhang \(2024\)Unifying fourteen post\-hoc attribution methods with taylor interactions\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(7\),pp\. 4625–4640\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- R\. C\. Fong and A\. Vedaldi \(2017\)Interpretable explanations of black boxes by meaningful perturbation\.InProceedings of the IEEE international conference on computer vision,pp\. 3429–3437\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- R\. Fong, M\. Patrick, and A\. Vedaldi \(2019\)Understanding deep networks via extremal perturbations and smooth masks\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 2950–2958\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- A\. Gokaslan, V\. Cohen, E\. Pavlick, and S\. Tellex \(2019\)Openwebtext corpus\.Cited by:[§F\.1](https://arxiv.org/html/2607.08776#A6.SS1.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024a\)MiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[4th item](https://arxiv.org/html/2607.08776#A5.I1.i4.p1.1),[§F\.2](https://arxiv.org/html/2607.08776#A6.SS2.p1.1),[§1](https://arxiv.org/html/2607.08776#S1.p1.1),[§2](https://arxiv.org/html/2607.08776#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- Y\. Gu, H\. Zhou, F\. Meng, J\. Zhou, and M\. Huang \(2024b\)Miniplm: knowledge distillation for pre\-training language models\.arXiv preprint arXiv:2410\.17215\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[1st item](https://arxiv.org/html/2607.08776#A5.I1.i1.p1.1),[§1](https://arxiv.org/html/2607.08776#S1.p1.1),[§2](https://arxiv.org/html/2607.08776#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. Liu \(2020\)Tinybert: distilling bert for natural language understanding\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 4163–4174\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- Y\. Jin, J\. Wang, and D\. Lin \(2023\)Multi\-level logit distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 24276–24285\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- G\. Kim, D\. Jang, and E\. Yang \(2024\)PromptKD: distilling student\-friendly knowledge for generative language models via prompt tuning\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 6266–6282\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- Y\. Kim and A\. M\. Rush \(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 1317–1327\.Cited by:[2nd item](https://arxiv.org/html/2607.08776#A5.I1.i2.p1.1),[§1](https://arxiv.org/html/2607.08776#S1.p1.1),[§2](https://arxiv.org/html/2607.08776#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- J\. Ko, T\. Chen, S\. Kim, T\. Ding, L\. Liang, I\. Zharkov, and S\. Yun \(2025\)Distillm\-2: a contrastive approach boosts the distillation of llms\.arXiv preprint arXiv:2503\.07067\.Cited by:[§1](https://arxiv.org/html/2607.08776#S1.p1.1),[§2](https://arxiv.org/html/2607.08776#S2.p1.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: towards streamlined distillation for large language models\.InInternational Conference on Machine Learning,pp\. 24872–24895\.Cited by:[6th item](https://arxiv.org/html/2607.08776#A5.I1.i6.p1.1),[3rd item](https://arxiv.org/html/2607.08776#A6.I1.i3.p1.1),[§F\.1](https://arxiv.org/html/2607.08776#A6.SS1.p1.1),[§F\.2](https://arxiv.org/html/2607.08776#A6.SS2.p1.1),[§1](https://arxiv.org/html/2607.08776#S1.p1.1),[§2](https://arxiv.org/html/2607.08776#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- H\. Lee, Y\. Park, H\. Seo, and M\. Kang \(2023\)Self\-knowledge distillation via dropout\.Computer Vision and Image Understanding233,pp\. 103720\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- L\. Li, Y\. Bao, P\. Dong, C\. Yang, A\. Li, W\. Luo, Q\. Liu, W\. Xue, and Y\. Guo \(2024\)Detkds: knowledge distillation search for object detectors\.InForty\-first International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- M\. Li and Q\. Zhang \(2023\)Does a neural network really encode symbolic concepts?\.InInternational conference on machine learning,pp\. 20452–20469\.Cited by:[§1](https://arxiv.org/html/2607.08776#S1.p3.9),[§2](https://arxiv.org/html/2607.08776#S2.p2.1),[§3](https://arxiv.org/html/2607.08776#S3.p1.15)\.
- Q\. Li, J\. Ruan, F\. Wu, Y\. Chen, Z\. Wei, and W\. Shen \(2025\)A unified approach to interpreting self\-supervised pre\-training methods for 3d point clouds via interactions\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 27315–27324\.Cited by:[§G\.3](https://arxiv.org/html/2607.08776#A7.SS3.p1.4),[§4\.4](https://arxiv.org/html/2607.08776#S4.SS4.p2.26),[§5](https://arxiv.org/html/2607.08776#S5.p2.10)\.
- C\. Liang, H\. Jiang, Z\. Li, X\. Tang, B\. Yin, and T\. Zhao \(2023a\)HomoDistil: homotopic task\-agnostic distillation of pre\-trained transformers\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- C\. Liang, S\. Zuo, Q\. Zhang, P\. He, W\. Chen, and T\. Zhao \(2023b\)Less is more: task\-aware layer\-wise distillation for language model compression\.InInternational Conference on Machine Learning,pp\. 20852–20867\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- A\. Lin, J\. Wohlwend, H\. Chen, and T\. Lei \(2020\)Autoregressive knowledge distillation through imitation learning\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 6121–6133\.Cited by:[3rd item](https://arxiv.org/html/2607.08776#A5.I1.i3.p1.1),[§1](https://arxiv.org/html/2607.08776#S1.p1.1),[§2](https://arxiv.org/html/2607.08776#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- C\. Lin \(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1),[§5](https://arxiv.org/html/2607.08776#S5.p4.1)\.
- D\. Liu, H\. Deng, X\. Cheng, Q\. Ren, K\. Wang, and Q\. Zhang \(2023\)Towards the difficulty for a deep neural network to learn concepts of different complexities\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- S\. M\. Lundberg and S\. Lee \(2017\)A unified approach to interpreting model predictions\.Advances in neural information processing systems30\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- J\. Lv, H\. Yang, and P\. Li \(2024\)Wasserstein distance rivals kullback\-leibler divergence for knowledge distillation\.Advances in Neural Information Processing Systems37,pp\. 65445–65475\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- S\. I\. Mirzadeh, M\. Farajtabar, A\. Li, N\. Levine, A\. Matsukawa, and H\. Ghasemzadeh \(2020\)Improved knowledge distillation via teacher assistant\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 5191–5198\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- U\. Ojha, Y\. Li, A\. Sundara Rajan, Y\. Liang, and Y\. J\. Lee \(2023\)What knowledge gets distilled in knowledge distillation?\.Advances in Neural Information Processing Systems36,pp\. 11037–11048\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[1st item](https://arxiv.org/html/2607.08776#A6.I1.i1.p1.1)\.
- M\. Phuong and C\. Lampert \(2019\)Towards understanding knowledge distillation\.InInternational conference on machine learning,pp\. 5142–5151\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- J\. Ren, M\. Li, Q\. Chen, H\. Deng, and Q\. Zhang \(2023a\)Defining and quantifying the emergence of sparse concepts in dnns\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 20280–20289\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p2.1),[§3](https://arxiv.org/html/2607.08776#S3.p1.15)\.
- J\. Ren, X\. Zheng, J\. Liu, A\. Lizarraga, Y\. N\. Wu, L\. Lin, and Q\. Zhang \(2025a\)Monitoring primitive interactions during the training of dnns\.Proceedings of the AAAI Conference on Artificial Intelligence39\(19\),pp\. 20183–20191\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/34223),[Document](https://dx.doi.org/10.1609/aaai.v39i19.34223)Cited by:[§1](https://arxiv.org/html/2607.08776#S1.p2.1),[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- Q\. Ren, H\. Deng, Y\. Chen, S\. Lou, and Q\. Zhang \(2023b\)Bayesian neural networks avoid encoding complex and perturbation\-sensitive concepts\.InInternational Conference on Machine Learning,pp\. 28889–28913\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- Q\. Ren, J\. Zhang, Y\. Xu, Y\. Xin, D\. Liu, and Q\. Zhang \(2025b\)Towards the dynamics of a dnn learning symbolic interactions\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2607.08776#S1.p2.1),[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- A\. Romero, N\. Ballas, S\. E\. Kahou, A\. Chassang, C\. Gatta, and Y\. Bengio \(2014\)Fitnets: hints for thin deep nets\. arxiv 2014\.arXiv preprint arXiv:1412\.6550\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- W\. Shen, L\. Cheng, Y\. Yang, M\. Li, and Q\. Zhang \(2023\)Can the inference logic of large language models be disentangled into symbolic concepts?\.arXiv preprint arXiv:2304\.01083\.Cited by:[Appendix E](https://arxiv.org/html/2607.08776#A5.p1.4)\.
- P\. Sturmfels, S\. Lundberg, and S\. Lee \(2020\)Visualizing the impact of feature attribution baselines\.Distill5\(1\),pp\. e22\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- S\. Sun, W\. Ren, J\. Li, R\. Wang, and X\. Cao \(2024\)Logit standardization in knowledge distillation\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 15731–15740\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- S\. Sun, Y\. Cheng, Z\. Gan, and J\. Liu \(2019\)Patient knowledge distillation for bert model compression\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 4323–4332\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- Z\. Sun, H\. Yu, X\. Song, R\. Liu, Y\. Yang, and D\. Zhou \(2020\)MobileBERT: a compact task\-agnostic bert for resource\-limited devices\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 2158–2170\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017\)Axiomatic attribution for deep networks\.InInternational conference on machine learning,pp\. 3319–3328\.Cited by:[Appendix B](https://arxiv.org/html/2607.08776#A2.p1.1)\.
- I\. Tenney, J\. Wexler, J\. Bastings, T\. Bolukbasi, A\. Coenen, S\. Gehrmann, E\. Jiang, M\. Pushkarna, C\. Radebaugh, E\. Reif,et al\.\(2020\)The language interpretability tool: extensible, interactive visualizations and analysis for nlp models\.arXiv preprint arXiv:2008\.05122\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- Y\. Tian, D\. Krishnan, and P\. Isola \(2019\)Contrastive representation distillation\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- G\. Wang, Z\. Yang, Z\. Wang, S\. Wang, Q\. Xu, and Q\. Huang \(2025\)ABKD: pursuing a proper allocation of the probability mass in knowledge distillation viaa​l​p​h​aalpha\-b​e​t​abeta\-divergence\.InForty\-second International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 13484–13508\.Cited by:[2nd item](https://arxiv.org/html/2607.08776#A6.I1.i2.p1.1),[§5](https://arxiv.org/html/2607.08776#S5.p4.1)\.
- Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap,et al\.\(2022\)Super\-naturalinstructions: generalization via declarative instructions on 1600\+ nlp tasks\.InEMNLP,Cited by:[4th item](https://arxiv.org/html/2607.08776#A6.I1.i4.p1.1),[§5](https://arxiv.org/html/2607.08776#S5.p4.1)\.
- Y\. Wen, Z\. Li, W\. Du, and L\. Mou \(2023\)F\-divergence minimization for sequence\-level knowledge distillation\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 10817–10834\.External Links:[Link](https://aclanthology.org/2023.acl-long.605/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.605)Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- C\. Wu, E\. S\. Lubana, B\. K\. Mlodozeniec, R\. Kirk, and D\. Krueger \(2024\)What mechanisms does knowledge distillation distill?\.InProceedings of UniReps: the First Workshop on Unifying Representations in Neural Models,pp\. 60–75\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p1.1)\.
- T\. Wu, C\. Tao, J\. Wang, R\. Yang, Z\. Zhao, and N\. Wong \(2025\)Rethinking kullback\-leibler divergence in knowledge distillation for large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 5737–5755\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- W\. Xu, R\. Han, Z\. Wang, L\. Le, D\. Madeka, L\. Li, W\. Y\. Wang, R\. Agarwal, C\. Lee, and T\. Pfister \(2024\)Speculative knowledge distillation: bridging the teacher\-student gap through interleaved sampling\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p1.1)\.
- S\. Zagoruyko and N\. Komodakis \(2017\)Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- H\. Zhang, S\. Li, Y\. Ma, M\. Li, Y\. Xie, and Q\. Zhang \(2020a\)Interpreting and boosting dropout from a game\-theoretic view\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2607.08776#S5.p2.10)\.
- Q\. Zhang, X\. Wang, R\. Cao, Y\. N\. Wu, F\. Shi, and S\. Zhu \(2020b\)Extraction of an explanatory graph to interpret a cnn\.IEEE transactions on pattern analysis and machine intelligence43\(11\),pp\. 3863–3877\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.\(2022\)Opt: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§4\.1](https://arxiv.org/html/2607.08776#S4.SS1.p2.1)\.
- B\. Zhao, Q\. Cui, R\. Song, Y\. Qiu, and J\. Liang \(2022\)Decoupled knowledge distillation\.InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition,pp\. 11953–11962\.Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- K\. Zheng and E\. YANG \(2024\)Knowledge distillation based on transformed teacher matching\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[Figure 11](https://arxiv.org/html/2607.08776#A6.F11),[Figure 11](https://arxiv.org/html/2607.08776#A6.F11.3.2),[§F\.2](https://arxiv.org/html/2607.08776#A6.SS2.p1.1),[§5](https://arxiv.org/html/2607.08776#S5.p4.1)\.
- H\. Zhou and L\. Song \(2021\)Rethinking soft labels for knowledge distillation: a bias–variance tradeoff perspective\.InProceedings of International Conference on Learning Representations \(ICLR\),Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.
- H\. Zhou, H\. Zhang, H\. Deng, D\. Liu, W\. Shen, S\. Chan, and Q\. Zhang \(2023\)Concept\-level explanation for the generalization of a dnn\.arXiv e\-prints,pp\. arXiv–2302\.Cited by:[§2](https://arxiv.org/html/2607.08776#S2.p2.1)\.
- H\. Zhou, H\. Zhang, H\. Deng, D\. Liu, W\. Shen, S\. Chan, and Q\. Zhang \(2024\)Explaining generalization power of a dnn using interactive concepts\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 17105–17113\.Cited by:[§1](https://arxiv.org/html/2607.08776#S1.p2.1),[§1](https://arxiv.org/html/2607.08776#S1.p5.1),[§2](https://arxiv.org/html/2607.08776#S2.p2.1),[§4\.4](https://arxiv.org/html/2607.08776#S4.SS4.p5.1)\.
- X\. Zhou, M\. Zhang, Z\. Lee, W\. Ye, and S\. Zhang \(2025\)Hademif: hallucination detection and mitigation in large language models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.08776#A1.p1.3)\.

## Appendix AAdditional Related Work

Knowledge distillation\.Beyond the output logits, intermediate representations contain rich information for knowledge transfer\. Early works like FitNets\(Romeroet al\.,[2014](https://arxiv.org/html/2607.08776#bib.bib25)\)introduced “hints” to align intermediate layers\. Subsequent research focused on transferring attention maps\(Zagoruyko and Komodakis,[2017](https://arxiv.org/html/2607.08776#bib.bib22)\)or compressing specific architectures like BERT using layer\-wise strategies, as seen in Patient\-KD\(Sunet al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib27)\), MobileBERT\(Sunet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib23)\), and TinyBERT\(Jiaoet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib24)\)\. More recent approaches have explored contrastive representation distillation\(Tianet al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib30)\)and task\-aware layer\-wise distillation\(Lianget al\.,[2023b](https://arxiv.org/html/2607.08776#bib.bib26)\)\. Others have proposed homotopic distillation for pre\-trained transformers\(Lianget al\.,[2023a](https://arxiv.org/html/2607.08776#bib.bib28)\)or utilized Wasserstein distance to match feature distributions\(Lvet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib29)\)\. While standard KD minimizes the KL divergence, numerous studies have sought to refine this objective\. Fundamental analyses of KD’s efficacy\(Cho and Hariharan,[2019](https://arxiv.org/html/2607.08776#bib.bib11)\)and the bias\-variance tradeoff\(Zhou and Song,[2021](https://arxiv.org/html/2607.08776#bib.bib13)\)have driven improvements such as Decoupled KD\(Zhaoet al\.,[2022](https://arxiv.org/html/2607.08776#bib.bib14)\)and the Teacher Assistant strategy\(Mirzadehet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib12)\)\. To address optimization challenges, researchers have proposed multi\-level logit distillation\(Jinet al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib15)\), logit standardization\(Sunet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib16)\), and transformed teacher matching\(Zheng and YANG,[2024](https://arxiv.org/html/2607.08776#bib.bib17)\)\. In the context of large language models and sequence generation, recent works have investigatedff\-divergence minimization\(Wenet al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib9)\),α\\alpha\-β\\beta\-divergence\(Wanget al\.,[2025](https://arxiv.org/html/2607.08776#bib.bib10)\), and re\-evaluated the role of KL divergence\(Wuet al\.,[2025](https://arxiv.org/html/2607.08776#bib.bib21)\)\. Additionally, self\-knowledge distillation via dropout\(Leeet al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib18)\)and pre\-training distillation methods like MiniPLM\(Guet al\.,[2024b](https://arxiv.org/html/2607.08776#bib.bib20)\)have been developed to enhance efficiency\. Knowledge distillation has been effectively adapted for various downstream applications beyond standard model compression\. In computer vision, techniques like DetKDS\(Liet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib32)\)optimize distillation search specifically for object detectors\. In the domain of generative language models, PromptKD\(Kimet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib19)\)utilizes prompt tuning to distill student\-friendly knowledge\. Furthermore, distillation is applied to critical safety tasks, such as the detection and mitigation of hallucinations in large language models\(Zhouet al\.,[2025](https://arxiv.org/html/2607.08776#bib.bib31)\)\.

## Appendix BMasking Strategies of Input Variables

In attribution method research, it is common to employ a specific token or embedding to mask the input variables of an LLM\(Lundberg and Lee,[2017](https://arxiv.org/html/2607.08776#bib.bib55); Anconaet al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib56); Fonget al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib58)\)and use changes in network outputs on the masked samples to estimate attributions of different input variables\. The selection of a masking approach is complex, as each method has its weakness\. For example, replacing input variables with the mean baseline value \(the average of all samples\) or the zero baseline value can introduce out\-of\-distribution signals, thereby providing the model with artificial information, such as uniform grey or black dots in an image\(Dabkowski and Gal,[2017](https://arxiv.org/html/2607.08776#bib.bib59); Anconaet al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib56); Sundararajanet al\.,[2017](https://arxiv.org/html/2607.08776#bib.bib60)\)\. Additionally, blurring image pixels using a Gaussian kernel\(Fong and Vedaldi,[2017](https://arxiv.org/html/2607.08776#bib.bib61); Fonget al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib58)\)as the masked state removes high\-frequency signals but fails to eliminate low\-frequency signals\(Covertet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib62); Sturmfelset al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib63)\)\.

Given these challenges, we adopt a token replacement strategy, which is standard for the text domain\. This involves substituting the target input word with a dedicated\[MASK\]token at the embedding level\. For example, to mask the word ”green” in the input ”He is a green hand,” we would provide the LLM with the modified input ”He is a \[MASK\] hand\.” This approach effectively nullifies the specific semantic contribution of the target word without introducing out\-of\-distribution artifacts, ensuring a clean and consistent baseline for our interaction analysis\. For the specific\[MASK\]token for each LLM, please refer to Section 5 for details\.

## Appendix CDetails about How to Computev​\(x\)v\(x\)for Text Generation Tasks

Prior studies typically calculatev​\(x\)v\(x\)based on the probability of generating a single next token\. For example, given an LLMvvand an input sentence𝒙\\bm\{x\}withnninput words indexed byN=\{1,2,…,n\}N=\\\{1,2,\\ldots,n\\\}, letv​\(𝒙\)∈ℝv\(\\bm\{x\}\)\\in\\mathbb\{R\}denote the scalar output of the LLM\. Here we setv​\(𝒙\)=log​p​\(y=y\*\|𝒙\)1−p​\(y=y\*\|𝒙\)∈ℝv\(\\bm\{x\}\)=\\text\{log\}\\frac\{p\(y=y^\{\\text\{\*\}\}\|\\bm\{x\}\)\}\{1\-p\(y=y^\{\\text\{\*\}\}\|\\bm\{x\}\)\}\\in\\mathbb\{R\}, wherep​\(y=y\*\|𝒙\)p\(y=y^\{\\text\{\*\}\}\|\\bm\{x\}\)represents the probability of generating the ground truth tokeny\*y^\{\\text\{\*\}\}given the input𝒙\\bm\{x\}\.

However, this token\-level approach fails to capture the complex reasoning process inherent in generating long sentences, making it insufficient for standard text generation tasks\. To address this limitation, we extend the definition ofv​\(x\)v\(x\)from a single\-token basis to a sequence\-level metric\.

For text generation task and given the ground truth sequencey∗=\(y1∗,…,yL∗\)y^\{\*\}=\(y^\{\*\}\_\{1\},\\ldots,y^\{\*\}\_\{L\}\), we definep¯\\bar\{p\}as the geometric mean of the probabilities of each ground truth token iny∗y^\{\*\},i\.e\.,p¯=\(∏l=1Lp​\(yl\*\|x,y<l\*\)\)1/L\\bar\{p\}=\\left\(\\prod\_\{l=1\}^\{L\}p\(y\_\{l\}^\{\\text\{\*\}\}\|x,y\_\{<l\}^\{\\text\{\*\}\}\)\\right\)^\{1/L\}Then scalar outputv​\(x\)v\(x\)is defined asv​\(x\)=log​p¯1−p¯∈ℝv\(x\)=\\text\{log\}\\frac\{\\bar\{p\}\}\{1\-\\bar\{p\}\}\\in\\mathbb\{R\}\. To ensure numerical stability, we clipp¯\\bar\{p\}to\[ϵ,1−ϵ\],ϵ=10−7\[\\epsilon,1\-\\epsilon\],\\epsilon=10^\{\-7\}in our implementation\.

Our choice of the scalar output functionv​\(x\)v\(x\)is theoretically grounded in the standard evaluation metric for generative models: Perplexity \(PPL\)\. Formally, the perplexity of a sequence is defined asPPL=\(∏l=1Lp​\(yl\*\|x,y<l\*\)\)−1/L\\text\{PPL\}=\\left\(\\prod\_\{l=1\}^\{L\}p\(y\_\{l\}^\{\\text\{\*\}\}\|x,y\_\{<l\}^\{\\text\{\*\}\}\)\\right\)^\{\-1/L\}\. Our geometric mean probabilityp¯\\bar\{p\}is strictly the inverse of perplexity,i\.e\.,p¯=PPL−1\\bar\{p\}=\\text\{PPL\}^\{\-1\}, representing the length\-normalized average likelihood assigned to the ground truth\. To satisfy the Universal Matching Property of Harsanyi interactions which requires the output space to be unbounded \(inℝ\\mathbb\{R\}rather than\[0,1\]\[0,1\]\), we apply the log\-odds transformation top¯\\bar\{p\}\. Thus,v​\(x\)v\(x\)quantifies the model’s confidence in generating the precise ground truth sequence, normalized by length and mapped to a theoretically rigorous additive space\.

## Appendix DProof of Theorem

### D\.1Proof of Universal Matching Property

In this section, we provide the formal proof for the Universal Matching Property of the complete AND interaction framework\. We will demonstrate that the output of the surrogate logical model, which is the sum of all AND interaction effects, can perfectly match the output of the LLM for any masked sample\.

Thesurrogate logical modelϕ​\(⋅\)\\phi\(\\cdot\)is defined as follows:

ϕ​\(xT\)≜ϕ​\(x∅\)\+∑S⊆N,S≠∅𝟙AND​\(S∣xT\)⋅ISAND\\displaystyle\\phi\(x\_\{T\}\)\\triangleq\\phi\(x\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq N,S\\neq\\emptyset\}\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\{\(S\\mid x\_\{T\}\)\}\\cdot I^\{\\text\{\\rm AND\}\}\_\{S\}\(8\)where the AND trigger function𝟙AND​\(S∣xT\)∈\{0,1\}\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\(\{S\\mid x\_\{T\}\}\)\\in\\\{0,1\\\}represents anAND relationshipbetween input variables inSS, which can also be termedAND interaction pattern\. The scalar weightISANDI^\{\\text\{\\rm AND\}\}\_\{S\}quantifies the effect of an AND relationship, which can also be termedAND interaction effect\. An AND relationship is activated only by the joint presence of all input variables in the setSS,i\.e\., all input variables inSSare not masked\. For instance, given the input sentencex=“He is a green hand,”x=\\textit\{\`\`He\\ is\\ a\\ green\\ hand,''\}the co\-occurrence of the input variables in the setS=\{green,hand\}S=\\\{\\textit\{green\},\\textit\{hand\}\\\}contributes a numerical effectISANDI^\{\\text\{\\rm AND\}\}\_\{S\}that pushes the surrogate logical model’s inference towards the semantic meaning of“beginner\.”If an AND interactionSSis triggered,i\.e\.,𝟙AND​\(S∣xT\)=1\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\(\{S\\mid x\_\{T\}\}\)=1, the corresponding interaction effectISANDI^\{\\text\{\\rm AND\}\}\_\{S\}is added to the output of the logical model\. Otherwise, if any word inSSis masked and the AND interaction is not triggered,i\.e\.,𝟙AND​\(S∣xT\)=0\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\(\{S\\mid x\_\{T\}\}\)=0, the interaction effectISANDI^\{\\text\{\\rm AND\}\}\_\{S\}is not added to the output of the logical model\.x∅x\_\{\\emptyset\}represents that all input variables inNNare masked\.

Definition of universal matching property for AND interactions\.When the scalar weights in the surrogate logical modelϕ​\(⋅\)\\phi\(\\cdot\)are set toISAND=∑S′⊆S\(−1\)\|S\|−\|S′\|​vand​\(xS′\)I^\{\\text\{\\rm AND\}\}\_\{S\}=\\sum\\nolimits\_\{S^\{\\prime\}\\subseteq S\}\(\-1\)^\{\|S\|\-\|S^\{\\prime\}\|\}v\_\{\\text\{and\}\}\(x\_\{S^\{\\prime\}\}\), the output ofϕ​\(⋅\)\\phi\(\\cdot\)can always match the output score of the LLMv​\(⋅\)v\(\\cdot\),i\.e\.,∀T⊆N,v​\(xT\)=ϕ​\(xT\)\\forall T\\subseteq N,v\(x\_\{T\}\)=\\phi\(x\_\{T\}\)\. Herevand​\(xT\)=v​\(xT\)v\_\{\\text\{and\}\}\(x\_\{T\}\)=v\(x\_\{T\}\)\.

We need to prove that given an input samplexx, for each masked sample\{xT\|T⊆N\}\\\{x\_\{T\}\|T\\subseteq N\\\}, the network output scorev​\(xT\)∈ℝv\(x\_\{T\}\)\\in\\mathbb\{R\}can be well matched by the surrogate logical modelϕ​\(xT\)\\phi\(x\_\{T\}\)\. The surrogate logical modelϕ​\(xT\)\\phi\(x\_\{T\}\)uses the sum of AND interactions to accurately explain/match the network output scorev​\(xT\)v\(x\_\{T\}\)\.

∀T⊆N,v​\(xT\)=ϕ​\(xT\)\.\\displaystyle\\forall T\\subseteq N,v\(x\_\{T\}\)=\\phi\(x\_\{T\}\)\.\(9\)ϕ​\(xT\)=ϕ​\(x∅\)\+∑S⊆N,S≠∅𝟙AND​\(S∣xT\)⋅ISAND\\displaystyle\\phi\(x\_\{T\}\)=\\phi\(x\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq N,S\\neq\\emptyset\}\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\{\(S\\mid x\_\{T\}\)\}\\cdot I^\{\\text\{\\rm AND\}\}\_\{S\}=v​\(x∅\)\+∑S⊆T,S≠∅ISAND⏟vand​\(xT\)\\displaystyle\\,\\kern 5\.0pt\\qquad=\\underbrace\{v\(x\_\{\\emptyset\}\)\+\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}I^\{\\text\{\\rm AND\}\}\_\{S\}\}\_\{v\_\{\\text\{\\rm and\}\}\(x\_\{T\}\)\}
###### Proof\.

Universal matching property of AND interactions\.For all2n2^\{n\}masked samples\{𝒙T∣T⊆N\}\\\{\\bm\{x\}\_\{T\}\\mid T\\subseteq N\\\}, what we need to prove is that the outputvand​\(𝒙T\)v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)of an LLM can be universally explained by all the interactions inT⊆NT\\subseteq N,i\.e\.,∀S⊆T,S≠∅,vand​\(𝒙T\)=∑S⊆T,S≠∅ISAND​\(𝒙\)=v​\(𝒙∅\)\+∑S⊆T,S≠∅ISAND\\forall S\\subseteq T,S\\neq\\emptyset,v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)=\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\. Here,v​\(𝒙∅\)=vand​\(𝒙∅\)v\(\\bm\{x\}\_\{\\emptyset\}\)=v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{\\emptyset\}\)\.

According to the definition of the AND interaction,ISAND​\(𝒙\)=∑L⊆S\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\. To simplify the computation of the sum of AND interactions∑S⊆T,S≠∅ISAND​\(𝒙\)=∑S⊆T,S≠∅∑L⊆S\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\), we exchange the order of summation of the setL⊆S⊆TL\\subseteq S\\subseteq Tand the setS⊇LS\\supseteq L\. Given a set of input variablesLL, we compute all linear combinations of all setsSScontainingLLwith respect to the model outputsvand​\(𝒙S\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{S\}\),i\.e\.,∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\. Then, we compute all summations over the setL⊆TL\\subseteq Tas∑S⊆T,S≠∅ISAND​\(𝒙\)=∑L⊆T∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)−vand​\(𝒙∅\)\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=\\sum\\nolimits\_\{L\\subseteq T\}\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\-v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{\\emptyset\}\)\. Then, we can compute different cases ofL⊆S⊆TL\\subseteq S\\subseteq Tas follows:

\(1\) WhenL=T=SL=T=S,∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)=\(−1\)\|T\|−\|T\|​vand​\(𝒙L\)=vand​\(𝒙L\)\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=\(\-1\)^\{\|T\|\-\|T\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\.

\(2\) WhenL⊆S⊆T,L≠TL\\subseteq S\\subseteq T,L\\neq T, let us consider the linear combinations of all setsSSwith number\|S\|\|S\|for the model outputvand​\(𝒙L\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\), respectively\. Letm:=\|S\|−\|L\|m:=\|S\|\-\|L\|, \(0≤m≤\|T\|−\|L\|0\\leq m\\leq\|T\|\-\|L\|\), then there are a total ofC\|T\|−\|L\|mC\_\{\|T\|\-\|L\|\}^\{m\}combinations of all setsSSof order\|S\|\|S\|\. GivenLL, accumulating the model outputsvand​\(𝒙L\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)corresponding to allS⊇LS\\supseteq L, we can get∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)=vand​\(𝒙L\)⋅∑m=0\|T\|−\|L\|C\|T\|−\|L\|m​\(−1\)m⏟=0=0\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\\cdot\\underbrace\{\\sum\\nolimits\_\{m=0\}^\{\|T\|\-\|L\|\}C\_\{\|T\|\-\|L\|\}^\{m\}\(\-1\)^\{m\}\}\_\{=0\}=0\.

Considering all the cases, the complete derivation of the sum of AND interactions is as follows\.

∑S⊆T,S≠∅ISAND\\displaystyle\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(10\)=\\displaystyle=∑S⊆T,S≠∅∑L⊆S\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)\\displaystyle\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=\\displaystyle=∑L⊆T∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)−vand​\(𝒙∅\)\\displaystyle\\sum\\nolimits\_\{L\\subseteq T\}\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\-v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=\\displaystyle=vand​\(𝒙T\)⏟L=T\+∑L⊆T,L≠Tvand​\(𝒙L\)⋅∑m=0\|T\|−\|L\|C\|T\|−\|L\|m​\(−1\)m⏟=0−vand​\(𝒙∅\)\\displaystyle\\underbrace\{v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)\}\_\{L=T\}\+\\sum\\nolimits\_\{L\\subseteq T,L\\neq T\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\\cdot\\underbrace\{\\sum\\nolimits\_\{m=0\}^\{\|T\|\-\|L\|\}C\_\{\|T\|\-\|L\|\}^\{m\}\(\-1\)^\{m\}\}\_\{=0\}\-v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=\\displaystyle=vand​\(𝒙T\)−v​\(𝒙∅\)\\displaystyle v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)\-v\(\\bm\{x\}\_\{\\emptyset\}\)
Therefore, we have proved that∀∅≠T⊆N,vand​\(𝒙T\)=v​\(𝒙∅\)\+∑S⊆T,S≠∅ISAND\\forall\\emptyset\\neq T\\subseteq N,v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)=v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\. We can easily getv​\(xT\)=ϕ​\(xT\)=vand​\(xT\)=v​\(x∅\)\+∑S⊆T,S≠∅ISANDv\(x\_\{T\}\)=\\phi\(x\_\{T\}\)=v\_\{\\text\{\\rm and\}\}\(x\_\{T\}\)=v\(x\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I^\{\\text\{\\rm AND\}\}\_\{S\}, thus, we obtain the universal matching property of AND interactions\.

∎

### D\.2Proof of Partition Size Constraint for the Sampling Method

Proposition\.To ensure that any non\-empty union of the partitioned subsets represents a complex interaction, the number of partitionsmmmust satisfym≤3m\\leq 3\.

###### Proof\.

Given an input sequencexxwithnnwords indexed byN=\{1,…,n\}N=\\\{1,\\dots,n\\\}, and let𝒫=\{S1,S2,…,Sm\}\\mathcal\{P\}=\\\{S\_\{1\},S\_\{2\},\\dots,S\_\{m\}\\\}be a partition ofNNsuch that⋃i=1mSi=N\\bigcup\_\{i=1\}^\{m\}S\_\{i\}=NandSi∩Sj=∅S\_\{i\}\\cap S\_\{j\}=\\emptysetfori≠ji\\neq j\. We define the set of complex interactions asΩcomplex=\{S⊆N∣⌈n/3⌉≤\|S\|≤n\}\\Omega^\{\\text\{complex\}\}=\\\{S\\subseteq N\\mid\\lceil n/3\\rceil\\leq\|S\|\\leq n\\\}\.

To guarantee that the sampling strategy is valid, we require that any non\-empty combination of these subsets constitutes a complex interaction\. Formally, this condition is expressed as:

∀K⊆\{1,…,m\},K≠∅⟹⋃i∈KSi∈Ωcomplex\\forall K\\subseteq\\\{1,\\dots,m\\\},K\\neq\\emptyset\\implies\\bigcup\_\{i\\in K\}S\_\{i\}\\in\\Omega^\{\\text\{complex\}\}\(11\)Substituting the definition ofΩcomplex\\Omega^\{\\text\{complex\}\}, this requires:

\|⋃i∈KSi\|≥⌈n/3⌉\\left\|\\bigcup\_\{i\\in K\}S\_\{i\}\\right\|\\geq\\lceil n/3\\rceil\(12\)Since this condition must hold forallvalidKK, it must specifically hold for the singleton sets where\|K\|=1\|K\|=1\. LetK=\{i\}K=\\\{i\\\}for anyi∈\{1,…,m\}i\\in\\\{1,\\dots,m\\\}\. The condition implies a lower bound on the size of each individual subset:

\|Si\|≥⌈n/3⌉,∀i∈\{1,…,m\}\|S\_\{i\}\|\\geq\\lceil n/3\\rceil,\\quad\\forall i\\in\\\{1,\\dots,m\\\}\(13\)On the other hand, since\{Si\}\\\{S\_\{i\}\\\}forms a disjoint partition ofNN, the sum of their cardinalities must equal the total number of wordsnn:

∑i=1m\|Si\|=\|N\|=n\\sum\_\{i=1\}^\{m\}\|S\_\{i\}\|=\|N\|=n\(14\)Combining this with the inequality in Eq\. \([13](https://arxiv.org/html/2607.08776#A4.E13)\), we obtain:

∑i=1m⌈n/3⌉≤∑i=1m\|Si\|=n\\sum\_\{i=1\}^\{m\}\\lceil n/3\\rceil\\leq\\sum\_\{i=1\}^\{m\}\|S\_\{i\}\|=n\(15\)Simplifying the summation:

m⋅⌈n/3⌉≤nm\\cdot\\lceil n/3\\rceil\\leq n\(16\)Solving formm:

m≤n⌈n/3⌉m\\leq\\frac\{n\}\{\\lceil n/3\\rceil\}\(17\)By the definition of the ceiling function, we know that⌈n/3⌉≥n/3\\lceil n/3\\rceil\\geq n/3\. Therefore:

n⌈n/3⌉≤nn/3=3\\frac\{n\}\{\\lceil n/3\\rceil\}\\leq\\frac\{n\}\{n/3\}=3\(18\)Consequently, we derive the strict upper boundm≤3m\\leq 3\. This proves that to strictly satisfy the definition of complex interactions for all sampled combinations, the partition sizemmis mathematically constrained to be a small constant \(specifically 2 or 3\)\. ∎

## Appendix EImplementation Details of Analysis

To construct the set of input variablesNNfor our interaction analysis, we extracted meaningful words from the “Instruction” segment of each input\. Following the preprocessing protocol ofShenet al\.\([2023](https://arxiv.org/html/2607.08776#bib.bib77)\), we defined meaningful words as tokens that are neither punctuation marks nor stop words listed in the NLTK library\(Birdet al\.,[2009](https://arxiv.org/html/2607.08776#bib.bib76)\)\. We standardized the number of input variables atn=\|N\|=13n=\|N\|=13\. As detailed in Appendix[F\.3](https://arxiv.org/html/2607.08776#A6.SS3), the instructions in thedatabricks\-dolly\-15kdataset are usually short; thus, a set size ofn=13n=13is empirically sufficient to encompass the core semantic content of the samples\. During the masking process, we exclusively applied masking to the selected tokens withinNN, while leaving all other “background” tokens unchanged\.

### E\.1Baselines

- •KD\(Hintonet al\.,[2015](https://arxiv.org/html/2607.08776#bib.bib1)\)minimizes the forward Kullback\-Leibler Divergence \(FKLD\) between the teacher and student distributions using the fixed training dataset\.
- •SeqKD\(Kim and Rush,[2016](https://arxiv.org/html/2607.08776#bib.bib2)\)maximizes the likelihood of high\-probability sequences generated by the teacher, effectively functioning as Supervised Fine\-Tuning \(SFT\) on teacher\-generated outputs\.
- •ImitKD\(Linet al\.,[2020](https://arxiv.org/html/2607.08776#bib.bib3)\)addresses the training\-inference mismatch by introducing Student\-Generated Outputs \(SGO\)\. It optimizes the standard KLD loss on sequences generated by the student model, forcing the student to mimic the teacher’s behavior on its own exploration path\.
- •MiniLLM\(Guet al\.,[2024a](https://arxiv.org/html/2607.08776#bib.bib5)\)also utilizes SGOs but employs an on\-policy gradient method\. It aims to minimize the Reverse KLD \(RKLD\) to prevent the student from assigning high probabilities to low\-probability teacher regions \(mode collapse\)\.
- •GKD\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib4)\)adopts the Generalized Jensen\-Shannon Divergence \(JSD\) as the objective\. It trains on a mixture of datasets, combining fixed data \(ground truth or teacher\-generated\) with on\-policy student\-generated sequences\.
- •DistiLLM\(Koet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib6)\)introduces the Skew KL \(or Skew RKL\) divergence to stabilize training\. It utilizes a mixed dataset and proposes an adaptive off\-policy method to selectively incorporate SGOs based on validation loss, thereby filtering out noisy generated data\.

## Appendix FImplementation Details of Training

### F\.1Training Details

For training the teacher and student models, we used eight RTX PRO 6000 96GB GPUs\. Our experimental setup for training LMs ondatabricks\-dolly\-15kprimarily follows the experimental setup forKoet al\.\([2024](https://arxiv.org/html/2607.08776#bib.bib6)\)\. For models within 1B parameters, we search for the learning rates in \{5e\-4, 1e\-4, 5e\-5\}, the batch sizes in \{8, 16, 32\} and train these models for 20 epochs\. For models that have more than 1B parameters, we search for the learning rate in \{5e\-5, 1e\-5, 5e\-6\}, the batch sizes of \{8, 16, 32\}, and train these models for 10 epochs\. We fully use the distillation loss for the instruction\-following dataset and language modeling loss for OpenWebText\(Gokaslanet al\.,[2019](https://arxiv.org/html/2607.08776#bib.bib74)\)corpus\. The checkpoints of each student are selected by the ROUGE\-L scores on the validation set\. To ensure the effectiveness of CIP, specifically when employing Student\-Generated Outputs \(SGO\), we calculate CIP based on the ground truth from the fixed dataset\. Furthermore, we only calculated CIP for samples where the number of input variables in the “Instruction” segment, maintaining consistency with the settings used in our interaction analysis\.

### F\.2Evaluation

For evaluating the teacher and student models, we applied two RTX PRO 6000 96GB GPUs\. FollowingGuet al\.\([2024a](https://arxiv.org/html/2607.08776#bib.bib5)\); Koet al\.\([2024](https://arxiv.org/html/2607.08776#bib.bib6)\), we adopt a prompt template as shown in[Figure10](https://arxiv.org/html/2607.08776#A6.F10)\. We sample the responses from each model using a temperature of 1\.0, a max\-length limit of 512\. For GPT\-5 feedback, we use a popular prompt introduced inZhenget al\.\([2023](https://arxiv.org/html/2607.08776#bib.bib78)\)which is illustrated in[Figure11](https://arxiv.org/html/2607.08776#A6.F11), the prompt asking GPT\-5 to compare model\-generated responses with the ground truth answers and give 1\-10 scores for both responses\. We report the ratio of the total score of model responses and ground truth answers by followingKoet al\.\([2024](https://arxiv.org/html/2607.08776#bib.bib6)\)\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x10.png)Figure 10:The prompt template for training and evaluation of instruction\-following task experiments\.![Refer to caption](https://arxiv.org/html/2607.08776v1/x11.png)Figure 11:The prompt template for single\-answer grading of GPT\-5 feedback fromZhenget al\.\([2023](https://arxiv.org/html/2607.08776#bib.bib78)\)\.
### F\.3Dataset Description

- •databricks\-dolly\-15k\(Conoveret al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib68)\): This is an open\-source dataset comprising instruction\-following records generated by Databricks employees\. It encompasses several behavioral categories outlined in\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.08776#bib.bib75)\), including brainstorming, classification, closed/open QA, generation, information extraction, and summarization\. Some specific examples provided in[Table2](https://arxiv.org/html/2607.08776#A6.T2)\.
- •Self\-Instruct\(Wanget al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib69)\): SelfInstruct is a framework designed to enhance the language model’s instruction\-following capabilities by leveraging the model’s own outputs to generate a vast set of instructional data\. It consists of 52k instructions and 82k instance inputs/outputs for fine\-tuning, supplemented by 252 expert\-written tasks for practical applications and an additional 50k examples from public datasets for benchmarking\.
- •Vicuna\(Chianget al\.,[2023](https://arxiv.org/html/2607.08776#bib.bib71)\): Following the evaluation protocol of\(Koet al\.,[2024](https://arxiv.org/html/2607.08776#bib.bib6)\), we incorporate the set of 80 challenging questions originally curated for evaluating the Vicuna model to assess model performance on complex queries\.
- •Super\-Natural Instructions\(Wanget al\.,[2022](https://arxiv.org/html/2607.08776#bib.bib70)\): This benchmark features a diverse collection of 1,616 NLP tasks paired with expert\-written instructions, spanning 76 distinct task types\. For evaluation, we utilize its test set, which comprises approximately 9k samples drawn from 119 different tasks\.

Table 2:Some cases from thedatabricks\-dolly\-15ktest set\.

## Appendix GMore Experiment Results

### G\.1Comparison of the training time with and without CIP loss

Table 3:Comparison of the training time per batch \(in seconds/batch\) between different KD methods without and with the CIP loss function across various model sizes\. The results are averaged over 20 epochs\. The overhead is calculated as the increase in time relative to the original method\.To assess the computational cost introduced by our proposed CIP loss, we compared the average training time per batch of various distillation methods with and without the CIP loss\. The experiments were conducted on a server equipped with 4×\\timesRTX PRO 6000 96GB GPUs\. We measured the average training time \(s/batch\) over 20 epochs for both GPT\-2\-0\.1B and GPT\-2\-0\.3B student models\. As shown in Table[3](https://arxiv.org/html/2607.08776#A7.T3), the integration of CIP introduces a marginal computational overhead, ranging from approximately 3\.03% to 11\.32% across different methods and model sizes\. This slight increase is primarily attributed to the sampling of subsets and the calculation of interaction terms\. Considering the significant performance improvements demonstrated in the main experiments, we consider this computational cost to be negligible and acceptable for practical applications\.

### G\.2More Results of the Main Experiments

[Figure12](https://arxiv.org/html/2607.08776#A7.F12)illustrates the distributions of Gini coefficientG​\(x\)G\(x\)and Shannon entropyH​\(x\)H\(x\)across all samples in the test set\. Results show that distilled student models demonstrate an obvious distributional shift towards higher Gini values and lower entropy compared to corresponding base models and SFT models\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x12.png)Figure 12:The distribution of the Gini coefficient and entropy across varying samples\.[Table4](https://arxiv.org/html/2607.08776#A7.T4)shows that distilled student models exhibit a higher overlap rate with the teacher model in terms of salient interactions than that of the non\-distilled student models\.

Table 4:Comparing the student\-teacher overlap rate of salient interactions before and after distillation\.O​v​e​r​l​a​p​@​k¯=𝔼x​\[O​v​e​r​l​a​p​@​k​\(x\)\]\\overline\{Overlap@k\}=\\mathbb\{E\}\_\{x\}\\left\[Overlap@k\(x\)\\right\]represents the average overlap rate across all samples\.[Figure13](https://arxiv.org/html/2607.08776#A7.F13)plots the change of Gini coefficients between distilled and base models for simple and complex interactions\. Results show that the sparsity of complex interactions exhibits an obvious increase after distillation\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x13.png)Figure 13:Comparison of sparsity changes in simple versus complex interactions between non\-distilled and distilled models\.[Figure14](https://arxiv.org/html/2607.08776#A7.F14)shows the overlap rate of salient simple interactions consistently surpasses that of complex interactions\. This indicates that the student model learn more simple salient interactions than complex salient interactions from the teacher model\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x14.png)Figure 14:Comparison of overlap rate between simple interactions and complex interactions after distillation\.[Figure15](https://arxiv.org/html/2607.08776#A7.F15)shows that the sparsity of complex interactions is positively correlated with model performance\. Student models that exhibit higher Gini coefficients and lower entropy \(indicating higher sparsity\) in their complex interactions generally achieve superior ROUGE\-L scores\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x15.png)Figure 15:Correlation between the sparsity of complex interactions and model performance\.[Figure16](https://arxiv.org/html/2607.08776#A7.F16)shows that there is no distinct relationship between the sparsity of simple interactions and the model performance\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x16.png)Figure 16:Correlation between the sparsity of simple interactions and model performance\.As shown in[Figure17](https://arxiv.org/html/2607.08776#A7.F17), the results reveal a positive correlation between model performance and the overlap rate of complex interactions\. This indicates that student models with higher performance learn more complex interactions from the teacher model compared to student models with lower performance\. Counter\-intuitively, the overlap rate of simple interactions shows no distinct correlation with model performance, as shown in[Figure18](https://arxiv.org/html/2607.08776#A7.F18)\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x17.png)Figure 17:Correlation between the student\-teacher overlap rate of complex interactions and model performance\.As shown in[Figure18](https://arxiv.org/html/2607.08776#A7.F18), simple interactions shows no distinct correlation with model performance\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x18.png)Figure 18:Correlation between the student\-teacher overlap rate of simple interactions and model performance\.Table 5:The comparison of ROUGE\-L and GPT\-5 Eval score between different KD methods without and with the CIP loss function across various model families and model sizes\. Results show that adding CIP consistently improves the performance of all original KD methods\.[Table6](https://arxiv.org/html/2607.08776#A7.T6)in shows that student models trained with CIP exhibit higher Gini coefficients and lower entropy\. This confirms that the CIP loss can truly increase the sparsity of complex interactions\.

Table 6:The comparison of the sparsity of complex interactions between different KD methods without and with the CIP loss function\.
### G\.3Robustness Check for Differentβ\\beta

In Section[4\.4](https://arxiv.org/html/2607.08776#S4.SS4), we utilizedβ=⌈n/3⌉\\beta=\\lceil n/3\\rceilfor interaction partitioning by followingLiet al\.\([2025](https://arxiv.org/html/2607.08776#bib.bib72)\)\. To assess the impact of this threshold, we conduct our experiments usingβ=⌈n/5⌉\\beta=\\lceil n/5\\rceilandβ=⌈n/2⌉\\beta=\\lceil n/2\\rceil\. As shown below, the experimental results align perfectly with those in Sections[4\.4](https://arxiv.org/html/2607.08776#S4.SS4)and[4\.5](https://arxiv.org/html/2607.08776#S4.SS5), confirming the robustness of our conclusions regarding the choice ofβ\\beta\.

Here are the results forβ=⌈n/5⌉\\beta=\\lceil n/5\\rceil

[Figure19](https://arxiv.org/html/2607.08776#A7.F19)plots the change of Gini coefficients between distilled and base models for simple and complex interactions\. Results show that the sparsity of complex interactions exhibits an obvious increase after distillation\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x19.png)Figure 19:Comparison of sparsity changes in simple versus complex interactions between non\-distilled and distilled models\. Hereβ=⌈n/5⌉\\beta=\\lceil n/5\\rceil\.[Figure20](https://arxiv.org/html/2607.08776#A7.F20)shows the overlap rate of salient simple interactions consistently surpasses that of complex interactions\. This indicates that the student model learn more simple salient interactions than complex salient interactions from the teacher model\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x20.png)Figure 20:Comparison of overlap rate between simple interactions and complex interactions after distillation\. Hereβ=⌈n/5⌉\\beta=\\lceil n/5\\rceil\.[Figure21](https://arxiv.org/html/2607.08776#A7.F21)shows that the sparsity of complex interactions is positively correlated with model performance\. Student models that exhibit higher Gini coefficients and lower entropy \(indicating higher sparsity\) in their complex interactions generally achieve superior ROUGE\-L scores\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x21.png)Figure 21:Correlation between the sparsity of complex interactions and model performance\. Hereβ=⌈n/5⌉\\beta=\\lceil n/5\\rceil\.As shown in[Figure22](https://arxiv.org/html/2607.08776#A7.F22), the results reveal a positive correlation between model performance and the overlap rate of complex interactions\. This indicates that student models with higher performance learn more complex interactions from the teacher model compared to student models with lower performance\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x22.png)Figure 22:Correlation between the student\-teacher overlap rate of complex interactions and model performance\. Hereβ=⌈n/5⌉\\beta=\\lceil n/5\\rceil\.Here are the results forβ=⌈n/2⌉\\beta=\\lceil n/2\\rceil

[Figure23](https://arxiv.org/html/2607.08776#A7.F23)plots the change of Gini coefficients between distilled and base models for simple and complex interactions\. Results show that the sparsity of complex interactions exhibits an obvious increase after distillation\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x23.png)Figure 23:Comparison of sparsity changes in simple versus complex interactions between non\-distilled and distilled models\. Hereβ=⌈n/2⌉\\beta=\\lceil n/2\\rceil\.[Figure24](https://arxiv.org/html/2607.08776#A7.F24)shows the overlap rate of salient simple interactions consistently surpasses that of complex interactions\. This indicates that the student model learn more simple salient interactions than complex salient interactions from the teacher model\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x24.png)Figure 24:Comparison of overlap rate between simple interactions and complex interactions after distillation\. Hereβ=⌈n/2⌉\\beta=\\lceil n/2\\rceil\.[Figure25](https://arxiv.org/html/2607.08776#A7.F25)shows that the sparsity of complex interactions is positively correlated with model performance\. Student models that exhibit higher Gini coefficients and lower entropy \(indicating higher sparsity\) in their complex interactions generally achieve superior ROUGE\-L scores\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x25.png)Figure 25:Correlation between the sparsity of complex interactions and model performance\. Hereβ=⌈n/2⌉\\beta=\\lceil n/2\\rceil\.As shown in[Figure26](https://arxiv.org/html/2607.08776#A7.F26), the results reveal a positive correlation between model performance and the overlap rate of complex interactions\. This indicates that student models with higher performance learn more complex interactions from the teacher model compared to student models with lower performance\.

![Refer to caption](https://arxiv.org/html/2607.08776v1/x26.png)Figure 26:Correlation between the student\-teacher overlap rate of complex interactions and model performance\. Hereβ=⌈n/2⌉\\beta=\\lceil n/2\\rceil\.

## Appendix HCase Study

We provide some cases generated by the models distilled by different methods based on the OPT\-1\.3B model in[AppendixH](https://arxiv.org/html/2607.08776#A8)\. We find that the KD methods with CIP generate more detailed and accurate responses compared with the baselines\.

Instruction\-following cases from thedatabricks\-dolly\-15kdataset\.ModelTypeContent\\endfirstheadContinued from previous pageModelTypeContent\\endheadContinued on next page…\\endfoot\\endlastfootCase \#1InstructionName some famous rock bands from the 1960sInputNoneGround\-truthThe 1960s had a number of famous rock bands including The Beatles, The Beach Boys, The Doors, The Rolling Stones, The Who, Led Zeppelin, The Grateful Dead, Pink Floyd, and many more\.KDw/o CIPIn the 1960s, there were a lot of famous rock bands, among them, The Rolling Stones, The Grateful Dead, The Who, KISS, Pink Floyd, Queen, The Doorsw/ CIPMany famous rock bands came from the late 1960s and early 1970s, including Led Zeppelin, The Who, Pink Floyd, The Doors and Jefferson AirplaneSeqKDw/o CIPFamous rock bands from the 1960’s include the Rolling Stones, the Beatles, and the Jimi Hendrixw/ CIPFamous rock bands from the 1960’s are the Rolling Stones, The Beatles, and Led ZeppelinImitKDw/o CIPThe Beatles, The Rolling Stones, The Jefferson Airplane, The Doors, Bob Dylan, Jagger, Oasis, Styx, You Know My Namew/ CIPLed Zeppelin, The Rolling Stones, The Who, Pink FloydMiniLLMw/o CIPOasis \- Wonderwall, The Beatles \- Rocky Raccoon, The Rolling Stones \-Its all over now, Eagles \- Take it easy, The Who \- Who, Pink Floyd \- Dark Side of the Moon, Led Zeppelin \- What is Dead May Never Die, Queen \- Another brick in the wall: Pink 3w/ CIPMany believe that the 1960s were the golden age of rock bands, and there are hundreds of beautiful songs from this time period\. Some of the most famous bands include The Beatles, The Rolling Stones, Led Zeppelin, The Who, Pink Floyd and R\.E\.M\.GKDw/o CIPThere are a lot of famous rock bands from the 60s, some of which are very well known today like The Beatles, The Rolling Stones, John Lennon and The Beatles\. Other bands that are very well known from the 1960s are The Beatles’ studio band, The Who, The Rolling Stones, The Grateful Dead and The Doorsw/ CIPThe Beatles, The Stones, The Rolling Stones, Led Zeppelin, The Moody BluesDISTILLMw/o CIPThe Rolling Stones, The Beatles, The Doors, Jefferson Airplane, The Rolling Stones, The Beatlesw/ CIPSome famous rock bands from the 1970s are Led Zeppelin, The Doors, Pink Floyd, The Rolling Stones, Eagles, The YardbirdsCase \#2InstructionWho won the 63rd staging of the Limerick Senior Hurling Championship?InputOn 29 September 1957, Claughaun won the championship after a 7\-07 to 3\-02 defeat of St\. Patrick’s in the final\.Ground\-truthThe 1957 Limerick Senior Hurling Championship was the 63rd staging of the Limerick Senior Hurling Championship since its establishment by the Limerick County Board in 1887\. Cappamore were the defending champions, however, they were defeated by St\. Patrick’s\. On 29 September 1957, Claughaun won the championship after a 7\-07 to 3\-02 defeat of St\. Patrick’s in the final\. It was their sixth championship title overall and their first championship title since 1926\.KDw/o CIPClaughaun won the championship on 29 September 1957 after a 7\-07 TO 3\-02 defeat of in the final\.w/ CIPClaughaun won the 57th edition of the Limerick Senior hurling championship after defeating St\. Patrick’s by 7\-01 to 3\-02 in the final\.SeqKDw/o CIPClaughaun won the championship by 7\-07 to 3\-02\.w/ CIPClaughaun won the championship following a 7\-07 to 3\-02 defeat of St Patrick’s in the final on Sunday 29 September 1957\.ImitKDw/o CIPClaughaun won the 63rd staging on September 29th, 1957\.w/ CIPClaughaun won the championship\.MiniLLMw/o CIPClaughaun won the 63rd staging an went on to win the next four senior hurling championships\.w/ CIPClaughaun won the Limerick Senior Hurting Championship on 29 September 1957, after a 7\-07\-3 victory over St\. Patrick’s\.GKDw/o CIPClaughaunw/ CIPClaughaun won the championship on 29 September 1957 after a 7\-07 to 3\-02 defeat of St\. Patrick’s in the final\.DISTILLMw/o CIPClaughaun won the 1957 Limerick Senior Football Championship after defeating St\. Patrick’s by 7\-07 to 3 \-02 in the final\.w/ CIPClaughaun won the championship After a 7\-07 to 3\-02 defeat of St Patrick’s in the final