Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting

# Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
Source: https://arxiv.org/html/2604.15794
###### Abstract

Large Language Models (LLMs) have achieved remarkable success, underpinning diverse AI applications. However, they often suffer from performance degradation due to factors such as catastrophic forgetting during Supervised Fine-Tuning (SFT), quantization, and pruning. In this work, we introduce a performance recovery framework based on Self-Distillation Fine-Tuning (SDFT) that effectively restores model capabilities. Complementing this practical contribution, we provide a rigorous theoretical explanation for the underlying recovery mechanism. We posit that an LLM's generative capability fundamentally relies on the high-dimensional manifold constructed by its hidden layers. To investigate this, we employ Centered Kernel Alignment (CKA) to quantify the alignment between student and teacher activation trajectories, leveraging its invariance to orthogonal transformations and scaling. Our experiments demonstrate a strong correlation between performance recovery and manifold alignment, substantiating the claim that self-distillation effectively aligns the student's high-dimensional manifold with the optimal structure represented by the teacher. This study bridges the gap between practical recovery frameworks and geometric representation theory, offering new insights into the internal mechanisms of self-distillation.

## 1 Introduction

Large Language Models (LLMs) have revolutionized natural language understanding, reasoning, and generation. However, deploying generic base models into real-world applications necessitates further adaptation. To align with specific downstream tasks, models typically undergo Supervised Fine-Tuning (SFT); simultaneously, to meet resource constraints, techniques such as pruning and quantization become indispensable.

However, these operations often incur significant performance degradation. In continuous learning, multi-round SFT frequently triggers Catastrophic Forgetting, where models lose original general knowledge and skills while acquiring new domain-specific knowledge and task capabilities. Similarly, aggressive compression disrupts internal parameter distributions, leading to declines in accuracy and logical consistency. This "capability trade-off" forces difficult choices between specialization and generalization. Once a model degrades, traditional repair methods are often computationally prohibitive, sometimes requiring retraining from scratch, a very inefficient solution in the context of scarce computational resources.

In this paper, we propose an effective "Recovery Mechanism" for model degradation, leveraging Self-Distillation Fine-Tuning (SDFT) (Shenfeld et al., 2026), a specialized paradigm of Self-Distillation (SD) (Hinton et al., 2015). While traditional SD focuses on improving generalization bounds through self-imitation, we argue that when a model suffers from distribution shift due to SFT or compression, the regularization effect of SDFT acts as an "anchor." This mechanism pulls degraded parameters back toward the original high-performance manifold. Crucially, our approach relies solely on the model's own historical states without relying on an external teacher, thereby facilitating efficient performance recovery.

Building on this insight, we establish a unified recovery framework and validate it across diverse degradation scenarios. With a primary emphasis on catastrophic forgetting in multi-round SFT, we further demonstrate the framework's efficacy against compression artifacts. Empirical results demonstrate that SDFT effectively restores model performance across multiple evaluation benchmarks, validating both its practical efficacy and theoretical foundation.

## 2 Related Work

#### Catastrophic Forgetting in LLMs

Catastrophic Forgetting (CF) refers to the phenomenon wherein neural networks suddenly and significantly lose previously learned knowledge when trained on new data (De Lange et al., 2021). In the context of LLMs, this phenomenon manifests when multi-round Supervised Fine-Tuning (SFT) overwrites the knowledge and skills acquired in previous trainings (Li and Hoiem, 2017). Existing mitigation strategies generally fall into three categories: (1) Replay-based methods, which store a subset of old data to interleave with new training (De Lange et al., 2021); (2) Regularization-based methods, such as Elastic Weight Consolidation (EWC), which penalize changes to important parameters (Kirkpatrick et al., 2017); and (3) Parameter-isolation methods, which allocate separate parameters for different tasks (Rusu et al., 2016). While effective to some extent, these approaches often incur high computational costs, require access to historical data, or complicate model architecture. Crucially, most existing work focuses on preventing forgetting during new training, rather than recovering performance after degradation has occurred.

#### Model Compression and Performance Degradation

To deploy LLMs efficiently, techniques such as pruning (Ma et al., 2024) and quantization (Dettmers et al., 2023) are widely adopted. However, these operations inevitably introduce performance degradation. Aggressive pruning removes redundant neurons but may disrupt critical knowledge pathways, while low-bit quantization introduces noise that affects logical consistency and factual accuracy (Frantar et al., 2023). Traditional remedies often rely on Knowledge Distillation (KD), where a compressed student model is trained to mimic a larger teacher (Hinton et al., 2015). While external strong teachers (e.g., larger LLMs or API-based models) are theoretically applicable, they often introduce distribution shifts, high computational overhead, or privacy constraints that limit their practicality for post-degradation recovery. In contrast, Self-Distillation offers a self-contained alternative that leverages the model's own historical states, avoiding external dependencies while preserving task alignment. This makes SD particularly suitable for lightweight, privacy-sensitive, or distribution-consistent recovery scenarios.

#### Self-Distillation Fine-Tuning

Self-Distillation (SD) has emerged as a powerful technique for enhancing model generalization without relying on an external teacher. Early works demonstrated that training a model to mimic its own deeper layers or earlier checkpoints acts as an effective regularizer, reducing overfitting and improving accuracy (Furlanello et al., 2018). More recently, studies have extended SD to Self-Distillation Fine-Tuning (SDFT), enabling on-policy learning directly from demonstrations. By leveraging in-context learning, SDFT uses the model itself as a teacher to generate training signals that preserve prior capabilities while acquiring new skills. Across various tasks, SDFT consistently outperforms conventional SFT, achieving higher new-task accuracy while mitigating catastrophic forgetting. However, existing SDFT approaches primarily focus on preventing forgetting during the training process, often assuming the teacher and student are synchronized. In this paper, we extend SDFT to a more general framework where the teacher can be any historical state of the model, not just the current iteration. Crucially, we reposition this generalized SDFT as a post-hoc recovery mechanism, designed to restore performance after degradation has occurred, rather than merely preventing it during training.

## 3 Recovery Framework

### 3.1 Problem Formulation

Let LLM_θ denote the original base model with parameters θ. After undergoing degradation processes such as multi-round SFT or compression, the model becomes LLM_θ₁ with parameters θ₁, exhibiting performance drops in general knowledge and skills. Our goal is to obtain a recovered model LLM_θ₂ with parameters θ₂ that maximizes performance on both original capabilities and new tasks.

### 3.2 Recovery Solutions

**Figure 1: The Self-Distillation Recovery Framework for Catastrophic Forgetting**

Figure 1 illustrates the overall architecture of our proposed Self-Distillation Recovery Framework in the catastrophic forgetting scenario. Unlike traditional Fine-Tuning pipelines that solely optimize for new task performance, our framework introduces a dual-objective optimization process aimed at both capability recovery and task adaptation. The framework consists of three main components: (1) the Teacher LLM_θ, constructed from the model's own historical checkpoints or earlier training states; (2) the Degraded Model θ₁, which serves as the initial student state suffering from performance loss due to prior multiple rounds of SFTs; and (3) the SDFT Recovery Process, where the student learns to mimic the teacher's output distribution while adapting to the datasets used in previous multiple rounds of SFTs. This self-contained process ensures that performance recovery is achieved without relying on external high-performance models or any external datasets.

**Figure 2: The Self-Distillation Recovery Framework for Compression**

Figure 2 extends the proposed recovery framework to compression scenarios. When an LLM is subjected to pruning or quantization, it inevitably incurs varying degrees of performance degradation. To facilitate recovery, the framework necessitates the curation of expert demonstration datasets aligned with the degraded capabilities. For example, if the tool-calling task shows performance degradation, related datasets are needed for recovery; if general knowledge shows degradation, then SFT datasets used in post-training are required. Notably, apart from this data selection strategy, the underlying recovery mechanism remains identical to the catastrophic forgetting scenario, demonstrating the unified nature of our approach across different degradation types.

**Figure 3: The Self-Distillation Recovery Framework for Small-Scale LLMs**

However, the original SDFT formulation exhibits a significant limitation at smaller scales (e.g., 3B variants), where insufficient in-context learning (ICL) capabilities fail to provide meaningful self-guidance, resulting in performance inferior to standard SFT. To address this, we propose an extended recovery strategy that introduces a single preliminary step while preserving the unified nature of our framework.

Figure 3 illustrates this enhanced workflow. The ineffectiveness of SDFT in small-scale models stems from its heavy reliance on robust ICL, which is typically underdeveloped in smaller architectures. Consequently, we first employ off-policy distillation using a large-scale LLM as the teacher to bootstrap the small model's ICL capabilities. While this step enhances ICL, it inevitably leads to degradation in general and domain-specific capabilities. Subsequently, we apply our SDFT recovery mechanism to restore these degraded capabilities. Ultimately, this two-stage process enables the small-scale model to retain its original capabilities while achieving improved ICL performance, effectively extending the applicability of our recovery framework to resource-constrained scenarios.

The external teacher is used only once to bootstrap ICL capabilities (enabling SDFT), whereas the core recovery process remains self-contained via SDFT. This hybrid approach balances practicality with the efficiency of self-distillation.

## 4 Theoretical Analysis of Self-Distillation via High-Dimensional Manifold Alignment

### 4.1 Introduction

Previous chapters have primarily focused on the empirical analysis of the recovery framework, leaving the underlying theoretical mechanisms unexplored. Why does self-distillation effectively recover model performance, and is there a geometric metric aligned with this phenomenon?

In this chapter, we answer these questions by shifting the focus from output distributions to internal representations. We posit that the generative capability of an LLM fundamentally relies on the high-dimensional manifold constructed by its hidden layers, and consequently the core function of self-distillation is not merely optimizing output probabilities but regularizing the spatial structure of hidden states to align the student's manifold with the teacher's. Building on this premise, we propose a theoretical framework grounded in high-dimensional manifold geometry.

To validate this theoretical framework, we employ Centered Kernel Alignment (CKA) (Kornblith et al., 2019) as a metric to quantify the alignment of manifold structures between the student and the teacher, leveraging its critical advantage over metrics like Mean Squared Error (MSE) — namely, invariance to orthogonal transformations and scaling.

### 4.2 Problem Formulation and Manifold Definition

Given an input sequence X = (x₁, x₂, ..., x_L), where L denotes the sequence length. For a certain hidden layer of an LLM (e.g., the last hidden layer), each token x_t corresponds to a d-dimensional activation vector h_t ∈ ℝ^d. We stack the activation vectors of all tokens from a complete forward pass to form the Activation Matrix H ∈ ℝ^(L×d):

H = [h₁^T; h₂^T; ...; h_L^T]

From the perspective of manifold learning, each row in H represents a sample point on the high-dimensional semantic manifold M, and the entire matrix H constitutes a discrete trajectory of the sequence on this manifold. The student model S and the teacher model T generate activation matrices H_S and H_T, respectively. Our objective is to measure the geometric alignment between these two trajectories. It is important to clarify that we do not compare the complete underlying manifolds of the student and teacher models directly. Instead, we utilize activation trajectories, which serve as discrete samples from these manifolds. This approach is both theoretically representative and computationally feasible.

Directly comparing the element values of activation matrices H_S and H_T (e.g., using MSE) is inappropriate because neural network representations possess rotation invariance. Semantically identical features may exist along different coordinate axes in the hidden space. To capture the intrinsic structure of the manifold, we must measure the relative relationships between tokens rather than their absolute coordinates.

We compute the Linear Kernel Matrix K ∈ ℝ^(L×L):

Here, the element K_ij = h_i · h_j represents the similarity between the i-th and j-th tokens in the hidden space. The matrix K encodes the semantic dependency structure within the sequence and serves as a representation of the geometric properties of the manifold.

### 4.3 Calculation Procedure

We follow the six steps below to calculate the manifold alignment degree between H_S and H_T:

1. **Input Consistency**: Input the identical sequence (Prompt + Ground Truth) into