ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery

# Calibrating Structured Pruning forOn-Policy Distillation Recovery
Source: [https://arxiv.org/html/2610.11332](https://arxiv.org/html/2610.11332)
Houcheng JiangAffiliation:Zhongguancun AcademyAffiliation:Foundation Model Department, TencentAffiliation:University of Science and Technology of ChinaMingyang SongAffiliation:Foundation Model Department, TencentQiyong ZhongAffiliation:Foundation Model Department, TencentJie SunAffiliation:University of Science and Technology of ChinaTianyu ZhangAffiliation:University of Science and Technology of ChinaJunfeng FangAffiliation:National University of Singapore\*Equal Contribution†Corresponding author

###### Abstract

Structured pruning reduces the deployment cost of reasoning language models, but capability degradation can hinder subsequent on\-policy distillation \(OPD\) recovery\. Because OPD relies on student\-generated trajectories, damage that persists after offline distillation can limit its effectiveness\. We proposeReCal, Recovery\-Aware Calibration, a simple plug\-and\-play approach that improves OPD recovery by adjusting calibration before pruning\.ReCaluses forward KL between an unpruned teacher and a pruned probe to identify disrupted teacher\-supported predictions, then reweights calibration statistics to guide existing pruning criteria toward preserving them\. Across models and pruning methods,ReCalimproves mathematical reasoning after OPD in most comparisons, with gains of up to 16\.7 percentage points on AIME and improvements in most code\-generation comparisons\. Further analysis shows reduced residual damage at heavily affected tokens and performance advantages that persist through recovery\. These results support calibrating structured pruning for OPD recovery\. Our code:[https://github\.com/jianghoucheng/ReCal](https://github.com/jianghoucheng/ReCal)\.

## 1Introduction

Large language models \(LLMs\) increasingly support complex reasoning and autonomous agent tasks, but their growing size and long generations make deployment expensive\([Anthropic, 2026](https://arxiv.org/html/2610.11332#bib.bib31);[OpenAI, 2026](https://arxiv.org/html/2610.11332#bib.bib32);[Yang et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib29)\)\. Structured pruning reduces this cost by removing entire channels, heads, or layers, producing smaller models that run on standard hardware\([Ma et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib16);[An et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib4);[Ashkboos et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib5)\)\. However, pruning often requires subsequent training to recover lost capabilities\([Xia et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib27);[Muralidharan et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib20);[Sreenivas et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib24)\)\. Recent distillation advances support a two\-stage recovery pipeline: offline distillation through supervised fine\-tuning \(SFT\) on teacher responses, followed by on\-policy distillation \(OPD\) on student\-generated trajectories\([Kim and Rush, 2016](https://arxiv.org/html/2610.11332#bib.bib1);[Agarwal et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib2);[Zhang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib30)\)\. The unpruned model serves as teacher and the pruned model as student\.

Figure 1:Pruning and the limitations of subsequent recovery\.\(a\)ReCaladjusts calibration before SFT and OPD\. \(b\) AIME accuracy before pruning, after pruning, and after offline SFT\. \(c\) The fraction of responses reaching the length limit after SFT and OPD\. Both data panels show measured 8B baselines; Appendix[B\.4](https://arxiv.org/html/2610.11332#A2.SS4)gives the protocol\.Despite this progress, pruning can severely degrade reasoning, and offline SFT only partly restores accuracy \(Figure[1](https://arxiv.org/html/2610.11332#S1.F1)b\)\. Pruned models can produce repetitive, unfinished, or incorrect responses\([Wang et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib26);[Zhang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib30)\)\. Worse still, OPD obtains teacher supervision at these student\-generated prefixes, where poor generations can compromise the feedback\([Xu et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib42)\)\. A teacher can agree with the student’s next\-token predictions at a flawed prefix even when the reasoning fails; teacher\-confirmed repetition is one documented example\([Zhang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib30)\)\. Low reverse KL on student rollouts therefore need not indicate successful reasoning\. In our baseline evaluations, many responses still reach the length limit after OPD \(Figure[1](https://arxiv.org/html/2610.11332#S1.F1)c\), leaving substantial generation failures unresolved\.

Since recovery depends on student trajectories, pruning should account for damage that may hinder recovery, complementing training\-aware structure selection\([Tang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib39);[Ai et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib3)\)\. We examine this damage on teacher\-generated prefixes, where the teacher and a pruned model receive the same context\. Forward KL emphasizes predictions that the teacher supports but the pruned model assigns little probability\. Our analysis shows that tokens most affected by pruning remain a major source of discrepancy after recovery\. Giving these tokens greater calibration weight could therefore protect vulnerable predictions before recovery begins\.

We therefore proposeReCal, a simple*Recovery\-Aware Calibration*method for structured pruning\. As shown in Figure[1](https://arxiv.org/html/2610.11332#S1.F1)\(a\),ReCalfirst uses a pruned probe to measure token\-level forward KL on teacher trajectories\. These scores reweight calibration statistics used to recompute channel importance on the unpruned model\. Applying the new mask produces a student for the same SFT and OPD recovery pipeline\.ReCalis plug\-and\-play: it preserves the base pruning criterion and target widths while changing the evidence used to select channels\.

To evaluateReCal, we compare its recovery performance with representative structured pruning baselines on Qwen3\-8B and Qwen3\-4B\-Instruct\. After on\-policy distillation,ReCalachieves stronger mathematical reasoning than the best\-performing baseline on both models\. It also improves mathematical reasoning in seven of eight model–criterion comparisons, with gains of up to 16\.7 percentage points, and benefits code generation in most comparisons\. These results show that calibrating which structures survive pruning can improve the capabilities recovered by the subsequent distillation pipeline\.

## 2Preliminaries

We first introduce calibration\-based structured pruning in Section[2\.1](https://arxiv.org/html/2610.11332#S2.SS1), then describe the on\-policy distillation recovery process that follows it in Section[2\.2](https://arxiv.org/html/2610.11332#S2.SS2)\.

### 2\.1Structured Pruning

Structured pruning removes parameter groups to obtain a smaller model\([Ma et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib16);[An et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib4);[Muralidharan et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib20)\)\. We consider feed\-forward network \(FFN\) width pruning, where each selected channel is removed together with its associated input and output weights\. LetpTp\_\{T\}denote the original model andmℓ∈\{0,1\}dℓm^\{\\ell\}\\in\\\{0,1\\\}^\{d\_\{\\ell\}\}the channel mask in layerℓ\\ell\. A value of one retains a channel\. Applying these masks to the original weights gives the pruned modelpmp\_\{m\}\.

Calibration\-based criteria use a small set of inputs to estimate channel importance\. For an additive criterion, letci​jℓc\_\{ij\}^\{\\ell\}be the contribution of calibration tokeniito channeljjin layerℓ\\ell\. Aggregating overNNtokens and retaining thekℓk\_\{\\ell\}highest\-scoring channels gives

sjℓ=1N∑i=1Nci​jℓ,mjℓ=𝟙\[j∈Topkℓ\(sℓ\)\]\.s\_\{j\}^\{\\ell\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}c\_\{ij\}^\{\\ell\},\\qquad m\_\{j\}^\{\\ell\}=\\mathbb\{1\}\\bigl\[j\\in\\operatorname\{Top\}\_\{k\_\{\\ell\}\}\(s^\{\\ell\}\)\\bigr\]\.\(1\)The contribution can use activation magnitude, norms, fluctuations, or gradients\([Sun et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib25);[An et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib4);[Ma et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib16)\)\. Equation[1](https://arxiv.org/html/2610.11332#S2.E1)covers additive scores; see Appendix[A\.2](https://arxiv.org/html/2610.11332#A1.SS2)for other criteria\.

We reweight token evidence while preserving the criterion and target widths\.

### 2\.2On\-Policy Distillation Recovery

Offline sequence distillation first initializes recovery by fitting fixed responses from the unpruned teacher\([Kim and Rush, 2016](https://arxiv.org/html/2610.11332#bib.bib1)\)\. We implement this stage as supervised fine\-tuning \(SFT\) on the teacher responses; we use SFT to denote this offline distillation stage throughout the paper\. On\-policy distillation \(OPD\) then trains the student on responses sampled from its own policy, with a frozen teacher providing the supervision\([Agarwal et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib2);[Li et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib15)\)\. Letpθp\_\{\\theta\}be the trainable pruned student\. For promptxx, a rollout checkpointpθ¯p\_\{\\bar\{\\theta\}\}generates responseyy, and both teacher and student evaluate the prefixesht=\(x,y<t\)h\_\{t\}=\(x,y\_\{<t\}\)\. A full\-vocabulary reverse\-KL matching objective on these rollouts is

ℒOPD\(θ;θ¯\)=𝔼x,y∼pθ¯\(⋅∣x\)\[1\|y\|∑tKL\(pθ\(⋅∣ht\)∥pT\(⋅∣ht\)\)\],\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta;\\bar\{\\theta\}\)=\\mathbb\{E\}\_\{x,\\,y\\sim p\_\{\\bar\{\\theta\}\}\(\\cdot\\mid x\)\}\\left\[\\frac\{1\}\{\|y\|\}\\sum\_\{t\}\\mathop\{\\mathrm\{KL\}\}\\bigl\(p\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)\\\|p\_\{T\}\(\\cdot\\mid h\_\{t\}\)\\bigr\)\\right\],\(2\)whereKL\(p∥q\)=∑vp\(v\)log\[p\(v\)/q\(v\)\]\\mathop\{\\mathrm\{KL\}\}\(p\\\|q\)=\\sum\_\{v\}p\(v\)\\log\[p\(v\)/q\(v\)\]\. Fresh responses are collected as the student updates, so the teacher supervises prefixes reached by the student’s current policy\.

The retained structure determines the starting student for this process\. We hold recovery fixed and examine whether pruning damage can inform structure selection\. Appendix[A](https://arxiv.org/html/2610.11332#A1)specifies the practical OPD signal and training settings\.

## 3Understanding Pruning Damage and Recovery

![Refer to caption](https://arxiv.org/html/2610.11332v1/analysis.png)Figure 2:Locating damage before recovery\.\(a\) Within\-trajectory damage concentration across ordinary pruning criteria\. \(b\) Mean KL through baseline recovery in groups fixed by initial damage; color is logarithmic\. \(c\) The fraction of each response’s damage carried by its initially highest\-damage group \(Q5\), averaged across responses\. Q1–Q5 are equal\-count groups ordered by initial damage\. Token groups are never reselected after recovery\. SFT denotes offline distillation through supervised fine\-tuning\. The pale vertical band marks Q5; curve bands and error bars denote 95% trajectory\-bootstrap intervals\.We analyze several structure pruning methods on held\-out teacher responses to locate pruning damage and track its recovery\. At tokenttin responserr, both models receive the same prefixhr,th\_\{r,t\}\. Token\-level damage is

dr,t\(m\)=KL\(pT\(⋅∣hr,t\)∥pm\(⋅∣hr,t\)\)\.d\_\{r,t\}\(m\)=\\mathop\{\\mathrm\{KL\}\}\\bigl\(p\_\{T\}\(\\cdot\\mid h\_\{r,t\}\)\\\|p\_\{m\}\(\\cdot\\mid h\_\{r,t\}\)\\bigr\)\.\(3\)This measures the change in next\-token predictions, weighted by the teacher distribution\. Within each response, we rank tokens by raw\-pruned baseline KL and divide them into five equal\-count groups, Q1–Q5, from lowest to highest damage\. Each group contains one fifth of the response’s tokens\. We keep memberships and teacher prefixes fixed through offline SFT and OPD, averaging KL within each response/group and then across responses\. Concentration is examined in both models; recovery tracking uses Wanda\. Full settings are in Appendix[A\.6](https://arxiv.org/html/2610.11332#A1.SS6)\.

Pruning distorts some token predictions much more than others\.Figure[2](https://arxiv.org/html/2610.11332#S3.F2)\(a\) shows that the highest\-damage 20% of tokens contribute 81\.5–82\.9% of within\-response damage across the three criteria\. The second model exhibits the same pattern, with shares of 80\.9–83\.1% \(Appendix[A\.6](https://arxiv.org/html/2610.11332#A1.SS6)\)\. Channel removal therefore affects token predictions unevenly, which uniform aggregation does not explicitly distinguish\. We next ask whether these predictions remain damaged after recovery\.

Recovery reduces token\-level damage but leaves substantial residual errors\.Using the fixed groups defined above, Figure[2](https://arxiv.org/html/2610.11332#S3.F2)\(b\) shows that mean KL in the highest\-damage group decreases from 2\.110 after pruning to 0\.705 after SFT and 0\.627 after OPD\. The full pipeline removes 70\.3% of this group’s initial discrepancy, with most of that reduction occurring during SFT\. Nevertheless, its post\-OPD KL remains nearly three times that of the next\-highest group \(0\.214\)\. Substantial repair therefore leaves these tokens disproportionately damaged\.

Tokens most affected by pruning still account for most damage after recovery\.To quantify how much of the remaining damage can be located before recovery, we measure the share carried by the original highest\-damage group\. Without reselecting any tokens, this group accounts for 64\.6% of within\-response damage after SFT and 69\.3% after OPD \(Figure[2](https://arxiv.org/html/2610.11332#S3.F2)c\)\. The increase in share from SFT to OPD does not mean its damage grows: its absolute KL decreases, while its share of the remaining discrepancy rises\. Initial damage thus identifies tokens that remain poorly matched after recovery, using a signal already available before training\.

These observations motivate accounting for teacher\-prefix damage during structure selection\. A base\-pruned model can identify affected tokens; emphasizing their calibration evidence may reduce the discrepancy passed to recovery\. Section[4](https://arxiv.org/html/2610.11332#S4)defines this intervention, and Section[5](https://arxiv.org/html/2610.11332#S5)tests whether it improves recovered accuracy\.

## 4Recovery\-Aware Calibration for Pruning

ReCaladjusts calibration to address the persistent damage identified in Section[3](https://arxiv.org/html/2610.11332#S3)\. We first locate token\-level damage, use it to calibrate structure selection, and then recover the selected student through SFT and OPD\. Figure[3](https://arxiv.org/html/2610.11332#S4.F3)summarizes this procedure\.

Figure 3:Overview ofReCal\.*Locate damage:*compare the teacher and a pruned probe on teacher trajectories\.*Calibrate & prune:*use token\-level damage to reweight the base criterion and prune the original model\.*OPD recovery:*recover the selected student with the unchanged training procedure\.#### Locate damage\.

We apply the base pruning criterion at the target widths to obtain a probepm0p\_\{m\_\{0\}\}\. The teacher and probe evaluate identical prefixes from teacher\-generated responses\. We estimate the forward\-KL damage in Equation[3](https://arxiv.org/html/2610.11332#S3.E3)using teacher\-selected tokens and a residual\-probability bucket \(Appendix[A\.2](https://arxiv.org/html/2610.11332#A1.SS2)\), and denote the resulting scores bydr,td\_\{r,t\}\. Teacher trajectories expose damage independently of the probe’s own rollouts\. The probe does not restrict the final mask\.

#### Calibrate & prune\.

We convert the probe’s damage into calibration weights\. For responserrwithTrT\_\{r\}valid tokens, letDr=∑tdr,tD\_\{r\}=\\sum\_\{t\}d\_\{r,t\}\. The token weights and resulting additive channel scores are

w~r,t=dr,tDr,∑t=1Trw~r,t=1\.\\widetilde\{w\}\_\{r,t\}=\\frac\{d\_\{r,t\}\}\{D\_\{r\}\},\\qquad\\sum\_\{t=1\}^\{T\_\{r\}\}\\widetilde\{w\}\_\{r,t\}=1\.\(4\)sjℓ=1R∑r=1R∑t=1Trw~r,tcr,t,jℓ,\(m⋆\)jℓ=𝟙\[j∈Topkℓ\(sℓ\)\]\.s\_\{j\}^\{\\ell\}=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\sum\_\{t=1\}^\{T\_\{r\}\}\\widetilde\{w\}\_\{r,t\}c\_\{r,t,j\}^\{\\ell\},\\qquad\(m^\{\\star\}\)\_\{j\}^\{\\ell\}=\\mathbb\{1\}\\bigl\[j\\in\\operatorname\{Top\}\_\{k\_\{\\ell\}\}\(s^\{\\ell\}\)\\bigr\]\.\(5\)HereRRis the number of calibration responses andcr,t,jℓc\_\{r,t,j\}^\{\\ell\}is the base contribution defined in Section[2\.1](https://arxiv.org/html/2610.11332#S2.SS1)\. Each response has equal total weight, with greater weight assigned to its more damaged tokens\. For negligible total damage, we use uniform weights \(Appendix[A\.2](https://arxiv.org/html/2610.11332#A1.SS2)\)\.

The weighted statistics are computed on the original unpruned model, and the selected mask is applied to its original weights\. Thus, a channel removed by the probe remains eligible for the final student\. Non\-additive criteria incorporate the weights into their activation statistics or token losses before aggregation \(Appendix[A\.2](https://arxiv.org/html/2610.11332#A1.SS2)\)\. The base criterion and retained widths remain unchanged\. Appendix[C](https://arxiv.org/html/2610.11332#A3)connects teacher\-prefix forward KL to sequence preservation and states its limits\.

#### OPD recovery\.

The selected student first undergoes offline distillation through SFT on fixed teacher responses and then OPD with the same unpruned teacher\. Both stages use the same training procedure as the baseline\.ReCalchanges the structure supplied to recovery, while OPD continues to learn from student\-generated trajectories as in Section[2\.2](https://arxiv.org/html/2610.11332#S2.SS2)\. Appendix[A](https://arxiv.org/html/2610.11332#A1)gives training details\.

## 5Experiments

Table 1:Recovery at approximately 25% parameter reduction\.ReCaldenotes Wanda \+ReCal\. Pruned: no recovery; SFT: offline distillation; OPD: SFT followed by OPD\. Scores are percentages; Avg\. averages all six benchmarks\. Each column’s best/second\-best pruned scores within a model arebold/underlined, excluding unpruned references\.MethodStageMathScienceInstructionCodeAvg\.AIME24AIME25AIME26GPQAIFB\-SLCB\-v5Qwen3\-8BUnpruned—66\.255\.862\.154\.523\.046\.451\.3MinitronPruned0\.00\.00\.01\.016\.90\.03\.0SFT9\.23\.38\.730\.318\.015\.714\.2OPD16\.710\.015\.435\.921\.219\.919\.9FLAPPruned0\.00\.00\.023\.216\.30\.06\.6SFT27\.520\.820\.433\.319\.215\.122\.7OPD31\.725\.430\.043\.420\.124\.729\.2LLM\-PrunerPruned0\.40\.00\.022\.216\.00\.06\.4SFT11\.76\.25\.835\.919\.512\.715\.3OPD18\.87\.910\.434\.822\.417\.518\.6WandaPruned2\.52\.12\.126\.820\.64\.29\.7SFT41\.734\.634\.245\.521\.525\.333\.8OPD46\.736\.741\.742\.419\.525\.935\.5ReCalPruned33\.323\.720\.035\.420\.116\.924\.9SFT51\.739\.646\.343\.421\.526\.538\.2OPD52\.942\.949\.640\.418\.328\.338\.7Qwen3\-4B\-InstructUnpruned—63\.345\.854\.260\.634\.331\.948\.4MinitronPruned0\.00\.40\.030\.326\.71\.29\.8SFT6\.20\.85\.442\.928\.215\.716\.5OPD6\.22\.16\.235\.430\.221\.717\.0FLAPPruned4\.20\.80\.831\.827\.94\.211\.6SFT21\.711\.713\.836\.926\.221\.722\.0OPD27\.917\.519\.238\.930\.822\.326\.1LLM\-PrunerPruned0\.00\.00\.026\.825\.00\.68\.7SFT6\.72\.56\.731\.825\.915\.114\.8OPD8\.70\.88\.735\.927\.915\.116\.2WandaPruned13\.35\.46\.740\.429\.75\.416\.8SFT29\.623\.720\.447\.027\.320\.528\.1OPD35\.832\.528\.744\.429\.724\.732\.6ReCalPruned20\.410\.410\.837\.930\.215\.120\.8SFT41\.731\.729\.243\.927\.322\.932\.8OPD42\.534\.235\.047\.028\.825\.335\.5After introducing the setup in Section[5\.1](https://arxiv.org/html/2610.11332#S5.SS1), we address four questions in Sections[5\.2](https://arxiv.org/html/2610.11332#S5.SS2)–[5\.5](https://arxiv.org/html/2610.11332#S5.SS5):

- •RQ1:How doesReCalcompare with existing pruning methods before and after recovery? Does it improve the performance reached after OPD?
- •RQ2:CanReCalimprove different pruning criteria without changing their scoring rules? Do its benefits extend beyond the default Wanda configuration?
- •RQ3:How does the benefit ofReCalchange as more parameters are removed? Does it persist across compression ratios, models, and base criteria?
- •RQ4:How doesReCalchange the selected channels and the damage identified in Section[3](https://arxiv.org/html/2610.11332#S3)? How do these changes relate to recovery?

### 5\.1Experimental Setup

Appendix[A](https://arxiv.org/html/2610.11332#A1)details the pruning adapters, data sources, training, and evaluation\.

#### Models and baselines\.

We use Qwen3\-8B in thinking mode and Qwen3\-4B\-Instruct\-2507 without thinking\([Yang et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib29)\)\. We compare structured FFN\-channel adapters of Minitron, Wanda, FLAP, and LLM\-Pruner\([Muralidharan et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib20);[Sun et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib25);[An et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib4);[Ma et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib16)\)\. All methods retain the same widths at a given compression ratio\. The main comparison removes approximately 25% of total parameters; the ratio sweep also includes 15% and 35%\.

#### Recovery and evaluation\.

Every student undergoes offline distillation through SFT on responses from its unpruned teacher, followed by OPD on student\-generated trajectories\. Paired methods share recovery data and training settings\. We evaluate mathematics on AIME 2024–2026, science on GPQA\-Diamond\([Rein et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib23)\), instruction following on IFBench\([Pyatkin et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib22)\), and code generation on LiveCodeBench\-v5\([Jain et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib11)\)\. We report AIME accuracy averaged over eight samples per question and the three years, GPQA accuracy, IFBench instruction\-level strict accuracy, and LiveCodeBench pass@1\. Supplementary experiments and additional results appear in Appendix[B](https://arxiv.org/html/2610.11332#A2)\.

Figure 4:Post\-OPD performance across pruning criteria\.Measured results on AIME24–26 and LiveCodeBench\-v5 \(LCB\-v5\)\. Wide blue bars show Base \+ReCal; narrow red bars show Base\. Rows identify models and columns identify criteria\. See Appendix[B\.1](https://arxiv.org/html/2610.11332#A2.SS1)for LLM\-Pruner\.

### 5\.2Recovery Performance \(RQ1\)

We use Wanda as the base criterion for the defaultReCalconfiguration\. Table[1](https://arxiv.org/html/2610.11332#S5.T1)compares the three stages: Pruned, SFT, and SFT followed by OPD\. We have the following observations:

- •Obs\. 1: ReCal leads in mathematics after recovery\.After OPD,ReCalreaches 48\.5% AIME accuracy in 8B and 37\.2% in 4B, exceeding the strongest baseline, Wanda, at 41\.7% and 32\.4%\. It leads on each AIME year in both models\. After SFT, it already reaches 45\.8% and 34\.2%, respectively, establishing an advantage that persists through OPD\.
- •Obs\. 2: Calibration preserves reasoning capability before recovery\.Immediately after pruning, the 8B default retains 25\.7% AIME accuracy, compared with 2\.2% for Wanda\. The 4B scores are 13\.9% and 8\.5%\. The advantage is already present before any recovery training, consistent with calibration preserving more of the teacher’s reasoning capability in the selected structure\.

### 5\.3Plug\-and\-Play Improvement \(RQ2\)

We next applyReCalto each pruning criterion while keeping its channel\-scoring rule, target widths, and recovery procedure fixed\. Figure[4](https://arxiv.org/html/2610.11332#S5.F4)compares each fully recovered baseline with its calibrated counterpart\. Baselines use prompt calibration, whereasReCaluses weighted teacher trajectories; the comparison evaluates this combined intervention\. We can draw the following observations:

- •Obs\. 3: ReCal improves mathematics in seven of eight pairs\.All six activation\-based pairs improve on AIME \(Figure[4](https://arxiv.org/html/2610.11332#S5.F4)\), with gains up to 16\.7 points\. LLM\-Pruner improves from 6\.1% to 12\.8% in 4B but declines from 12\.4% to 9\.2% in 8B \(Appendix[B\.1](https://arxiv.org/html/2610.11332#A2.SS1)\)\. The benefit therefore extends across criteria, with a model\-dependent exception\.
- •Obs\. 4: Code improves broadly, with trade\-offs on other tasks\.LiveCodeBench improves in seven of eight pairs and GPQA in five, while IFBench declines in all eight \(Figure[7](https://arxiv.org/html/2610.11332#S5.F7)c\)\. For 8B LLM\-Pruner, science and code gain 10\.1 and 10\.8 points despite the mathematics loss\. These trade\-offs limit claims of uniform capability preservation \(Appendices[B\.1](https://arxiv.org/html/2610.11332#A2.SS1)and[B\.5](https://arxiv.org/html/2610.11332#A2.SS5)\)\.

### 5\.4Sensitivity to Compression Ratio \(RQ3\)

Figure 5:Base versus Base \+ReCalacross compression ratios\.Solid: SFT; dashed: OPD\. Ratios denote whole\-model parameter reduction\.We vary whole\-model parameter reduction from 15% to 35% for Minitron, Wanda, and FLAP in both models\. Figure[5](https://arxiv.org/html/2610.11332#S5.F5)shows both recovery stages\. We can draw the following observations:

- •Obs\. 5: ReCal improves SFT recovery at every measured ratio\.ReCalexceeds Base in all 18 model–criterion–ratio comparisons\. Wanda gains 3\.6, 9\.0, and 14\.6 points in 8B at 15%, 25%, and 35% reduction; 4B gains are 1\.8, 9\.6, and 7\.6 points\. Benefits persist across ratios but need not increase monotonically\.
- •Obs\. 6: More aggressive pruning still limits recovery\.All measured SFT curves decline with compression\. At 35% reduction, 8B Wanda \+ReCalretains 27\.1% accuracy versus 12\.5% for Base, while calibrated Minitron reaches only 3\.6%\. In 4B, calibrated Wanda also outperforms calibrated Minitron and FLAP at this ratio\. Both the criterion and retained capacity still matter\.

### 5\.5Structure and Recovery Analysis \(RQ4\)

![Refer to caption](https://arxiv.org/html/2610.11332v1/rq4_masks.png)

Figure 6:Changes in channel selection\.\(a\) Shared \(gray\) and replaced \(blue\) baseline deletions\. \(b\) Replacement by layer depth\. \(c\) Spearman correlation of complete channel rankings\.![Refer to caption](https://arxiv.org/html/2610.11332v1/rq4_damage_recovery.png)

Figure 7:Damage reduction and recovery\.\(a\) Q5 KL through recovery\. \(b\) Post\-OPD KL by fixed group \(log scale\)\. Both use 8B Wanda and Section[3](https://arxiv.org/html/2610.11332#S3)teacher prefixes; error bars are 95% trajectory\-bootstrap intervals\. \(c\) Post\-OPD gains across eight pairs and four tasks: blue indicates gains and red indicates losses\. AIME is the three\-year mean\.We compare masks at identical widths \(Figure[7](https://arxiv.org/html/2610.11332#S5.F7)\) and revisit the teacher prefixes and baseline\-defined Q1–Q5 groups from Section[3](https://arxiv.org/html/2610.11332#S3), now adding the calibrated checkpoints \(Figure[7](https://arxiv.org/html/2610.11332#S5.F7)\)\. We have the following observations:

- •Obs\. 7: ReCal preserves most selections, with larger changes in deeper layers\.Activation\-based pairs retain 75\.8–79\.3% of baseline deletions, with ranking correlations of 0\.784–0\.839\. Replacement is greater in middle and late layers; 4B LLM\-Pruner replaces 37\.6% of its deletion set\. With widths fixed,ReCalchanges which channels survive while retaining most base selections\.
- •Obs\. 8: ReCal reduces the residual damage identified before recovery\.On the fixed Q5 tokens,ReCallowers KL at every stage, ending at 0\.470 versus 0\.627 for Base, a 25\.1% reduction\. The paired difference is−0\.157\-0\.157nats \(95% interval:\[−0\.177,−0\.140\]\[\-0\.177,\-0\.140\]\)\. Q3–Q5 improve after OPD, while Q1/Q2 intervals include zero\. The reduction thus reaches the vulnerable tokens identified in Section[3](https://arxiv.org/html/2610.11332#S3), without improving every group\.
- •Obs\. 9: The average SFT advantage persists after OPD\.Across all eight pairs, the mean AIME advantage is 7\.9 points after SFT and 7\.7 after OPD\. The signed advantage increases in four pairs and decreases in four\. In 8B LLM\-Pruner, OPD narrows the SFT deficit without reversing it\. Final gains therefore do not imply uniformly larger OPD increments\. Appendix[B\.3](https://arxiv.org/html/2610.11332#A2.SS3)adds completion and alignment diagnostics\.

## 6Related Work

#### Structured pruning and capability recovery\.

Structured pruning selects channels or heads using gradient, activation, or reconstruction criteria\([Ma et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib16);[An et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib4);[van der Ouderaa et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib33);[Meng et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib34);[Ling et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib38);[Guo et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib9)\)\. Continued training and distillation recover the resulting models\([Xia et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib27);[Muralidharan et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib20);[Sreenivas et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib24)\)\. DarwinLM evaluates candidate structures after brief training, while NIRVANA models fine\-tuning dynamics and selects calibration data with KL\([Tang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib39);[Ai et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib3)\)\. Reasoning\-oriented approaches use generated traces, preserve sampling diversity, or improve student initialization\([Wang et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib26);[Nguyen et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib21);[Monjur et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib19);[He et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib40)\)\.ReCalmeasures teacher–probe damage at individual tokens and reweights their contribution within each teacher trajectory\. It preserves the base criterion and widths, changing the calibration evidence used to select a student for SFT and OPD\. Appendix[D](https://arxiv.org/html/2610.11332#A4)compares these mechanisms and broader pruning approaches\.

#### Offline and on\-policy distillation\.

Offline distillation fits teacher responses\([Kim and Rush, 2016](https://arxiv.org/html/2610.11332#bib.bib1)\); OPD supervises student\-generated prefixes to address training–generation mismatch\([Agarwal et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib2);[Gu et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib8)\)\. DistiLLM uses skew divergences and adaptive reuse, DistiLLM\-2 distinguishes teacher and student responses, and speculative distillation interleaves their tokens\([Ko et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib13);[Ko et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib14);[Xu et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib42)\)\. Recent work studies thinking\-pattern compatibility, entropy\-aware forward KL, and privileged\-context self\-distillation\([Li et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib15);[Jin et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib12);[Zhao et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib43)\)\. For pruned students, ShortOPD schedules short\-to\-long rollouts to reduce computation spent on teacher\-confirmed repetition\([Zhang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib30)\)\. These methods modify recovery supervision or sampling\.ReCaluses forward KL on teacher prefixes to select the starting structure, leaving the OPD loss unchanged\.

## 7Limitations

#### Generalization beyond the current pruning setting\.

Our experiments focus on FFN\-width pruning for two Qwen3 models followed by a fixed SFT–OPD recovery pipeline\. WhileReCalconsistently improves recovered reasoning performance across several pruning criteria, its effectiveness may depend on the type of structure being removed and the capability being evaluated\. In particular, the mixed results on instruction following and the 8B LLM\-Pruner case indicate that recovery\-aware calibration does not uniformly benefit every criterion or task\. An important next step is to extendReCalbeyond FFN channels to other structured compression settings, such as attention\-head and layer pruning, and to study whether the same recovery\-aware signal transfers across model families and more interactive agent workloads\.

#### Understanding what makes recovery\-aware calibration effective\.

ReCalcombines teacher trajectories with damage\-aware weighting based on forward KL, so the current experiments evaluate the complete calibration procedure rather than isolating each design choice independently\. Although the fixed\-prefix analysis shows thatReCalreduces residual discrepancy on tokens that are heavily affected by pruning, the precise connection between this reduction and downstream recovery remains only partially understood\. We are therefore interested in disentangling the roles of trajectory choice, divergence direction, and token weighting, and in developing diagnostics that follow the student states encountered during OPD rather than only fixed teacher prefixes\. These directions may help identify when recovery\-aware calibration is most useful and lead to more general principles for selecting structures that remain recoverable after pruning\.

## 8Conclusion

We introducedReCal, a recovery\-aware calibration approach for structured pruning followed by on\-policy distillation\. By using teacher–probe disagreement to reweight calibration evidence before structure selection,ReCalimproves recovered mathematical reasoning and reduces residual damage at vulnerable tokens\. More broadly, our results suggest that pruning should account not only for what a model preserves immediately after compression, but also for what the resulting student can recover afterward\.

## AI Use Statement

Generative AI tools were used solely as assistants for coding and writing\. For coding, they supported the implementation, debugging, refactoring, and documentation of author\-designed training and evaluation pipelines, data\-processing utilities, and visualization scripts\. For writing, they were used to improve clarity, concision, phrasing, and copy\-editing of content and claims developed by the authors\. Generative AI was not used for research ideation, problem formulation, methodological or experimental design, theoretical development or proofs, literature search or reference selection, result interpretation, or the generation of experimental results\. All AI\-assisted code was reviewed and tested by the authors, and all reported results were independently verified against the underlying per\-sample records\. The authors reviewed all AI\-assisted outputs and take full responsibility for the final manuscript, claims, code, and associated artifacts\.

## Ethics Statement

This study examines compression and recovery of existing language models using public benchmarks and teacher\-generated responses\. Compression may reduce the resources needed to deploy capable models, including in applications with potential for misuse\. The reported recovery gains do not establish safety, fairness, or reliability in deployment\. In particular, instruction\-following and scientific reasoning do not improve uniformly, and the compressed students retain failure modes of both pruning and their teachers\. Models and data should be used under their applicable licenses, and intended applications require separate capability and safety evaluation\.

## Reproducibility Statement

Our code:[https://github\.com/jianghoucheng/ReCal](https://github.com/jianghoucheng/ReCal)\. Sections[4](https://arxiv.org/html/2610.11332#S4)and[5\.1](https://arxiv.org/html/2610.11332#S5.SS1)describe the method and evaluation comparisons\. Appendix[A](https://arxiv.org/html/2610.11332#A1)documents the models, calibration adapters, training data, recovery settings, and evaluation protocol; Appendix[A\.6](https://arxiv.org/html/2610.11332#A1.SS6)specifies the fixed\-prefix analysis\. The accompanying source bundle includes aggregate measurements and table/figure scripts\. Appendix[C](https://arxiv.org/html/2610.11332#A3)states the theoretical assumptions and provides the proof\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. BachemOn\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2306.13649)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.11332#S2.SS2.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Aiet al\.\(2026\)M\. Ai, T\. Wei, S\. Chen, and J\. HeNIRVANA: Structured Pruning Reimagined for Large Language Model Compression\.InConference on Language Modeling,External Links:[Link](https://arxiv.org/abs/2509.14230)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p3.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Anet al\.\(2024\)Y\. An, X\. Zhao, T\. Yu, M\. Tang, and J\. WangFluctuation\-based Adaptive Structured Pruning for Large Language Models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 10865–10873\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i10.28960),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/28960)Cited by:[§A\.2](https://arxiv.org/html/2610.11332#A1.SS2.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.11332#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.11332#S2.SS1.p2.2),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2026\)AnthropicIntroducing Claude Fable 5\.1 and Claude Mythos 5\.1\.Note:Official model releaseAccessed September 23, 2026External Links:[Link](https://www.anthropic.com/claude-fable-and-mythos-5-1)Cited by:[§1](https://arxiv.org/html/2610.11332#S1.p1.1)\.
- Ashkbooset al\.\(2024\)S\. Ashkboos, M\. L\. Croci, M\. G\. d\. Nascimento, T\. Hoefler, and J\. HensmanSliceGPT: Compress Large Language Models by Deleting Rows and Columns\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2401.15024)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1)\.
- Cover and Thomas \(2006\)T\. M\. Cover and J\. A\. ThomasElements of information theory\.Second edition,John Wiley & Sons\.External Links:[Document](https://dx.doi.org/10.1002/047174882X),[Link](https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X)Cited by:[§C\.1](https://arxiv.org/html/2610.11332#A3.SS1.p2.1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu,et al\.DeepSeek\-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\.arXiv preprint arXiv:2501\.12948\.External Links:[Link](https://arxiv.org/abs/2501.12948)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px3.p1.1)\.
- Dutta and Aditya \(2026\)A\. Dutta and S\. AdityaMuCRASP: Multimodal Chain\-of\-thought Reasoning aware Structured Pruning\.arXiv preprint arXiv:2605\.25842\.External Links:[Link](https://arxiv.org/abs/2605.25842)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px3.p1.1)\.
- Frantar and Alistarh \(2023\)E\. Frantar and D\. AlistarhSparseGPT: Massive Language Models Can Be Accurately Pruned in One\-Shot\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 10323–10337\.External Links:[Link](https://proceedings.mlr.press/v202/frantar23a.html)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMiniLLM: Knowledge Distillation of Large Language Models\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/8ac015d409635f196f9e3e9dcfb9a94e-Abstract-Conference.html)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)J\. Guo, X\. Chen, Y\. Tang, and Y\. WangSlimLLM: Accurate Structured Pruning for Large Language Models\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2505.22689)Cited by:[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2026\)J\. He, Y\. Tang, T\. Nie, G\. Li, B\. Yang, J\. Du, L\. Sun, and W\. MaReasoning\-preserved Efficient Distillation of Large Language Models via Activation\-aware Initialization\.arXiv preprint arXiv:2605\.29327\.External Links:[Link](https://arxiv.org/abs/2605.29327)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the Knowledge in a Neural Network\.arXiv preprint arXiv:1503\.02531\.External Links:[Link](https://arxiv.org/abs/1503.02531)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/94074dd5a072d28ff75a76dabed43767-Abstract-Conference.html)Cited by:[§A\.5](https://arxiv.org/html/2610.11332#A1.SS5.SSS0.Px4.p1.1),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px2.p1.1)\.
- Jinet al\.\(2026\)W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. LeeEntropy\-Aware On\-Policy Distillation of Language Models\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2603.07079)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Kim and Rush \(2016\)Y\. Kim and A\. M\. RushSequence\-level knowledge distillation\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 1317–1327\.External Links:[Document](https://dx.doi.org/10.18653/v1/D16-1139),[Link](https://aclanthology.org/D16-1139/)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.11332#S2.SS2.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Koet al\.\(2025\)J\. Ko, T\. Chen, S\. Kim, T\. Ding, L\. Liang, I\. Zharkov, and S\. YunDistiLLM\-2: A Contrastive Approach Boosts the Distillation of LLMs\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2503.07067)Cited by:[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Koet al\.\(2024\)J\. Ko, S\. Kim, T\. Chen, and S\. YunDistiLLM: Towards Streamlined Distillation for Large Language Models\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2402.03898)Cited by:[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Kolawoleet al\.\(2024\)S\. Kolawole, L\. Dery, J\. Kagy, V\. Smith, G\. Neubig, and A\. TalwalkarEverybody Prune Now: Structured Pruning of LLMs with only Forward Passes\.arXiv preprint arXiv:2402\.05406\.External Links:[Link](https://arxiv.org/abs/2402.05406)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p1.1)\.
- Kurticet al\.\(2023\)E\. Kurtic, E\. Frantar, and D\. AlistarhZipLM: Inference\-Aware Structured Pruning of Language Models\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://arxiv.org/abs/2302.04089)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p2.1)\.
- Liet al\.\(2026\)Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. DingRethinking On\-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe\.arXiv preprint arXiv:2604\.13016\.External Links:[Link](https://arxiv.org/abs/2604.13016)Cited by:[§2\.2](https://arxiv.org/html/2610.11332#S2.SS2.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Linget al\.\(2024\)G\. Ling, Z\. Wang, Y\. Yan, and Q\. LiuSlimGPT: Layer\-wise Structured Pruning for Large Language Models\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2024/hash/c1c44e46358e0fb94dc94ec495a7fb1a-Abstract-Conference.html)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled Weight Decay Regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1711.05101)Cited by:[§A\.4](https://arxiv.org/html/2610.11332#A1.SS4.p1.1)\.
- Maet al\.\(2023\)X\. Ma, G\. Fang, and X\. WangLLM\-Pruner: On the Structural Pruning of Large Language Models\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2023/hash/44956951349095f74492a5471128a7e0-Abstract-Conference.html)Cited by:[§A\.2](https://arxiv.org/html/2610.11332#A1.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.11332#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.11332#S2.SS1.p2.2),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Menet al\.\(2024\)X\. Men, M\. Xu, Q\. Zhang, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. ChenShortGPT: Layers in Large Language Models are More Redundant Than You Expect\.arXiv preprint arXiv:2403\.03853\.External Links:[Link](https://arxiv.org/abs/2403.03853)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p2.1)\.
- Menget al\.\(2024\)X\. Meng, S\. Ibrahim, K\. Behdin, H\. Hazimeh, N\. Ponomareva, and R\. MazumderOSSCAR: One\-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 35354–35377\.External Links:[Link](https://proceedings.mlr.press/v235/meng24a.html)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Monjuret al\.\(2026\)O\. Monjur, S\. K\. Nahin, and A\. ChhabraRevisiting the Effectiveness of LLM Pruning for Test\-Time Scaling\.InFindings of the Association for Computational Linguistics: EMNLP 2026,External Links:[Link](https://arxiv.org/abs/2604.25098)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Muralidharanet al\.\(2024\)S\. Muralidharan, S\. T\. Sreenivas, R\. Joshi, M\. Chochowski, M\. Patwary, M\. Shoeybi, B\. Catanzaro, J\. Kautz, and P\. MolchanovCompact Language Models via Pruning and Knowledge Distillation\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2024/hash/4822991365c962105b1b95b1107d30e5-Abstract-Conference.html)Cited by:[§A\.2](https://arxiv.org/html/2610.11332#A1.SS2.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.11332#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Nguyenet al\.\(2025\)H\. T\. Nguyen, B\. Nguyen, and V\. A\. NguyenStructured Pruning for Diverse Best\-of\-N Reasoning Optimization\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 23911–23922\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1225),[Link](https://aclanthology.org/2025.findings-acl.1225/)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2026\)OpenAIGPT\-6 Astra\.Note:Official model documentationAccessed September 23, 2026External Links:[Link](https://developers.openai.com/api/docs/models/gpt-6-astra)Cited by:[§1](https://arxiv.org/html/2610.11332#S1.p1.1)\.
- Pyatkinet al\.\(2025\)V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. HajishirziGeneralizing Verifiable Instruction Following\.InAdvances in Neural Information Processing Systems,Vol\.38\.Note:Datasets and Benchmarks TrackExternal Links:[Link](https://arxiv.org/abs/2507.02833)Cited by:[§A\.5](https://arxiv.org/html/2610.11332#A1.SS5.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px2.p1.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: A Graduate\-Level Google\-Proof Q&A Benchmark\.arXiv preprint arXiv:2311\.12022\.External Links:[Link](https://arxiv.org/abs/2311.12022)Cited by:[§A\.5](https://arxiv.org/html/2610.11332#A1.SS5.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px2.p1.1)\.
- Rosset al\.\(2011\)S\. Ross, G\. Gordon, and D\. BagnellA reduction of imitation learning and structured prediction to no\-regret online learning\.InInternational Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.15,pp\. 627–635\.External Links:[Link](https://proceedings.mlr.press/v15/ross11a.html)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1)\.
- Songet al\.\(2024\)J\. Song, K\. Oh, T\. Kim, H\. Kim, Y\. Kim, and J\. KimSLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 46136–46155\.External Links:[Link](https://proceedings.mlr.press/v235/song24f.html)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p2.1)\.
- Sreenivaset al\.\(2024\)S\. T\. Sreenivas, S\. Muralidharan, R\. Joshi, M\. Chochowski, A\. S\. Mahabaleshwarkar, G\. Shen, J\. Zeng,et al\.LLM Pruning and Distillation in Practice: The Minitron Approach\.arXiv preprint arXiv:2408\.11796\.External Links:[Link](https://arxiv.org/abs/2408.11796)Cited by:[§A\.2](https://arxiv.org/html/2610.11332#A1.SS2.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Sunet al\.\(2024\)M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. KolterA Simple and Effective Pruning Approach for Large Language Models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2306.11695)Cited by:[§A\.2](https://arxiv.org/html/2610.11332#A1.SS2.SSS0.Px2.p1.1),[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2610.11332#S2.SS1.p2.2),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px1.p1.1)\.
- Tanget al\.\(2026\)S\. Tang, O\. Sieberling, E\. Kurtic, Z\. Shen, and D\. AlistarhDarwinLM: Evolutionary Structured Pruning of Large Language Models\.InConference on Language Modeling,External Links:[Link](https://arxiv.org/abs/2502.07780)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p3.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- van der Ouderaaet al\.\(2024\)T\. F\. A\. van der Ouderaa, M\. Nagel, M\. van Baalen, Y\. M\. Asano, and T\. BlankevoortThe LLM Surgeon\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2312.17244)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, E\. Diao, Q\. Le, P\. Wang, G\. Wang, M\. Lee, S\. Yeh, and L\. YangThink Before You Prune: Self\-Reflective Structured Pruning for Reasoning Language Models\.arXiv preprint arXiv:2512\.02185\.External Links:[Link](https://arxiv.org/abs/2512.02185)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p2.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Xiaet al\.\(2024\)M\. Xia, T\. Gao, Z\. Zeng, and D\. ChenSheared LLaMA: Accelerating Language Model Pre\-training via Structured Pruning\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/160adf2dc118a920e7858484b92a37d8-Abstract-Conference.html)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2025\)W\. Xu, R\. Han, Z\. Wang, L\. T\. Le, D\. Madeka, L\. Li, W\. Y\. Wang, R\. Agarwal, C\. Lee, and T\. PfisterSpeculative Knowledge Distillation: Bridging the Teacher\-Student Gap Through Interleaved Sampling\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2410.11325)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p2.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu,et al\.Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§A\.1](https://arxiv.org/html/2610.11332#A1.SS1.p1.1),[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.11332#S5.SS1.SSS0.Px1.p1.1)\.
- Yanget al\.\(2024\)Y\. Yang, Z\. Cao, and H\. ZhaoLaCo: Large Language Model Pruning via Layer Collapse\.InFindings of the Association for Computational Linguistics: EMNLP 2024,External Links:[Link](https://arxiv.org/abs/2402.11187)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p2.1)\.
- Zhanget al\.\(2026\)Q\. Zhang, Q\. Yuan, H\. Lin, Y\. Lu, X\. Han, L\. Sun, M\. Xu, and J\. LiShortOPD: Recovering Pruned LLMs with Short\-to\-Long On\-Policy Distillation\.arXiv preprint arXiv:2607\.13124\.External Links:[Link](https://arxiv.org/abs/2607.13124)Cited by:[§B\.4](https://arxiv.org/html/2610.11332#A2.SS4.p2.1),[§1](https://arxiv.org/html/2610.11332#S1.p1.1),[§1](https://arxiv.org/html/2610.11332#S1.p2.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026\)S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. GroverSelf\-Distilled Reasoner: On\-Policy Self\-Distillation for Large Language Models\.arXiv preprint arXiv:2601\.18734\.External Links:[Link](https://arxiv.org/abs/2601.18734)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.11332#S6.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2023\)Y\. Zhao, A\. Gu, R\. Varma, L\. Luo, C\. Huang, M\. Xu, L\. Wright, H\. Shojanazeri, M\. Ott, S\. Shleifer, A\. Desmaison, C\. Balioglu, P\. Damania, B\. Nguyen, G\. Chauhan, Y\. Hao, A\. Mathews, and S\. LiPyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel\.arXiv preprint arXiv:2304\.11277\.External Links:[Link](https://arxiv.org/abs/2304.11277)Cited by:[§A\.4](https://arxiv.org/html/2610.11332#A1.SS4.SSS0.Px2.p1.1)\.
- Zhonget al\.\(2025\)L\. Zhong, F\. Wan, R\. Chen, X\. Quan, and L\. LiBlockPruner: Fine\-grained Pruning for Large Language Models\.InFindings of the Association for Computational Linguistics: ACL 2025,External Links:[Link](https://arxiv.org/abs/2406.10594)Cited by:[Appendix D](https://arxiv.org/html/2610.11332#A4.SS0.SSS0.Px1.p2.1)\.

## Appendix AExperimental Details

We describe the models and channel\-scoring adapters in Sections[A\.1](https://arxiv.org/html/2610.11332#A1.SS1)–[A\.2](https://arxiv.org/html/2610.11332#A1.SS2), followed by the training data \(Section[A\.3](https://arxiv.org/html/2610.11332#A1.SS3)\) and recovery settings \(Section[A\.4](https://arxiv.org/html/2610.11332#A1.SS4)\)\. Sections[A\.5](https://arxiv.org/html/2610.11332#A1.SS5)and[A\.6](https://arxiv.org/html/2610.11332#A1.SS6)specify benchmark evaluation and fixed\-prefix analysis\.

### A\.1Models and Pruned Architecture

We use Qwen3\-8B with thinking enabled and Qwen3\-4B\-Instruct\-2507 in non\-thinking mode\([Yang et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib29)\)\. Each student’s corresponding unpruned checkpoint serves as its frozen teacher\. Structured pruning removes channels from the gated FFN\. For hidden inputzzand intermediate activation

a=SiLU⁡\(Wgate​z\)⊙Wup​z,o=Wdown​a,a=\\operatorname\{SiLU\}\(W^\{\\mathrm\{gate\}\}z\)\\odot W^\{\\mathrm\{up\}\}z,\\qquad o=W^\{\\mathrm\{down\}\}a,\(6\)removing channeljjremoves rowjjofWgateW^\{\\mathrm\{gate\}\}andWupW^\{\\mathrm\{up\}\}and columnjjofWdownW^\{\\mathrm\{down\}\}\. Attention, embeddings, normalization layers, and the language\-model head are retained\. Every layer uses the same retained width, aligned to a multiple of 128\. Table[2](https://arxiv.org/html/2610.11332#A1.T2)gives the exact widths; whole\-model parameter reduction is smaller than FFN\-width reduction because only FFNs are pruned\.

Table 2:Retained FFN widths and whole\-model parameter reduction\. The original widths are 12,288 for 8B and 9,728 for 4B\.ModelTarget reductionRetained widthActual reductionParameters8B15%9,47215\.21%6\.945B8B25%7,68024\.89%6\.152B8B35%5,76035\.26%5\.303B4B15%7,55214\.96%3\.421B4B25%6,14424\.63%3\.032B4B35%4,60835\.19%2\.607B
### A\.2Pruning Criteria and Calibration

The compared methods share the architecture in Section[A\.1](https://arxiv.org/html/2610.11332#A1.SS1)\. They differ in channel scores, rather than layerwise width allocation\. For responserr, letar,t,ja\_\{r,t,j\}be channeljj’s activation at tokenttandWdown:,jW^\{\\mathrm\{down\}\}\_\{:,j\}its output\-weight column\. We suppress the layer index below\. ForReCal, define the weighted activation expectation

𝔼w​\[f⁡\(aj\)\]=1R​∑r=1R∑t∈ℛrw~r,t​f​\(ar,t,j\),\\mathbb\{E\}\_\{w\}\[f\(a\_\{j\}\)\]=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\sum\_\{t\\in\\mathcal\{R\}\_\{r\}\}\\widetilde\{w\}\_\{r,t\}f\(a\_\{r,t,j\}\),\(7\)whereℛr\\mathcal\{R\}\_\{r\}contains valid response tokens and each response has total weight one\. The ordinary adapters aggregate uniformly over their prompt calibration inputs\.ReCalinstead uses teacher responses and the damage weights in Equation[4](https://arxiv.org/html/2610.11332#S4.E4)\.

#### Minitron\.

Following activation\-based width pruning\([Muralidharan et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib20);[Sreenivas et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib24)\), the implemented adapter first averages absolute activations within each calibration sequence, then aggregates their squares across sequences:

sj=∑r\(1\|𝒯r\|​∑t∈𝒯r\|ar,t,j\|\)2\.s\_\{j\}=\\sqrt\{\\sum\_\{r\}\\left\(\\frac\{1\}\{\|\\mathcal\{T\}\_\{r\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{r\}\}\|a\_\{r,t,j\}\|\\right\)^\{2\}\}\.\(8\)Here𝒯r\\mathcal\{T\}\_\{r\}is the baseline sequence’s token set\.ReCalreplaces the inner mean with∑t∈ℛrw~r,t​\|ar,t,j\|\\sum\_\{t\\in\\mathcal\{R\}\_\{r\}\}\\widetilde\{w\}\_\{r,t\}\|a\_\{r,t,j\}\|\. Squaring remains outside the within\-sequence aggregation; this is not equivalent to a global mean\-absolute\-activation score\.

#### Wanda\.

Wanda combines weight magnitude and activation statistics\([Sun et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib25)\)\. Our structured channel adapter, Wanda\-SP, scores

sj=∥Wdown:,j∥2𝔼⁡\[aj2\]\.s\_\{j\}=\\\|W^\{\\mathrm\{down\}\}\_\{:,j\}\\\|\_\{2\}\\sqrt\{\\mathbb\{E\}\[a\_\{j\}^\{2\}\]\}\.\(9\)ReCalsubstitutes𝔼w​\[aj2\]\\mathbb\{E\}\_\{w\}\[a\_\{j\}^\{2\}\]for the uniform second moment\. This preserves the weight factor and channel grouping\. It is a structured adaptation of Wanda, whose original formulation prunes individual weights\.

#### FLAP\.

The reported FLAP adapter uses the WIFN score\([An et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib4)\):

sj=∥W:,jdown∥2𝔼\[\|aj\|\]\.s\_\{j\}=\\\|W^\{\\mathrm\{down\}\}\_\{:,j\}\\\|\_\{2\}\\mathbb\{E\}\[\|a\_\{j\}\|\]\.\(10\)ReCalreplaces the activation expectation with𝔼w​\[\|aj\|\]\\mathbb\{E\}\_\{w\}\[\|a\_\{j\}\|\]\. Although the implementation also records the WIFV statistic∥W:,jdown∥22Var\(aj\)\\\|W^\{\\mathrm\{down\}\}\_\{:,j\}\\\|\_\{2\}^\{2\}\\operatorname\{Var\}\(a\_\{j\}\), WIFV does not select the reported masks\. We do not use FLAP’s adaptive layerwise sparsity or bias compensation; fixed per\-layer widths and the same recovery pipeline allow comparison of channel selection across all adapters\.

#### LLM\-Pruner\.

The first\-order LLM\-Pruner adapter groups the gate, up, and down weights associated with each channel, denoted byGjG\_\{j\}, and uses the first\-order score\([Ma et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib16)\)

sj=∑u∈Gj\|u​∂ℒ∂u\|\.s\_\{j\}=\\sum\_\{u\\in G\_\{j\}\}\\left\|u\\frac\{\\partial\\mathcal\{L\}\}\{\\partial u\}\\right\|\.\(11\)The baseline uses next\-token cross entropy on calibration prompts\.ReCaldifferentiates a weighted response\-token loss,ℒ=∑twt​CEt/∑twt\\mathcal\{L\}=\\sum\_\{t\}w\_\{t\}\\operatorname\{CE\}\_\{t\}/\\sum\_\{t\}w\_\{t\}, and applies the same group score\. Only FFN projection parameters receive gradients during scoring\. The maximum scoring sequence length is 2,048 for both variants\. Weights enter the loss before differentiation and before the absolute\-value aggregation\.

#### Calibration data and divergence\.

Both calibration pools are balanced equally across mathematics, code, science, and instruction following\. The baseline pool contains prompts, whereasReCaluses responses from the unpruned teacher, with prompt and padding tokens excluded from weighted statistics\. The probe is the corresponding baseline pruned to the same target widths, without recovery training\. Final scores are recomputed on the original unpruned model; probe weights and masks are not inherited by the final student\.

For efficient damage estimation, letSr,tS\_\{r,t\}be the teacher’s Top\-kktoken set at a shared prefix\. We retain the original probabilities on these tokens and collect the remaining mass in a single bucket:

d^r,t=∑v∈Sr,tpT​\(v\)​log⁡pT​\(v\)pP​\(v\)\+pT​\(other\)​log⁡pT​\(other\)pP​\(other\),\\widehat\{d\}\_\{r,t\}=\\sum\_\{v\\in S\_\{r,t\}\}p\_\{T\}\(v\)\\log\\frac\{p\_\{T\}\(v\)\}\{p\_\{P\}\(v\)\}\+p\_\{T\}\(\\mathrm\{other\}\)\\log\\frac\{p\_\{T\}\(\\mathrm\{other\}\)\}\{p\_\{P\}\(\\mathrm\{other\}\)\},\(12\)wherepM​\(other\)=1−∑v∈Sr,tpM​\(v\)p\_\{M\}\(\\mathrm\{other\}\)=1\-\\sum\_\{v\\in S\_\{r,t\}\}p\_\{M\}\(v\)forM∈\{T,P\}M\\in\\\{T,P\\\}\. We usek=16k=16and a maximum calibration length of 8,192 tokens for the activation\-based adapters\. This is KL on a shared coarsened vocabulary, not KL after renormalizing only the Top\-kktokens\. The per\-response weights arew~r,t=d^r,t/∑ud^r,u\\widetilde\{w\}\_\{r,t\}=\\widehat\{d\}\_\{r,t\}/\\sum\_\{u\}\\widehat\{d\}\_\{r,u\}\. If the denominator is at most10−910^\{\-9\}, we use uniform weights over valid response tokens\. Each response therefore has equal total weight regardless of its length\. Damage scores are aligned with the response\-token activations that predict the affected next token before statistics are accumulated\. These comparisons evaluate the combined use of teacher trajectories and damage weighting\.

### A\.3Training Data

SFT and OPD use separate prompt pools drawn from mathematics, code, science, and instruction following, initially mixed in equal proportions \(1:1:1:1\)\. Table[3](https://arxiv.org/html/2610.11332#A1.T3)lists the sources\. Science prompts balance physics, chemistry, and biology in a 1:1:1 ratio; OPD instruction prompts balance one through five constraints\. Mathematics prompts added from the SFT source use a separate source partition, and the final SFT and OPD pools have no normalized\-prompt overlap\. Prompts undergo deduplication and benchmark\-overlap screening\.

Table 3:Public prompt sources\. Dataset identifiers are linked to their source pages\. NVIDIA datasets carry thenvidia/namespace; IF\-multi is fromallenai/\. The initial domain mixture is 1:1:1:1 in both stages\.DomainSFT prompt sourceOPD prompt sourceMath[Nemotron\-SFT\-Math\-v4](https://huggingface.co/datasets/nvidia/Nemotron-SFT-Math-v4)[Nemotron\-RL\-Math\-v2](https://huggingface.co/datasets/nvidia/Nemotron-RL-Math-v2); disjoint supplement from SFT\-Math\-v4Code[Nemotron\-SFT\-Competitive\-Programming\-v2](https://huggingface.co/datasets/nvidia/Nemotron-SFT-Competitive-Programming-v2)[Nemotron\-RL\-coding\-competitive\_coding](https://huggingface.co/datasets/nvidia/Nemotron-RL-coding-competitive_coding)Science[Nemotron\-SFT\-Science\-v2](https://huggingface.co/datasets/nvidia/Nemotron-SFT-Science-v2)[Nemotron\-RL\-Science\-v1](https://huggingface.co/datasets/nvidia/Nemotron-RL-Science-v1)Instruction[Nemotron\-SFT\-Instruction\-Following\-Chat\-v3](https://huggingface.co/datasets/nvidia/Nemotron-SFT-Instruction-Following-Chat-v3)[IF\_multi\_constraints\_upto5](https://huggingface.co/datasets/allenai/IF_multi_constraints_upto5)Only prompts are retained from these sources\. For SFT, the corresponding unpruned teacher generates new targets with temperature 0\.6 and top\-pp0\.95\. Truncated or malformed responses are excluded, so the final SFT response mixture is not exactly domain\-balanced\. All compared methods for a given model share this same target set\. Mathematics and science use a common final\-answer format, code follows the LiveCodeBench Python template, and instruction\-following prompts retain their original constraints\. OPD uses prompts only; it does not consume source answers or external task rewards\.

### A\.4Recovery Hyperparameters

Table[4](https://arxiv.org/html/2610.11332#A1.T4)summarizes the settings that determine training and sampling\. Both stages use AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.11332#bib.bib44)\)\. SFT minimizes next\-token cross entropy on assistant responses, including reasoning tokens when thinking is enabled\. User and system tokens do not contribute to the loss\. OPD then samples fresh responses from the current student and evaluates the teacher at those student\-generated prefixes\. All completed main runs use 200 OPD updates\.

Table 4:Recovery settings shared by each Base/ReCal pair\. SFT is offline distillation on teacher responses\. OPD uses student rollouts\.SettingSFTOPDOptimizerAdamWAdamWLearning rate10−510^\{\-5\}10−610^\{\-6\}ScheduleCosine, 5% warmupConstantBatch size32 sequences64 promptsTraining budget2 epochs200 updatesResponses per prompt1 teacher target4 student rolloutsSampling temperature0\.6 \(teacher\)1\.0 \(student\)Sampling top\-pp0\.951\.0Response limit20,48020,480Sequence / prompt limit21,504 sequence2,048 promptDistillation supportSampled teacher tokensStudent Top\-16Teacher temperature—1\.0Repetition penalty—1\.05Optimization passes per update—1Policy clip ratio—0\.2Gradient clipping1\.01\.0Weight decay00#### Restricted\-support OPD signal\.

Equation[2](https://arxiv.org/html/2610.11332#S2.E2)describes the full\-vocabulary reference objective\. The implemented signal instead uses the student’s Top\-16 token IDsKtK\_\{t\}at each prefixhth\_\{t\}\. Student probabilities are normalized on this support:

p~S​\(v∣ht\)=pS​\(v∣ht\)∑u∈KtpS​\(u∣ht\),v∈Kt\.\\widetilde\{p\}\_\{S\}\(v\\mid h\_\{t\}\)=\\frac\{p\_\{S\}\(v\\mid h\_\{t\}\)\}\{\\sum\_\{u\\in K\_\{t\}\}p\_\{S\}\(u\\mid h\_\{t\}\)\},\\qquad v\\in K\_\{t\}\.\(13\)The token\-level teacher signal is

rt,v=−p~S​\(v∣ht\)​\[log⁡pS​\(v∣ht\)−log⁡pT​\(v∣ht\)\]\.r\_\{t,v\}=\-\\widetilde\{p\}\_\{S\}\(v\\mid h\_\{t\}\)\\bigl\[\\log p\_\{S\}\(v\\mid h\_\{t\}\)\-\\log p\_\{T\}\(v\\mid h\_\{t\}\)\\bigr\]\.\(14\)The implementation uses this signal directly as an advantage in its clipped policy update, without GAE or an external task reward\. The log probabilities retain their original vocabulary normalization\. Thus, this is a restricted\-support reverse\-KL\-style signal, not the full KL in Equation[2](https://arxiv.org/html/2610.11332#S2.E2), and it differs from the teacher\-support\-plus\-residual calculation used for calibration\.

#### Shared optimization conditions\.

Paired runs start from the same unpruned checkpoint, retain identical per\-layer widths, and share the same SFT targets, OPD prompts, data ordering, and optimization settings\. Training uses FSDP\([Zhao et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib45)\), BF16 forward computation, and FP32 master weights\. The recorded environment uses eight NVIDIA H20 GPUs\. Matching update counts does not imply identical sampled token counts or GPU time, since students can generate different response lengths\.

### A\.5Benchmarks and Evaluation

We evaluate every available checkpoint before recovery, after SFT, and after SFT plus OPD, using the same response budget of 20,480 tokens\. AIME is sampled at temperature 0\.6 and top\-pp0\.95; the other benchmarks use greedy decoding\. Metrics are defined below\.

#### AIME 2024–2026\.

Each annual set contains competition mathematics problems with exact numerical answers\. We average correctness over eight sampled responses per question and then average the three yearly scores\. This is average sampled accuracy, not pass@8\. The final boxed answer is extracted and normalized before comparison\.

#### GPQA\-Diamond\.

GPQA evaluates graduate\-level science questions in biology, chemistry, and physics\([Rein et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib23)\)\. We use the Diamond test split and report multiple\-choice accuracy, extracting the final A/B/C/D answer from the common boxed\-answer format\.

#### IFBench\.

IFBench evaluates compliance with verifiable instructions\([Pyatkin et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib22)\)\. We use the official test set and verifier and report instruction\-level strict accuracy \(IFB\-S\)\. A response can satisfy some constraints while violating others, so this metric is distinct from prompt\-level strict accuracy\.

#### LiveCodeBench\.

We use the code\-generation task in LiveCodeBench\-v5, covering problems released from October 2024 through February 2025\([Jain et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib11)\)\. We report pass@1 under the official input/output or function\-call templates, with a 10\-second execution timeout per problem\.

#### Reporting conventions\.

Scores are percentages\. Within each main\-table model block, boldface marks the highest displayed score in each column and underlining the second\-highest distinct score, across all pruned stages, excluding the unpruned reference\. Ties receive the same formatting\. The final Avg\. column is the unweighted arithmetic mean of AIME24, AIME25, AIME26, GPQA, IFBench, and LiveCodeBench\-v5\. AIME averages elsewhere use only the three AIME benchmarks\. The original seven paired gains were reported before rounding and can differ by 0\.1 percentage point from displayed\-score differences\. The additional 8B LLM\-Pruner gain is computed from the reported three\-decimal scores\. Recovery\-stage averages use full\-precision repository records for all eight pairs; final\-performance comparisons also use all eight\. The unpruned 8B AIME/GPQA reference was regraded at the common response limit; IFBench and LiveCodeBench use fresh generations\. Individual\-year AIME scores are available for every main\-table checkpoint and both LLM\-Pruner OPD pairs\. The 8B LLM\-Pruner \+ReCalPruned/SFT scores are included in Table[5](https://arxiv.org/html/2610.11332#A2.T5)\.

### A\.6Fixed\-Prefix Diagnostic Protocol

We use held\-out teacher responses balanced across the four domains to measure distributional damage\. Both teacher and student evaluate the same prefix at every response token\. Within each response, tokens are ranked by raw\-pruned baseline KL and divided into five equal\-count groups, Q1–Q5, from lowest to highest damage\. The groups are fixed thereafter; no checkpoint selects its own favorable token subset\.

Concentration curves average each response’s fraction of damage carried by its highest\-damage tokens\. The fixed\-group plots first average KL within each response/group, then across responses\. For the Q5 damage share, each response’s group\-average KL is multiplied by its token count, divided by total response damage, and then averaged across responses\. This is an average of per\-response ratios, not a ratio of group means\.

The diagnostic pool contains 128 responses per model, equally split across domains, with no filtering by answer correctness or damage\. Recovery tracking uses 8B Wanda at the main compression ratio\. Section[3](https://arxiv.org/html/2610.11332#S3)uses only baseline checkpoints; Section[5\.5](https://arxiv.org/html/2610.11332#S5.SS5)adds the calibrated counterparts on the same responses and token groups\. Intervals use 2,000 trajectory\-bootstrap resamples; differences use paired resampling\. Tokens within a response are not treated as independent samples\.

## Appendix BSupplementary Experiments

We supplement the main results with the LLM\-Pruner integration in Section[B\.1](https://arxiv.org/html/2610.11332#A2.SS1), additional damage analysis in Section[B\.2](https://arxiv.org/html/2610.11332#A2.SS2), and generation diagnostics in Section[B\.3](https://arxiv.org/html/2610.11332#A2.SS3)\. Section[B\.4](https://arxiv.org/html/2610.11332#A2.SS4)documents the introduction diagnostics, and Section[B\.5](https://arxiv.org/html/2610.11332#A2.SS5)reports benchmark results not displayed in the main paper\.

### B\.1LLM\-Pruner Integration

Figure[8](https://arxiv.org/html/2610.11332#A2.F8)extends the comparison to both LLM\-Pruner models\. In 4B, post\-OPD AIME, GPQA, and code improve, while IFBench declines slightly\. In 8B, AIME falls from 12\.4% to 9\.2% and IFBench from 22\.4% to 19\.5%, whereas GPQA rises from 34\.8% to 44\.9% and code from 17\.5% to 28\.3%\. The 8B AIME24/25/26 scores change from 18\.8/7\.9/10\.4% to 13\.8/5\.0/8\.7%, so the mathematics decline occurs in every year\. Thus, the positive 4B result does not generalize uniformly to 8B\. The 8B mathematics deficit is already present after SFT: 3\.2% for the calibrated student versus 7\.9% for Base\. OPD narrows this deficit from 4\.7 to 3\.2 points but does not reverse it\. As with the other adapters, the comparison changes both calibration context and weighting\.

Figure 8:LLM\-Pruner with and withoutReCal\.\(a–c\) The three 4B recovery stages\. \(d\) The completed 8B OPD pair\. AIME denotes the three\-year average\. Wide blue bars show Base \+ReCal; narrow red bars show Base\.
### B\.2Additional Damage Analysis

Figure[9](https://arxiv.org/html/2610.11332#A2.F9)adds the 4B concentration curves and paired KL changes before and after SFT\. The concentration pattern extends to 4B\. In 8B Wanda, the paired changes reveal increased discrepancy in some lower\-damage groups after pruning and SFT, despite the reduction in Q5\. These earlier stages complement the post\-OPD comparison in Figure[7](https://arxiv.org/html/2610.11332#S5.F7)\(b\)\.

Figure 9:Additional concentration and redistribution results\.\(a\) Within\-response damage concentration in 4B\. \(b,c\) PairedReCalminus Base KL after pruning and SFT\. Groups remain fixed by the raw\-pruned baseline\. Intervals are 95% paired trajectory\-bootstrap intervals\.
### B\.3Generation and Alignment Diagnostics

Figure[10](https://arxiv.org/html/2610.11332#A2.F10)\(a\) reports net completion rescue on paired AIME generations: the number where Base truncates butReCalcompletes, minus the number with the reverse outcome\. Pairing uses the same question and sample index, not an identical sampled trajectory\. All seven post\-OPD pairs with completion diagnostics are positive, ranging from 12 to 116 net completions per 720 generations\. However, 4B Wanda and FLAP initially have fewer completed generations immediately after pruning\. Completion after pruning is therefore an incomplete indicator of the eventual recovered model\. Its aggregate post\-OPD truncation rate nevertheless increases from 76\.8% to 79\.4% withReCal, consistent with its mathematics exception\.

![Refer to caption](https://arxiv.org/html/2610.11332v1/supp_recovery.png)Figure 10:Supplementary recovery diagnostics\.\(a\) Net completion rescue across the three stages\. \(b,c\) Top\-16 teacher–student overlap gains during the first 20 OPD updates and at the final update\. Each model generates its own trajectories\.The three 8B comparisons enter OPD with 1\.9–3\.6 percentage points higher teacher–student Top\-16 overlap and finish 1\.0–2\.9 points higher\. The 4B changes are smaller and not uniformly positive\. Because these statistics use each student’s own rollouts, they describe the trajectories encountered during recovery and do not isolate agreement at matched states\.

Trajectory\-level diagnostics further distinguish response length from token damage\. Across probes, response length correlates strongly with summed KL \(Spearman 0\.783–0\.886\), motivating equal total weight per response\. Cross\-probe mean\-damage rankings are highly correlated \(0\.963–0\.975\), so the necessity of a criterion\-specific probe is not established\. Instruction responses have high mean damage, yet IFBench does not consistently improve\. A high divergence is therefore not interchangeable with task importance\.

### B\.4Introduction Figure Measurement

Figure[1](https://arxiv.org/html/2610.11332#S1.F1)\(b,c\) uses the measured Qwen3\-8B Minitron, Wanda, and FLAP baselines at the main compression ratio\. Panel \(b\) reports mean AIME accuracy across 2024–2026 for the unpruned teacher, raw\-pruned student, and post\-SFT student\. The shared teacher score is 61\.4%\. Panel \(c\) reports the fraction of AIME generations reaching the common response\-length limit after SFT and after OPD\. Scores follow the sampling and decoding settings in Section[A\.5](https://arxiv.org/html/2610.11332#A1.SS5)\.

Truncation measures failure to finish within the evaluation budget; it does not distinguish repetition, an unfinished derivation, or a completed answer followed by excess text\. It is therefore reported separately from answer accuracy\. These aggregate measurements do not estimate reverse KL conditional on failure type\. The introduction’s discussion of teacher confirmation at flawed prefixes draws on prior work\([Zhang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib30)\); our fixed\-prefix experiments in Section[3](https://arxiv.org/html/2610.11332#S3)examine pruning damage directly\.

### B\.5Additional Benchmark Results

Tables[5](https://arxiv.org/html/2610.11332#A2.T5)and[6](https://arxiv.org/html/2610.11332#A2.T6)report the calibrated Minitron, FLAP, and additional LLM\-Pruner scores not displayed in Table[1](https://arxiv.org/html/2610.11332#S5.T1)or Figure[4](https://arxiv.org/html/2610.11332#S5.F4)\. The first table adds Pruned/SFT checkpoints; the second adds post\-OPD science and instruction scores\. Matching baseline scores are in Table[1](https://arxiv.org/html/2610.11332#S5.T1)\. The 4B/8B LLM\-Pruner comparisons are shown in Figure[8](https://arxiv.org/html/2610.11332#A2.F8)\. Complete numerical records, including the ratio sweep, are retained in the accompanying data files\.

Table 5:Additional Pruned and SFT scores at 25% parameter reduction\. These calibrated variants are absent from the main table; their post\-OPD scores appear in Figures[4](https://arxiv.org/html/2610.11332#S5.F4)and[8](https://arxiv.org/html/2610.11332#A2.F8)\. Scores are percentages\.MethodStageAIME24AIME25AIME26GPQAIFB\-SLCB\-v5Qwen3\-8BMinitron \+ReCalPruned4\.60\.81\.730\.818\.65\.4SFT25\.816\.718\.335\.418\.617\.5FLAP \+ReCalPruned15\.09\.67\.528\.818\.68\.4SFT41\.230\.832\.138\.420\.121\.1LLM\-Pruner \+ReCalPruned0\.00\.00\.027\.318\.00\.0SFT6\.30\.42\.941\.917\.219\.3Qwen3\-4B\-InstructMinitron \+ReCalPruned4\.21\.73\.730\.323\.36\.6SFT22\.915\.015\.837\.925\.616\.3FLAP \+ReCalPruned14\.610\.08\.332\.825\.311\.4SFT27\.923\.718\.334\.824\.418\.7Table 6:Additional post\-OPD science and instruction results at 25% reduction, complementing the AIME and code results in Figure[4](https://arxiv.org/html/2610.11332#S5.F4)\. Scores are percentages\.ModelMethodGPQAIFB\-S8BMinitron \+ReCal36\.419\.58BFLAP \+ReCal37\.419\.84BMinitron \+ReCal36\.427\.04BFLAP \+ReCal35\.427\.0

## Appendix CForward KL and Teacher\-Trajectory Preservation

We first relate full\-vocabulary forward KL to preservation of teacher sequence probabilities in Section[C\.1](https://arxiv.org/html/2610.11332#A3.SS1)\. Section[C\.2](https://arxiv.org/html/2610.11332#A3.SS2)explains why this connection motivates the method without guaranteeing recovery gains\.

### C\.1Sequence\-Distribution Connection

For a fixed promptxx, writePTP\_\{T\}andPmP\_\{m\}for teacher and pruned\-model sequence distributions up to a shared horizonHH, with an absorbing EOS state\. Define

D\(m\)=𝔼y∼PT∑t=1HKL\(pT\(⋅∣x,y<t\)∥pm\(⋅∣x,y<t\)\)\.D\(m\)=\\mathbb\{E\}\_\{y\\sim P\_\{T\}\}\\sum\_\{t=1\}^\{H\}\\mathop\{\\mathrm\{KL\}\}\\bigl\(p\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\)\\\|p\_\{m\}\(\\cdot\\mid x,y\_\{<t\}\)\\bigr\)\.\(15\)
###### Proposition 1\(Teacher\-trajectory preservation\)\.

AssumePmP\_\{m\}is positive whereverPTP\_\{T\}is positive\. For any setAAof completed responses,

KL\(PT∥Pm\)=D\(m\),Pm\(A\)≥PT\(A\)−D⁡\(m\)/2\.\\mathop\{\\mathrm\{KL\}\}\(P\_\{T\}\\\|P\_\{m\}\)=D\(m\),\\qquad P\_\{m\}\(A\)\\geq P\_\{T\}\(A\)\-\\sqrt\{D\(m\)/2\}\.\(16\)

###### Proof\.

Both distributions use the same vocabulary and stopping convention\. The autoregressive factorization and the KL chain rule\([Cover and Thomas, 2006](https://arxiv.org/html/2610.11332#bib.bib46)\)give

KL\(PT∥Pm\)\\displaystyle\\mathop\{\\mathrm\{KL\}\}\(P\_\{T\}\\\|P\_\{m\}\)=𝔼y∼PT​log⁡∏t=1HpT​\(yt∣x,y<t\)∏t=1Hpm​\(yt∣x,y<t\)\\displaystyle=\\mathbb\{E\}\_\{y\\sim P\_\{T\}\}\\log\\frac\{\\prod\_\{t=1\}^\{H\}p\_\{T\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\{\\prod\_\{t=1\}^\{H\}p\_\{m\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\(17\)=∑t=1H𝔼y<t∼PTKL\(pT\(⋅∣x,y<t\)∥pm\(⋅∣x,y<t\)\)=D\(m\)\.\\displaystyle=\\sum\_\{t=1\}^\{H\}\\mathbb\{E\}\_\{y\_\{<t\}\\sim P\_\{T\}\}\\mathop\{\\mathrm\{KL\}\}\\bigl\(p\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\)\\\|p\_\{m\}\(\\cdot\\mid x,y\_\{<t\}\)\\bigr\)=D\(m\)\.\(18\)Terms after EOS are zero\. With natural logarithms, Pinsker’s inequality yields

\|PT​\(A\)−Pm​\(A\)\|≤TV⁡\(PT,Pm\)≤KL\(PT∥Pm\)/2,\|P\_\{T\}\(A\)\-P\_\{m\}\(A\)\|\\leq\\TV\(P\_\{T\},P\_\{m\}\)\\leq\\sqrt\{\\mathop\{\\mathrm\{KL\}\}\(P\_\{T\}\\\|P\_\{m\}\)/2\},\(19\)which proves the claim\. The same argument applies to a shared distribution over prompts by considering the joint prompt–response distribution\. This proposition applies standard information\-theoretic identities to the pruning setting; it is not a new KL inequality\. ∎

### C\.2Interpretation and Scope

The connection explains why teacher\-generated prefixes and teacher\-weighted disagreement are relevant to retaining teacher\-supported continuations\. It does not show that KL weighting directly minimizesD⁡\(m\)D\(m\), that smaller bounds order actual task accuracies, or that SFT and OPD preserve any ordering of masks\. The teacher may also generate incorrect answers\.

The population quantityD⁡\(m\)D\(m\)sums token divergences before averaging over responses\. The fixed\-group means and length\-normalized response averages in our diagnostics are different statistics\. Moreover, merging tail tokens into the residual bucket in Equation[12](https://arxiv.org/html/2610.11332#A1.E12)can only decrease KL by the data\-processing inequality\. The implemented coarsened KL is thus not an upper bound on full\-vocabulary damage and cannot be substituted into Proposition[1](https://arxiv.org/html/2610.11332#Thmproposition1)to certify sequence preservation\. Recovery benefits are established empirically through the paired experiments\.

## Appendix DAdditional Related\-Work Comparisons

We extend Section[6](https://arxiv.org/html/2610.11332#S6)by distinguishing the structural decisions, calibration signals, and recovery interventions in prior work\. These distinctions determine which methods are direct comparisons and which address complementary problems\.

#### Pruning granularity and selection criteria\.

SparseGPT removes individual weights using an approximate second\-order reconstruction procedure, while Wanda combines weight magnitude with input activations\([Frantar and Alistarh, 2023](https://arxiv.org/html/2610.11332#bib.bib7);[Sun et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib25)\)\. Their original weight\-level formulations differ from removing complete FFN channels; our Wanda\-SP adaptation is specified in Appendix[A\.2](https://arxiv.org/html/2610.11332#A1.SS2)\. Among structured methods, the LLM Surgeon uses curvature information and weight updates, OSSCAR formulates layer reconstruction as combinatorial optimization, and SlimGPT applies batched greedy pruning within an Optimal Brain Surgeon framework\([van der Ouderaa et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib33);[Meng et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib34);[Ling et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib38)\)\. Bonsai estimates importance through forward\-pass perturbations, avoiding backward passes\([Kolawole et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib35)\)\. These methods primarily differ in how they estimate or optimize the cost of deleting a structure\.ReCalchanges the token evidence supplied to a criterion\. Compatibility with the four evaluated adapters does not establish compatibility with every reconstruction or curvature\-based method\.

Layer and block methods offer a different compression axis\. ShortGPT ranks layer redundancy, LaCo collapses layers, BlockPruner distinguishes attention and feed\-forward blocks, and SLEB eliminates redundant transformer blocks\([Men et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib18);[Yang et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib17);[Zhong et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib28);[Song et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib37)\)\. SliceGPT rotates representations before removing rows and columns\([Ashkboos et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib5)\)\. ZipLM incorporates inference cost into structural decisions\([Kurtic et al\., 2023](https://arxiv.org/html/2610.11332#bib.bib36)\)\. Our experiments retain all layers and attention modules and compare FFN channels at identical widths\. Consequently, they do not establish the best compression axis or measured deployment speedups\.

#### Accounting for recovery during compression\.

Sheared LLaMA combines targeted structured pruning with continued pretraining; Minitron studies pruning and distillation across multiple architectural dimensions\([Xia et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib27);[Muralidharan et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib20);[Sreenivas et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib24)\)\. More directly, DarwinLM incorporates brief candidate training into evolutionary architecture search\([Tang et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib39)\)\. NIRVANA uses function\-space saliency to account for fine\-tuning dynamics and KL\-based calibration\-data selection\([Ai et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib3)\)\. These works already establish that compression should consider subsequent training\. Our contribution is the specific teacher–probe token signal and its integration into calibration for an unchanged SFT–OPD pipeline, rather than the general idea of recovery\-aware pruning\. Unlike candidate training or layer\-budget search,ReCalfixes the target widths and uses an unrecovered probe to redistribute calibration weight within responses\.

#### Preserving reasoning\.

Reasoning models can exploit long generated traces and test\-time computation\([DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib6);[Yang et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib29)\), making next\-token fit alone an incomplete evaluation of compression\. RESP uses self\-generated reasoning traces and decode\-only gradient scoring, SPRINT considers diverse best\-of\-NNreasoning, and recent evaluations examine pruning under test\-time scaling\([Wang et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib26);[Nguyen et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib21);[Monjur et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib19)\)\. RED uses activation\-aware initialization for reasoning\-model distillation\([He et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib40)\)\. MuCRASP extends reasoning\-aware pruning to multimodal models using reasoning and modality signals\([Dutta and Aditya, 2026](https://arxiv.org/html/2610.11332#bib.bib41)\)\. Our fixed\-prefix analysis instead tracks baseline\-defined damaged tokens through recovery\. It supports an empirical calibration strategy, without showing that large token divergence always identifies task\-critical reasoning steps\. Our mathematics and code gains, instruction\-following losses, and the 8B LLM\-Pruner exception bound that interpretation\.

#### Distillation objectives and rollout distributions\.

Classical distillation transfers teacher predictions, while sequence\-level distillation fits generated teacher responses\([Hinton et al\., 2015](https://arxiv.org/html/2610.11332#bib.bib10);[Kim and Rush, 2016](https://arxiv.org/html/2610.11332#bib.bib1)\)\. Learning on learner\-induced states addresses a general distribution\-shift problem in sequential prediction\([Ross et al\., 2011](https://arxiv.org/html/2610.11332#bib.bib47)\); language\-model OPD obtains teacher supervision on these states\([Agarwal et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib2);[Gu et al\., 2024](https://arxiv.org/html/2610.11332#bib.bib8)\)\. Speculative distillation interleaves student proposals and teacher replacements to balance on\-policy coverage with response quality\([Xu et al\., 2025](https://arxiv.org/html/2610.11332#bib.bib42)\)\. Self\-Distilled Reasoner obtains supervision from a solution\-conditioned version of the same model\([Zhao et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib43)\)\. These supervision designs differ from our frozen unpruned teacher\. Entropy\-aware OPD also employs forward KL, but uses teacher uncertainty to adapt the recovery objective\([Jin et al\., 2026](https://arxiv.org/html/2610.11332#bib.bib12)\)\.ReCaluses teacher–probe disagreement before recovery to weight structural calibration\. Thus, the divergence direction alone is not the novelty; its role in pre\-recovery structure selection is the relevant distinction\.