Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment

# A Cost-Normalized Study of Post-Compression Adjustment
Source: [https://arxiv.org/html/2609.06076](https://arxiv.org/html/2609.06076)
## Beyond Retraining\-Free MoE Compression: A Cost\-Normalized Study of Post\-Compression Adjustment

###### Abstract

Retraining\-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact\. We argue that this view is incomplete: compressed MoE checkpoints are better understood as compressed initializations that benefit from a tiny post\-compression adjustment stage\. Across two MoE LLM backbones, four pruning/merging methods, three expert\-retention ratios, and 28 benchmarks, we compare LM fine\-tuning and teacher\-based KD under matched small\-data budgets and measured GPU costs\. Using only 3,000 C4 examples and a single epoch of adjustment,Full FTrecovers 37\.3% of the original\-to\-compressed performance gap on average\. Moreover, LM fine\-tuning is more cost\-effective than standard token\-level KD, and full\-parameter adjustment gives the strongest cost–recovery trade\-off among the tested scopes\. These results suggest that retraining\-free compression should be paired with small post\-compression adjustment to recover a substantial portion of the performance lost during compression\.

## 1Introduction

Large language models exhibit strong performance, but as the number of parameters increases, their memory footprint and deployment cost grow rapidly[Kaplan et al\. \(2020\)](https://arxiv.org/html/2609.06076#bib.bib1)\. Mixture\-of\-Experts \(MoE\)[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.06076#bib.bib2);[Lepikhin et al\. \(2020\)](https://arxiv.org/html/2609.06076#bib.bib3)reduces computation cost by activating only a subset of experts for each token, but it still requires all expert parameters to be loaded into memory during inference, making deployment difficult in resource\-constrained environments\.

Retraining\-free MoE compression is an attractive approach for alleviating this memory bottleneck\. For ordinary researchers or developers, preparing large\-scale training data and sufficient GPU resources to deploy or compress a large\-scale MoE LLM is burdensome\. For this reason, many recent studies have emphasized retraining\-free or calibration\-free compression that reduces the memory footprint of MoE LLMs without full\-scale retraining\. In particular, because most MoE parameters are concentrated in experts, prior work has focused on expert\-side compression, such as Expert Pruning[Liu et al\. \(2026b\)](https://arxiv.org/html/2609.06076#bib.bib57)and Expert Merging[Li et al\. \(2024b\)](https://arxiv.org/html/2609.06076#bib.bib27)\.

However, retraining\-free does not necessarily mean cost\-free\. Many methods still use calibration samples, model inference, and GPU computation to calculate expert importance[Lu et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib4), expert similarity[Chen et al\. \(2025a\)](https://arxiv.org/html/2609.06076#bib.bib16), routing statistics[Li et al\. \(2024b\)](https://arxiv.org/html/2609.06076#bib.bib27), and so on\. This point leads us to ask again whether retraining\-free compression alone is sufficient\.If a small amount of data and GPU computation is already allowed, is it truly optimal to use them only for the compression process itself?

In this paper, we revisit the existing perspective on retraining\-free MoE compression\. We argue that a compressed checkpoint should be viewed not as the final artifact, but as a compressed initialization\. Expert Pruning and Expert Merging change the expert landscape of the original MoE, leaving residual errors such as routing shift, expert function distortion, and router–expert interaction error\. Therefore, if the goal is near\-original recovery, a small post\-compression adjustment after retraining\-free compression is not merely an additional step, but a practically important recovery stage\.

Here, post\-compression adjustment refers to a procedure that applies a small amount of general\-domain text and short gradient updates to a compressed checkpoint in order to reduce the residual performance gap\. This is clearly distinguished from full\-scale retraining, which requires a large\-scale corpus or a long training schedule\. Our goal is not to introduce another compressor, but to define and evaluate post\-compression adjustment as a missing design axis in MoE compression\. Rather than asking only which retraining\-free compressor produces the best immediate checkpoint, we ask which compression\-and\-adjustment pipeline yields the best recovery under the same small data budget and measured GPU cost\.

To verify the necessity of post\-compression adjustment, we conduct large\-scale comparative experiments across Expert Pruning and Expert Merging\. Specifically, we construct a variety of compressed checkpoints using two MoE LLM backbones, four primary compression methods from Expert Pruning and Expert Merging, and three expert\-parameter retention ratios, and apply the same small adjustment data budget and hyperparameters to all checkpoints\. We then evaluate both the degree of performance recovery and GPU cost on 28 downstream benchmarks, thereby analyzing a general post\-compression recovery stage that is not dependent on a specific compressor\.

In addition, to closely analyze the design choices of post\-compression adjustment, we compare two objectives, causal language modeling \(LM\) loss[Radford et al\. \(2018\)](https://arxiv.org/html/2609.06076#bib.bib59)and teacher\-based knowledge distillation \(KD\)[Hinton et al\. \(2015\)](https://arxiv.org/html/2609.06076#bib.bib60), and evaluate several trainable parameter scopes: router\-only, router\+top\-k experts, router\+all\-experts, and full\-parameter adjustment\. In particular, for top\-k experts, we expand k to 8, 16, and 50, and analyze the trade\-off among the number of selected experts, recovery gain, and GPU cost\. Through this, we test whether carefully selecting and adjusting only a small subset of parameters is truly cost\-efficient\.

Our experiments present three practical lessons\.

\(1\) With a single post\-compression adjustment epoch over 3,000 C4 examples, compressed MoE LLMs recover 37\.3% of the original\-to\-compressed performance gap on average\. This implies that retraining\-free compression should be viewed not as producing a final artifact, but as producing a post\-compression\-adjustment\-friendly initialization under a tiny adjustment budget\.

\(2\) Under a small general\-domain adjustment budget, causal LM fine\-tuning is more cost\-effective than standard token\-level teacher KD in our setting\. After compression, when the student’s expert landscape and feasible function class have changed, directly imitating the teacher distribution may not always be the most efficient local repair direction\. In contrast, LM fine\-tuning directly improves the observed next\-token likelihood without teacher forward passes, providing a simpler and more cost\-effective repair signal\.

\(3\) Full\-parameter causal LM adjustment is simple but highly competitive\. Although it updates more parameters and may require more optimizer memory, its end\-to\-end GPU time can be competitive because it avoids expert\-selection overhead and yields substantially larger recovery per run\. Under our measured GPU\-cost protocol, it emerges as the strongest default strategy among the tested scopes\. Router\-only adjustment can alleviate some routing mismatch, but it is not sufficient to recover expert function distortion and router–expert interaction error\. Router\+top\-k experts adjustment limits the number of trainable parameters, but additionally incurs selection cost for identifying frequently activated experts\.

As a result, this paper does not reject the recent trend of retraining\-free MoE compression, but instead reinterprets it as a more practical two\-stage pipeline\. Retraining\-free is a strong start, but not necessarily the end\. The central lesson of this paper is that, for near\-original MoE recovery, not only the compressor itself but also the post\-compression adjustment stage should be designed together\.

## 2Related Works

Expert Pruningremoves a subset of redundant experts\. Strategies include minimizing reconstruction loss[Lu et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib4), differentiable selection[Bai et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib5), and leveraging router magnitudes[Lasby et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib6)\. Other approaches utilize output discrepancy bounds[Liu et al\. \(2026a\)](https://arxiv.org/html/2609.06076#bib.bib15), token variation[Dong et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib7), trajectory\-based importance[Yang et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib10), or coarser layer\-level pruning[He et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib9)\.Expert Mergingis grounded in model merging hypotheses[Wortsman et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib19);[Matena and Raffel \(2022\)](https://arxiv.org/html/2609.06076#bib.bib20)\. This approach synthesizes experts via output similarity clustering[Chen et al\. \(2025a\)](https://arxiv.org/html/2609.06076#bib.bib16), selective dual\-masks[Zhao et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib17), or compression matrices[Miao et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib18)\.

Some MoE compression studies include a recovery stage, such as expert\-wise KD after pruning[Xie et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib52), condensed\-layer fine\-tuning[Cao et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib53), LoRA\-based recovery after hybrid compression[Yang et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib25), or fine\-tuning after subspace expert merging[Li et al\. \(2025a\)](https://arxiv.org/html/2609.06076#bib.bib56); larger systems further rely on distillation or continual training at much larger data scales[Li et al\. \(2025c\)](https://arxiv.org/html/2609.06076#bib.bib54);[Tang et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib55)\. However, these works do not systematically ask whether retraining\-free compression itself is sufficient across already\-deployable expert\-compressed MoE checkpoints, nor which post\-compression objective, trainable scope, and GPU\-cost trade\-off is optimal under a tiny calibration budget\. Our work fills this gap by treating post\-compression adjustment as a first\-class component of MoE compression rather than a method\-specific add\-on\. See Appendix[H](https://arxiv.org/html/2609.06076#A8)for extended related works\.

## 3Causes of Performance Degradation

Retraining\-free MoE compression changes the computation path of the original model\. This change can come from removing experts or merging multiple experts into a smaller set\. As a result, the compressed model can deviate from the original model through two coupled sources: the router may assign different mixture weights, and the selected expert functions or expert paths may no longer match the original ones\. In this section, we use Expert Pruning as a representative case and keep only the core equations in the main text\. Full derivations for Expert Pruning and Expert Merging are provided in Appendices[C](https://arxiv.org/html/2609.06076#A3)and[D](https://arxiv.org/html/2609.06076#A4), respectively\.

We adopt the MoE notation and pruning scenario analysis of[Hyeon and Do \(2026\)](https://arxiv.org/html/2609.06076#bib.bib61)\. To isolate the local compression source, the derivation below evaluates the original and compressed MoE layer maps at the same reference hidden\-state inputxx\. Differences caused by the realized end\-to\-end state shift are handled separately in the propagation analysis below\.

Let𝒮\\mathcal\{S\}be the top\-kkexpert set selected by the original MoE layer, let𝒫⊆\{0,…,N−1\}\\mathcal\{P\}\\subseteq\\\{0,\\ldots,N\-1\\\}be the set of experts retained after pruning, and let𝒮′⊆𝒫\\mathcal\{S\}^\{\\prime\}\\subseteq\\mathcal\{P\}be the top\-kkset selected by the pruned layer at the same reference inputxx\. For any active set𝒜\\mathcal\{A\}, letg~io,𝒜​\(x\)\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{A\}\}\(x\)andg~ip,𝒜​\(x\)\\widetilde\{g\}\_\{i\}^\{p,\\mathcal\{A\}\}\(x\)denote the original and pruned router weights renormalized over𝒜\\mathcal\{A\}, respectively\. For readability, we omitxxbelow and also writeEi=Ei​\(x\)E\_\{i\}=E\_\{i\}\(x\)\.

Building on the prior three\-part pruning analysis, we regroup the residual into shared\-path router reweighting and active\-path replacement\. The overlap is defined between the expert sets that are actually selected:

𝒯\\displaystyle\\mathcal\{T\}=𝒮∩𝒮′,\\displaystyle=\\mathcal\{S\}\\cap\\mathcal\{S\}^\{\\prime\},𝒟\\displaystyle\\mathcal\{D\}=𝒮∖𝒮′,\\displaystyle=\\mathcal\{S\}\\setminus\\mathcal\{S\}^\{\\prime\},ℛ\\displaystyle\\mathcal\{R\}=𝒮′∖𝒮\.\\displaystyle=\\mathcal\{S\}^\{\\prime\}\\setminus\\mathcal\{S\}\.Thus,𝒮=𝒯∪˙𝒟\\mathcal\{S\}=\\mathcal\{T\}\\mathbin\{\\dot\{\\cup\}\}\\mathcal\{D\}and𝒮′=𝒯∪˙ℛ\\mathcal\{S\}^\{\\prime\}=\\mathcal\{T\}\\mathbin\{\\dot\{\\cup\}\}\\mathcal\{R\}\.

##### Best scenario\.

The most favorable case is𝒮′=𝒮\\mathcal\{S\}^\{\\prime\}=\\mathcal\{S\}, which already implies𝒮⊆𝒫\\mathcal\{S\}\\subseteq\\mathcal\{P\}\. Then

‖yorig−yprunedbest‖=‖∑i∈𝒮\(g~io,𝒮−g~ip,𝒮\)​Ei‖\.\\left\\lVert y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{pruned\}\}^\{\\mathrm\{best\}\}\\right\\rVert=\\left\\lVert\\sum\_\{i\\in\\mathcal\{S\}\}\\left\(\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\-\\widetilde\{g\}\_\{i\}^\{p,\\mathcal\{S\}\}\\right\)E\_\{i\}\\right\\rVert\.At the common reference input, the active expert functions are identical, so the local residual can contain only shared\-path router reweighting\. If pure pruning leaves the router and gate implementation unchanged, theng~ip,S​\(x\)=g~io,S​\(x\)\\tilde\{g\}\_\{i\}^\{p,S\}\(x\)=\\tilde\{g\}\_\{i\}^\{o,S\}\(x\)and this same\-input best\-case residual is exactly zero\. Realized end\-to\-end deviations may nevertheless persist through upstream hidden\-state shifts\. If the gate implementation has changed or the router is subsequently adjusted, router alignment is the only local correction needed whileS′=SS^\{\\prime\}=S\.

##### Most common scenario\.

In the partial\-overlap case,0<\|𝒯\|<k0<\|\\mathcal\{T\}\|<k, and

‖yorig−yprunedcommon‖=‖eroutep\+epathp‖,\\left\\lVert y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{pruned\}\}^\{\\mathrm\{common\}\}\\right\\rVert=\\left\\lVert e\_\{\\mathrm\{route\}\}^\{p\}\+e\_\{\\mathrm\{path\}\}^\{p\}\\right\\rVert,\(1\)where

eroutep\\displaystyle e\_\{\\mathrm\{route\}\}^\{p\}=∑i∈𝒯\(g~io,𝒮−g~ip,𝒮′\)​Ei,\\displaystyle=\\sum\_\{i\\in\\mathcal\{T\}\}\\left\(\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\-\\widetilde\{g\}\_\{i\}^\{p,\\mathcal\{S\}^\{\\prime\}\}\\right\)E\_\{i\},\(2\)epathp\\displaystyle e\_\{\\mathrm\{path\}\}^\{p\}=∑i∈𝒟g~io,𝒮​Ei−∑i∈ℛg~ip,𝒮′​Ei\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{D\}\}\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}E\_\{i\}\-\\sum\_\{i\\in\\mathcal\{R\}\}\\widetilde\{g\}\_\{i\}^\{p,\\mathcal\{S\}^\{\\prime\}\}E\_\{i\}\.\(3\)Within a fixed active\-set region,eroutepe\_\{\\mathrm\{route\}\}^\{p\}is the component that router\-only adjustment directly targets by changing mixture weights on shared experts\. Router updates may also move the top\-kkboundary and thereby change𝒯\\mathcal\{T\},𝒟\\mathcal\{D\}, andℛ\\mathcal\{R\}\. However, with expert parameters frozen, router\-only adjustment cannot change the expert functions themselves and therefore cannot in general reconstruct a missing expert contribution unless the available replacements already provide an adequate functional approximation\. Appendix[C](https://arxiv.org/html/2609.06076#A3)gives the unified derivation\. The cluster\-space counterpart for Expert Merging is given in Appendix[D](https://arxiv.org/html/2609.06076#A4)\.

##### Scenario frequency\.

The decomposition suggests that router\-only adjustment is most favorable when the active path is preserved and only mixture weights require correction\. Using 100 questions sampled from ELI5\([Fan et al\., 2019](https://arxiv.org/html/2609.06076#bib.bib28)\), we categorize token–layer observations into Best, Most Common, and Worst scenarios using the active sets from the original and compressed end\-to\-end forward passes\. These realized frequencies include upstream hidden\-state perturbations and therefore diagnose path preservation; they do not by themselves isolate the magnitude of each local residual term\. For pruning, overlap is measured between the original and pruned active sets\. For merging, both paths are compared in the physical cluster space defined by the mappingϕ\\phiin Appendix[D](https://arxiv.org/html/2609.06076#A4)\.

Table 1:Scenario\-frequency summary by backbone and expert retention ratio\. Results are based on 100 samples from the ELI5 dataset\. Best denotes cases where the original selected expert set is preserved, Most denotes partial overlap between the original and compressed selected expert sets, and Worst denotes disjoint selected expert sets\. For each model–ratio pair, results are averaged across the corresponding compression methods\.Table[1](https://arxiv.org/html/2609.06076#S3.T1)shows that the best scenario is relatively rare, while partial overlap dominates across both backbones and all expert retention ratios\. This frequency analysis does not determine whether routing error or expert/path error is larger in norm\. Rather, it shows that the case in which router\-only correction can account for the entire local residual is not the typical realized path\. In the dominant partial\-overlap regime, botheroutepe\_\{\\mathrm\{route\}\}^\{p\}andepathpe\_\{\\mathrm\{path\}\}^\{p\}can contribute, motivating our comparison of router\-only, expert\-inclusive, and full\-parameter adjustment\.

##### Noise propagation caused by compression\.

LetFℓF\_\{\\ell\}andF^ℓ\\widehat\{F\}\_\{\\ell\}denote the original and compressed layer maps, respectively\. Lethℓh\_\{\\ell\}andh^ℓ\\widehat\{h\}\_\{\\ell\}be their hidden states before layerℓ\\ell, and defineδℓ=h^ℓ−hℓ\\delta\_\{\\ell\}=\\widehat\{h\}\_\{\\ell\}\-h\_\{\\ell\}\. The same\-input local compression source is

ϵℓ​\(hℓ\):=F^ℓ​\(hℓ\)−Fℓ​\(hℓ\)\.\\epsilon\_\{\\ell\}\(h\_\{\\ell\}\):=\\widehat\{F\}\_\{\\ell\}\(h\_\{\\ell\}\)\-F\_\{\\ell\}\(h\_\{\\ell\}\)\.For the pruning contribution in Eqs\. \([2](https://arxiv.org/html/2609.06076#S3.E2)\)– \([3](https://arxiv.org/html/2609.06076#S3.E3)\), which uses the original\-minus\-compressed convention,

ϵℓp=−\(eroute,ℓp\+epath,ℓp\)\.\\epsilon\_\{\\ell\}^\{p\}=\-\\left\(e\_\{\\mathrm\{route\},\\ell\}^\{p\}\+e\_\{\\mathrm\{path\},\\ell\}^\{p\}\\right\)\.Within a fixed\-routing neighborhood, or away from top\-kkdecision boundaries, first\-order linearization gives

δℓ\+1≈J^ℓ​δℓ\+ϵℓ​\(hℓ\),J^ℓ:=∂F^ℓ​\(h\)∂h\|h=hℓ\.\\delta\_\{\\ell\+1\}\\approx\\widehat\{J\}\_\{\\ell\}\\delta\_\{\\ell\}\+\\epsilon\_\{\\ell\}\(h\_\{\\ell\}\),\\qquad\\widehat\{J\}\_\{\\ell\}:=\\left\.\\frac\{\\partial\\widehat\{F\}\_\{\\ell\}\(h\)\}\{\\partial h\}\\right\|\_\{h=h\_\{\\ell\}\}\.Assuming both models receive the same input,δ0=0\\delta\_\{0\}=0, and unrolling the recurrence yields

δL≈∑ℓ=0L−1Pℓ→Lϵℓ,Pℓ→L:=J^L−1J^L−2⋯J^ℓ\+1,\\delta\_\{L\}\\approx\\sum\_\{\\ell=0\}^\{L\-1\}P\_\{\\ell\\rightarrow L\}\\epsilon\_\{\\ell\}\\quad,P\_\{\\ell\\rightarrow L\}:=\\widehat\{J\}\_\{L\-1\}\\widehat\{J\}\_\{L\-2\}\\cdots\\widehat\{J\}\_\{\\ell\+1\},withPL−1→L=IP\_\{L\-1\\rightarrow L\}=Ifor the empty product\. Thus, even when only MoE experts are compressed, the induced local source enters the residual stream and is transformed by subsequent attention, normalization, dense, and MoE modules\.

This propagation view motivates a local sensitivity comparison of trainable parameter scopes\. LetθR\\theta\_\{R\},θE\\theta\_\{E\}, andθD\\theta\_\{D\}denote router, expert, and remaining dense/shared parameters of the compressed model, and letθ0\\theta^\{0\}be the unadjusted compressed checkpoint\. Define the final\-state discrepancydL​\(θ\)=h^L​\(θ\)−hLd\_\{L\}\(\\theta\)=\\widehat\{h\}\_\{L\}\(\\theta\)\-h\_\{L\}and, within a fixed\-routing local region,

Gb:=∂dL∂θb\|θ=θ0,b∈\{R,E,D\}\.G\_\{b\}:=\\left\.\\frac\{\\partial d\_\{L\}\}\{\\partial\\theta\_\{b\}\}\\right\|\_\{\\theta=\\theta^\{0\}\},\\qquad b\\in\\\{R,E,D\\\}\.A first\-order parameter perturbation then satisfies

Δ​dL≈GR​Δ​θR\+GE​Δ​θE\+GD​Δ​θD\.\\Delta d\_\{L\}\\approx G\_\{R\}\\Delta\\theta\_\{R\}\+G\_\{E\}\\Delta\\theta\_\{E\}\+G\_\{D\}\\Delta\\theta\_\{D\}\.
Let𝒦\\mathcal\{K\}denote the selected trainable experts and letGE,𝒦G\_\{E,\\mathcal\{K\}\}be the sub\-Jacobian formed by the corresponding columns ofGEG\_\{E\}\. Then the locally reachable spaces satisfy

col⁡\(GR\)⊆col⁡\(\[GR​GE,𝒦\]\)⊆col⁡\(\[GR​GE\]\)\\displaystyle\\operatorname\{col\}\(G\_\{R\}\)\\subseteq\\operatorname\{col\}\(\[G\_\{R\}\\;G\_\{E,\\mathcal\{K\}\}\]\)\\subseteq\\operatorname\{col\}\(\[G\_\{R\}\\;G\_\{E\}\]\)⊆col⁡\(\[GR​GE​GD\]\)\.\\displaystyle\\subseteq\\operatorname\{col\}\(\[G\_\{R\}\\;G\_\{E\}\\;G\_\{D\}\]\)\.
Thus, a router\+top\-kk\-expert scope retains all router directions and adds only the expert directions associated with the selected trainable experts\. These inclusions describe first\-order reachable directions, not optimization success or the relative causal importance of the parameter groups\.

## 4Experiment Setups

Our experiments are designed to answer three questions:\(1\) Can a small amount of post\-compression adjustment substantially recover the performance of compressed MoE LLMs? \(2\) When performance recovery and GPU cost are considered, which adjustment strategy is most efficient? \(3\) Which training objective is more suitable for post\-compression adjustment?To answer these questions, we evaluate multiple MoE compression paradigms, compression ratios, adjustment objectives, and trainable parameter scopes under a matched adjustment budget\.

### 4\.1Backbones and Compression Baselines

We use two MoE LLM backbones:Qwen3\-30B\-A3B\-Instruct\-2507[Qwen Team \(2025\)](https://arxiv.org/html/2609.06076#bib.bib29)andgemma\-4\-26B\-A4B\-it[Google DeepMind \(2026\)](https://arxiv.org/html/2609.06076#bib.bib58)\. For each backbone, we construct compressed checkpoints using representative methods from two primary parameter\-level MoE compression paradigms: Expert Pruning and Expert Merging\.

ForQwen3\-30B\-A3B\-Instruct\-2507, we use the following methods\. For Expert Pruning, we adopt REAP[Lasby et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib6), which estimates expert importance using both router gate values and expert output magnitudes\. For Expert Merging, we use HC\-SMoE[Chen et al\. \(2025a\)](https://arxiv.org/html/2609.06076#bib.bib16), which performs hierarchical clustering based on expert\-output similarity computed on calibration data\.

Forgemma\-4\-26B\-A4B\-it, we use a different set of representative methods\. For Expert Pruning, we use AIMER[Liu et al\. \(2026b\)](https://arxiv.org/html/2609.06076#bib.bib57), a calibration\-free pruning method based on RMS\-normalized weight magnitudes\. For Expert Merging, we use M\-SMoE[Li et al\. \(2024b\)](https://arxiv.org/html/2609.06076#bib.bib27), which groups experts using routing\-policy statistics, aligns their neurons by permutation, and then merges each group through activation\-frequency\-weighted averaging\.

For each compression method, we evaluate three expert\-parameter retention ratios: 50%, 62\.5%, and 75%\. In the pruning setting, for example, a 50% retention ratio means that half of the expert parameters are removed; for a layer with 128 experts, this corresponds to retaining 64 experts\. All ratios are applied only to expert parameters, while non\-expert parameters are kept unchanged\.

### 4\.2Benchmarks and Evaluation Protocol

We evaluate each model on 28 downstream benchmarks spanning five categories: general reasoning/QA, mathematical reasoning, coding, chain\-of\-thought \(CoT\) reasoning, and multiple\-choice question answering \(MCQA\)\. All benchmarks are detailed in Appendix[A\.4](https://arxiv.org/html/2609.06076#A1.SS4)\.

All benchmark evaluations are conducted usinglm\-evaluation\-harness[Gao et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib49)andvLLM[Kwon et al\. \(2023\)](https://arxiv.org/html/2609.06076#bib.bib50)\. We use greedy decoding with temperature 0 and fix the random seed to 42 for all experiments\. We report each task using the accuracy or exact\-match metric provided by the evaluation harness\. We conduct every post\-compression adjustment run on exactly two NVIDIA H200 GPUs \(140 GB each\), regardless of the backbone, compression method, retention ratio, objective, or trainable scope\. For post\-compression adjustment, we use the same C4[Dodge et al\. \(2021\)](https://arxiv.org/html/2609.06076#bib.bib51)text subset, data budget, and hyperparameters across all models, compression methods, ratios, objectives, and trainable scopes\. To specifically observe if performance recovery is achievable at a minimal scale, we restrict our adjustment to a single epoch over just 3,000 sampled examples with a batch size of 2\. More detailed hyperparameters and runtime settings are provided in Appendix[A](https://arxiv.org/html/2609.06076#A1)\.

### 4\.3Adjustment Strategies and Metrics

#### 4\.3\.1Objectives

We study post\-compression adjustment along two axes: the training objective and the trainable parameter scope\. Letθc\\theta\_\{c\}denote the parameters of a compressed model and let𝒰\\mathcal\{U\}denote the subset of parameters updated during adjustment\. All parameters outside𝒰\\mathcal\{U\}are frozen\. We compare the following objectives\.

##### Causal language modeling fine\-tuning\.

The first objective is standard causal LM loss[Radford et al\. \(2018\)](https://arxiv.org/html/2609.06076#bib.bib59)on a small general\-domain text subset\. For a sequencex=\(x1,…,xL\)x=\(x\_\{1\},\\ldots,x\_\{L\}\)and a loss maskmtm\_\{t\}, the LM objective is

ℒLM=−1Nx∑t=1L−1mt\+1logpθc\(xt\+1∣x≤t\),\\mathcal\{L\}\_\{\\mathrm\{LM\}\}=\-\\frac\{1\}\{N\_\{x\}\}\\sum\_\{t=1\}^\{L\-1\}m\_\{t\+1\}\\log p\_\{\\theta\_\{c\}\}\(x\_\{t\+1\}\\mid x\_\{\\leq t\}\),whereNx=∑t=1L−1mt\+1N\_\{x\}=\\sum\_\{t=1\}^\{L\-1\}m\_\{t\+1\}is the number of valid target tokens\. This objective directly improves the likelihood of observed text and does not require a teacher forward pass\.

##### Teacher\-based knowledge distillation\.

The second objective is token\-level knowledge distillation \(KD\)[Hinton et al\. \(2015\)](https://arxiv.org/html/2609.06076#bib.bib60)from the original uncompressed model\. LetzT\(t\)z\_\{T\}^\{\(t\)\}andzS\(t\)z\_\{S\}^\{\(t\)\}denote the teacher and student vocabulary logits at positiontt, and letτ\\taube the distillation temperature\. We define

pT\(t\)=softmax⁡\(zT\(t\)/τ\),pS\(t\)=softmax⁡\(zS\(t\)/τ\)\.p\_\{T\}^\{\(t\)\}=\\mathrm\{softmax\}\(z\_\{T\}^\{\(t\)\}/\\tau\),\\quad p\_\{S\}^\{\(t\)\}=\\mathrm\{softmax\}\(z\_\{S\}^\{\(t\)\}/\\tau\)\.The KD objective is

ℒKD=τ2Nx∑t=1L−1mt\+1DKL\(pT\(t\)∥pS\(t\)\)\.\\mathcal\{L\}\_\{\\mathrm\{KD\}\}=\\frac\{\\tau^\{2\}\}\{N\_\{x\}\}\\sum\_\{t=1\}^\{L\-1\}m\_\{t\+1\}D\_\{\\mathrm\{KL\}\}\\left\(p\_\{T\}^\{\(t\)\}\\\|p\_\{S\}^\{\(t\)\}\\right\)\.In all experiments, the teacher is the original model before compression and the student is the compressed model after adjustment\.

##### Direct router logit matching\.

We also include Direct Router Logit Matching \(DRLM\) as a diagnostic baseline when router coordinates can be aligned through the compression mapping\. We use DRLM only as a diagnostic router\-alignment baseline and provide its formulation in Appendix[A\.5](https://arxiv.org/html/2609.06076#A1.SS5)\.

#### 4\.3\.2Trainable parameter scopes\.

For LM and KD, we evaluate multiple trainable parameter scopes:

- •Router\-only:update only router/gating parameters while freezing experts and all shared parameters\.
- •Router\+top\-kkexperts:update router parameters and thekkmost frequently activated experts per MoE layer\. We select top\-kkexperts using the same adjustment data before training\. We evaluatek∈\{8,16,50\}k\\in\\\{8,16,50\\\}\.
- •Router\+all\-experts:update router parameters and all expert parameters, while freezing non\-expert dense/shared parameters\.
- •Full\-parameter:update all parameters in the compressed checkpoint\.

The router\+top\-kksetting is intended to test whether carefully selecting a small subset of frequently used experts is more cost\-efficient than full adjustment\. However, this strategy also introduces an expert\-selection stage, which requires additional calibration inference before the actual adjustment training\.

#### 4\.3\.3Adjustment strategies\.

Combining LM/KD objectives with the trainable scopes yields the main set of post\-compression adjustment strategies evaluated in this work\. DRLM is reported separately as a diagnostic router\-matching baseline\. For example,Router\-only FTdenotes causal LM fine\-tuning with only router parameters trainable, whileRouter\+top\-8 Expert KDdenotes KD where the router and the top\-8 selected experts per layer are trainable\.Full FTandFull KDupdate all parameters of the compressed checkpoint using LM loss and KD loss, respectively\. We also compare against the unadjusted compressed baseline to measure recovery gain\.

#### 4\.3\.4Performance and GPU\-cost metrics\.

For performance, we report the macro\-average score over the 28 benchmarks and define recovery gain as the difference, in percentage points \(pp\), between the adjusted model’s score and that of the unadjusted compressed baseline; the recovered fraction of the gap divides this gain by the Original\-to\-Compressed gap\. For GPU cost, we report three complementary metrics: utilization\-weighted GPU time, time\-integrated GPU\-memory occupancy, and GPU energy\. LetThrT\_\{\\mathrm\{hr\}\}be the total wall\-clock duration, in hours, of the measured loop segments,P¯g\\bar\{P\}\_\{g\}be the mean power draw of GPUggover its valid retained readings,GGbe the number of assigned GPUs,u¯\\bar\{u\}be the mean GPU\-utilization percentage over valid retained readings, andM¯obs\\bar\{M\}\_\{\\mathrm\{obs\}\}be the mean observed per\-poll sum ofmemory\.usedacross assigned GPUs with valid readings\. We compute

Heff=G​Thr​u¯100,H\_\{\\mathrm\{eff\}\}=G\\,T\_\{\\mathrm\{hr\}\}\\frac\{\\bar\{u\}\}\{100\},measured in effective GPU\-hours,

Hmem=Thr​M¯obs,H\_\{\\mathrm\{mem\}\}=T\_\{\\mathrm\{hr\}\}\\bar\{M\}\_\{\\mathrm\{obs\}\},measured in GPU\-memory GiB\-hours, and

EGPU=Thr1000​∑g=1GP¯g,E\_\{\\mathrm\{GPU\}\}=\\frac\{T\_\{\\mathrm\{hr\}\}\}\{1000\}\\sum\_\{g=1\}^\{G\}\\bar\{P\}\_\{g\},measured in kWh\. All three metrics use the same measurement boundaries\. For KD and DRLM, the monitored loop includes both teacher and student forward passes\. When a strategy requires a separate expert\-selection phase, all three metrics include both the expert\-selection and training\-loop segments\.

## 5Experiment Results

##### Overall recovery\.

Figure[1](https://arxiv.org/html/2609.06076#S5.F1)summarizes the mean recovery gain of each post\-compression adjustment strategy, averaged over all backbone, compressor, retention\-ratio, and benchmark combinations\. Appendix[E](https://arxiv.org/html/2609.06076#A5)further reports the same comparison broken down by compression method\. The strongest strategy isFull FT, which achieves a mean recovery gain of\+3\.49\+3\.49pp over the unadjusted compressed baseline\. In absolute mean score, the original model reaches 48\.37%, the unadjusted compressed baseline drops to 39\.01%, andFull FTimproves it to 42\.50%, recovering 37\.3% of the original\-to\-compressed gap\. It is followed byRouter\+All\-Expert FT\(\+2\.31\+2\.31pp\) andFull KD\(\+2\.19\+2\.19pp\)\. A clear pattern is that, under the same trainable parameter scope, causal LM fine\-tuning consistently outperforms teacher\-based KD\. For example,Router\+Top\-50 Expert FToutperformsRouter\+All\-Expert KD, andRouter\+Top\-8 Expert FToutperformsRouter\+Top\-16 Expert KD\. This suggests that, under a tiny adjustment budget, the choice of objective can be more important than simply increasing the number of trainable experts\.

Importantly, this result should not be read as a claim that distillation is generally ineffective for post\-compression recovery\. Our comparison focuses on standard token\-level KD under a tiny general\-domain adjustment budget, and counts the additional teacher\-forward cost required by KD\. Under this protocol, causal LM fine\-tuning provides a more cost\-effective repair signal\.

Router\-only adjustment also improves performance, but only marginally\.Router\-only FTandRouter\-only KDyield small gains of\+0\.23\+0\.23pp and\+0\.17\+0\.17pp, respectively, while direct router logit matching yields only\+0\.06\+0\.06pp\. This supports the analysis in Section[3](https://arxiv.org/html/2609.06076#S3): router reweighting is a real source of error, but the best scenario where router\-only correction can fully explain the residual is relatively rare\. In the more common partial\-overlap scenario, expert\-path and expert\-function errors remain, and these cannot be sufficiently reduced by modifying the router alone\.

![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/main_01_total_restoration_gain_bar.png)Figure 1:Overall recovery gain of post\-compression adjustment strategies\. Bars show the average\-score recovery gain, measured in percentage points \(pp\), over the unadjusted compressed baseline\. Causal LM fine\-tuning consistently outperforms KD under matched trainable scopes, andFull FTachieves the largest recovery\.Figure 2:GPU cost–performance trade\-off of post\-compression adjustment strategies\. The y\-axis shows mean average\-score recovery gain, while the x\-axis shows effective GPU\-hours \(top\), GPU\-memory occupancy in GiB\-hours \(middle\), and GPU energy in kWh \(bottom\)\. Each point is an unweighted arithmetic mean over 12 method–ratio settings \(four compression methods×\\timesthree retention ratios\)\. Both expert\-selection cost and training\-loop cost are included when applicable\. Upper\-left is better\.
##### GPU cost–performance trade\-off\.

Figure[2](https://arxiv.org/html/2609.06076#S5.F2)compares recovery gain against three GPU\-cost metrics: effective GPU\-hours, GPU\-memory occupancy in GiB\-hours, and GPU energy in kWh\. The desirable region is the upper\-left corner, where a strategy achieves high recovery with low cost\.Full FTachieves the highest recovery among all strategies\. Although it is not the cheapest option in absolute GPU cost, it provides the best recovery per end\-to\-end adjustment run among the high\-recovery strategies\. This is because expert\-subset strategies incur additional selection inference and still recover less than full adjustment\. In particular, the gap betweenRouter\+All\-Expert FTandFull FTshows that updating only the compressed MoE modules is not necessarily the most efficient use of the adjustment budget\.

The router\+top\-kkstrategies are also less cost\-efficient than their trainable\-parameter count might suggest\. These strategies update only a subset of frequently activated experts, but they require a separate expert\-selection stage before training\. This additional calibration inference increases GPU cost, while the recovery gain remains lower than that ofRouter\+All\-Expert FTorFull FT\. Thus, unless the selected experts can be identified almost perfectly at negligible cost, restricting adjustment to a small top\-kkexpert subset is not clearly more efficient than adjusting all experts or the full model\.

Router\-only strategies occupy the low\-cost region, but their recovery gains are also small\. They may be useful when GPU resources are extremely limited, but they should not be expected to approach the recovery of full\-parameter adjustment\. Overall, when performance recovery and GPU cost are considered,Full FTemerges as the strongest default strategy among the tested methods\.

Figure 3:Benchmark\-family recovery ofFull FT\. Bars show the post\-adjustment performance, and dashed lines denote the original model and the unadjusted compressed baseline\.Full FTyields the largest recovery on chain\-of\-thought and MCQA benchmarks, while math and coding show only marginal changes\.
##### Which benchmark families recover?

Figure[3](https://arxiv.org/html/2609.06076#S5.F3)shows the benchmark\-family recovery obtained byFull FT\. The largest gains appear in chain\-of\-thought reasoning and multiple\-choice QA, followed by general reasoning and QA\. In contrast, math and coding benchmarks show only marginal changes\. This pattern suggests that a small general\-domain LM adjustment is especially effective at restoring the model’s general generation behavior and long\-form reasoning stability, while highly specialized domains such as mathematics and coding may require more task\-specific knowledge or longer adaptation\.

We also qualitatively observe generation\-level failures after compression, including token repetition, premature termination, and loss of coherent continuation\. Appendix[G](https://arxiv.org/html/2609.06076#A7)provides representative examples whereFull FTrestores fluent continuation without task\-specific supervision\. This indicates that compression can damage not only factual knowledge but also the model’s ability to express and compose its remaining knowledge fluently\. Causal LM fine\-tuning directly optimizes observed next\-token likelihood on natural text, and therefore provides a simple signal for repairing such generation\-level degradation\. This helps explain whyFull FTis particularly effective on general, CoT, and MCQA\-style benchmarks, where fluent continuation and stable reasoning traces are important\.

![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/main_04_scope_progression_ft_kd_combined.png)Figure 4:Recovery gain as the trainable parameter scope expands\. For both LM fine\-tuning and KD, recovery increases from router\-only to router\+top\-kkexperts, router\+all\-experts, and full\-parameter adjustment\. The large gain from router\+all\-experts to full adjustment suggests that compression\-induced noise propagates beyond MoE modules into dense/shared components\.
##### Effect of expanding the trainable scope\.

Figure[4](https://arxiv.org/html/2609.06076#S5.F4)analyzes how recovery changes as the trainable parameter scope expands\. For both FT and KD, recovery increases as we move from router\-only adjustment to router\+top\-kkexperts, router\+all\-experts, and full\-parameter adjustment\. This monotonic trend indicates that compression\-induced degradation is not confined to a single component\. Router\-only adjustment can reduce part of the routing error, and expert\-inclusive adjustment can further reduce expert\-path or expert\-function errors\.

The most notable pattern is the large jump fromRouter\+All\-Expertadjustment toFulladjustment\. Even though the compression is applied to expert parameters, allowing non\-expert dense/shared parameters to update provides a substantial additional gain\. This suggests that local perturbations introduced in MoE layers propagate through the residual stream and affect subsequent attention, normalization, and dense transformations\. Consequently, correcting only the compressed MoE modules may leave downstream propagated noise unresolved\. Full\-parameter fine\-tuning can compensate for both the local MoE error and its downstream propagation, which explains why it achieves the strongest recovery in Figure[4](https://arxiv.org/html/2609.06076#S5.F4)\.

## 6Robustness Check

Figure 5:Recovery with 1,024 C4 calibration examples\. Bars show the mean score change from the unadjusted compressed model across 28 benchmarks and 12 backbone–compressor–retention settings\. Full FT remains the strongest strategy, followed by Router All\-Expert FT and Full KD, showing that reducing the main study’s 3,000\-example budget preserves the central strategy\-level conclusion\.Reducing the C4 calibration budget from 3,000 to 1,024 examples preserves the main strategy\-level pattern \(Figure[5](https://arxiv.org/html/2609.06076#S6.F5)\)\. Across four backbone–compressor pairs and three retention ratios, Full FT gives the largest mean recovery \(\+2\.78\+2\.78pp\), followed by Router All\-Expert FT \(\+1\.86\+1\.86pp\) and Full KD \(\+1\.77\+1\.77pp\)\. Causal LM fine\-tuning also remains ahead of KD at each matched trainable scope\. Thus, the central conclusion is not contingent on the 3,000\-example calibration budget used in the main study\.

We further test calibration\-domain sensitivity under matched 1,024\-example budgets by replacing C4 with the math\-domain OpenR1\-Math\-220k dataset[Lozhkov et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib69)for six Qwen3/HC\-SMoE and Gemma4/AIMER settings, while keeping the 28\-task evaluation suite fixed \(Figure[6](https://arxiv.org/html/2609.06076#S6.F6)\)\. Points farther toward the upper right indicate stronger recovery under both domains\. Full FT has the largest recovery averaged across the two domains \(4\.04 pp, versus 3\.40 pp for Full KD\)\. Across the 13 adjustment strategies, C4\- and OpenR1\-Math\-calibrated gains are strongly associated \(Spearmanρ=0\.973\\rho=0\.973; Kendallτb=0\.897\\tau\_\{b\}=0\.897\), and Full FT, Full KD, and Router All\-Expert FT form the same top\-three set\. We interpret this as robustness of the aggregate strategy trend, not as invariant ranking in every individual setting; Appendix[I](https://arxiv.org/html/2609.06076#A9)reports the full aggregate results, costs, and setting\-level variation\.

Figure 6:Calibration\-domain robustness with 1,024 examples each from C4 and OpenR1\-Math\-220k, averaged over six matched settings and 28 benchmarks\. Each point is an adjustment strategy; upper\-right indicates stronger recovery in both domains\. Full FT has the largest two\-domain mean, and aggregate rankings are strongly associated \(ρ=0\.973\\rho=0\.973,τb=0\.897\\tau\_\{b\}=0\.897\)\.
## 7Conclusion

We show that retraining\-free MoE compression should be treated as a compressed initialization rather than a final artifact\. With a tiny general\-domain adjustment budget, compressed MoE LLMs can recover a substantial portion of the original\-to\-compressed performance gap\. Our results indicate that causal LM fine\-tuning, especially full\-parameter adjustment, offers the strongest cost–recovery trade\-off among the strategies\. Retraining\-free compression is a strong start, and recovery toward the original model can be substantially improved through a small post\-compression adjustment stage\.

## Limitations

First, our GPU\-cost comparison is controlled but not fully system\-optimized\. For fair comparison, the GPU costs reported for post\-compression adjustment are measured under a unifiedPyTorch/Transformers\-based implementation for expert selection and adjustment training, using the same hardware, data budget, and monitoring protocol\. Benchmark evaluation is performed separately withlm\-evaluation\-harnessandvLLM, and is not included in the adjustment\-training cost\. This design avoids mixing strategy\-level effects with implementation\-specific optimization differences\. However, each stage could be further optimized with specialized systems\. For example, expert\-selection inference may be accelerated with optimized serving engines such asvLLM, while adjustment training may benefit from optimized distributed\-training stacks such as DeepSpeed, FSDP/Accelerate, Megatron\-style training, or custom fused kernels\. We do not evaluate how such system\-level optimizations would change the cost–recovery trade\-off\. Second, our main experiments focus on Expert Pruning and Expert Merging\. Expert Editing is an important MoE compression paradigm, but many editing methods rely on native factorized or tensor\-decomposed expert representations that are not yet efficiently supported by the same inference and training stack used for standard MoE checkpoints\. In practice, methods such as MoBE or TD\-MoE often provide reconstructed standard\-MoE realizations for compatibility with existing serving engines\. Post\-adjusting these reconstructed checkpoints is not parameterization\-equivalent to updating the native factorized editing model\. For this reason, we report Expert Editing results separately in Appendix[F](https://arxiv.org/html/2609.06076#A6), and do not use them as primary evidence for deployable compressed\-artifact efficiency\. Third, our study does not evaluate models above the 100B\-parameter scale\. Larger MoE models may exhibit different routing behavior, different expert redundancy patterns, and different cost trade\-offs\. Extending post\-compression adjustment to such models is an important direction for future work\. Finally, we use a tiny general\-domain C4 adjustment budget\. Task\-specific data, multilingual data, safety\-alignment data, or longer adjustment schedules may change which objective and trainable scope is optimal\.

Our conclusions should therefore be interpreted as evidence for a small\-budget, general\-domain post\-compression adjustment regime rather than as a universal optimum for all deployment settings\.

## Ethical Considerations

This work studies post\-compression adjustment for MoE LLMs, with the goal of reducing the memory and compute burden required to deploy large models\. We do not collect human\-subject data, private user data, or personally identifiable information\. The main adjustment experiments use a sampled C4 subset, while the calibration\-domain robustness check additionally uses 1,024 examples from OpenR1\-Math\-220k; all evaluations use public benchmark datasets\. Appendix[J](https://arxiv.org/html/2609.06076#A10)reports metric\-specific behavioral proxies, which we treat as diagnostics rather than as a safety certification\. A positive implication of this work is that small post\-compression adjustment may make compressed MoE models more accessible to researchers and practitioners with limited GPU resources\. At the same time, improving the efficiency of LLM deployment can also lower the cost of deploying models for harmful or unintended applications\. Our work does not introduce new safety\-alignment methods, and we do not claim that compression or post\-compression adjustment improves the safety, fairness, or reliability of the underlying models\. Compressed and adjusted models may inherit biases, hallucination behavior, unsafe generations, or misuse risks from their original base models\. We therefore encourage practitioners to treat post\-compression adjustment as an efficiency technique rather than a safety intervention\. Any deployment of compressed MoE LLMs should follow the license and usage restrictions of the original models and datasets, and should be accompanied by appropriate safety evaluation for the intended application domain\. Finally, because our experiments involve large GPU workloads, we report effective GPU\-hours, GPU\-memory occupancy in GiB\-hours, and GPU energy in kWh to make the computational cost of different adjustment strategies transparent\.

In accordance with the ACL Policy on AI Assistance, we acknowledge the use of Gemini 3\.1 Pro111[https://deepmind\.google/technologies/gemini/](https://deepmind.google/technologies/gemini/)to assist with code debugging and writing polishing\. All experimental designs, data analyses, and scientific claims presented in this work were verified by the authors\.

## Acknowledgments

This work was supported in part by the National Research Foundation of Korea \(NRF\) grant \(RS\-2025\-00560762\), the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grants \(RS\-2025\-25442338, RS\-2026\-25518317, RS\-2026\-25551943, RS\-2021\-II211343\), the AI Seoul Tech Research Support Program of the Seoul Future Foundation, Samsung Electronics Co\., Ltd\. \(IO250817\-13428\-01\), and the Ministry of Science and ICT under the Advanced GPU Utilization Support Program\. The authors thank Samsung MAX Lab for providing research infrastructure\. J\. Do is with ASRI, Seoul National University\.

## References

- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Baiet al\.\(2025\)S\. Bai, H\. Li, J\. Zhang, Z\. Hong, and S\. GuoDiEP: adaptive mixture\-of\-experts compression through differentiable expert pruning\.Advances in neural information processing systems\.Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.InThirty\-Fourth AAAI Conference on Artificial Intelligence,Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Caoet al\.\(2026\)M\. Cao, G\. Li, J\. Ji, J\. Zhang, A\. Jaiswal, L\. Shen, X\. Ma, S\. Liu, and L\. YinCondense, don’t just prune: enhancing efficiency and performance in moe layer pruning\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=BQe6j6sAu6)Cited by:[§H\.2](https://arxiv.org/html/2609.06076#A8.SS2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p2.1)\.
- Chenet al\.\(2025a\)I\. Chen, H\. Liu, W\. Sun, C\. Chao, Y\. Hsu, and C\. LeeRetraining\-free merging of sparse moe via hierarchical clustering\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=hslOzRxzXL)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.06076#S1.p3.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.06076#S4.SS1.p2.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Chenet al\.\(2025b\)X\. Chen, M\. Ha, Z\. Lan, J\. Zhang, and J\. LiMoBE: mixture\-of\-basis\-experts for compressing moe\-based llms\.External Links:2508\.05257,[Link](https://arxiv.org/abs/2508.05257)Cited by:[§F\.1](https://arxiv.org/html/2609.06076#A6.SS1.SSS0.Px1.p1.1),[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px3.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.ArXivabs/1803\.05457\.Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Dodgeet al\.\(2021\)J\. Dodge, M\. Sap, A\. Marasović, W\. Agnew, G\. Ilharco, D\. Groeneveld, M\. Mitchell, and M\. GardnerDocumenting large webtext corpora: a case study on the colossal clean crawled corpus\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 1286–1305\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.98/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.98)Cited by:[§4\.2](https://arxiv.org/html/2609.06076#S4.SS2.p2.1)\.
- Donget al\.\(2025\)Z\. Dong, H\. Peng, P\. Liu, W\. X\. Zhao, D\. Wu, F\. Xiao, and Z\. WangDomain\-specific pruning of large mixture\-of\-experts models with few\-shot demonstrations\.External Links:2504\.06792,[Link](https://arxiv.org/abs/2504.06792)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Fanet al\.\(2019\)A\. Fan, Y\. Jernite, E\. Perez, D\. Grangier, J\. Weston, and M\. AuliELI5: long form question answering\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 3558–3567\.External Links:[Link](https://aclanthology.org/P19-1346/),[Document](https://dx.doi.org/10.18653/v1/P19-1346)Cited by:[§3](https://arxiv.org/html/2609.06076#S3.SS0.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2024\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouThe language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§4\.2](https://arxiv.org/html/2609.06076#S4.SS2.p2.1)\.
- Google DeepMind \(2026\)Google DeepMindGemma 4 model card\.Note:[https://ai\.google\.dev/gemma/docs/core/model\_card\_4](https://ai.google.dev/gemma/docs/core/model_card_4)Cited by:[§4\.1](https://arxiv.org/html/2609.06076#S4.SS1.p1.1)\.
- Guet al\.\(2025\)H\. Gu, W\. Li, L\. Li, Z\. Qiyuan, M\. G\. Lee, S\. Sun, W\. Xue, and Y\. GuoDelta decompression for moe\-based LLMs compression\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=ziezViPoN1)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px4.p1.1)\.
- Hartvigsenet al\.\(2022\)T\. Hartvigsen, S\. Gabriel, H\. Palangi, M\. Sap, D\. Ray, and E\. KamarToxiGen: a large\-scale machine\-generated dataset for adversarial and implicit hate speech detection\.External Links:2203\.09509,[Link](https://arxiv.org/abs/2203.09509)Cited by:[Appendix J](https://arxiv.org/html/2609.06076#A10.p1.1)\.
- Heet al\.\(2025\)S\. He, D\. Dong, L\. Ding, and A\. LiTowards efficient mixture of experts: a holistic study of compression techniques\.External Links:2406\.02500,[Link](https://arxiv.org/abs/2406.02500)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.Advances in neural information processing systems\.Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.External Links:1503\.02531,[Link](https://arxiv.org/abs/1503.02531)Cited by:[§1](https://arxiv.org/html/2609.06076#S1.p7.1),[§4\.3\.1](https://arxiv.org/html/2609.06076#S4.SS3.SSS1.Px2.p1.1)\.
- Honget al\.\(2024\)S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. SchmidhuberMetaGPT: meta programming for a multi\-agent collaborative framework\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VtmBAGCN7o)Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1)\.
- Hyeon and Do \(2026\)S\. Hyeon and J\. DoIs retraining\-free enough? the necessity of router calibration for efficient moe compression\.External Links:2603\.02217,[Link](https://arxiv.org/abs/2603.02217)Cited by:[Appendix B](https://arxiv.org/html/2609.06076#A2.p1.1),[Appendix C](https://arxiv.org/html/2609.06076#A3.p1.1),[Appendix D](https://arxiv.org/html/2609.06076#A4.p1.1),[§F\.2](https://arxiv.org/html/2609.06076#A6.SS2.p1.1),[§H\.2](https://arxiv.org/html/2609.06076#A8.SS2.p1.1),[Appendix H](https://arxiv.org/html/2609.06076#A8.p2.1),[§3](https://arxiv.org/html/2609.06076#S3.p2.1)\.
- Hyeonet al\.\(2026\)S\. Hyeon, J\. Oh, S\. S\. Cho, and J\. DoMATA: multi\-agent framework for reliable and flexible table question answering\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 33436–33476\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1672/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1672),ISBN 979\-8\-89176\-395\-1Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1)\.
- Jaiswalet al\.\(2025\)A\. Jaiswal, J\. Wang, Y\. Li, P\. Li, T\. Chen, Z\. Wang, C\. Wang, R\. Pang, and X\. DuFinding fantastic experts in moes: a unified study for expert dropping strategies and observations\.External Links:2504\.05586,[Link](https://arxiv.org/abs/2504.05586)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1)\.
- Jinet al\.\(2020\)D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. SzolovitsWhat disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.External Links:2009\.13081,[Link](https://arxiv.org/abs/2009.13081)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2609.06076#S1.p1.1)\.
- Kimet al\.\(2026\)J\. Kim, W\. Kim, J\. Hong, Y\. Lee, S\. Hyeon, M\. Lim, Y\. Han, D\. Kim, H\. Lee, H\. Kim, and J\. DoDynin\-omni: omnimodal unified large diffusion language model\.External Links:2604\.00007,[Link](https://arxiv.org/abs/2604.00007)Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1),[§4\.2](https://arxiv.org/html/2609.06076#S4.SS2.p2.1)\.
- Kydliceket al\.\(2025\)H\. Kydlicek, A\. Lozovskaya, N\. Habib, and C\. FourrierFixing open llm leaderboard with math\-verify\.Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Lasbyet al\.\(2025\)M\. Lasby, I\. Lazarevich, N\. Sinnadurai, S\. Lie, Y\. Ioannou, and V\. ThangarasaREAP the experts: why pruning prevails for one\-shot moe compression\.External Links:2510\.13999,[Link](https://arxiv.org/abs/2510.13999)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.06076#S4.SS1.p2.1)\.
- Lepikhinet al\.\(2020\)D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. ChenGShard: scaling giant models with conditional computation and automatic sharding\.External Links:2006\.16668,[Link](https://arxiv.org/abs/2006.16668)Cited by:[§1](https://arxiv.org/html/2609.06076#S1.p1.1)\.
- Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Liet al\.\(2023\)G\. Li, H\. Hammoud, H\. Itani, D\. Khizbullin, and B\. GhanemCAMEL: communicative agents for "mind" exploration of large language model society\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 51991–52008\.External Links:[Document](https://dx.doi.org/10.52202/075280-2264),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a3621ee907def47c1b952ade25c67698-Paper-Conference.pdf)Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1)\.
- Liet al\.\(2026\)L\. Li, D\. Li, Q\. Zhu, J\. Wang, X\. Qin, W\. Li, H\. Gu, S\. Han, and Y\. GuoDrop or merge? hybrid moe LLMs compressors via metric\-driven adaptive allocation\.External Links:[Link](https://openreview.net/forum?id=9ps2joMieU)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px4.p1.1)\.
- Liet al\.\(2025a\)L\. Li, Z\. Qiyuan, J\. Wang, W\. Li, H\. Gu, S\. Han, and Y\. GuoSub\-moe: efficient mixture\-of\-expert llms compression via subspace expert merging\.External Links:2506\.23266,[Link](https://arxiv.org/abs/2506.23266)Cited by:[§H\.2](https://arxiv.org/html/2609.06076#A8.SS2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p2.1)\.
- Liet al\.\(2024a\)N\. Li, A\. Pan, A\. Gopal, S\. Yue, D\. Berrios, A\. Gatti, J\. D\. Li, A\. Dombrowski, S\. Goel, L\. Phan, G\. Mukobi, N\. Helm\-Burger, R\. Lababidi, L\. Justen, A\. B\. Liu, M\. Chen, I\. Barrass, O\. Zhang, X\. Zhu, R\. Tamirisa, B\. Bharathi, A\. Khoja, Z\. Zhao, A\. Herbert\-Voss, C\. B\. Breuer, S\. Marks, O\. Patel, A\. Zou, M\. Mazeika, Z\. Wang, P\. Oswal, W\. Lin, A\. A\. Hunt, J\. Tienken\-Harder, K\. Y\. Shih, K\. Talley, J\. Guan, R\. Kaplan, I\. Steneker, D\. Campbell, B\. Jokubaitis, A\. Levinson, J\. Wang, W\. Qian, K\. K\. Karmakar, S\. Basart, S\. Fitz, M\. Levine, P\. Kumaraguru, U\. Tupakula, V\. Varadharajan, R\. Wang, Y\. Shoshitaishvili, J\. Ba, K\. M\. Esvelt, A\. Wang, and D\. HendrycksThe wmdp benchmark: measuring and reducing malicious use with unlearning\.External Links:2403\.03218,[Link](https://arxiv.org/abs/2403.03218)Cited by:[Appendix J](https://arxiv.org/html/2609.06076#A10.p1.1)\.
- Liet al\.\(2024b\)P\. Li, Z\. Zhang, P\. Yadav, Y\. Sung, Y\. Cheng, M\. Bansal, and T\. ChenMerge, then compress: demystify efficient SMoe with hints from its routing policy\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=eFWG9Cy3WK)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.06076#S1.p2.1),[§1](https://arxiv.org/html/2609.06076#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.06076#S4.SS1.p3.1)\.
- Liet al\.\(2025b\)W\. Li, L\. Li, H\. Gu, Y\. Huang, M\. G\. Lee, S\. Sun, W\. Xue, and Y\. GuoMoE\-SVD: structured mixture\-of\-experts LLMs compression via singular value decomposition\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 35209–35230\.External Links:[Link](https://proceedings.mlr.press/v267/li25az.html)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px3.p1.1)\.
- Liet al\.\(2025c\)Z\. Li, C\. Liang, Z\. Zhang, I\. Hong, Y\. J\. Kim, W\. Chen, and T\. ZhaoSlimMoE: structured compression of large moe models via expert slimming and distillation\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=oaCUsn391F)Cited by:[§H\.2](https://arxiv.org/html/2609.06076#A8.SS2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p2.1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTruthfulQA: measuring how models mimic human falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 3214–3252\.External Links:[Link](https://aclanthology.org/2022.acl-long.229/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.229)Cited by:[Appendix J](https://arxiv.org/html/2609.06076#A10.p1.1)\.
- Liuet al\.\(2024\)E\. Liu, J\. Zhu, Z\. Lin, X\. Ning, M\. B\. Blaschko, S\. Yan, G\. Dai, H\. Yang, and Y\. WangEfficient expert pruning for sparse mixture\-of\-experts language models: enhancing performance and reducing inference costs\.External Links:2407\.00945,[Link](https://arxiv.org/abs/2407.00945)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px4.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZHANGIs your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 21558–21572\.External Links:[Document](https://dx.doi.org/10.52202/075280-0943),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/43e9d647ccd3e4b7b5baab53f0368686-Paper-Conference.pdf)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Liuet al\.\(2026a\)P\. Liu, C\. Li, T\. Wei, S\. Tu, Y\. Liu, and J\. Z\. YangCompressing large moe models via efficient pruning and data\-aware calibration\.External Links:[Link](https://openreview.net/forum?id=dJy6z9peC7)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, H\. Wu, R\. She, X\. Fu, X\. Han, T\. Zhong, and M\. YuanMoLAE: mixture of latent experts for parameter\-efficient language models\.External Links:2503\.23100,[Link](https://arxiv.org/abs/2503.23100)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px3.p1.1)\.
- Liuet al\.\(2026b\)Z\. Liu, S\. Tang, Y\. Shen, H\. Wang, and X\. YuanAIMER: calibration\-free task\-agnostic moe pruning\.External Links:2603\.18492,[Link](https://arxiv.org/abs/2603.18492)Cited by:[§1](https://arxiv.org/html/2609.06076#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.06076#S4.SS1.p3.1)\.
- Lozhkovet al\.\(2025\)A\. Lozhkov, H\. Kydlíček, L\. B\. Allal, G\. Penedo, E\. Beeching, Q\. Gallouédec, N\. Habib, L\. Tunstall, and L\. von WerraOpenR1\-Math\-220k\.Hugging Face\.Note:[https://huggingface\.co/datasets/open\-r1/OpenR1\-Math\-220k](https://huggingface.co/datasets/open-r1/OpenR1-Math-220k)Cited by:[Appendix I](https://arxiv.org/html/2609.06076#A9.SS0.SSS0.Px2.p1.1),[Table 6](https://arxiv.org/html/2609.06076#A9.T6),[§6](https://arxiv.org/html/2609.06076#S6.p2.1)\.
- Luet al\.\(2024\)X\. Lu, Q\. Liu, Y\. Xu, A\. Zhou, S\. Huang, B\. Zhang, J\. Yan, and H\. LiNot all experts are equal: efficient expert pruning and skipping for mixture\-of\-experts large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 6159–6172\.External Links:[Link](https://aclanthology.org/2024.acl-long.334/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.334)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.06076#S1.p3.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Matena and Raffel \(2022\)M\. Matena and C\. RaffelMerging models with fisher\-weighted averaging\.External Links:2111\.09832,[Link](https://arxiv.org/abs/2111.09832)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- math\-ai \(2025\)math\-aiAIME problem set 2025\.Huggingface\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime25)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Miaoet al\.\(2025\)R\. Miao, Y\. Yao, Z\. Wang, Z\. Wang, B\. Yi, L\. Liu, Y\. Zhao, and T\. YangMergeMoE: efficient compression of moe models via expert output merging\.External Links:2510\.14436,[Link](https://arxiv.org/abs/2510.14436)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.External Links:1809\.02789,[Link](https://arxiv.org/abs/1809.02789)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Nangiaet al\.\(2020\)N\. Nangia, C\. Vania, R\. Bhalerao, and S\. R\. BowmanCrowS\-pairs: a challenge dataset for measuring social biases in masked language models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1953–1967\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.154/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.154)Cited by:[Appendix J](https://arxiv.org/html/2609.06076#A10.p1.1)\.
- Palet al\.\(2022\)A\. Pal, L\. K\. Umapathi, and M\. SankarasubbuMedMCQA: a large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.InProceedings of the Conference on Health, Inference, and Learning,G\. Flores, G\. H\. Chen, T\. Pollard, J\. C\. Ho, and T\. Naumann \(Eds\.\),Proceedings of Machine Learning Research, Vol\.174,pp\. 248–260\.External Links:[Link](https://proceedings.mlr.press/v174/pal22a.html)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.1](https://arxiv.org/html/2609.06076#S4.SS1.p1.1)\.
- Radfordet al\.\(2018\)A\. Radford, K\. Narasimhan, T\. Salimans, I\. Sutskever,et al\.Improving language understanding by generative pre\-training\.Cited by:[§1](https://arxiv.org/html/2609.06076#S1.p7.1),[§4\.3\.1](https://arxiv.org/html/2609.06076#S4.SS3.SSS1.Px1.p1.1)\.
- Reddyet al\.\(2019\)S\. Reddy, D\. Chen, and C\. D\. ManningCoQA: a conversational question answering challenge\.Transactions of the Association for Computational Linguistics7,pp\. 249–266\.External Links:[Link](https://aclanthology.org/Q19-1016/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00266)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Rudingeret al\.\(2018\)R\. Rudinger, J\. Naradowsky, B\. Leonard, and B\. Van DurmeGender bias in coreference resolution\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 8–14\.External Links:[Link](https://aclanthology.org/N18-2002/),[Document](https://dx.doi.org/10.18653/v1/N18-2002)Cited by:[Appendix J](https://arxiv.org/html/2609.06076#A10.p1.1)\.
- Sakaguchiet al\.\(2019\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.External Links:1907\.10641,[Link](https://arxiv.org/abs/1907.10641)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.External Links:1701\.06538,[Link](https://arxiv.org/abs/1701.06538)Cited by:[§1](https://arxiv.org/html/2609.06076#S1.p1.1)\.
- Shenget al\.\(2023\)Y\. Sheng, L\. Zheng, B\. Yuan, Z\. Li, M\. Ryabinin, B\. Chen, P\. Liang, C\. Re, I\. Stoica, and C\. ZhangFlexGen: high\-throughput generative inference of large language models with a single GPU\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 31094–31116\.External Links:[Link](https://proceedings.mlr.press/v202/sheng23a.html)Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1)\.
- Suzgunet al\.\(2022\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. WeiChallenging big\-bench tasks and whether chain\-of\-thought can solve them\.arXiv preprint arXiv:2210\.09261\.Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Tanget al\.\(2026\)S\. Tang, Z\. Wang, B\. Zheng, L\. Wang, R\. Men, S\. Zhang, X\. Yuan, Z\. Qiu, Z\. Shen, and D\. LiuSlimQwen: exploring the pruning and distillation in large moe model pre\-training\.External Links:2605\.08738,[Link](https://arxiv.org/abs/2605.08738)Cited by:[§H\.2](https://arxiv.org/html/2609.06076#A8.SS2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p2.1)\.
- Veeraboina \(2024\)H\. VeeraboinaAIME problem set 1983\-2024\.Kaggle\.External Links:[Link](https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Vendrowet al\.\(2025\)J\. Vendrow, E\. Vendrow, S\. Beery, and A\. MadryDo large language model benchmarks test reliability?\.External Links:2502\.03461,[Link](https://arxiv.org/abs/2502.03461)Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Wortsmanet al\.\(2022\)M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith, and L\. SchmidtModel soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.External Links:2203\.05482,[Link](https://arxiv.org/abs/2203.05482)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Wuet al\.\(2024\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. \(\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, A\. Awadallah, R\. W\. White, D\. Burger, and C\. WangAutoGen: enabling next\-gen llm applications via multi\-agent conversation\.InCOLM 2024,External Links:[Link](https://www.microsoft.com/en-us/research/publication/autogen-enabling-next-gen-llm-applications-via-multi-agent-conversation-framework/)Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1)\.
- Xieet al\.\(2024\)Y\. Xie, Z\. Zhang, D\. Zhou, C\. Xie, Z\. Song, X\. Liu, Y\. Wang, X\. Lin, and A\. XuMoE\-pruner: pruning mixture\-of\-experts large language model using the hints from its router\.External Links:2410\.12013,[Link](https://arxiv.org/abs/2410.12013)Cited by:[§H\.2](https://arxiv.org/html/2609.06076#A8.SS2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p2.1)\.
- XUet al\.\(2026\)Y\. XU, Y\. WANG, X\. Peng, H\. Zang, C\. Minghao, P\. Xia, and Z\. WenTD\-moe: tensor decomposition for moe models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=D9cnZNZfxX)Cited by:[§F\.1](https://arxiv.org/html/2609.06076#A6.SS1.SSS0.Px1.p1.1),[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)C\. Yang, Y\. Sui, J\. Xiao, L\. Huang, Y\. Gong, Y\. Duan, W\. Jia, M\. Yin, Y\. Cheng, and B\. YuanMoE\-i2\{\}^\{2\}: compressing mixture of experts models through inter\-expert pruning and intra\-expert low\-rank decomposition\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10456–10466\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.612/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.612)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px4.p1.1),[§H\.2](https://arxiv.org/html/2609.06076#A8.SS2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p2.1)\.
- Yanget al\.\(2025\)X\. Yang, Y\. Tian, and Y\. SongMoE pathfinder: trajectory\-driven expert pruning\.External Links:2512\.18425,[Link](https://arxiv.org/abs/2512.18425)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Yuet al\.\(2022\)G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. ChunOrca: a distributed serving system for Transformer\-Based generative models\.In16th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 22\),Carlsbad, CA,pp\. 521–538\.External Links:ISBN 978\-1\-939133\-28\-1,[Link](https://www.usenix.org/conference/osdi22/presentation/yu)Cited by:[Appendix H](https://arxiv.org/html/2609.06076#A8.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Cited by:[§A\.4](https://arxiv.org/html/2609.06076#A1.SS4.p1.1)\.
- Zhanget al\.\(2025\)G\. Zhang, Y\. Han, Y\. Lou, W\. Zhao, Y\. Zhang, and Y\. YouMoNE: replacing redundant experts with lightweight novices for structured pruning of moe\.External Links:2507\.00390,[Link](https://arxiv.org/abs/2507.00390)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px4.p1.1)\.
- Zhaoet al\.\(2025\)Y\. Zhao, Z\. Wang, and M\. ZhangPuzzleMoE: efficient compression of large mixture\-of\-experts models via sparse expert merging and bit\-packed inference\.External Links:2511\.04805,[Link](https://arxiv.org/abs/2511.04805)Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.06076#S2.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[Appendix J](https://arxiv.org/html/2609.06076#A10.p1.1)\.
- Zhouet al\.\(2025\)Y\. Zhou, Z\. Zhao, D\. Cheng, Z\. Wu, J\. Gui, Y\. Yang, F\. Wu, Y\. Cheng, and H\. FanDropping experts, recombining neurons: retraining\-free pruning for sparse mixture\-of\-experts LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15169–15186\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.820/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.820),ISBN 979\-8\-89176\-335\-7Cited by:[§H\.1](https://arxiv.org/html/2609.06076#A8.SS1.SSS0.Px4.p1.1)\.

## Appendix AExperiment Settings

This appendix reports the implementation settings used for compression, post\-compression adjustment, and downstream evaluation\. Unless otherwise stated, random seeds were fixed to 42, models were loaded and saved in bfloat16, tokenizer and generation sidecar files were copied from the source checkpoint, and dense non\-expert parameters were kept unchanged by the compression method itself\. The reported expert\-retention ratios are 50%, 62\.5%, and 75%\. For pruning and merging methods, these correspond to 64, 80, and 96 retained or physical routed experts per MoE layer, respectively, for models with 128 routed experts per layer\.

### A\.1Backbones and Compression Targets

We use two instruction\-tuned MoE backbones: Qwen3\-30B\-A3B\-Instruct\-2507 and gemma\-4\-26B\-A4B\-it\. Qwen3 contains 48 sparse MoE blocks with 128 routed experts per block, while the Gemma4 text stack contains 30 routed MoE layers with 128 routed experts per layer\. For all pruning and merging methods, the router, attention layers, embeddings, layer normalizations, and other dense components are preserved; only routed expert capacity is reduced\.

### A\.2Post\-Compression Adjustment

All Post\-Compression Adjustment runs use the same text\-only calibration corpus, optimizer settings, and tokenization settings across backbones, methods, retention ratios, objectives, and trainable scopes\. The adjustment data are the first 3000 non\-empty text examples read from the C4\. Examples are tokenized with padding and truncation to 512 tokens\. The dataloader is shuffled with a seed\-42torch\.Generator, usesdrop\_last=false, and uses zero worker processes\.

Table 2:Common Post Compression Adjustment hyperparameters\.##### Objectives\.

For fine\-tuning \(FT\), we optimize the standard causal language\-modeling loss, masking padding tokens with label value−100\-100\. For knowledge distillation \(KD\), the teacher is the corresponding uncompressed backbone, either Qwen3\-30B\-A3B\-Instruct\-2507 or gemma\-4\-26B\-A4B\-it\. The teacher is evaluated in no\-gradient mode, and the student is optimized with token\-level KL divergence between shifted teacher and student logits at temperature 1\.0\. Both teacher and student forward passes are included in the monitored KD training\-loop segment and therefore in effective GPU\-hours, GPU\-memory GiB\-hours, and GPU energy\. Direct router logit matching \(DRLM\) is used as a diagnostic router\-only objective when a teacher\-to\-student router\-coordinate mapping is available; it uses the same data, optimizer, epoch count, batch size, gradient accumulation, temperature, and gradient clipping as the other Post\-Compression Adjustment objectives\.

##### Trainable scopes\.

The unadjusted baseline is evaluated immediately after compression\. Post\-Compression Adjustment then uses the following trainable scopes:

- •Router\-only:only MoE router parameters are trainable\.
- •Router \+ top\-kkexperts:routers and a fixed per\-layer subset of physical experts are trainable\. The main experiments usek=8k=8\. Additional scope ablations usek=16k=16andk=50k=50\.
- •Router \+ all experts:routers and all routed expert parameters are trainable while dense non\-expert parameters remain frozen\. This is stored asrouter\_top128\_expert\_finetuneandrouter\_top128\_expert\_kdbecause each original layer has 128 logical routed experts\.
- •Full\-parameter:all student parameters are trainable\. This corresponds tofull\_finetuneandfull\_kd\.

For top\-kkexpert scopes, the selected experts are chosen before training using the same adjustment data\. We count router top\-8 assignments, map logical expert IDs to physical expert IDs when the compressed model has a logical\-to\-physical mapping, and keep the most frequently activatedkkphysical experts per layer\. The expert\-selection inference is monitored as a separate segment; whenever a strategy requires selection, its samples and wall\-clock duration are aggregated with those of the subsequent training loop for effective GPU\-hours, GPU\-memory GiB\-hours, and GPU energy\.

##### Hardware and cost logging\.

All adjustment runs used two NVIDIA H200 140GB GPUs\. GPU power draw, utilization, andmemory\.usedwere polled periodically withnvidia\-smion all assigned GPUs, using a nominal one\-second sleep interval between successive poll attempts\. Including query overhead, the observed median effective polling interval was 1\.685 seconds across runs\. LetThrT\_\{\\mathrm\{hr\}\}be the sum, in hours, of the included measured\-segment durations and letG=2G=2\. LetP¯g\\bar\{P\}\_\{g\}be the mean power draw of GPUggover its valid retained readings, and letu¯\\bar\{u\}be the mean utilization percentage over valid retained GPU readings\. At pollii, let𝒢i\\mathcal\{G\}\_\{i\}denote the subset of assigned GPUs with a valid retained reading and letMg,iM\_\{g,i\}denotememory\.usedin GiB\. Across theNMN\_\{M\}polls with at least one valid memory reading, we defineM¯obs=1NM​∑i=1NM∑g∈𝒢iMg,i\.\\bar\{M\}\_\{\\mathrm\{obs\}\}=\\frac\{1\}\{N\_\{M\}\}\\sum\_\{i=1\}^\{N\_\{M\}\}\\sum\_\{g\\in\\mathcal\{G\}\_\{i\}\}M\_\{g,i\}\.We report

Heff=G​Thr​u¯100,H\_\{\\mathrm\{eff\}\}=GT\_\{\\mathrm\{hr\}\}\\frac\{\\bar\{u\}\}\{100\},Hmem=Thr​M¯obs,H\_\{\\mathrm\{mem\}\}=T\_\{\\mathrm\{hr\}\}\\bar\{M\}\_\{\\mathrm\{obs\}\},EGPU=Thr1000​∑g=1GP¯g,E\_\{\\mathrm\{GPU\}\}=\\frac\{T\_\{\\mathrm\{hr\}\}\}\{1000\}\\sum\_\{g=1\}^\{G\}\\bar\{P\}\_\{g\},measured in effective GPU\-hours, GPU\-memory GiB\-hours, and kWh, respectively\. All three metrics use the same measured\-segment boundaries\. Metric averages are computed from their valid retained readings; 147 of 704,576 poll attempts \(0\.021%\) contained only one of the two assigned GPU rows\. Model loading occurs before monitoring and is excluded, whereas resident\-model memory during the measured segments is included inmemory\.used\. The monitored KD and DRLM loops include both teacher and student forward passes\. For router\+top\-kkstrategies, separately monitored expert\-selection and training\-loop segments are both included; gaps between measured segments are not counted\.

### A\.3Evaluation withlm\-evaluation\-harness

All downstream evaluations uselm\-evaluation\-harnesswith the vLLM backend\. The recorded harness version is0\.4\.13\.dev0, and the recorded Transformers version is5\.8\.1\. Each evaluation job usesmodel=vllm, bfloat16 weights,tensor\_parallel\_size=1,max\_model\_len=4096,trust\_remote\_code=true, automatic harness batch sizing, chat\-template application, multi\-turn few\-shot formatting, and sample logging\. The limit is unset, so all task examples are evaluated\. Random, NumPy, PyTorch, and few\-shot seeds are all 42\. GPU memory utilization is 0\.9 for reasoning, math, chain\-of\-thought, code, and AIME categories, and 0\.8 for the multiple\-choice category\.

For Gemma4 evaluation,enable\_thinking=falseis forced for all categories\. For Qwen3,enable\_thinking=falseis explicitly set for the AIME category\. HC\-SMoE Qwen checkpoints are evaluated withenforce\_eager=true\. Gemma4 M\-SMoE checkpoints are evaluated withmodel\_impl=transformers,enforce\_eager=true, andattention\_backend=TRITON\_ATTN\. Code tasks are run withHF\_ALLOW\_CODE\_EVAL=1and harness unsafe\-code confirmation enabled\. For AIME tasks, generation is non\-sampling withdo\_sample=false, temperature 0\.0, andmax\_gen\_toks=2780\. Other categories use the deterministic generation or scoring defaults defined by the corresponding harness task YAML files\.

Table 3:Downstream evaluation categories and task\-level settings\.
### A\.4Benchmarks

We evaluate each model on 28 downstream benchmarks covering general reasoning, mathematical reasoning, coding, chain\-of\-thought reasoning, and multiple\-choice question answering\. For general reasoning and QA, we use BBH[Suzgun et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib30)in both few\-shot and zero\-shot settings, CoQA[Reddy et al\. \(2019\)](https://arxiv.org/html/2609.06076#bib.bib31), and HellaSwag[Zellers et al\. \(2019\)](https://arxiv.org/html/2609.06076#bib.bib42)\. For mathematical reasoning, we use GSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.06076#bib.bib32), GSM8K Platinum[Vendrow et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib33), MATH[Hendrycks et al\. \(2021b\)](https://arxiv.org/html/2609.06076#bib.bib34);[Lewkowycz et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib35);[Kydlicek et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib36), AIME 1983–2024[Veeraboina \(2024\)](https://arxiv.org/html/2609.06076#bib.bib37), and AIME 2025[math\-ai \(2025\)](https://arxiv.org/html/2609.06076#bib.bib38)\. For coding, we evaluate MBPP[Austin et al\. \(2021\)](https://arxiv.org/html/2609.06076#bib.bib39), HumanEval[Chen et al\. \(2021\)](https://arxiv.org/html/2609.06076#bib.bib40), MBPP\+ & HumanEval\+[Liu et al\. \(2023\)](https://arxiv.org/html/2609.06076#bib.bib76)\. We further evaluate chain\-of\-thought reasoning on BBH, GSM8K, and GSM8K Platinum under both few\-shot and zero\-shot settings\. Finally, we evaluate multiple\-choice question answering on ARC\-Challenge, ARC\-Easy, MedMCQA, MedQA, OpenBookQA, PIQA, WinoGrande, and MMLU[Clark et al\. \(2018\)](https://arxiv.org/html/2609.06076#bib.bib41);[Pal et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib43);[Jin et al\. \(2020\)](https://arxiv.org/html/2609.06076#bib.bib44);[Mihaylov et al\. \(2018\)](https://arxiv.org/html/2609.06076#bib.bib45);[Bisk et al\. \(2020\)](https://arxiv.org/html/2609.06076#bib.bib46);[Sakaguchi et al\. \(2019\)](https://arxiv.org/html/2609.06076#bib.bib47);[Hendrycks et al\. \(2021a\)](https://arxiv.org/html/2609.06076#bib.bib48)\.

### A\.5Direct router logit matching

We also include Direct Router Logit Matching \(DRLM\) as a diagnostic baseline when router coordinates can be aligned through the compression mapping\. LetrT,ℓ,tr\_\{T,\\ell,t\}andrS,ℓ,tr\_\{S,\\ell,t\}be the teacher and student router logits at layerℓ\\elland token positiontt, and letΠℓ\\Pi\_\{\\ell\}map teacher router coordinates to the student router space\. The objective is

ℒDRLM=1\|ℐ\|​∑\(ℓ,t\)∈ℐ‖Πℓ​\(rT,ℓ,t\)−rS,ℓ,t‖22,\\mathcal\{L\}\_\{\\mathrm\{DRLM\}\}=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{\(\\ell,t\)\\in\\mathcal\{I\}\}\\left\\\|\\Pi\_\{\\ell\}\(r\_\{T,\\ell,t\}\)\-r\_\{S,\\ell,t\}\\right\\\|\_\{2\}^\{2\},\(4\)whereℐ\\mathcal\{I\}is the set of valid layer\-token observations\. Unlike KD, DRLM directly aligns router behavior rather than output\-token distributions\. We use it only as a diagnostic comparison because it requires a well\-defined router\-coordinate mapping, which may not exist uniformly across all compression paradigms\.

Table 4:Compression settings\.

## Appendix BOriginal MoE \(before Compression\)

We adopt the MoE notation of[Hyeon and Do \(2026\)](https://arxiv.org/html/2609.06076#bib.bib61)\. We restate only the definitions needed for the residual decompositions below; the underlying MoE formulation and the original scenario analysis are not claimed as new contributions of the present paper\.

Throughout Appendices B–D,xxdenotes a common local hidden\-state input supplied to the original and compressed layer maps\. This same\-input convention isolates the local compression source\. The end\-to\-end input shift between the two networks is handled separately by the propagation analysis in Section[3](https://arxiv.org/html/2609.06076#S3)\.

Consider one sparse MoE layer with routed\-expert index setℰ=\{0,…,N−1\}\\mathcal\{E\}=\\\{0,\\ldots,N\-1\\\}\. Let𝒮⊆ℰ\\mathcal\{S\}\\subseteq\\mathcal\{E\}be the top\-kkactive set selected by the original router, with\|𝒮\|=k<N\|\\mathcal\{S\}\|=k<N\. Letgiorig​\(x\)≥0g\_\{i\}^\{\\mathrm\{orig\}\}\(x\)\\geq 0denote the nonnegative gate mass used for mixture weighting, such as a softmax probability\. This avoids normalizing raw router logits, which need not be nonnegative\. We assume that the gate masses are positive on the selected set\.

For any nonempty active set𝒜⊆ℰ\\mathcal\{A\}\\subseteq\\mathcal\{E\}, define

g~io,𝒜​\(x\):=giorig​\(x\)∑j∈𝒜gjorig​\(x\),i∈𝒜\.\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{A\}\}\(x\):=\\frac\{g\_\{i\}^\{\\mathrm\{orig\}\}\(x\)\}\{\\sum\_\{j\\in\\mathcal\{A\}\}g\_\{j\}^\{\\mathrm\{orig\}\}\(x\)\},\\qquad i\\in\\mathcal\{A\}\.\(5\)The routed MoE contribution is

yorig​\(x\)=∑i∈𝒮g~io,𝒮​\(x\)​Ei​\(x\)\.y\_\{\\mathrm\{orig\}\}\(x\)=\\sum\_\{i\\in\\mathcal\{S\}\}\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\(x\)E\_\{i\}\(x\)\.\(6\)Any unchanged shared\-expert, residual, or dense branch is omitted from this local expression because it cancels in the same\-input difference\. In the following appendices, superscriptspp,mm, andeedenote the pruned, merged, and edited models, respectively\.

## Appendix CExpert Pruning \(N→N−αN\\rightarrow N\-\\alpha\)

The exhaustive best/partial/disjoint pruning analysis was introduced in prior work[Hyeon and Do \(2026\)](https://arxiv.org/html/2609.06076#bib.bib61)\. Here we replace the repeated scenario\-wise algebra with one exact active\-path decomposition that directly supports the routing\-versus\-expert\-path interpretation in Section[3](https://arxiv.org/html/2609.06076#S3)\.

Let𝒫⊆ℰ\\mathcal\{P\}\\subseteq\\mathcal\{E\}be the set of retained experts, with\|𝒫\|≥k\|\\mathcal\{P\}\|\\geq k, and let𝒮′⊆𝒫\\mathcal\{S\}^\{\\prime\}\\subseteq\\mathcal\{P\}be the active set selected by the pruned layer at the common reference inputxx\. We assume\|𝒮\|=\|𝒮′\|=k\|\\mathcal\{S\}\|=\|\\mathcal\{S\}^\{\\prime\}\|=k\. For any nonempty active setA⊆PA\\subseteq Psatisfying∑j∈Agjpruned​\(x\)\>0\\sum\_\{j\\in A\}g\_\{j\}^\{\\mathrm\{pruned\}\}\(x\)\>0, define

g~ip,𝒜​\(x\):=gipruned​\(x\)∑j∈𝒜gjpruned​\(x\),i∈𝒜\.\\widetilde\{g\}\_\{i\}^\{p,\\mathcal\{A\}\}\(x\):=\\frac\{g\_\{i\}^\{\\mathrm\{pruned\}\}\(x\)\}\{\\sum\_\{j\\in\\mathcal\{A\}\}g\_\{j\}^\{\\mathrm\{pruned\}\}\(x\)\},\\qquad i\\in\\mathcal\{A\}\.\(7\)The two routed outputs are

yorig​\(x\)\\displaystyle y\_\{\\mathrm\{orig\}\}\(x\)=∑i∈𝒮g~io,𝒮​\(x\)​Ei​\(x\),\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\}\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\(x\)E\_\{i\}\(x\),\(8\)ypruned​\(x\)\\displaystyle y\_\{\\mathrm\{pruned\}\}\(x\)=∑i∈𝒮′g~ip,𝒮′​\(x\)​Ei​\(x\)\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}^\{\\prime\}\}\\widetilde\{g\}\_\{i\}^\{p,\\mathcal\{S\}^\{\\prime\}\}\(x\)E\_\{i\}\(x\)\.\(9\)For compact displays, write

g~io:=g~io,𝒮​\(x\),g~ip:=g~ip,𝒮′​\(x\),Ei:=Ei​\(x\)\.\\widetilde\{g\}\_\{i\}^\{o\}:=\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\(x\),\\quad\\widetilde\{g\}\_\{i\}^\{p\}:=\\widetilde\{g\}\_\{i\}^\{p,\\mathcal\{S\}^\{\\prime\}\}\(x\),\\quad E\_\{i\}:=E\_\{i\}\(x\)\.\(10\)
##### Active\-set decomposition\.

The availability set𝒫\\mathcal\{P\}and the active set𝒮′\\mathcal\{S\}^\{\\prime\}have different roles\. Define

𝒯\\displaystyle\\mathcal\{T\}=𝒮∩𝒮′,\\displaystyle=\\mathcal\{S\}\\cap\\mathcal\{S\}^\{\\prime\},𝒟\\displaystyle\\mathcal\{D\}=𝒮∖𝒮′,\\displaystyle=\\mathcal\{S\}\\setminus\\mathcal\{S\}^\{\\prime\},ℛ\\displaystyle\\mathcal\{R\}=𝒮′∖𝒮\.\\displaystyle=\\mathcal\{S\}^\{\\prime\}\\setminus\\mathcal\{S\}\.\(11\)Then

𝒮=𝒯∪˙𝒟,𝒮′=𝒯∪˙ℛ\.\\mathcal\{S\}=\\mathcal\{T\}\\mathbin\{\\dot\{\\cup\}\}\\mathcal\{D\},\\qquad\\mathcal\{S\}^\{\\prime\}=\\mathcal\{T\}\\mathbin\{\\dot\{\\cup\}\}\\mathcal\{R\}\.\(12\)Moreover,

𝒟\\displaystyle\\mathcal\{D\}=\(𝒮∖𝒫\)∪˙\(\(𝒮∩𝒫\)∖𝒮′\)\.\\displaystyle=\(\\mathcal\{S\}\\setminus\\mathcal\{P\}\)\\mathbin\{\\dot\{\\cup\}\}\\bigl\(\(\\mathcal\{S\}\\cap\\mathcal\{P\}\)\\setminus\\mathcal\{S\}^\{\\prime\}\\bigr\)\.\(13\)The first subset contains physically removed experts\. The second contains retained experts absent from the compressed active path\. In a realized end\-to\-end comparison, the latter can arise from an upstream hidden\-state shift even when the router parameters themselves are unchanged\.

Using Eq\. \([12](https://arxiv.org/html/2609.06076#A3.E12)\), the exact vector residual is

yorig−ypruned\\displaystyle y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{pruned\}\}=∑i∈𝒯\(g~io−g~ip\)​Ei\\displaystyle=\\sum\_\{i\\in\\mathcal\{T\}\}\(\\widetilde\{g\}\_\{i\}^\{o\}\-\\widetilde\{g\}\_\{i\}^\{p\}\)E\_\{i\}\(14\)\+∑i∈𝒟g~ioEi−∑i∈ℛg~ipEi\.\\displaystyle\+\\sum\_\{i\\in\\mathcal\{D\}\}\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\sum\_\{i\\in\\mathcal\{R\}\}\\widetilde\{g\}\_\{i\}^\{p\}E\_\{i\}\.Define

eroutep​\(x\)\\displaystyle e\_\{\\mathrm\{route\}\}^\{p\}\(x\):=∑i∈𝒯\(g~io−g~ip\)​Ei,\\displaystyle:=\\sum\_\{i\\in\\mathcal\{T\}\}\(\\widetilde\{g\}\_\{i\}^\{o\}\-\\widetilde\{g\}\_\{i\}^\{p\}\)E\_\{i\},\(15\)epathp​\(x\)\\displaystyle e\_\{\\mathrm\{path\}\}^\{p\}\(x\):=∑i∈𝒟g~io​Ei−∑i∈ℛg~ip​Ei\.\\displaystyle:=\\sum\_\{i\\in\\mathcal\{D\}\}\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\sum\_\{i\\in\\mathcal\{R\}\}\\widetilde\{g\}\_\{i\}^\{p\}E\_\{i\}\.\(16\)For any vector norm,

∥yorig−ypruned∥\\displaystyle\\lVert y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{pruned\}\}\\rVert=∥eroutep\+epathp∥\\displaystyle=\\lVert e\_\{\\mathrm\{route\}\}^\{p\}\+e\_\{\\mathrm\{path\}\}^\{p\}\\rVert\(17\)≤∥eroutep∥\+∥epathp∥\.\\displaystyle\\leq\\lVert e\_\{\\mathrm\{route\}\}^\{p\}\\rVert\+\\lVert e\_\{\\mathrm\{path\}\}^\{p\}\\rVert\.

##### Best scenario: preserved active path\.

The favorable condition is𝒮′=𝒮\\mathcal\{S\}^\{\\prime\}=\\mathcal\{S\}, which implies𝒮⊆𝒫\\mathcal\{S\}\\subseteq\\mathcal\{P\}\. Then𝒯=𝒮\\mathcal\{T\}=\\mathcal\{S\}and𝒟=ℛ=∅\\mathcal\{D\}=\\mathcal\{R\}=\\emptyset, so

yorig−ypruned\\displaystyle y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{pruned\}\}=eroutep​\(x\)\\displaystyle=e\_\{\\mathrm\{route\}\}^\{p\}\(x\)\(18\)=∑i∈𝒮\(g~io−g~ip\)​Ei\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\}\(\\widetilde\{g\}\_\{i\}^\{o\}\-\\widetilde\{g\}\_\{i\}^\{p\}\)E\_\{i\}\.The normalized router weights can differ if the gate implementation is changed or after the router itself is adjusted\. For pure expert pruning that leaves the router and gate implementation unchanged, however, the common\-input condition together withS′=SS^\{\\prime\}=Simpliesg~ip,S​\(x\)=g~io,S​\(x\)\\tilde\{g\}\_\{i\}^\{p,S\}\(x\)=\\tilde\{g\}\_\{i\}^\{o,S\}\(x\)for everyi∈Si\\in S; hence the initial same\-input residual in Eq\.[18](https://arxiv.org/html/2609.06076#A3.E18)is exactly zero\. Realized end\-to\-end deviations can still occur because upstream compression changes the hidden state supplied to deeper routers\. More generally, Eq\.[18](https://arxiv.org/html/2609.06076#A3.E18)is the only active\-set case whose local residual has no expert\-path replacement term\.

##### Most common scenario: partial active\-set overlap\.

If0<\|𝒯\|<k0<\|\\mathcal\{T\}\|<k, then𝒟≠∅\\mathcal\{D\}\\neq\\emptysetandℛ≠∅\\mathcal\{R\}\\neq\\emptyset, with\|𝒟\|=\|ℛ\|=k−\|𝒯\|\|\\mathcal\{D\}\|=\|\\mathcal\{R\}\|=k\-\|\\mathcal\{T\}\|\. Both residual components may contribute\. Router\-only adjustment can change shared\-path weights and may move the top\-kkboundary, but it cannot directly change the expert functions\.

##### Worst scenario: disjoint active paths\.

If𝒯=∅\\mathcal\{T\}=\\emptyset, then

yorig−ypruned\\displaystyle y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{pruned\}\}=epathp​\(x\)\\displaystyle=e\_\{\\mathrm\{path\}\}^\{p\}\(x\)\(19\)=∑i∈𝒮g~io​Ei−∑i∈𝒮′g~ip​Ei\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\}\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\sum\_\{i\\in\\mathcal\{S\}^\{\\prime\}\}\\widetilde\{g\}\_\{i\}^\{p\}E\_\{i\}\.Here, “worst” denotes complete active\-set replacement\. Disjointness alone does not prove that the residual norm is maximal for every input\.

##### Sign convention for the propagation analysis\.

Equations \([14](https://arxiv.org/html/2609.06076#A3.E14)\)– \([16](https://arxiv.org/html/2609.06076#A3.E16)\) use the original\-minus\-compressed convention\. Because Section[3](https://arxiv.org/html/2609.06076#S3)definesδℓ=h^ℓ−hℓ\\delta\_\{\\ell\}=\\widehat\{h\}\_\{\\ell\}\-h\_\{\\ell\}, the same\-input source injected into that recurrence is

ϵℓp:=ypruned,ℓ−yorig,ℓ=−\(eroute,ℓp\+epath,ℓp\)\.\\epsilon\_\{\\ell\}^\{p\}:=y\_\{\\mathrm\{pruned\},\\ell\}\-y\_\{\\mathrm\{orig\},\\ell\}=\-\\left\(e\_\{\\mathrm\{route\},\\ell\}^\{p\}\+e\_\{\\mathrm\{path\},\\ell\}^\{p\}\\right\)\.\(20\)

## Appendix DExpert Merging \(N→MN\\rightarrow M, whereM<NM<N\)

We adopt the cluster mapping and projected\-path notation introduced in[Hyeon and Do \(2026, Appendix D\)](https://arxiv.org/html/2609.06076#bib.bib61)\. Rather than repeating the prior nine\-case enumeration, we give one cluster\-space identity tailored to the present adjustment\-scope analysis\.

Let𝒞=\{0,…,M−1\}\\mathcal\{C\}=\\\{0,\\ldots,M\-1\\\}be the physical merged\-expert index set, and let

ϕ:ℰ→𝒞\\phi:\\mathcal\{E\}\\rightarrow\\mathcal\{C\}\(21\)be a surjective map assigning each original routed expert to a merged cluster\. The function implemented by physical merged expertccis denoted byMc​\(x\)M\_\{c\}\(x\)\. We do not assume thatMcM\_\{c\}is exactly equal to every original expert in its preimage; it is the synthesized function produced by the merging method\.

For the original active set𝒮\\mathcal\{S\}, define

𝒞proj:=ϕ⁡\(𝒮\)=\{ϕ⁡\(i\):i∈𝒮\}\.\\mathcal\{C\}\_\{\\mathrm\{proj\}\}:=\\phi\(\\mathcal\{S\}\)=\\\{\\phi\(i\):i\\in\\mathcal\{S\}\\\}\.\(22\)Let𝒮m:=𝒮merge⊆𝒞\\mathcal\{S\}\_\{m\}:=\\mathcal\{S\}^\{\\mathrm\{merge\}\}\\subseteq\\mathcal\{C\}be the set of physical merged experts receiving nonzero active gate mass\. This covers both routers withMMphysical coordinates and routers that retainNNlogical coordinates and map several active logical indices to one physical expert\. In the latter case,gcmerge​\(x\)g\_\{c\}^\{\\mathrm\{merge\}\}\(x\)is the sum of the active logical gate masses mapped tocc\.

For any nonempty active setA⊆𝒞A\\subseteq\\mathcal\{C\}satisfying∑d∈Agdmerge​\(x\)\>0\\sum\_\{d\\in A\}g\_\{d\}^\{\\mathrm\{merge\}\}\(x\)\>0, define

g~cm,𝒜​\(x\):=gcmerge​\(x\)∑d∈𝒜gdmerge​\(x\),c∈𝒜\.\\widetilde\{g\}\_\{c\}^\{m,\\mathcal\{A\}\}\(x\):=\\frac\{g\_\{c\}^\{\\mathrm\{merge\}\}\(x\)\}\{\\sum\_\{d\\in\\mathcal\{A\}\}g\_\{d\}^\{\\mathrm\{merge\}\}\(x\)\},\\qquad c\\in\\mathcal\{A\}\.\(23\)The merged output is

ymerge​\(x\)=∑c∈𝒮mg~cm,𝒮m​\(x\)​Mc​\(x\)\.y\_\{\\mathrm\{merge\}\}\(x\)=\\sum\_\{c\\in\\mathcal\{S\}\_\{m\}\}\\widetilde\{g\}\_\{c\}^\{m,\\mathcal\{S\}\_\{m\}\}\(x\)M\_\{c\}\(x\)\.\(24\)
##### Projecting the original output into cluster space\.

For eachc∈𝒞projc\\in\\mathcal\{C\}\_\{\\mathrm\{proj\}\}, define the original gate mass assigned to that cluster by

αco,𝒮​\(x\):=∑i∈𝒮ϕ⁡\(i\)=cg~io,𝒮​\(x\)\.\\alpha\_\{c\}^\{o,\\mathcal\{S\}\}\(x\):=\\sum\_\{\\begin\{subarray\}\{c\}i\\in\\mathcal\{S\}\\\\ \\phi\(i\)=c\\end\{subarray\}\}\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\(x\)\.\(25\)Because the selected gate masses are positive,αco,𝒮​\(x\)\>0\\alpha\_\{c\}^\{o,\\mathcal\{S\}\}\(x\)\>0\. Define the corresponding gate\-weighted original cluster output by

E¯co,𝒮​\(x\):=∑i∈𝒮ϕ⁡\(i\)=cg~io,𝒮​\(x\)​Ei​\(x\)αco,𝒮​\(x\)\.\\overline\{E\}\_\{c\}^\{o,\\mathcal\{S\}\}\(x\):=\\frac\{\\sum\_\{\\begin\{subarray\}\{c\}i\\in\\mathcal\{S\}\\\\ \\phi\(i\)=c\\end\{subarray\}\}\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\(x\)E\_\{i\}\(x\)\}\{\\alpha\_\{c\}^\{o,\\mathcal\{S\}\}\(x\)\}\.\(26\)Then

yorig​\(x\)=∑c∈𝒞projαco,𝒮​\(x\)​E¯co,𝒮​\(x\)\.y\_\{\\mathrm\{orig\}\}\(x\)=\\sum\_\{c\\in\\mathcal\{C\}\_\{\\mathrm\{proj\}\}\}\\alpha\_\{c\}^\{o,\\mathcal\{S\}\}\(x\)\\overline\{E\}\_\{c\}^\{o,\\mathcal\{S\}\}\(x\)\.\(27\)For compact displays below, suppressxxand write

αco\\displaystyle\\alpha\_\{c\}^\{o\}:=αco,𝒮​\(x\),\\displaystyle:=\\alpha\_\{c\}^\{o,\\mathcal\{S\}\}\(x\),E¯co\\displaystyle\\overline\{E\}\_\{c\}^\{o\}:=E¯co,𝒮​\(x\),\\displaystyle:=\\overline\{E\}\_\{c\}^\{o,\\mathcal\{S\}\}\(x\),\(28\)g~cm\\displaystyle\\widetilde\{g\}\_\{c\}^\{m\}:=g~cm,𝒮m​\(x\),\\displaystyle:=\\widetilde\{g\}\_\{c\}^\{m,\\mathcal\{S\}\_\{m\}\}\(x\),Mc\\displaystyle M\_\{c\}:=Mc​\(x\)\.\\displaystyle:=M\_\{c\}\(x\)\.
Define the shared, missing, and newly active physical clusters by

𝒯\\displaystyle\\mathcal\{T\}=𝒞proj∩𝒮m,\\displaystyle=\\mathcal\{C\}\_\{\\mathrm\{proj\}\}\\cap\\mathcal\{S\}\_\{m\},\(29\)𝒟\\displaystyle\\mathcal\{D\}=𝒞proj∖𝒮m,\\displaystyle=\\mathcal\{C\}\_\{\\mathrm\{proj\}\}\\setminus\\mathcal\{S\}\_\{m\},ℛ\\displaystyle\\mathcal\{R\}=𝒮m∖𝒞proj\.\\displaystyle=\\mathcal\{S\}\_\{m\}\\setminus\\mathcal\{C\}\_\{\\mathrm\{proj\}\}\.The exact residual is

yorig−ymerge\\displaystyle y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{merge\}\}=∑c∈𝒯\(αco​E¯co−g~cm​Mc\)\\displaystyle=\\sum\_\{c\\in\\mathcal\{T\}\}\\left\(\\alpha\_\{c\}^\{o\}\\overline\{E\}\_\{c\}^\{o\}\-\\widetilde\{g\}\_\{c\}^\{m\}M\_\{c\}\\right\)\(30\)\+∑c∈𝒟αcoE¯co−∑c∈ℛg~cmMc\.\\displaystyle\+\\sum\_\{c\\in\\mathcal\{D\}\}\\alpha\_\{c\}^\{o\}\\overline\{E\}\_\{c\}^\{o\}\-\\sum\_\{c\\in\\mathcal\{R\}\}\\widetilde\{g\}\_\{c\}^\{m\}M\_\{c\}\.For every shared clusterc∈𝒯c\\in\\mathcal\{T\},

αco​E¯co−g~cm​Mc\\displaystyle\\alpha\_\{c\}^\{o\}\\overline\{E\}\_\{c\}^\{o\}\-\\widetilde\{g\}\_\{c\}^\{m\}M\_\{c\}=\(αco−g~cm\)​Mc\\displaystyle=\(\\alpha\_\{c\}^\{o\}\-\\widetilde\{g\}\_\{c\}^\{m\}\)M\_\{c\}\(31\)\+αco​\(E¯co−Mc\)\.\\displaystyle\+\\alpha\_\{c\}^\{o\}\(\\overline\{E\}\_\{c\}^\{o\}\-M\_\{c\}\)\.This yields

eroutem​\(x\)\\displaystyle e\_\{\\mathrm\{route\}\}^\{m\}\(x\):=∑c∈𝒯\(αco−g~cm\)​Mc,\\displaystyle:=\\sum\_\{c\\in\\mathcal\{T\}\}\(\\alpha\_\{c\}^\{o\}\-\\widetilde\{g\}\_\{c\}^\{m\}\)M\_\{c\},\(32\)eapproxm​\(x\)\\displaystyle e\_\{\\mathrm\{approx\}\}^\{m\}\(x\):=∑c∈𝒯αco​\(E¯co−Mc\),\\displaystyle:=\\sum\_\{c\\in\\mathcal\{T\}\}\\alpha\_\{c\}^\{o\}\(\\overline\{E\}\_\{c\}^\{o\}\-M\_\{c\}\),\(33\)epathm​\(x\)\\displaystyle e\_\{\\mathrm\{path\}\}^\{m\}\(x\):=∑c∈𝒟αco​E¯co−∑c∈ℛg~cm​Mc\.\\displaystyle:=\\sum\_\{c\\in\\mathcal\{D\}\}\\alpha\_\{c\}^\{o\}\\overline\{E\}\_\{c\}^\{o\}\-\\sum\_\{c\\in\\mathcal\{R\}\}\\widetilde\{g\}\_\{c\}^\{m\}M\_\{c\}\.\(34\)Consequently,

yorig−ymerge=eroutem\+eapproxm\+epathm\.y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{merge\}\}=e\_\{\\mathrm\{route\}\}^\{m\}\+e\_\{\\mathrm\{approx\}\}^\{m\}\+e\_\{\\mathrm\{path\}\}^\{m\}\.\(35\)To match the two\-source interpretation in Section[3](https://arxiv.org/html/2609.06076#S3), define

eexpert/pathm:=eapproxm\+epathm\.e\_\{\\mathrm\{expert/path\}\}^\{m\}:=e\_\{\\mathrm\{approx\}\}^\{m\}\+e\_\{\\mathrm\{path\}\}^\{m\}\.\(36\)Then

∥yorig−ymerge∥=‖eroutem\+eexpert/pathm‖\.\\lVert y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{merge\}\}\\rVert=\\left\\lVert e\_\{\\mathrm\{route\}\}^\{m\}\+e\_\{\\mathrm\{expert/path\}\}^\{m\}\\right\\rVert\.\(37\)

##### Full projected\-path coverage\.

If𝒟=∅\\mathcal\{D\}=\\emptyset, every cluster implied by the original active set is selected\. If alsoℛ=∅\\mathcal\{R\}=\\emptyset, only router reweighting and merged\-function approximation remain\. Ifℛ≠∅\\mathcal\{R\}\\neq\\emptyset, every required cluster is covered but additional clusters are activated\.

##### Partial projected\-path coverage\.

If𝒯≠∅\\mathcal\{T\}\\neq\\emptysetand𝒟≠∅\\mathcal\{D\}\\neq\\emptyset, all three terms in Eq\. \([35](https://arxiv.org/html/2609.06076#A4.E35)\) may be nonzero\.

##### Disjoint projected paths\.

If𝒯=∅\\mathcal\{T\}=\\emptyset, the shared\-cluster routing and approximation terms vanish under this decomposition, and the discrepancy is represented byepathme\_\{\\mathrm\{path\}\}^\{m\}\. Disjointness does not prove a universally maximal residual norm\.

##### Capacity\-limited coverage\.

Letkphysk\_\{\\mathrm\{phys\}\}be the maximum number of distinct physical merged experts that can be active for one token\. If\|𝒞proj\|\>kphys\|\\mathcal\{C\}\_\{\\mathrm\{proj\}\}\|\>k\_\{\\mathrm\{phys\}\}, then𝒟≠∅\\mathcal\{D\}\\neq\\emptysetfor every feasible𝒮m\\mathcal\{S\}\_\{m\}\. Thus, part of the projected original path is structurally unrepresentable by the merged routing capacity\.

Router\-only adjustment can alter physical cluster gate masses and may change𝒮m\\mathcal\{S\}\_\{m\}, but it cannot directly makeMcM\_\{c\}equal to the input\-dependent original cluster outputE¯co\\overline\{E\}\_\{c\}^\{o\}\. Expert\-inclusive adjustment can also change the merged functions, while full\-parameter adjustment can further compensate for their downstream effects\.

##### Sign convention for the propagation analysis\.

Underδℓ=h^ℓ−hℓ\\delta\_\{\\ell\}=\\widehat\{h\}\_\{\\ell\}\-h\_\{\\ell\}, the same\-input merged\-model source is

ϵℓm:=ymerge,ℓ−yorig,ℓ\\displaystyle\\epsilon\_\{\\ell\}^\{m\}:=y\_\{\\mathrm\{merge\},\\ell\}\-y\_\{\\mathrm\{orig\},\\ell\}\(38\)=−\(eroute,ℓm\+eexpert/path,ℓm\)\.\\displaystyle=\-\\left\(e\_\{\\mathrm\{route\},\\ell\}^\{m\}\+e\_\{\\mathrm\{expert/path\},\\ell\}^\{m\}\\right\)\.

## Appendix EExperiment Results by Compression Paradigm

This appendix breaks down the main results by compression paradigm\. While Section[5](https://arxiv.org/html/2609.06076#S5)reports averages over all pruning and merging checkpoints, here we separately analyze Expert Pruning and Expert Merging to verify that the observed trends are not driven by a single compression family\. Across both paradigms, the same qualitative pattern appears: causal LM fine\-tuning is generally stronger than KD under matched trainable scopes, router\-only adjustment provides only limited recovery, and full\-parameter fine\-tuning achieves the largest recovery gain\.

### E\.1Expert Pruning

![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/appendix/only_prune/prune_01a_total_restoration_gain_bar.png)Figure 7:Expert Pruning: Overall recovery gain of post\-compression adjustment strategies\. Bars show mean average\-score recovery gain, measured in percentage points \(pp\), over the unadjusted pruning baseline\. Results are averaged over pruning checkpoints, retention ratios, and benchmarks\.Full FTachieves the largest recovery, and causal LM fine\-tuning outperforms KD under matched trainable scopes\.Figures[7](https://arxiv.org/html/2609.06076#A5.F7),[8](https://arxiv.org/html/2609.06076#A5.F8),[9](https://arxiv.org/html/2609.06076#A5.F9), and[10](https://arxiv.org/html/2609.06076#A5.F10)summarize the results for Expert Pruning checkpoints\. The overall trend matches the main result:Full FTgives the largest recovery gain, followed byFull KDand expert\-inclusive FT strategies\. Router\-only adjustment remains a low\-cost but low\-recovery option, indicating that pruning\-induced degradation cannot be fully repaired by router reweighting alone\. The benchmark\-family view shows thatFull FTis especially effective for CoT and MCQA\. Finally, the scope\-progression plot shows a monotonic improvement as the trainable scope expands, with causal LM fine\-tuning outperforming KD at every matched scope\.

Figure 8:Expert Pruning: GPU cost–performance trade\-off of post\-compression adjustment strategies\. The y\-axis shows mean average\-score recovery gain, while the x\-axis shows effective GPU\-hours \(top\), GPU\-memory occupancy in GiB\-hours \(middle\), and GPU energy in kWh \(bottom\)\. Each point is an unweighted arithmetic mean over six pruning method–ratio settings \(two pruning methods×\\timesthree retention ratios\)\. Both expert\-selection cost and training\-loop cost are included when applicable\. Upper\-left is better\.Figure 9:Expert Pruning: Benchmark\-family recovery ofFull FT\. Bars show post\-adjustment performance, and dashed lines denote the original model and the unadjusted pruning baseline\.Full FTyields gains on CoT, MCQA, and general benchmarks, while math and coding decrease slightly\.![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/appendix/only_prune/prune_05_scope_progression_ft_kd_combined.png)Figure 10:Expert Pruning: Recovery gain as the trainable parameter scope expands\. For both LM fine\-tuning and KD, recovery increases from router\-only to router\+top\-kkexperts, router\+all\-experts, and full\-parameter adjustment\. LM fine\-tuning remains stronger than KD at each matched scope\.
### E\.2Expert Merging

![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/appendix/only_merge/merge_01a_total_restoration_gain_bar.png)Figure 11:Expert Merging: Overall recovery gain of post\-compression adjustment strategies\. Bars show mean average\-score recovery gain, measured in percentage points \(pp\), over the unadjusted merging baseline\. Results are averaged over merging checkpoints, retention ratios, and benchmarks\.Full FTachieves the largest recovery, and causal LM fine\-tuning consistently outperforms KD under matched trainable scopes\.Figures[11](https://arxiv.org/html/2609.06076#A5.F11),[12](https://arxiv.org/html/2609.06076#A5.F12),[13](https://arxiv.org/html/2609.06076#A5.F13)and[14](https://arxiv.org/html/2609.06076#A5.F14)report the same analysis for Expert Merging checkpoints\. Again,Full FTachieves the strongest recovery, and LM fine\-tuning consistently provides higher gains than KD under the same trainable scope\. The recovery gain is larger than in the pruning\-only breakdown, suggesting that merged checkpoints also leave substantial recoverable residual errors\. Router\+top\-kkstrategies improve over router\-only adjustment, but they do not dominate router\+all\-experts or full\-parameter adjustment once recovery gain and expert\-selection overhead are considered together\. The benchmark\-family and scope\-progression plots further confirm the same conclusion as the main text: small post\-compression adjustment is broadly useful, but expanding the trainable scope and using LM loss are key to strong recovery\.

Figure 12:Expert Merging: GPU cost–performance trade\-off of post\-compression adjustment strategies\. The y\-axis shows mean average\-score recovery gain, while the x\-axis shows effective GPU\-hours \(top\), GPU\-memory occupancy in GiB\-hours \(middle\), and GPU energy in kWh \(bottom\)\. Each point is an unweighted arithmetic mean over six merging method–ratio settings \(two merging methods×\\timesthree retention ratios\)\. Both expert\-selection cost and training\-loop cost are included when applicable\. Upper\-left is better\.Figure 13:Expert Merging: Benchmark\-family recovery ofFull FT\. Bars show post\-adjustment performance, and dashed lines denote the original model and the unadjusted merging baseline\.Full FTrecovers strongly on CoT, MCQA, general, and math benchmarks, while coding is essentially unchanged\.![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/appendix/only_merge/merge_05_scope_progression_ft_kd_combined.png)Figure 14:Expert Merging: Recovery gain as the trainable parameter scope expands\. For both LM fine\-tuning and KD, recovery increases as the trainable scope expands from router\-only to router\+top\-kkexperts, router\+all\-experts, and full\-parameter adjustment\. The large final jump to full\-parameter adjustment mirrors the main result and suggests that merging\-induced residuals are not confined to MoE expert parameters alone\.

## Appendix FExpert Editing

### F\.1Compatibility\-Mode Analysis for Expert Editing

##### Why Expert Editing is reported separately\.

Expert Editing methods reduce the number of expert parameters by factorizing or tensor\-decomposing expert weights while keeping the number of expert indices unchanged\. Representative methods include MoBE[Chen et al\. \(2025b\)](https://arxiv.org/html/2609.06076#bib.bib13)and TD\-MoE[XU et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib14), which compress expert internals through rank or tensor decomposition\. In principle, post\-compression adjustment should be performed directly on the native factorized representation in order to preserve the compressed parameterization\. However, native Expert Editing implementations often lack the optimized training and inference support available for standard MoE checkpoints\. Efficient end\-to\-end execution may require custom fused or mega\-kernel implementations for the factorized expert operators; without such kernels, native checkpoints can be substantially slower to train and evaluate than materialized standard\-MoE checkpoints\.

For this reason, we evaluate Expert Editing in a compatibility\-mode setting: we use the reconstructed standard\-MoE realization supplied by the editing method so that it can run in the same training, serving, and evaluation stack as the pruning and merging checkpoints\. These experiments test whether the materialized function produced by an editing compressor is amenable to post\-compression adjustment\. They should not be interpreted as showing that expert\-inclusive adjustment of the reconstructed checkpoint is parameterization\-equivalent to adjustment of the native factorization\.

##### Forward equivalence before adjustment\.

Letψ\\psidenote the native factor parameters, and letRRdenote the reconstruction map that materializes those factors as standard expert weights\. For expertiiand projection typeqq,

W^iq=Riq​\(ψ\)\.\\widehat\{W\}\_\{i\}^\{q\}=R\_\{i\}^\{q\}\(\\psi\)\.\(39\)For example, in a basis\-decomposition method,Riq​\(ψ\)R\_\{i\}^\{q\}\(\\psi\)may combine an expert\-specific factor with a linear combination of shared basis matrices\. The reconstructed checkpoint explicitly storesW^iq\\widehat\{W\}\_\{i\}^\{q\}\.

Before adjustment, the native and reconstructed operators implement the same materialized expert function up to numerical precision:

Einative​\(h,ψ\)=Eirecon​\(h,R⁡\(ψ\)\)\.E\_\{i\}^\{\\mathrm\{native\}\}\(h;\\psi\)=E\_\{i\}^\{\\mathrm\{recon\}\}\(h;R\(\\psi\)\)\.\(40\)Hence, when the router and all non\-expert parameters are identical,

fnative​\(x,ψ,θR\)=frecon​\(x,R⁡\(ψ\),θR\)f\_\{\\mathrm\{native\}\}\(x;\\psi,\\theta\_\{R\}\)=f\_\{\\mathrm\{recon\}\}\(x;R\(\\psi\),\\theta\_\{R\}\)\(41\)up to numerical precision\. The reconstructed checkpoint is therefore a valid functional realization of the native edited model at initialization\.

##### Router\-only adjustment is equivalent under frozen experts\.

If only router parameters are updated, bothψ\\psiandR⁡\(ψ\)R\(\\psi\)remain fixed\. Provided the native and reconstructed expert operators implement the same differentiable function, their loss values and router gradients are equal up to numerical precision:

∇θRℒnative​\(θR,ψ\)=∇θRℒrecon​\(θR,R⁡\(ψ\)\)\.\\nabla\_\{\\theta\_\{R\}\}\\mathcal\{L\}\_\{\\mathrm\{native\}\}\(\\theta\_\{R\};\\psi\)=\\nabla\_\{\\theta\_\{R\}\}\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\(\\theta\_\{R\};R\(\\psi\)\)\.\(42\)Thus, router\-only adjustment of the reconstructed standard\-MoE realization is functionally equivalent to router\-only adjustment of the native factorized model, subject to ordinary numerical differences between the two implementations\.

##### Expert\-inclusive and full adjustment are not generally equivalent\.

The equivalence does not extend to updates of expert parameters\. The native model moves in factor space, whereas the reconstructed model moves directly in materialized weight space\. To make this distinction explicit, consider one Euclidean gradient step; adaptive optimizers or weight decay introduce additional differences\.

For the reconstructed checkpoint,

W^t\+1=W^t−η​∇W^ℒ​\(W^t\)\.\\widehat\{W\}\_\{t\+1\}=\\widehat\{W\}\_\{t\}\-\\eta\\nabla\_\{\\widehat\{W\}\}\\mathcal\{L\}\(\\widehat\{W\}\_\{t\}\)\.\(43\)For the native factorization,

ψt\+1=ψt−η​∇ψℒ​\(R⁡\(ψt\)\)\.\\psi\_\{t\+1\}=\\psi\_\{t\}\-\\eta\\nabla\_\{\\psi\}\\mathcal\{L\}\(R\(\\psi\_\{t\}\)\)\.\(44\)WritingJR​\(ψt\)J\_\{R\}\(\\psi\_\{t\}\)for the Jacobian of the reconstruction map andW^t=R⁡\(ψt\)\\widehat\{W\}\_\{t\}=R\(\\psi\_\{t\}\), the chain rule gives

∇ψℒ​\(R⁡\(ψt\)\)=JR​\(ψt\)⊤​∇W^ℒ​\(W^t\)\.\\nabla\_\{\\psi\}\\mathcal\{L\}\(R\(\\psi\_\{t\}\)\)=J\_\{R\}\(\\psi\_\{t\}\)^\{\\top\}\\nabla\_\{\\widehat\{W\}\}\\mathcal\{L\}\(\\widehat\{W\}\_\{t\}\)\.\(45\)A first\-order Taylor expansion of the induced materialized update yields

R⁡\(ψt\+1\)\\displaystyle R\(\\psi\_\{t\+1\}\)=W^t−η​JR​\(ψt\)​JR​\(ψt\)⊤\\displaystyle=\\widehat\{W\}\_\{t\}\-\\eta J\_\{R\}\(\\psi\_\{t\}\)J\_\{R\}\(\\psi\_\{t\}\)^\{\\top\}\(46\)⋅∇W^ℒ​\(W^t\)\+O⁡\(η2\)\.\\displaystyle\\cdot\\nabla\_\{\\widehat\{W\}\}\\mathcal\{L\}\(\\widehat\{W\}\_\{t\}\)\+O\(\\eta^\{2\}\)\.The first\-order native update equals the direct materialized update only if

JR​\(ψt\)​JR​\(ψt\)⊤​∇W^ℒ​\(W^t\)\\displaystyle J\_\{R\}\(\\psi\_\{t\}\)J\_\{R\}\(\\psi\_\{t\}\)^\{\\top\}\\nabla\_\{\\widehat\{W\}\}\\mathcal\{L\}\(\\widehat\{W\}\_\{t\}\)=∇W^ℒ​\(W^t\)\.\\displaystyle=\\nabla\_\{\\widehat\{W\}\}\\mathcal\{L\}\(\\widehat\{W\}\_\{t\}\)\.\(47\)This is a restrictive fixed\-point condition\. Membership of the loss gradient in the tangent spacerange⁡\(JR​\(ψt\)\)\\operatorname\{range\}\(J\_\{R\}\(\\psi\_\{t\}\)\)is necessary, but is not sufficient in general, becauseJR​JR⊤J\_\{R\}J\_\{R\}^\{\\top\}need not be the orthogonal projector or the identity on that tangent space\. Equality additionally requiresJR​JR⊤J\_\{R\}J\_\{R\}^\{\\top\}to act as the identity on the particular gradient direction\. Shared bases or tensor factors can also couple the updates of multiple experts, whereas direct reconstructed\-weight adjustment can move materialized expert matrices more freely\.

Accordingly, expert\-inclusive and full\-parameter adjustment of a reconstructed Expert Editing checkpoint should be interpreted as an unconstrained compatibility\-mode update in materialized weight space, not as an update equivalent to optimizing the native compressed parameterization\.

##### Empirical trend and interpretation\.

Although compatibility\-mode Expert Editing is excluded from the primary compressed\-artifact cost comparison, we report it for completeness\. The observed trend is broadly consistent with the pruning and merging results: small post\-compression adjustment improves the reconstructed checkpoints, causal LM fine\-tuning tends to outperform pure teacher KD under the same small budget, router\-only adjustment gives limited but nonzero gains, and recovery increases as the trainable scope expands\. Because expert\-inclusive and full updates do not preserve the native factorized constraint, these results are supplementary evidence about the materialized edited function rather than primary evidence about deployable native Expert Editing artifacts\.

### F\.2Active\-Set Formulation for Expert Editing

We adopt the Expert Editing notation and active\-set analysis of[Hyeon and Do \(2026\)](https://arxiv.org/html/2609.06076#bib.bib61), but reorganize the residual into routing, expert\-function, and active\-path components that match the adjustment\-scope analysis in Section[3](https://arxiv.org/html/2609.06076#S3)\. To connect the formulation directly to Appendix[F\.1](https://arxiv.org/html/2609.06076#A6.SS1), we denote the materialized edited expert byE^i\\widehat\{E\}\_\{i\}\.

Throughout Appendices F\.2–F\.6,xxdenotes a common local hidden\-state input supplied to both the original and edited layer maps\. The end\-to\-end state shift between the two networks is handled by the propagation analysis in Section[3](https://arxiv.org/html/2609.06076#S3)\. Expert Editing preserves the routed\-expert index setℰ=\{0,…,N−1\}\\mathcal\{E\}=\\\{0,\\ldots,N\-1\\\}but replacesEiE\_\{i\}byE^i\\widehat\{E\}\_\{i\}\.

Let𝒮\\mathcal\{S\}and𝒮e:=𝒮edit\\mathcal\{S\}\_\{e\}:=\\mathcal\{S\}^\{\\mathrm\{edit\}\}be the top\-kkactive sets in the original and edited layer maps at the reference inputxx, respectively\. Letgiedit​\(x\)≥0g\_\{i\}^\{\\mathrm\{edit\}\}\(x\)\\geq 0denote the edited model’s gate mass, not a raw router logit\. For any nonempty active setA⊆ℰA\\subseteq\\mathcal\{E\}satisfying∑j∈Agjedit​\(x\)\>0\\sum\_\{j\\in A\}g\_\{j\}^\{\\mathrm\{edit\}\}\(x\)\>0, define

g~ie,𝒜​\(x\):=giedit​\(x\)∑j∈𝒜gjedit​\(x\),i∈𝒜\.\\widetilde\{g\}\_\{i\}^\{e,\\mathcal\{A\}\}\(x\):=\\frac\{g\_\{i\}^\{\\mathrm\{edit\}\}\(x\)\}\{\\sum\_\{j\\in\\mathcal\{A\}\}g\_\{j\}^\{\\mathrm\{edit\}\}\(x\)\},\\qquad i\\in\\mathcal\{A\}\.\(48\)The two routed outputs are

yorig​\(x\)\\displaystyle y\_\{\\mathrm\{orig\}\}\(x\)=∑i∈𝒮g~io,𝒮​\(x\)​Ei​\(x\),\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\}\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\(x\)E\_\{i\}\(x\),\(49\)yedit​\(x\)\\displaystyle y\_\{\\mathrm\{edit\}\}\(x\)=∑i∈𝒮eg~ie,𝒮e​\(x\)​E^i​\(x\)\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\_\{e\}\}\\widetilde\{g\}\_\{i\}^\{e,\\mathcal\{S\}\_\{e\}\}\(x\)\\widehat\{E\}\_\{i\}\(x\)\.\(50\)For compact displays, write

g~io\\displaystyle\\widetilde\{g\}\_\{i\}^\{o\}:=g~io,𝒮​\(x\),\\displaystyle:=\\widetilde\{g\}\_\{i\}^\{o,\\mathcal\{S\}\}\(x\),g~ie\\displaystyle\\widetilde\{g\}\_\{i\}^\{e\}:=g~ie,𝒮e​\(x\),\\displaystyle:=\\widetilde\{g\}\_\{i\}^\{e,\\mathcal\{S\}\_\{e\}\}\(x\),\(51\)Ei\\displaystyle E\_\{i\}:=Ei​\(x\),\\displaystyle:=E\_\{i\}\(x\),E^i\\displaystyle\\widehat\{E\}\_\{i\}:=E^i​\(x\)\.\\displaystyle:=\\widehat\{E\}\_\{i\}\(x\)\.Define

𝒯\\displaystyle\\mathcal\{T\}=𝒮∩𝒮e,\\displaystyle=\\mathcal\{S\}\\cap\\mathcal\{S\}\_\{e\},𝒟\\displaystyle\\mathcal\{D\}=𝒮∖𝒮e,\\displaystyle=\\mathcal\{S\}\\setminus\\mathcal\{S\}\_\{e\},\(52\)ℛ\\displaystyle\\mathcal\{R\}=𝒮e∖𝒮\.\\displaystyle=\\mathcal\{S\}\_\{e\}\\setminus\\mathcal\{S\}\.The exact residual is

yorig−yedit\\displaystyle y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{edit\}\}=∑i∈𝒯\(g~io​Ei−g~ie​E^i\)\\displaystyle=\\sum\_\{i\\in\\mathcal\{T\}\}\\left\(\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\widetilde\{g\}\_\{i\}^\{e\}\\widehat\{E\}\_\{i\}\\right\)\(53\)\+∑i∈𝒟g~ioEi−∑i∈ℛg~ieE^i\.\\displaystyle\+\\sum\_\{i\\in\\mathcal\{D\}\}\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\sum\_\{i\\in\\mathcal\{R\}\}\\widetilde\{g\}\_\{i\}^\{e\}\\widehat\{E\}\_\{i\}\.For every shared indexi∈𝒯i\\in\\mathcal\{T\},

g~io​Ei−g~ie​E^i\\displaystyle\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\widetilde\{g\}\_\{i\}^\{e\}\\widehat\{E\}\_\{i\}=\(g~io−g~ie\)​E^i\\displaystyle=\(\\widetilde\{g\}\_\{i\}^\{o\}\-\\widetilde\{g\}\_\{i\}^\{e\}\)\\widehat\{E\}\_\{i\}\(54\)\+g~io​\(Ei−E^i\)\.\\displaystyle\+\\widetilde\{g\}\_\{i\}^\{o\}\(E\_\{i\}\-\\widehat\{E\}\_\{i\}\)\.We therefore define

eroutee​\(x\)\\displaystyle e\_\{\\mathrm\{route\}\}^\{e\}\(x\):=∑i∈𝒯\(g~io−g~ie\)​E^i,\\displaystyle:=\\sum\_\{i\\in\\mathcal\{T\}\}\(\\widetilde\{g\}\_\{i\}^\{o\}\-\\widetilde\{g\}\_\{i\}^\{e\}\)\\widehat\{E\}\_\{i\},\(55\)efunce​\(x\)\\displaystyle e\_\{\\mathrm\{func\}\}^\{e\}\(x\):=∑i∈𝒯g~io​\(Ei−E^i\),\\displaystyle:=\\sum\_\{i\\in\\mathcal\{T\}\}\\widetilde\{g\}\_\{i\}^\{o\}\(E\_\{i\}\-\\widehat\{E\}\_\{i\}\),\(56\)epathe​\(x\)\\displaystyle e\_\{\\mathrm\{path\}\}^\{e\}\(x\):=∑i∈𝒟g~io​Ei−∑i∈ℛg~ie​E^i\.\\displaystyle:=\\sum\_\{i\\in\\mathcal\{D\}\}\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\sum\_\{i\\in\\mathcal\{R\}\}\\widetilde\{g\}\_\{i\}^\{e\}\\widehat\{E\}\_\{i\}\.\(57\)Thus,

yorig−yedit=eroutee\+efunce\+epathe\.y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{edit\}\}=e\_\{\\mathrm\{route\}\}^\{e\}\+e\_\{\\mathrm\{func\}\}^\{e\}\+e\_\{\\mathrm\{path\}\}^\{e\}\.\(58\)If the editing compressor leaves the router and its gate implementation unchanged, then at a common inputxx,g~ie=g~io\\widetilde\{g\}\_\{i\}^\{e\}=\\widetilde\{g\}\_\{i\}^\{o\}and𝒮e=𝒮\\mathcal\{S\}\_\{e\}=\\mathcal\{S\}before adjustment\. In that special case,eroutee=epathe=0e\_\{\\mathrm\{route\}\}^\{e\}=e\_\{\\mathrm\{path\}\}^\{e\}=0locally, and the initial same\-input residual is purely an expert\-function approximation term\. Realized routing shifts in deeper layers can still arise through the upstream hidden\-state error described in Section[3](https://arxiv.org/html/2609.06076#S3)\. We retain the general three\-way identity because it also covers altered gate implementations and post\-compression router adjustment\.

### F\.3Best Scenario: Preserved Active Path

If𝒮e=𝒮\\mathcal\{S\}\_\{e\}=\\mathcal\{S\}, then𝒯=𝒮\\mathcal\{T\}=\\mathcal\{S\}and𝒟=ℛ=∅\\mathcal\{D\}=\\mathcal\{R\}=\\emptyset\. Therefore,

yorig−yedit=eroutee\+efunce,y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{edit\}\}=e\_\{\\mathrm\{route\}\}^\{e\}\+e\_\{\\mathrm\{func\}\}^\{e\},\(59\)with

yorig−yedit\\displaystyle y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{edit\}\}=∑i∈𝒮\(g~io−g~ie\)​E^i\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\}\(\\widetilde\{g\}\_\{i\}^\{o\}\-\\widetilde\{g\}\_\{i\}^\{e\}\)\\widehat\{E\}\_\{i\}\(60\)\+∑i∈𝒮g~io\(Ei−E^i\)\.\\displaystyle\+\\sum\_\{i\\in\\mathcal\{S\}\}\\widetilde\{g\}\_\{i\}^\{o\}\(E\_\{i\}\-\\widehat\{E\}\_\{i\}\)\.Preserving the selected indices does not imply a zero residual\. In the general identity, shared\-index router reweighting and expert\-function approximation may both remain\. If the router is unchanged and both layer maps receive the same input, the first term vanishes and only the function approximation term remains\. No ordering between the two terms is implied without additional assumptions\.

### F\.4Most Common Scenario: Partial Active\-Set Overlap

If0<\|𝒯\|<k0<\|\\mathcal\{T\}\|<k, then𝒟≠∅\\mathcal\{D\}\\neq\\emptysetandℛ≠∅\\mathcal\{R\}\\neq\\emptyset\. All three terms in Eq\. \([58](https://arxiv.org/html/2609.06076#A6.E58)\) may be nonzero:erouteee\_\{\\mathrm\{route\}\}^\{e\}measures gate\-weight disagreement on shared indices,efuncee\_\{\\mathrm\{func\}\}^\{e\}measures function mismatch for those shared indices, andepathee\_\{\\mathrm\{path\}\}^\{e\}compares original contributions that leave the active path with newly activated edited\-expert contributions\. This separates within\-index editing error from cross\-index path replacement\.

### F\.5Worst Scenario: Disjoint Active Paths

If𝒮∩𝒮e=∅\\mathcal\{S\}\\cap\\mathcal\{S\}\_\{e\}=\\emptyset, then𝒯=∅\\mathcal\{T\}=\\emptyset,𝒟=𝒮\\mathcal\{D\}=\\mathcal\{S\}, andℛ=𝒮e\\mathcal\{R\}=\\mathcal\{S\}\_\{e\}\. Under the shared\-index decomposition,

eroutee​\(x\)=efunce​\(x\)=0,e\_\{\\mathrm\{route\}\}^\{e\}\(x\)=e\_\{\\mathrm\{func\}\}^\{e\}\(x\)=0,\(61\)and

yorig−yedit\\displaystyle y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{edit\}\}=epathe​\(x\)\\displaystyle=e\_\{\\mathrm\{path\}\}^\{e\}\(x\)\(62\)=∑i∈𝒮g~io​Ei−∑i∈𝒮eg~ie​E^i\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\}\\widetilde\{g\}\_\{i\}^\{o\}E\_\{i\}\-\\sum\_\{i\\in\\mathcal\{S\}\_\{e\}\}\\widetilde\{g\}\_\{i\}^\{e\}\\widehat\{E\}\_\{i\}\.This is complete active\-path replacement\. The term “worst” names the disjoint\-set scenario; it does not establish that the residual norm is maximal for every input\.

### F\.6Connection to Adjustment Scope

To match the two\-source interpretation in Section[3](https://arxiv.org/html/2609.06076#S3), define

eexpert/pathe:=efunce\+epathe\.e\_\{\\mathrm\{expert/path\}\}^\{e\}:=e\_\{\\mathrm\{func\}\}^\{e\}\+e\_\{\\mathrm\{path\}\}^\{e\}\.\(63\)Then

yorig−yedit=eroutee\+eexpert/pathe\.y\_\{\\mathrm\{orig\}\}\-y\_\{\\mathrm\{edit\}\}=e\_\{\\mathrm\{route\}\}^\{e\}\+e\_\{\\mathrm\{expert/path\}\}^\{e\}\.\(64\)Because Section[3](https://arxiv.org/html/2609.06076#S3)usesδℓ=h^ℓ−hℓ\\delta\_\{\\ell\}=\\widehat\{h\}\_\{\\ell\}\-h\_\{\\ell\}, the same\-input edited\-model source is

ϵℓe:=yedit,ℓ−yorig,ℓ=−\(eroute,ℓe\+eexpert/path,ℓe\)\.\\epsilon\_\{\\ell\}^\{e\}:=y\_\{\\mathrm\{edit\},\\ell\}\-y\_\{\\mathrm\{orig\},\\ell\}=\-\\left\(e\_\{\\mathrm\{route\},\\ell\}^\{e\}\+e\_\{\\mathrm\{expert/path\},\\ell\}^\{e\}\\right\)\.\(65\)
Under router\-only adjustment, the materialized expert functionsE^i\\widehat\{E\}\_\{i\}remain fixed\. The router can changeerouteee\_\{\\mathrm\{route\}\}^\{e\}and, by moving top\-kkboundaries, may also change𝒯\\mathcal\{T\},𝒟\\mathcal\{D\}, andℛ\\mathcal\{R\}\. It cannot, however, change the expert functions themselves or directly eliminateEi−E^iE\_\{i\}\-\\widehat\{E\}\_\{i\}\. As established in Appendix[F\.1](https://arxiv.org/html/2609.06076#A6.SS1), router\-only adjustment is functionally equivalent between the native factorized model and its reconstructed standard\-MoE realization, up to numerical precision\.

Expert\-inclusive or full adjustment of the reconstructed checkpoint can also changeE^i\\widehat\{E\}\_\{i\}, but these updates occur in unconstrained materialized weight space and are not generally equivalent to native factor updates\. Full\-parameter adjustment additionally exposes dense/shared correction directions for downstream effects ofϵℓe\\epsilon\_\{\\ell\}^\{e\}, consistent with Section[3](https://arxiv.org/html/2609.06076#S3)\.

### F\.7Expert Editing Experiment Results

![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/appendix/only_edit/edit_01a_total_restoration_gain_bar.png)Figure 15:Compatibility\-mode Expert Editing: Overall recovery gain of post\-compression adjustment strategies\. Bars show the mean average\-score recovery gain, measured in percentage points \(pp\), over the unadjusted reconstructed Expert Editing baseline\.Full FTachieves the largest recovery\. LM fine\-tuning outperforms KD at every expert\-inclusive and full\-parameter matched scope; the router\-only pair is the sole exception, whereRouter\-only KDyields\+0\.36\+0\.36pp versus\+0\.25\+0\.25pp forRouter\-only FT\. Router\-only adjustment provides only limited gains\. These results are reported as supplementary compatibility\-mode evidence because adjustment is performed on reconstructed standard\-MoE weights rather than native factorized Expert Editing parameters\.Figures[15](https://arxiv.org/html/2609.06076#A6.F15),[16](https://arxiv.org/html/2609.06076#A6.F16)and[17](https://arxiv.org/html/2609.06076#A6.F17)summarize the compatibility\-mode Expert Editing results\. Although these results are excluded from the main deployable compressed\-artifact comparison, they show trends broadly consistent with the main Expert Pruning and Expert Merging experiments\. First, small post\-compression adjustment improves the reconstructed Expert Editing checkpoints, withFull FTachieving the largest average recovery gain\. Second, causal LM fine\-tuning is stronger than token\-level KD at every expert\-inclusive and full\-parameter matched scope; the router\-only pair is the sole exception, whereRouter\-only KDyields\+0\.36\+0\.36pp versus\+0\.25\+0\.25pp forRouter\-only FT\. Third, router\-only adjustment gives limited but nonzero gains, while expanding the trainable scope from router\-only to router\+experts and full\-parameter adjustment monotonically improves recovery\. These observations suggest that Expert Editing checkpoints are also adjustment\-friendly\. However, because these experiments update reconstructed standard\-MoE weights rather than native factorized Expert Editing parameters, they should be interpreted as supplementary compatibility\-mode evidence rather than primary evidence for compressed\-artifact efficiency\.

Figure 16:Compatibility\-mode Expert Editing: Benchmark\-family recovery ofFull FT\. Bars show post\-adjustment performance, and dashed lines denote the original model and the unadjusted reconstructed Expert Editing baseline\.Full FTyields large gains on CoT and MCQA benchmarks, while math shows the weakest recovery in this setting\.![Refer to caption](https://arxiv.org/html/2609.06076v1/Figures/appendix/only_edit/edit_05_scope_progression_ft_kd_combined.png)Figure 17:Compatibility\-mode Expert Editing: Recovery gain as the trainable parameter scope expands\. For both LM fine\-tuning and KD, recovery increases from router\-only adjustment to router\+top\-kkexperts, router\+all\-experts, and full\-parameter adjustment\. The same monotonic scope\-expansion trend observed in the main pruning/merging experiments also appears for reconstructed Expert Editing checkpoints\.

## Appendix GQualitative Examples of Generation\-Level Recovery

In Section[5](https://arxiv.org/html/2609.06076#S5), we observe thatFull FTrecovers most strongly on general, CoT, and MCQA\-style benchmarks, where fluent continuation and stable natural\-language generation are important\. Here we provide representative qualitative examples illustrating the type of generation\-level degradation that appears after compression and how causal LM fine\-tuning repairs it\.

Figure[18](https://arxiv.org/html/2609.06076#A7.F18)shows two cases: an Expert Pruning checkpoint on Gemma4 with AIMER at 50% retention, and an Expert Merging checkpoint on Qwen3 with HC\-SMoE at 62\.5% retention\. Before adjustment, both compressed models exhibit degenerate continuation patterns, such as repeated tokens or broken word fragments\. AfterFull FT, the same prompts produce substantially more fluent and coherent explanations without any task\-specific supervision\. These examples are not intended as a comprehensive evaluation of generation quality, but they support the interpretation that small causal LM adjustment can restore the model’s ability to express and compose its remaining knowledge in natural language\.

Example 1: Expert pruning, Gemma4 AIMER 50%\. Prompt:what are forces \(gravity, magnetism\) made of?

Compressed model beforeFull FT

> Imagine you are playing in a playground\. You see a ball rolling down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down down ……

AfterFull FT

> Imagine you are playing in a playground with a big, heavy ball\. If you throw that ball down, it does not just go away and disappear\. It stays in the ground, and if you throw it up, it stays in the sky\. It does not just float on its own; it stays there because it is “stuck” to the ground or the sky\. That “stickiness” is what we call gravity\. …… When we talk about forces like gravity or magnetism, we are talking about the rules of the universe\. These rules are made of the way things are connected\.

Example 2: Expert merging, Qwen3 HC\-SMoE 62\.5%\. Prompt:How do overlays in NFL look so real? Like paint? How are they so accurate?

Compressed model beforeFull FT

> Okay, let us talk about this like you are a kid who is just learning how to talk\. So imagine you are watching a video of a football game, and you see this thing called an “overlay\.” It looks like a flat layer of color that just sits on the ground, like paint\. But here is the cool part: even though it looks like paint, it is not really paint at all\. It is not like paint that you can touch or that you can walk on and it will fl fl fl fl fl fl fl fl fl fl fl fl fl fl fl ……

AfterFull FT

> Okay, imagine you are playing football and you are on the field\. You are running, jumping, dodging, and suddenly you see these bright lines and colors that look like paint on the grass\. They are not just lines; they are little signs that show where the ball is going and where the players are\. …… Instead, they use a digital overlay technology\. The computer knows exactly where every player is, where the ball is, and where the field lines are, so it draws the graphics in real time and makes them look as if they belong to the field\.

Figure 18:Qualitative examples of generation\-level recovery afterFull FT\. We compare outputs before and after causal LM full\-parameter fine\-tuning for two compressed checkpoints: Gemma4 AIMER at 50% retention and Qwen3 HC\-SMoE at 62\.5% retention\. Before adjustment, the compressed models exhibit degenerate generation patterns such as token repetition and broken fragments\. AfterFull FT, the same prompts yield more fluent, coherent, and complete natural\-language explanations without task\-specific supervision\.
## Appendix HRelated Works

Efficient LLM execution is increasingly important not only for serving and deploying individual models[Kwon et al\. \(2023\)](https://arxiv.org/html/2609.06076#bib.bib50);[Sheng et al\. \(2023\)](https://arxiv.org/html/2609.06076#bib.bib62);[Kim et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib68);[Yu et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib63), but also for multi\-agent systems and frameworks, where a single task may require multiple model invocations[Li et al\. \(2023\)](https://arxiv.org/html/2609.06076#bib.bib64);[Wu et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib65);[Hyeon et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib67);[Hong et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib66)\. Against this broader efficiency backdrop, we focus on reducing the deployment cost of Mixture\-of\-Experts \(MoE\) LLMs through parameter\-level compression and small post\-compression adjustment\.

We consider methods that reduce the number of expert parameters rather than methods that change the bit representation of individual parameters\. Following the taxonomy introduced in the previous study[Hyeon and Do \(2026\)](https://arxiv.org/html/2609.06076#bib.bib61), we useExpert Pruning,Expert Editing, andExpert Mergingas descriptive categories for parameter\-level MoE compression\. These categories differ in whether they remove experts, re\-parameterize expert internals, or aggregate multiple experts into a smaller expert set\.

### H\.1Compression Operators and Their Consequences

##### Removing experts\.

Expert Pruning reduces the feasible expert set by deleting experts that are estimated to be redundant or weakly contributive\. Existing approaches differ mainly in how they estimate expert importance\. NAEE[Lu et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib4)searches for the retained expert combination that minimizes layer\-wise reconstruction loss on calibration data, whereas DiEP[Bai et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib5)relaxes discrete expert selection into a differentiable optimization problem\. REAP[Lasby et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib6)argues that one\-shot merging can collapse functional subspaces and instead scores experts using both router gate values and expert\-output magnitudes\. CFES[Liu et al\. \(2026a\)](https://arxiv.org/html/2609.06076#bib.bib15)derives a layer\-wise output\-discrepancy bound and uses it to replace an expensive global search with a coarse\-to\-fine layer\-wise procedure\. Other criteria exploit domain\-conditioned expert\-output statistics and token variation[Dong et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib7), activation trajectories accumulated across layers[Yang et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib10), or unified evaluations of alternative expert\-dropping signals[Jaiswal et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib8)\. Beyond removing individual experts, coarser approaches also consider deleting complete MoE layers or transformer blocks[He et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib9)\.

##### Aggregating experts\.

Expert Merging maps multiple original experts to a smaller set of synthesized experts\. Its motivation is related to model\-merging results showing that related parameter sets can often be combined while retaining useful behavior[Wortsman et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib19);[Matena and Raffel \(2022\)](https://arxiv.org/html/2609.06076#bib.bib20)\. HC\-SMoE[Chen et al\. \(2025a\)](https://arxiv.org/html/2609.06076#bib.bib16)clusters experts using output similarity measured on calibration data\. PuzzleMoE[Zhao et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib17)uses an entry\-wise similarity mask together with an activation\-weighted saliency mask to merge selected parameters while retaining expert\-specific information\. MergeMoE[Miao et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib18)formulates merging in output space and estimates a compression matrix by least squares\. In each case, aggregation changes the correspondence between router decisions and the expert functions that are ultimately executed\.

##### Re\-parameterizing experts\.

Expert Editing keeps the logical expert count fixed while replacing each expert with a lower\-dimensional, shared, or tensorized representation\. MoE\-SVD[Li et al\. \(2025b\)](https://arxiv.org/html/2609.06076#bib.bib11)applies singular value decomposition to expert weight matrices\. MoLAE[Liu et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib12)separates a shared latent mapping from expert\-specific transformations, while MoBE[Chen et al\. \(2025b\)](https://arxiv.org/html/2609.06076#bib.bib13)represents each expert through an expert\-specific component and a learned combination of shared basis matrices\. TD\-MoE[XU et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib14)models experts jointly as correlated tensors and applies whitening followed by Tucker\-style decomposition\. Although these methods preserve routing slots, the functions associated with those slots are modified, so the pre\-compression router may no longer be optimally matched to the edited experts\.

##### Combining compression operators\.

Hybrid methods combine removal, aggregation, and low\-rank re\-parameterization in a single pipeline\. DM\-MoE[Li et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib21)adaptively drops experts before merging the remaining ones, while DERN[Zhou et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib22)prunes experts, reallocates their neurons, and recombines the resulting components\. EEP[Liu et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib23)couples gradient\-free expert pruning with knowledge\-preserving merging\. MoNE[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib24)replaces redundant experts with lightweight, input\-agnostic novice vectors, and MoE\-I2[Yang et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib25)combines inter\-expert pruning, intra\-expert low\-rank decomposition, and LoRA\-based recovery\.D2D^\{2\}\-MoE[Gu et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib26)represents experts as low\-rank edits around a shared base, whereas MC\-SMoE[Li et al\. \(2024b\)](https://arxiv.org/html/2609.06076#bib.bib27)first integrates experts using routing\-policy statistics and then applies low\-rank compression\. These methods illustrate that the residual error after compression can depend on several coupled transformations rather than on a single operator\.

### H\.2Post\-Compression Adjustment and Recovery Training

Several MoE compression methods attach a recovery procedure to a particular compressor\. MoE\-Pruner[Xie et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib52)applies expert\-wise knowledge distillation after pruning\. CD\-MoE[Cao et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib53)condenses sparse MoE layers and fine\-tunes only the condensed components\. MoE\-I2[Yang et al\. \(2024\)](https://arxiv.org/html/2609.06076#bib.bib25)uses LoRA after its pruning\-and\-decomposition pipeline, while Sub\-MoE[Li et al\. \(2025a\)](https://arxiv.org/html/2609.06076#bib.bib56)reports additional gains from fine\-tuning after subspace expert merging\. At substantially larger training scales, SlimMoE[Li et al\. \(2025c\)](https://arxiv.org/html/2609.06076#bib.bib54)combines structured expert slimming with distillation, and SlimQwen[Tang et al\. \(2026\)](https://arxiv.org/html/2609.06076#bib.bib55)studies pruning, distillation, and continued training as part of large\-scale MoE model construction\. Beyond these compressor\-specific recovery procedures, prior work analyzes router–expert mismatch across pruning, editing, and merging and introduces Router Knowledge Distillation \(Router KD\), which distills the original model’s next\-token distribution while updating only the compressed model’s router[Hyeon and Do \(2026\)](https://arxiv.org/html/2609.06076#bib.bib61)\. The router\-only KD setting evaluated here corresponds to Router KD in its objective and trainable parameter scope; we include it as a baseline within a broader comparison of training objectives and parameter\-update scopes\.

These studies show that compressed MoE checkpoints can remain recoverable, but their recovery procedures are generally coupled to one compressor, one objective, or one predefined trainable scope\. To our knowledge, the works reviewed above do not provide a matched comparison on the same compressed checkpoints and under the same small data budget of \(i\) causal LM fine\-tuning versus teacher\-based KD, \(ii\) router\-only, selected\-expert, all\-expert, and full\-parameter updates, and \(iii\) recovery gain versus end\-to\-end GPU cost\. This distinction matters because a smaller trainable parameter set does not automatically imply lower total cost: expert\-subset strategies can require a separate selection pass, and KD additionally requires teacher forward passes\.

The present paper therefore treatsPost\-Compression Adjustmentitself as the object of study rather than as a method\-specific add\-on\. Under a matched small\-data protocol, we compare causal LM fine\-tuning and teacher\-based KD across multiple trainable scopes, and we measure both performance recovery and GPU cost\. The main experiments apply this protocol to pruning and merging checkpoints across multiple retention ratios and MoE backbones, while the Expert Editing appendix reports supplementary compatibility\-mode evidence\. This design asks a practical question left open by compressor\-specific recovery studies:given an already\-compressed MoE LLM, which adjustment objective and parameter scope provide the strongest cost–recovery trade\-off?

## Appendix IAdditional Calibration Robustness

We conduct two complementary checks of whether the aggregate Post\-Compression Adjustment \(PCA\) conclusions depend on the size or domain of the calibration corpus\. Both checks retain the one\-epoch adjustment protocol, adjustment objectives and scopes, evaluation procedure, and the same 28 downstream benchmarks; only the calibration budget or corpus is changed\. Figures[5](https://arxiv.org/html/2609.06076#S6.F5)and[6](https://arxiv.org/html/2609.06076#S6.F6)summarize the main patterns, while Tables[5](https://arxiv.org/html/2609.06076#A9.T5)and[6](https://arxiv.org/html/2609.06076#A9.T6)report the strategy\-level performance and measured costs\.

##### Smaller calibration budget\.

We reduce the C4 adjustment budget from 3,000 to 1,024 examples for four backbone–compressor pairs—Qwen3/REAP, Qwen3/HC\-SMoE, Gemma4/AIMER, and Gemma4/M\-SMoE—at 50%, 62\.5%, and 75% expert\-retention ratios\. Table[5](https://arxiv.org/html/2609.06076#A9.T5)averages the resulting 12 settings over all 28 evaluation benchmarks\. Although the smaller budget reduces the amount of available adjustment data, the leading pattern is preserved: Full FT gives the largest mean recovery \(\+2\.78\+2\.78pp\), followed by Router All\-Expert FT \(\+1\.86\+1\.86pp\) and Full KD \(\+1\.77\+1\.77pp\), and FT remains ahead of KD at every matched trainable scope\. The table additionally reports end\-to\-end GPU energy, effective GPU\-hours, and GPU\-memory occupancy in GiB\-hours, including expert\-selection and online\-teacher stages when applicable\.

Table 5:Adjustment with a 1,024\-example C4 calibration budget\. Values are macro\-averaged over 28 evaluation benchmarks and 12 backbone–compressor–retention settings \(four backbone–compressor pairs at three retention ratios\)\.Δ\\Deltais the score change from the unadjusted compressed model in percentage points\. Costs are end\-to\-end means; Mem\. is GPU\-memory occupancy in GiB\-hours\. Original and Compressed Only require no adjustment and therefore have no adjustment cost\. The Original and Compressed Only scores for this check were re\-evaluated; their aggregate gap differs from Table[8](https://arxiv.org/html/2609.06076#A10.T8)by evaluation\-level noise of about 0\.05 pp\.
##### Calibration\-domain shift\.

We next compare C4 with the math\-domain OpenR1\-Math\-220k dataset[Lozhkov et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib69)under the same 1,024\-example calibration budget\. The OpenR1\-Math runs use the first 1,024 examples from the dataset’sall/trainsplit\. We evaluate six matched settings: Qwen3/HC\-SMoE and Gemma4/AIMER, each at 50%, 62\.5%, and 75% expert retention\. The unadjusted checkpoints and the 28\-task evaluation suite are held fixed, so the comparison isolates the calibration\-corpus change within these settings\.

Table[6](https://arxiv.org/html/2609.06076#A9.T6)shows that the aggregate strategy\-level ordering is strongly associated across the two domains \(Spearmanρ=0\.973\\rho=0\.973; Kendallτb=0\.897\\tau\_\{b\}=0\.897\)\. Full FT has the largest two\-domain mean recovery \(4\.04 pp\), and the same three strategies—Full FT, Full KD, and Router All\-Expert FT—form the top\-three set in both domain aggregates\. This result supports robustness of the paper’s overall strategy trend to a calibration\-domain change\. It does not imply that the ranking is invariant for every compressed checkpoint: the six setting\-level Spearman correlations average 0\.672 and range from 0\.071 to 0\.962\. Accordingly, we treat the domain\-shift result as aggregate evidence and retain the setting\-level variation as an explicit limitation\.

Table 6:Calibration\-domain comparison on six matched settings \(Qwen3/HC\-SMoE and Gemma4/AIMER, each at three retention ratios\)\. Both C4 and OpenR1\-Math\-220k[Lozhkov et al\. \(2025\)](https://arxiv.org/html/2609.06076#bib.bib69)use 1,024 calibration examples, and scores average the same 28 evaluation benchmarks\. Recovery is measured from the unadjusted compressed model\. The last three columns report end\-to\-end adjustment costs for OpenR1\-Math calibration; GiB\-h is GPU\-memory occupancy in GiB\-hours\.

## Appendix JBehavioral Proxies

We complement the performance analysis with six heterogeneous behavioral proxies, evaluated for the Original model, the unadjusted Compressed Only checkpoint, all LM fine\-tuning and KD scopes, and Direct Router Logit Match over the same 12 backbone–compressor–retention settings, using the checkpoints adjusted with the 1,024\-example C4 budget of Appendix[I](https://arxiv.org/html/2609.06076#A9)\(Table[7](https://arxiv.org/html/2609.06076#A10.T7)\)\. These measurements are diagnostic proxies rather than a comprehensive safety evaluation\. Their meanings and preferred directions differ, so we report them separately and neither average them into a scalar safety score nor identify a single “safest” strategy\. Higher is better for IFEval[Zhou et al\. \(2023\)](https://arxiv.org/html/2609.06076#bib.bib70), TruthfulQA MC1[Lin et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib71), and WinoGender[Rudinger et al\. \(2018\)](https://arxiv.org/html/2609.06076#bib.bib75)accuracy; ToxiGen[Hartvigsen et al\. \(2022\)](https://arxiv.org/html/2609.06076#bib.bib72)here is classification accuracy rather than a direct measure of generated toxicity; higher WMDP[Li et al\. \(2024a\)](https://arxiv.org/html/2609.06076#bib.bib73)indicates greater hazardous\-knowledge accuracy; and a CrowS\-Pairs[Nangia et al\. \(2020\)](https://arxiv.org/html/2609.06076#bib.bib74)score closer to 0\.5 is more neutral\.

##### Compression and subsequent adjustment\.

On IFEval, TruthfulQA, and ToxiGen, compression changes the scores from 0\.853 to 0\.761, from 0\.421 to 0\.365, and from 0\.812 to 0\.708, respectively\. The means across the 13 adjusted strategies are 0\.756, 0\.362, and 0\.705, corresponding to additional changes of only−0\.47\-0\.47,−0\.24\-0\.24, and−0\.23\-0\.23pp relative to Compressed Only when computed from the unrounded scores\. Thus, at the strategy\-aggregate level, compression is the larger source of change on these three proxies\. This is not uniform across strategies or metrics: Full FT further changes IFEval by−4\.13\-4\.13pp, while its TruthfulQA and ToxiGen changes are−0\.38\-0\.38and−1\.49\-1\.49pp\. We therefore do not interpret the aggregate result as evidence that every PCA strategy is behaviorally neutral\.

##### Restoration has a dual character\.

For WMDP, CrowS\-Pairs, and WinoGender, adjustment tends to move the strategy\-mean score back toward, or slightly beyond, the Original model’s level\. From Compressed Only, the 13\-strategy means change from 0\.437 to 0\.450 on WMDP, from 0\.556 to 0\.565 on CrowS\-Pairs, and from 0\.567 to 0\.585 on WinoGender; the corresponding Original scores are 0\.508, 0\.576, and 0\.581\. Full FT produces larger shifts to 0\.477, 0\.577, and 0\.610\. This pattern is consistent with PCA restoring parts of the behavioral and knowledge profile attenuated by compression, but the restoration is not selectively beneficial: it can recover useful behavior such as coreference accuracy while also recovering hazardous knowledge measured by WMDP or stereotype\-related associations reflected by CrowS\-Pairs\. These proxies do not establish the underlying causal mechanism, and they should not be read as showing that PCA improves safety\.

##### Implications\.

PCA is designed here as an efficiency and capability\-recovery stage, not as a safety\-alignment intervention\. A compressed and adjusted model should therefore undergo separate, application\-specific safety and alignment evaluation after PCA, with additional mitigation or alignment when needed\. Designing safety\-aware PCA objectives is an important direction for future work\.

Table 7:Behavioral\-proxy results averaged over 12 backbone–compressor–retention settings\. Adjusted strategies are the 1,024\-example C4 runs of Table[5](https://arxiv.org/html/2609.06076#A9.T5)\. The table jointly reports the two baselines, all LM fine\-tuning and KD scopes, and Direct Router Logit Match\. Metric directions differ: higher is better for IFEval, TruthfulQA MC1, and WinoGender accuracy; ToxiGen is classification accuracy rather than a monotone safety score; higher WMDP means greater hazardous\-knowledge accuracy; and CrowS is more neutral near 0\.5\. We therefore neither average across columns nor designate a single “safest” condition\.Table 8:Overall performance and measured cost of the 13 Post\-Compression Adjustment strategies\. Score is the unweighted mean over 28 benchmarks and 12 compressor–retention settings \(four compressors×\\timesthree retention ratios\)\.Δ\\Deltais the score change from Compressed Only in percentage points, and Gap is the recovered fraction of the aggregate Original\-to\-Compressed gap\. Cost columns are unweighted means over the same 12 settings; GPU\-memory values are time\-integrated occupancy\. Expert\-selection and online\-teacher passes are included when applicable\.Table 9:Retention\-ratio sensitivity with all three measured GPU\-cost metrics\. Each cell reports Score on the first line andE/H/ME/H/Mon the second, whereEEis GPU energy \(kWh\),HHis effective GPU\-hours, andMMis GPU\-memory occupancy \(GiB\-hours\)\. Score is an unweighted mean over four compressor–backbone settings and 28 benchmarks at the indicated retention ratio; each cost is the unweighted mean of the corresponding four strategy runs\. Expert\-selection and online\-teacher passes are included when applicable\.Table 10:Integrated compressor\-level performance and cost, averaged over the three retention ratios\. Full FT and Full KD cells report Score and the change from that compressor’s unadjusted checkpoint on the first line, followed by GPU energy \(kWh\), effective GPU\-hours, and GPU\-memory occupancy \(GiB\-hours\) asE/H/ME/H/M\. Performance additionally averages 28 benchmarks; each cost averages three runs\. Each compressor is paired with one backbone, so compressor and backbone effects are not separately identifiable\.Table 11:Performance–cost summary by compression paradigm with the complete measured cost tuple\. For Expert Pruning and Expert Merging,Δ\\Deltais the 28\-benchmark score change from the unadjusted compressed checkpoint, averaged over two compressors and three retention ratios;E/H/ME/H/Mdenotes GPU energy \(kWh\), effective GPU\-hours, and GPU\-memory occupancy \(GiB\-hours\), each averaged over the corresponding six runs\. Expert Editing reports the same performance change over six compatibility\-mode MoBE/TD\-MoE settings; its cost tuple is shown as unavailable because those reconstructed checkpoints use a different parameterization and measurement boundary\.\(a\) Causal\-LM fine\-tuning scopes

\(b\) Knowledge\-distillation scopes and router\-logit matching

Table 12:Benchmark\-family performance averaged over all 12 compressor–retention settings\. General contains BBH\-Fewshot, BBH\-Zeroshot, CoQA, and HellaSwag; the remaining families contain 6 Math, 4 Coding, 6 chain\-of\-thought \(CoT\), and 8 multiple\-choice QA \(MCQA\) benchmarks\. Cost is not allocated by benchmark because every strategy run evaluates all benchmark families after a single shared adjustment procedure; aggregate run\-level costs are reported in Table[8](https://arxiv.org/html/2609.06076#A10.T8)\.