Shape Mutating Expert Compression:LorExperts and BTExperts

# Shape Mutating Expert Compression: LorExperts and BTExperts
Source: [https://arxiv.org/html/2608.07814](https://arxiv.org/html/2608.07814)
Inesh Chakrabarti Sourjya Roy11footnotemark:1Bowen Bao Thiago Crepaldi Spandan Tiwari Ashish Sirasao Advanced Micro Devices \{inesh\.chakrabarti, sourjya\.roy, bowen\.bao\}@amd\.com \{thiago\.crepaldi, spandan\.tiwari, ashish\.sirasao\}@amd\.com

\(August 7, 2026\)

###### Abstract

Mixture\-of\-Experts \(MoE\) language models deliver high capacity at low per\-token compute, but deploying them cheaply requires compressing their many expert weight matrices\. Expert*pruning*\(e\.g\., REAP\) and*merging*reduce cost but sacrifice accuracy and require*retraining the router*; low\-rank*delta decomposition*of experts \(e\.g\., D2\-MoE\) preserves all experts and the router, but degrades sharply as the expert count grows because a single shared component cannot approximate many near\-orthogonal experts\.

Because MoE expert weights are near\-orthogonal, a single shared component \(as in prior delta decomposition\) scales poorly with the expert count; we show that experts nonetheless organize into functional*co\-activation communities*that are decoupled from weight similarity\. Building on this, we introduceLorExperts, a*router\-preserving*compression method that clusters experts, keeps one full\-precision*dominant*per cluster, and represents the remaining members as low\-rank corrections to their local dominant\. LorExperts retains*all*experts and the*original router*\(no router retraining\)\. At∼\\sim50% expert compression on Qwen3\-30B\-A3B and Gemma\-4\-26B\-A4B, LorExperts preserves downstream accuracy and perplexity better than the baselines on most of the tasks; the margin over D2\-MoE grows with expert countEE\. We further give a reconstruction fine\-tuning procedure for LorExperts, and BTExperts, a tree organization of dominants and corrections that enables inference\-time amortization of shared computation\.

Shape Mutating Expert Compression: LorExperts and BTExperts

## 1Introduction

Large language models \(LLMs\) have improved rapidly as dense transformers have scaled to hundreds of billions of parameters\(Brown et al\.,[2020](https://arxiv.org/html/2608.07814#bib.bib2); Chowdhery et al\.,[2023](https://arxiv.org/html/2608.07814#bib.bib4); Touvron et al\.,[2023](https://arxiv.org/html/2608.07814#bib.bib13)\)\. Scaling a dense model, however, means paying for every parameter on every token\. Sparse Mixture\-of\-Experts \(MoE\) architectures ease this by adding capacity while holding the cost of each forward pass roughly constant\. An MoE layer keepsEEseparate experts, and for every token a small router activates onlyk≪Ek\\ll Eof them, leaving the rest idle\(Jiang et al\.,[2024](https://arxiv.org/html/2608.07814#bib.bib10); Yang et al\.,[2025](https://arxiv.org/html/2608.07814#bib.bib14); Dai et al\.,[2024](https://arxiv.org/html/2608.07814#bib.bib5)\)\. Because total capacity is decoupled from the compute spent per token, models can grow to hundreds of billions of parameters without a comparable rise in inference floating\-point operations \(FLOPs\)\.

The difficulty is that this capacity lives almost entirely in the experts, and each expert is as large as a dense feed\-forward block\. Since only a few are active per token, MoE inference is limited by the memory traffic of loading expert weights rather than by arithmetic\(Eliseev and Mazur,[2023](https://arxiv.org/html/2608.07814#bib.bib6)\)\. Recent architectures push this further, favoring more experts that are each smaller and more specialized, and configurations ofE=128E\{=\}128are now common\(Dai et al\.,[2024](https://arxiv.org/html/2608.07814#bib.bib5); Yang et al\.,[2025](https://arxiv.org/html/2608.07814#bib.bib14)\)\. As this expert budget grows, serving the models affordably comes down to compressing the experts\. The aim is to shrink their memory footprint without losing quality, and without disturbing the routing the model has already learned\.

### 1\.1Related Work

#### Expert pruning\.

Pruning removes low\-importance experts\. Candidates are chosen by activation frequency or by an importance criterion, as in REAP\(Lasby et al\.,[2026](https://arxiv.org/html/2608.07814#bib.bib11)\)\. The approach is simple and reduces both storage and per\-token compute\. However, it*discards model capacity*and*changes the set of experts the router can select*\. The router is therefore miscalibrated after pruning and must be adjusted or retrained\. Accuracy also degrades on the capabilities carried by the removed experts\. The relative merit of pruning and merging is benchmark\-dependent\. On discriminative metrics such as perplexity and multiple\-choice accuracy, merging has been reported to do better\(Li et al\.,[2024](https://arxiv.org/html/2608.07814#bib.bib12)\)\. On generative tasks, REAP\(Lasby et al\.,[2026](https://arxiv.org/html/2608.07814#bib.bib11)\)finds the opposite\. It attributes the gap to an irreducible error in merging, which arises from the loss of independent routing control over the combined experts\.

#### Expert merging\.

Merging fuses similar experts into fewer experts\. M\-SMoE\(Li et al\.,[2024](https://arxiv.org/html/2608.07814#bib.bib12)\)groups experts by the cosine similarity of their router logits and merges each group by frequency\-weighted averaging\. HC\-SMoE\(Chen et al\.,[2025](https://arxiv.org/html/2608.07814#bib.bib3)\)instead clusters experts by the similarity of their output activations, which makes the grouping less dependent on routing statistics\. Like pruning, merging shrinks the expert set and usually requires the router to be re\-calibrated or retrained\. Its quality depends on how losslessly similar experts can be fused\.

#### Low\-rank / delta decomposition\.

A third family preserves*all*experts and the*router*\. It writes each expert as a shared component plus a low\-rank per\-expert delta, in the spirit of LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2608.07814#bib.bib8)\)\. D2\-MoE\(Gu et al\.,[2025](https://arxiv.org/html/2608.07814#bib.bib7)\)uses a shared \(Fisher\) mean with per\-expert singular value decomposition \(SVD\) deltas\. SD\-MoE\(Huang et al\.,[2026](https://arxiv.org/html/2608.07814#bib.bib9)\)instead uses a spectral shared component\. Because the expert set and the routing are unchanged,*no router retraining is required*, which is attractive in practice\. Their quality, though, is bounded by how well a*single*shared component plus a small delta can approximate each expert\.

#### Neuron permutation symmetry\.

Feed\-forward neurons have no canonical ordering\. Two functionally related networks can match closely only after a permutation of their hidden units\. This symmetry is well studied in model merging, for example the activation and weight matching used in Git Re\-Basin\(Ainsworth et al\.,[2023](https://arxiv.org/html/2608.07814#bib.bib1)\)\. We reuse this alignment machinery when forming inter\-expert residuals\.

### 1\.2Motivation

We build on delta decomposition, the most deployment\-friendly of these families, because it leaves every expert and the router in place\. Its limitation is scale\. A*single*shared component cannot represent a growing expert set, since one global anchor sits far from most experts in a large, diverse pool\. MoE expert weights are also*near\-orthogonal*, so a small low\-rank delta still leaves a large residual\. As a result, single\-anchor methods degrade asEEgrows \(Sec\.[2](https://arxiv.org/html/2608.07814#S2)\)\.

We address this with two ideas\. The first is to use*multiple local anchors*\. Instead of one global root, we cluster the experts and keep a full\-precision*dominant*per cluster, so every anchor stays close to its members and the method scales withEE\. The second is to exploit permutation symmetry\. Two experts may be similar only*up to a neuron permutation*, so we*align*each member’s neurons to its dominant before forming the low\-rank residual \(Sec\.[3](https://arxiv.org/html/2608.07814#S3)\)\. Throughout, every expert and the original router are left untouched, with no rerouting and no router retraining\.

#### Contributions\.

- •Analysis \(Sec\.[2](https://arxiv.org/html/2608.07814#S2)\)\.We characterize the “orthogonal\-experts” problem: why a single shared component scales poorly asEEgrows, and we measure the functional structure that survives it—co\-activation communities, and their projection\-specific dissociation from weight similarity\. We then quantify the two things a compression budget actually buys \(Sec\.[2\.3](https://arxiv.org/html/2608.07814#S2.SS3)\): expert spectra decay only slowly, so rank is expensive, while routing is strongly concentrated—the 64 busiest of 128 experts absorb94\.5%94\.5\\%of routed visits—so retaining a well\-chosen minority exactly is cheap\. Together these motivate the asymmetric budget LorExperts uses\.
- •LorExperts \(Sec\.[3](https://arxiv.org/html/2608.07814#S3)\)\.A router\-preserving compression method that clusters experts, keeps one full\-precision dominant per cluster, and encodes the remaining members as permutation\-aligned low\-rank corrections\. Every expert and the original router are retained\.
- •Fine\-tuning \(Sec\.[4](https://arxiv.org/html/2608.07814#S4)\)\.A lightweight distillation step that fits only the low\-rank factors to the original expert outputs, while the dominants and the router stay frozen\.
- •BTExperts \(Sec\.[5](https://arxiv.org/html/2608.07814#S5)\)\.A tree organization of dominants and corrections that amortizes shared computation at inference time\.

## 2Analysis: The Orthogonal\-Experts Problem

Router: top\-kkover allEEexperts*\(unchanged — every expert stays individually addressable\)*Wd1W\_\{d\_\{1\}\}Wd1\+B1​A1W\_\{d\_\{1\}\}\\\!\+\\\!B\_\{1\}A\_\{1\}cluster 1Wd2W\_\{d\_\{2\}\}Wd2\+B2​A2W\_\{d\_\{2\}\}\\\!\+\\\!B\_\{2\}A\_\{2\}cluster 2Wd3W\_\{d\_\{3\}\}Wd3\+B3​A3W\_\{d\_\{3\}\}\\\!\+\\\!B\_\{3\}A\_\{3\}clusterkk■\\blacksquarefull\-precision dominant \(kept\)■\\blacksquaremember≈\\approxdominant\+\+rank\-rrcorrectionBm​AmB\_\{m\}A\_\{m\}

Figure 1:LorExperts overview\.Experts are clustered \(by co\-activation or weight similarity\); each cluster keeps its highest\-firing expert as a full\-precision*dominant*WdW\_\{d\}and represents every other member asWd\+Bm​AmW\_\{d\}\+B\_\{m\}A\_\{m\}with a rank\-rrcorrection\. Unlike single\-shared\-component decomposition, which uses one global root, LorExperts useskk*local*dominants, so approximation quality does not collapse as the expert countEEgrows\. The router and allEEexperts are preserved—no rerouting or retraining\.This section motivates LorExperts \(overview in Fig\.[1](https://arxiv.org/html/2608.07814#S2.F1)\) by characterizing*why*shared\-component low\-rank decomposition of MoE experts is hard, and identifying the structure that makes it tractable\.

#### Setup: the single\-anchor budget\.

Let an MoE layer haveEEexperts with weight matricesWe∈ℝI×HW\_\{e\}\\in\\mathbb\{R\}^\{I\\times H\}\. A single\-shared\-component method approximatesWe≈Wshared\+ΔeW\_\{e\}\\approx W\_\{\\text\{shared\}\}\+\\Delta\_\{e\}withΔe\\Delta\_\{e\}low\-rank, so its quality is bounded by how much of each expert the one shared component can carry and how low\-rank the leftover residual is\. The difficulty is an accounting one: a single anchor is amortized across the whole pool, so asEEgrows it must sit close to ever more, ever more diverse experts at once\. For experts that are close to mutually orthogonal in weight space—as MoE experts are, and as the co\-activation analysis below takes as its starting point—any one direction can align with only about1/E1/Eof the pool’s inter\-expert variance, leaving a near\-full\-rank residual for the per\-expert delta to absorb\. Adding rank toΔe\\Delta\_\{e\}does not fix this, because the residual is not low\-rank to begin with\. The remedy we pursue is therefore not a better global anchor but*more*anchors:kklocal dominants, each close to its own members \(Sec\.[3](https://arxiv.org/html/2608.07814#S3)\)\. What remains is to decide which experts belong together\.

### 2\.1What the co\-activation structure reveals

Although expert*weights*are near\-orthogonal, the router induces rich, non\-random*functional*structure\. The expert co\-activation graph \(edges weighted by normalized pointwise mutual information, NPMI\), built across 13 datasets on Qwen3\-30B\-A3B \(Fig\.[2](https://arxiv.org/html/2608.07814#S2.F2)\), exposes several properties that bear directly on the design choices—and failure modes—of prior compression methods\.

![Refer to caption](https://arxiv.org/html/2608.07814v1/figs/coact_graph_L23.png)Figure 2:Expert co\-activation structure at Qwen3\-30B\-A3B layer 23 as a*backbone graph*: experts are nodes, and each strong co\-activation edge is colored by the dataset with the largest NPMI \(code=green, math/GSM8K=red, web/wikitext=cyan, …\) and widened by its strength\. Experts partition into stable communities even though their weights are near\-orthogonal, and distinct domains induce visibly specialized structure\. A raw heatmap view and clique statistics appear in Appendix[B](https://arxiv.org/html/2608.07814#A2)\.- •Stable communities\.Experts partition into consistent co\-activation communities that persist across datasets—natural groups that LorExperts clusters over \(motivating the Coact\-NPMI /MsoftM\_\{\\text\{soft\}\}metrics, Sec\.[3](https://arxiv.org/html/2608.07814#S3)\)\.
- •Depth\-varying granularity\.The number of co\-activation communities grows with network depth\. A*uniform*compression budget is therefore suboptimal, motivating per\-layer treatment\.
- •Domain specialists\.Some communities are domain\-general while others are strongly domain\-specific: code and math \(GSM8K\) induce the most distinctive routing \(visible as the colored communities in Fig\.[2](https://arxiv.org/html/2608.07814#S2.F2)\), whereas web text \(C4, Wikipedia\) tracks the consensus\. These specialists are*rare*yet load\-bearing for their domain\.

#### Consequences for prior methods—and for LorExperts\.

This structure is exactly what the prior families endanger\. Frequency\-based*pruning*\(Sec\.[1\.1](https://arxiv.org/html/2608.07814#S1.SS1)\) discards precisely the rare domain specialists that carry code/math ability, and*merging*\(Sec\.[1\.1](https://arxiv.org/html/2608.07814#S1.SS1)\) blurs distinct communities—both then force router re\-calibration\. LorExperts instead keeps every expert and the router intact, preserving this functional structure by construction\. Moreover, the structure lives in the*routing*while the*weights*are near\-orthogonal—a decoupling we measure directly \(Sec\.[2\.2](https://arxiv.org/html/2608.07814#S2.SS2), Table[1](https://arxiv.org/html/2608.07814#S2.T1)\)—so single\-shared\-component weight decomposition \(Sec\.[1\.1](https://arxiv.org/html/2608.07814#S1.SS1)\) cannot exploit it, which is why LorExperts uses local per\-cluster dominants rather than one global root\.

### 2\.2The functional–weight dissociation is projection\-specific

Do co\-firing experts also have similar*weights*? We correlate each expert pair’s co\-activation \(NPMI\) with its weight cosine similarity, per layer and per projection \(Table[1](https://arxiv.org/html/2608.07814#S2.T1)\)\. The correlation is small but consistently significant for thegateprojection \(r≈0\.08r\\approx 0\.08–0\.210\.21,p<10−3p<10^\{\-3\}\) and effectively zero forupanddown\.

Table 1:Correlation between expert co\-activation \(NPMI\) and weight cosine similarity, by layer and projection \(Qwen3\-30B\-A3B\)\.p∗∗<10−3\{\}^\{\*\*\}p<10^\{\-3\},p∗<0\.05\{\}^\{\*\}p<0\.05\.The pattern is interpretable\. Thegateprojection selects*which inputs activate*an expert, so co\-firing experts share a*weak*alignment in what excites them; their actual*computation*\(up,down\) is essentially orthogonal\. Experts that fire together thus respond to similar contexts but compute different things—*complementary, not redundant*\. Even the gate correlation is small \(r≤0\.21r\\leq 0\.21\), so experts remain largely near\-orthogonal in weight space\. This is precisely why routing\-space structure is not recoverable by a shared\-component weight decomposition, and why LorExperts corrects each expert against a*local*dominant rather than assuming a single shared low\-rank basis across experts\.

### 2\.3What the budget can buy: spectra and routing mass

Two measurable quantities decide how a compression budget for an MoE layer is best spent: how fast expert spectra decay, which sets what a unit of rank buys, and how concentrated the routing is, which sets what keeping a single expert exact buys\. Measured on Qwen3\-30B\-A3B, together they argue for exactly the asymmetric allocation LorExperts adopts\.

#### Spectra decay, but slowly\.

Table[2](https://arxiv.org/html/2608.07814#S2.T2)reports the fraction of squared Frobenius energy a rank\-rrtruncation retains, averaged over the three projections of 12 experts at layers 5, 23 and 40\. There*is*genuine low\-rank structure: rank 64 retains26\.0%26\.0\\%where a perfectly flat spectrum would retain only64/768=8\.3%64/768=8\.3\\%\. But the decay is shallow—reaching72%72\\%of the energy already costs rank 311, i\.e\.40%40\\%of the maximum rank 768\. Rank is expensive on these matrices, so a budget spread thinly and uniformly across allEEexperts buys comparatively little per expert; it is worth asking whether some of it is better spent keeping selected experts exactly\.

Table 2:Fraction of squared Frobenius energy retained by a rank\-rrtruncation of an expert matrix \(I=768I\{=\}768,H=2048H\{=\}2048, so rank768768is lossless\), mean overgate/up/downof 12 experts per layer\. The last row is what a flat spectrum \(r/768r/768\) would give\.
#### Routing is strongly concentrated\.

Expert utilization is meanwhile far from uniform\. Over five layers, the 64 busiest of the 128 experts absorb94\.5%94\.5\\%of routed token–expert visits, against the50%50\\%that uniform routing would give, and 18–30 experts per layer are effectively dead \(<10−4<10^\{\-4\}of traffic\)\. Concentration this strong is what makes an asymmetric budget attractive: if a minority of experts carries almost all of the traffic, then holding exactly those experts in full precision costs a well\-understood share of the budget and removes approximation error where it is encountered most often\.

Table 3:Routed token–expert visits absorbed by the6464busiest ofE=128E\{=\}128experts, measured on WikiText\-2\. Uniform routing would give0\.5000\.500\. “Dead” counts experts receiving<10−4<10^\{\-4\}of traffic\.Taken together the two measurements motivate the shape of LorExperts: because rank is expensive, uniform low\-rank treatment of every expert is a poor use of the budget; because routing is concentrated, keeping a well\-chosen minority exact covers most of what the router actually does\. Retaining a full\-precision dominant per cluster and spending only low\-rank corrections on the remainder follows directly, and it is why LorExperts selects each cluster’s highest\-firing member rather than its geometric center\.

One caveat qualifies how the dominant set should be read\. Utilization here is measured on WikiText\-2, and Sec\.[2](https://arxiv.org/html/2608.07814#S2)shows that code and math induce visibly different communities, so the busy set is partly domain\-dependent\. This is a reason to keep every expert addressable rather than pruning the tail: the rare domain specialists are precisely the experts a single\-domain utilization estimate will rank last\.

### 2\.4Implication for method design

Two consequences drive LorExperts\. First, use*multiple local anchors*\(per\-cluster dominants\) rather than one global root, so each anchor is close to its members and the residual it must correct is small—and the approach scales withEEinstead of degrading \(ablated in Appendix[A](https://arxiv.org/html/2608.07814#A1)\)\. Second, by Sec\.[2\.3](https://arxiv.org/html/2608.07814#S2.SS3), split the budget asymmetrically rather than uniformly: rank buys little on these spectra, while routing mass is concentrated enough that full\-precision dominants pay for themselves—so a dominant should be chosen for the traffic it absorbs\.

## 3LorExperts: Router\-Preserving Low\-Rank Expert Compression

### 3\.1Representation

Consider one MoE layer withEEexperts, each a set of weight matrices \(gate, up, down\) that we write jointly asWeW\_\{e\}\. LorExperts partitions the experts intokkclusters\. Within each cluster it keeps one*dominant*expert at full precision and represents every other membermmas a low\-rank correction of its dominant,

Wm≈Wdominant\+Bm​Am,rank⁡\(Bm​Am\)=r,\\begin\{gathered\}W\_\{m\}\\;\\approx\\;W\_\{\\text\{dominant\}\}\+B\_\{m\}A\_\{m\},\\\\ \\operatorname\{rank\}\(B\_\{m\}A\_\{m\}\)=r,\\end\{gathered\}with factorsBm∈ℝd×rB\_\{m\}\\in\\mathbb\{R\}^\{d\\times r\},Am∈ℝr×d′A\_\{m\}\\in\\mathbb\{R\}^\{r\\times d^\{\\prime\}\}\. The dominant is stored exactly; each member costs onlyBm,AmB\_\{m\},A\_\{m\}\. Withk≈E/2k\\approx E/2andr=64r=64, expert parameters are reduced by roughly one half; the ratio is controlled bykkandrr\.

### 3\.2Compression pipeline

#### Stage 1 — Router profiling\.

We pass a small calibration set \(C4, 64 sequences of length 2048\) through the model with router hooks, accumulating per\-layer firing countsNeN\_\{e\}\. These counts seed clustering and dominant selection\. Profiling only reads the router: LorExperts keeps allEEexperts and leaves the gate unchanged, unlike pruning or merging\.

#### Stage 2 — Clustering\.

Per layer, we form a pairwise distance matrixD=1−MD=1\-Mwith the diagonal zeroed\. We then run frequency\-seededkk\-medoids: the medoids start at the highest\-firing experts, and the assign/update loop iterates to convergence\. Seeding from frequently\-used experts biases the dominants toward experts the router actually selects\. Each cluster’s dominant is its highest\-firing member, with ties broken by co\-activation centrality, and the remaining members are queued for reconstruction\. An optional*protection set*can be pulled out as singleton clusters and kept at full precision\. We compare three choices of the distanceMMempirically \(Sec\.[6](https://arxiv.org/html/2608.07814#S6)\): weight cosine \(WS\-Frob\), co\-activation NPMI \(Coact\-NPMI\), and soft co\-activation \(MsoftM\_\{\\text\{soft\}\}\)\.

#### Stage 3 — Alignment and low\-rank decomposition\.

We apply three steps to each non\-dominant member\.*\(i\) Neuron alignment\.*An expert’s hidden feed\-forward network \(FFN\) has no canonical ordering of its intermediate neurons, so a member may store corresponding neurons at different indices than its dominant\. This inflates the naive residualWm−WdominantW\_\{m\}\-W\_\{\\text\{dominant\}\}, which can be full\-rank even when the experts match up to a permutation\. Aligning networks by such permutations is well established in model merging\(Ainsworth et al\.,[2023](https://arxiv.org/html/2608.07814#bib.bib1)\), and has been applied to experts before merging in MoE compression\(Li et al\.,[2024](https://arxiv.org/html/2608.07814#bib.bib12)\)\. We therefore align the member to its dominant before decomposing\. Treating each neuron as the concatenation of its gate, up, and down columns, we score neuroniiof the dominant against neuronjjof the member by the squared distance

Ci​j=‖ai−bj‖22=∥ai∥2\+∥bj∥2−2​ai⊤​bj,C\_\{ij\}=\\bigl\\lVert a\_\{i\}\-b\_\{j\}\\bigr\\rVert\_\{2\}^\{2\}=\\lVert a\_\{i\}\\rVert^\{2\}\+\\lVert b\_\{j\}\\rVert^\{2\}\-2\\,a\_\{i\}^\{\\top\}b\_\{j\},\(1\)whereai=Wdom​\[:,i\]a\_\{i\}=W\_\{\\text\{dom\}\}\[:,i\]andbj=Wm​\[:,j\]b\_\{j\}=W\_\{m\}\[:,j\]are the concatenated neuron vectors\. We then seek the permutationPPof the member’s neurons that minimizes the total matching cost,

P⋆=arg⁡minP​∑iCi,P​\(i\),P^\{\\star\}=\\arg\\min\_\{P\}\\sum\_\{i\}C\_\{i,\\,P\(i\)\},\(2\)which we solve exactly with the Jonker–Volgenant algorithm\. ApplyingP⋆P^\{\\star\}gives the aligned memberWmPW\_\{m\}^\{P\}\. The reported results use this alignment\.

*\(ii\) Truncated\-SVD residual\.*We form the aligned residual and take its rank\-rrSVD,

R=WmP−Wdominant≈Ur​Σr​Vr⊤,R=W\_\{m\}^\{P\}\-W\_\{\\text\{dominant\}\}\\approx U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{\\top\},\(3\)computed with a randomized solver \(r=64r=64\)\. Splitting the singular values symmetrically yields the low\-rank factors

Bm=Ur​Σr1/2,Am=Σr1/2​Vr⊤,so​Bm​Am≈R\.\\begin\{gathered\}B\_\{m\}=U\_\{r\}\\Sigma\_\{r\}^\{1/2\},\\qquad A\_\{m\}=\\Sigma\_\{r\}^\{1/2\}V\_\{r\}^\{\\top\},\\\\ \\text\{so \}B\_\{m\}A\_\{m\}\\approx R\.\\end\{gathered\}\(4\)
*\(iii\) Installation\.*The member is replaced by a module that computesWdominant\+Bm​AmW\_\{\\text\{dominant\}\}\+B\_\{m\}A\_\{m\}at inference, while the dominants and the router remain untouched\.

#### Stage 4 — Reconstruction fine\-tuning\.

A short output\-matching pass refines the factors \(Sec\.[4](https://arxiv.org/html/2608.07814#S4)\); it updates onlyBm,AmB\_\{m\},A\_\{m\}\.

#### Variants\.

The framework is parameterized by the clustering distanceMMand an optional protection set, giving four variants we evaluate side by side \(Sec\.[6](https://arxiv.org/html/2608.07814#S6), Tables[4](https://arxiv.org/html/2608.07814#S6.T4)–[5](https://arxiv.org/html/2608.07814#S6.T5)\):

- •WS\-Frobclusters on the weight\-space Frobenius distance∥Wi−Wj∥F\\lVert W\_\{i\}\-W\_\{j\}\\rVert\_\{F\}\. It uses no router information, so it serves as a routing\-agnostic reference\.
- •WS\-Frob\+protK8uses the same weight\-space clustering but keeps eight full\-precision protected experts per layer\. This isolates the effect of protection on an otherwise routing\-agnostic variant\.
- •Coact\-NPMIclusters on single\-mixture routing co\-activation, measured as1−NPMI1\-\\mathrm\{NPMI\}over a mixed calibration set\. It is our first routing\-aware variant\.
- •MsoftM\_\{\\text\{soft\}\}clusters on the cross\-domain soft consensus1−Msoft1\-M\_\{\\text\{soft\}\}, optionally with the router\-identified protection list, and is our strongest variant\. Becausekkand the protection set are tunable, we report it at several operating points to trace the compression–quality frontier\.

All variants share the pipeline above and differ only in the Stage\-2 distance and whether a protection set is supplied\.

### 3\.3Storage

For a cluster with a dominant andccmembers, storage is one full matrix plusccrank\-rrfactor pairs, versusc\+1c\{\+\}1full matrices uncompressed\. Summed over clusters, the layer’s expert parameters scale with the number of dominants \(set bykk\) plusrrtimes the number of members\. Thuskkandrrjointly set the compression ratio, and every expert remains individually addressable by the unchanged router\.

## 4Reconstruction Fine\-Tuning for LorExperts

Truncated\-SVD \(Stage 3\) minimizes weight\-space error; a short output\-matching refinement then recovers the error that matters for the token distribution—at negligible cost\.

#### Objective\.

On*cached*calibration activationsxx, we fit each member’s factors so its output matches the original expert’s,

minBm,Am∥\(Wdominant\+BmAm\)x−Wmx∥F2,\\min\_\{B\_\{m\},A\_\{m\}\}\\;\\big\\lVert\(W\_\{\\text\{dominant\}\}\+B\_\{m\}A\_\{m\}\)\\,x\-W\_\{m\}x\\big\\rVert\_\{F\}^\{2\},training*only*the rank\-rrfactorsBm,AmB\_\{m\},A\_\{m\}; the dominant weights and the router are frozen\.

#### Why it is cheap\.

Three properties make this the least costly stage of LorExperts:*\(i\)*it updates only the small rank\-rrfactors—not the base model and not the router, so no gradients flow through either;*\(ii\)*the objective is*local and per\-member*and is evaluated on*pre\-cached*activations, so there are no full\-model forward/backward passes during fitting—just small matmuls;*\(iii\)*it is an*activation\-weighted low\-rank regression*, which admits a near\-closed\-form solution: whitening by the activation second moment and taking a \(generalized\) SVD of the residual recovers the optimal rank\-rrfactors directly, in place of iterative gradient steps\.

Its wall\-clock cost, and its quality contribution relative to the SVD initialization \(Stage\-3\-only vs\. Stage\-3\+\+4\), are reported with the calibration cost \(Sec\.[6](https://arxiv.org/html/2608.07814#S6)\)\. We expect the SVD initialization to do the heavy lifting and this stage to be a cheap refinement\.

## 5BTExperts: Tree Organization for Inference\-Time Amortization

BTExperts organizes the dominants and their corrections in a binary tree\. We emphasize up front what BTExperts is*not*: it does*not*improve compression*quality*\. On near\-orthogonal experts, sharing corrections across a hierarchy does not beat flat per\-cluster decomposition: in a controlled study on Mixtral\-8×\\times7B \(E=8E\{=\}8\) at matched7\.5×7\.5\\timescompression, the hierarchical organization matches flat decomposition within noise in perplexity \(PPL;Δ​PPL=0\.099\\Delta\\text\{PPL\}=0\.099vs\.0\.0980\.098after reconstruction fine\-tuning;0\.1120\.112vs\.0\.1050\.105zero\-shot, 3 seeds; Appendix[A](https://arxiv.org/html/2608.07814#A1)\)\. Its sole purpose is therefore*inference\-time amortization*: components shared by several selected experts are computed once and reused\.

### 5\.1Amortized forward pass

When multiple selected experts share a cluster \(dominant\) or a tree ancestor, the shared computation—x​Wdominant⊤xW\_\{\\text\{dominant\}\}^\{\\top\}and any shared correction factors—is computed*once*per token and broadcast to those experts, rather than recomputed per expert\. The tree \(Fig\.[3](https://arxiv.org/html/2608.07814#S5.F3)\) makes this sharing explicit: experts on the same path share the ancestors’ contributions\.

WrootW\_\{\\text\{root\}\}B6​A6B\_\{6\}A\_\{6\}B7​A7B\_\{7\}A\_\{7\}−1\-1\+1\+1B4​A4B\_\{4\}A\_\{4\}B5​A5B\_\{5\}A\_\{5\}B2​A2B\_\{2\}A\_\{2\}B3​A3B\_\{3\}A\_\{3\}−1\-1\+1\+1−1\-1\+1\+1e0e\_\{0\}e1e\_\{1\}e2e\_\{2\}e3e\_\{3\}e4e\_\{4\}e5e\_\{5\}e6e\_\{6\}e7e\_\{7\}

Figure 3:BTExperts tree \(E=8E\{=\}8\)\. Shared components near the root are computed once and amortized across experts that share a path; leaves are experts\. The highlighted red path is path\(e3e\_\{3\}\)\. The tree is an*inference\-time*organization, not a compression\-quality mechanism\.
### 5\.2Inference cost: theoretical FLOPs

Standard top\-kkrouting evaluateskkfull experts per token, costingk⋅2​I​Hk\\cdot 2IHFLOPs and loadingkkfull weight tensors\.*Pruning*\(REAP\) does not reduce this: top\-kkstill selectskkfull*survivors*, so per\-token FLOPs and high\-bandwidth memory \(HBM\) reads are unchanged \(a0×0\\timesreduction—pruning saves storage, not per\-token compute\)\. BTExperts instead*amortizes*: when co\-selected experts share a dominant, that dominant is computed*once*and only the cheap rank\-rrcorrections are added per expert \(Fig\.[4](https://arxiv.org/html/2608.07814#S5.F4)\)\. For such a group,

speedup=k⋅2​I​H2​I​H\+k⋅2​r​\(I\+H\)≈1\.9×\\text\{speedup\}\\;=\\;\\frac\{k\\cdot 2IH\}\{2IH\+k\\cdot 2r\(I\+H\)\}\\;\\approx\\;1\.9\\timesfor Mixtral\-8×\\times7B gate\_proj \(k=2k\{=\}2,I=14336I\{=\}14336,H=4096H\{=\}4096,r=64r\{=\}64\)\. Because this projection is memory\-bandwidth\-bound \(arithmetic intensity≈1\\approx 1FLOP/byte\), the reduction is*kernel\-independent*\. Two caveats keep it honest: the benefit is*routing\-dependent*\(it applies only when the router co\-selects experts from the same cluster\), and it is*shared with any single\-shared\-component method*—D2\-MoE can amortize its shared base likewise\.

\(a\) OriginaltokenxxWe1W\_\{e\_\{1\}\}117 MBWe2W\_\{e\_\{2\}\}117 MBout235M FLOPs, 235 MB\(b\) REAP \(pruned\)tokenxxWe1W\_\{e\_\{1\}\}117 MBWe2W\_\{e\_\{2\}\}117 MBout235M FLOPs, 235 MBsame per\-token cost\(c\) LorExperts\+\+BTExpertstokenxxWdomW\_\{\\text\{dom\}\}117 MBsharedBe1​Ae1B\_\{e\_\{1\}\}A\_\{e\_\{1\}\}2\.4 MBBe2​Ae2B\_\{e\_\{2\}\}A\_\{e\_\{2\}\}2\.4 MBout122M FLOPs, 122 MB≈\\approx1\.9×\\timesfewer

Figure 4:Per\-token forward pass \(gate\_proj, top\-2 routing\)\.\(a\)Original: two full expert matmuls\.\(b\)REAP: identical per\-token cost—pruning removes experts but top\-kkstill runskkfull survivors\.\(c\)LorExperts\+\+BTExperts for two co\-selected experts sharing a dominant: the dominant is computed once and reused, plus two cheap rank\-rrcorrections, cutting FLOPs and HBM reads≈1\.9×\\approx 1\.9\\times\(kernel\-independent\)\. Adapted from the prior draft’s FLOP analysis\.

## 6Results

We evaluate on Qwen3\-30B\-A3B \(E=128E\{=\}128\) and Gemma\-4\-26B\-A4B at∼\\sim50% expert compression, reporting perplexity on Wikitext\-103 and downstream accuracy via the LM Evaluation Harness \(MMLU, ARC\-C, HellaSwag, WinoGrande, PIQA\)\. We compare against merging \(M\-SMoE, HC\-SMoE\), pruning \(Freq\-Prune, REAP\), and delta decomposition \(D2\-MoE\), all at matched compression \(Tables[4](https://arxiv.org/html/2608.07814#S6.T4)–[5](https://arxiv.org/html/2608.07814#S6.T5)\)\.

Table 4:Qwen3\-30B\-A3B \(∼\\sim50% expert compression; uncompressed PPL=8\.50=8\.50\)\.Table 5:Gemma\-4\-26B\-A4B \(∼\\sim50% expert compression; uncompressed PPL≈7\.15\\approx 7\.15\)\.#### Reading the tables\.

Freq\-Prune is the strongest prior baseline, but LorExperts preserves*knowledge and reasoning*accuracy \(MMLU, ARC\-C, HellaSwag\) markedly better: it leads across the board on Qwen3, and on both models holds MMLU/ARC well above the pruning and merging baselines, whereas D2\-MoE degrades sharply at this expert count\. On Gemma, Freq\-Prune retains lower PPL and higher commonsense \(WinoGrande/PIQA\), so LorExperts’s advantage is in*preserved capability*rather than a clean sweep of every metric\. As a side observation consistent with the near\-orthogonality analysis \(Sec\.[2](https://arxiv.org/html/2608.07814#S2)\), LorExperts’s margin over single\-root decomposition \(D2\-MoE\) widens as the expert countEEgrows—the regime where one shared component is least able to fit the experts\.

#### Which variant to use\.

The routing\-aware variants win: the soft cross\-domain consensusMsoftM\_\{\\text\{soft\}\}is strongest \(best MMLU on both models at∼\\sim40–50% compression\), with Coact\-NPMI close behind, because clustering by co\-activation groups experts that are functionally related rather than merely close in weight space\. Protecting the highest\-firing experts also helps sharply \(WS\-Frob MMLU42\.4→53\.742\.4\\to 53\.7on Qwen3 with protK8\), since those load\-bearing specialists are the costliest to approximate\.

### 6\.1Compression–quality frontier

The tables fix a single operating point, but the compression ratio is a knob we can turn\. In Figure[5](https://arxiv.org/html/2608.07814#S6.F5)we sweep the Coact\-NPMI variant on Qwen3\-30B\-A3B from0to about50%50\\%expert compression and track perplexity alongside the five downstream tasks\. Accuracy changes little through low and moderate compression, and the clearer drops set in only as the ratio nears50%50\\%; perplexity climbs slowly over the same range\. Quality therefore falls off smoothly rather than all at once, and the50%50\\%end of the sweep reproduces the Coact\-NPMI row of Table[4](https://arxiv.org/html/2608.07814#S6.T4)\. In practice the ratio can then be tuned to a quality target rather than fixed in advance\.

![Refer to caption](https://arxiv.org/html/2608.07814v1/figs/qwen3_frontier.png)Figure 5:Compression–quality frontier for LorExperts \(Coact\-NPMI\) on Qwen3\-30B\-A3B\. As the expert compression ratio increases \(top to bottom\), downstream accuracy \(MMLU, ARC\-C, HellaSwag, WinoGrande, PIQA; bottom axis\) stays near its uncompressed value through moderate compression and falls off mainly near50%50\\%, while perplexity \(dashed, top axis\) rises gradually\.
### 6\.2Calibration cost

Calibration time is dominated by fitting the low\-rank factors, and this admits a large,*analytically guaranteed*speedup\. The reconstruction fine\-tuning of Sec\.[4](https://arxiv.org/html/2608.07814#S4)reaches the factors through∼\\sim500 iterative gradient steps over cached activations\. That objective, however, is an*activation\-weighted low\-rank regression*, which has a*closed\-form*optimum: whitening by the activation second moment and taking a single \(generalized\) SVD of the residual recovers the optimal rank\-rrfactors directly\. Replacing the∼\\sim500 optimization passes with one whitened SVD collapses the factor\-fitting stage fromO​\(T\)O\(T\)passes toO​\(1\)O\(1\)—roughly*two orders of magnitude*\(∼\\sim100×100\\times\) less compute for that stage—while attaining the*same*rank\-rroptimum, so accuracy is unchanged\. Because factor fitting dominates the pipeline, this turns whole\-model calibration into a short offline step rather than a bottleneck\. We report this speedup as a relative multiplier; absolute wall\-clock times are omitted\.

## 7Discussion

#### Why LorExperts works\.

The gains trace to one design choice grounded in Sec\.[2](https://arxiv.org/html/2608.07814#S2):*multiple local anchors*\(per\-cluster dominants\) instead of one global root, so each anchor is close to its members and the method scales withEEwhere single\-root decomposition degrades\. This preserves all experts and the original router\.

#### Hessian\-weighted fitting \(ongoing work\)\.

Sec\.[6](https://arxiv.org/html/2608.07814#S6)introduces the activation\-weighted closed\-form solve as a way to*replace*the iterative reconstruction pass at equal quality\. We note here that the same device is also a quality improvement in its own right over the plain\-SVD default, and that it generalizes beyond the corrections\.

The default fit minimizes weight\-space error∥W−W^∥F\\lVert W\-\\widehat\{W\}\\rVert\_\{F\}, which is not the quantity a layer emits\. The error reaching the next layer is∥X​\(W−W^\)⊤∥F\\lVert X\(W\-\\widehat\{W\}\)^\{\\top\}\\rVert\_\{F\}for the calibration activationsXXthe router actually sends to that expert—a per\-expert quantity in an MoE, since each expert sees only its own slice of the tokens\. WithGe=Xe⊤​Xe=Le​Le⊤G\_\{e\}=X\_\{e\}^\{\\top\}X\_\{e\}=L\_\{e\}L\_\{e\}^\{\\top\}, minimizing the emitted error is again a truncation problem—ofW​LeWL\_\{e\}rather thanWW—so it costs the same rank, the same storager​\(I\+H\)r\(I\+H\), and the same two matmuls at inference\. In preliminary runs on Qwen3\-30B\-A3B this improves perplexity at matched storage for every variant we have tried\. One MoE\-specific caveat:GeG\_\{e\}hasH2H^\{2\}entries estimated from the tokens a single expert receives, so it needs substantially more calibration data than a diagonal or norm\-based statistic, and experts that see fewer tokens thanHHrely on damping\.

The same reweighting extends to the shared components themselves, and there it also removes a structural weakness\. For a shared componentCCserving expert setSSwith correctionsReR\_\{e\}, the optimal Hessian\-weighted choice is closed\-form,

C⋆=\[∑e∈S\(We−Re\)​Ge\]​\[∑e∈SGe\]−1,C^\{\\star\}=\\Big\[\\textstyle\\sum\_\{e\\in S\}\(W\_\{e\}\-R\_\{e\}\)\\,G\_\{e\}\\Big\]\\Big\[\\textstyle\\sum\_\{e\\in S\}G\_\{e\}\\Big\]^\{\-1\},which is a traffic\-weighted combination of the cluster rather than a member of it\. BecauseC=0C=0lies in the feasible set, a solved anchor can never be worse than using no anchor at all—a guarantee that selecting an existing expert as the dominant does not provide, and one that matters precisely when experts are near\-orthogonal and the residual against another expert is larger than the expert itself\. Note that a solved anchor is no longer one of the experts, so the former dominant also needs a correction; atk=64k\{=\}64this is roughly a10%10\\%increase in the expert budget, which must be returned by a small reduction inrr\. We are pursuing the full treatment, including its interaction with the BTExperts tree, where the component values change but the sharing structure—and hence the amortization factor—does not\.

#### Limitations\.

1. 1\.Calibration cost\.Calibration is dominated by the per\-member SVD and the short reconstruction pass; a closed\-form activation\-weighted solve can replace the SVD \(Sec\.[6](https://arxiv.org/html/2608.07814#S6), Gate H\) but is not yet the default\.
2. 2\.Inference amortization is bounded\.The BTExperts amortization benefit is routing\-dependent and partly shared with any shared\-component method; we report only fairly\-baselined numbers \(Sec\.[5](https://arxiv.org/html/2608.07814#S5)\) and make no strawman speedup claims\.

## 8Conclusion

We presentedLorExperts, a router\-preserving method for compressing Mixture\-of\-Experts weight matrices\. Motivated by a mechanistic analysis of the “orthogonal experts” problem—experts are near\-orthogonal in weight space yet organize into functional co\-activation communities—LorExperts keeps a full dominant per cluster and represents other experts as low\-rank corrections to that local dominant, retaining all experts and the original router with no rerouting or router retraining\. A lightweight reconstruction fine\-tuning refines the factors, and BTExperts organizes dominants/corrections for inference\-time amortization\.

Across Qwen3\-30B\-A3B and Gemma\-4\-26B\-A4B at∼\\sim50% expert compression, LorExperts preserves downstream capability—particularly knowledge\- and reasoning\-heavy accuracy \(MMLU, ARC\-C, HellaSwag\)—better than pruning, merging, and single\-shared\-component decomposition \(D2\-MoE\), and its advantage over the latter grows with the expert count\.

#### Future work\.

Higher\-compression and full\-model operating points; generation\-benchmark evaluation; faster closed\-form factor fitting at scale; and using the BTExperts structure for upcycling / MoE construction from dense checkpoints\.

## References

- Ainsworth et al\. \(2023\)Samuel K\. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa\. 2023\.Git re\-basin: Merging models modulo permutation symmetries\.In*International Conference on Learning Representations \(ICLR\)*\.
- Brown et al\. \(2020\)Tom B\. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert\-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M\. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei\. 2020\.Language models are few\-shot learners\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Chen et al\. \(2025\)I\-Chun Chen, Hsu\-Shen Liu, Wei\-Fang Sun, Chen\-Hao Chao, Yen\-Chang Hsu, and Chun\-Yi Lee\. 2025\.Retraining\-free merging of sparse MoE via hierarchical clustering\.In*International Conference on Machine Learning \(ICML\)*, volume 267, pages 8594–8620\.
- Chowdhery et al\. \(2023\)Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur\-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M\. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier\-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel\. 2023\.PaLM: Scaling language modeling with pathways\.*Journal of Machine Learning Research*, 24\(240\):1–113\.
- Dai et al\. \(2024\)Damai Dai, Chengqi Deng, Chenggang Zhao, R\.X\. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y\. Wu, Zhenda Xie, Y\.K\. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang\. 2024\.DeepSeekMoE: Towards ultimate expert specialization in mixture\-of\-experts language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1280–1297\. Association for Computational Linguistics\.
- Eliseev and Mazur \(2023\)Artyom Eliseev and Denis Mazur\. 2023\.Fast inference of mixture\-of\-experts language models with offloading\.*arXiv preprint arXiv:2312\.17238*\.
- Gu et al\. \(2025\)Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark G\. Lee, Shengjie Sun, Wei Xue, and Yike Guo\. 2025\.Delta decompression for MoE\-based LLMs compression\.In*International Conference on Machine Learning \(ICML\)*, volume 267, pages 20497–20514\.
- Hu et al\. \(2022\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2022\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Huang et al\. \(2026\)Ruijun Huang, Fang Dong, Xin Zhang, Hengjie Cao, Zhendong Huang, Anrui Chen, Jixian Zhou, Mengyi Chen, Yifeng Yang, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P\. Dick, Yuan Cheng, Fan Yang, Tun Lu, Chun Zhang, and Li Shang\. 2026\.SD\-MoE: Spectral decomposition for effective expert specialization\.*arXiv preprint arXiv:2602\.12556*\.
- Jiang et al\. \(2024\)Albert Q\. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie\-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\. 2024\.Mixtral of experts\.*arXiv preprint arXiv:2401\.04088*\.
- Lasby et al\. \(2026\)Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa\. 2026\.REAP the experts: Why pruning prevails for one\-shot MoE compression\.In*International Conference on Learning Representations \(ICLR\)*\.
- Li et al\. \(2024\)Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi\-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen\. 2024\.Merge, then compress: Demystify efficient SMoE with hints from its routing policy\.In*International Conference on Learning Representations \(ICLR\)*\.
- Touvron et al\. \(2023\)Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample\. 2023\.LLaMA: Open and efficient foundation language models\.*arXiv preprint arXiv:2302\.13971*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.

## Appendix AAblations

#### Tree vs\. flat \(quality\)\.

A recurring question is whether organizing the low\-rank corrections as a hierarchy \(BTExperts\) helps compression*quality*relative to flat per\-cluster decomposition\. Table[6](https://arxiv.org/html/2608.07814#A1.T6)answers it in a controlled study on Mixtral\-8×\\times7B \(E=8E\{=\}8\) at matched compression: the two are indistinguishable within noise, both zero\-shot and after reconstruction fine\-tuning\. This confirms that BTExperts is a quality\-neutral*inference\-time*organization \(Sec\.[5](https://arxiv.org/html/2608.07814#S5)\), not a compression\-quality mechanism—its value is amortization, not accuracy\.

Table 6:Tree \(BTExperts\) vs\. flat per\-cluster decomposition on Mixtral\-8×\\times7B \(E=8E\{=\}8; layer 16, rank 32, 3 seeds\), at matched compression\.Δ\\DeltaPPL is the increase over the uncompressed baseline \(lower is better\); the hierarchy matches flat decomposition within noise\.

## Appendix BAdditional Co\-Activation Views

The backbone graph in the main text \(Fig\.[2](https://arxiv.org/html/2608.07814#S2.F2)\) summarizes the co\-activation structure at layer 23\. Figure[6](https://arxiv.org/html/2608.07814#A2.F6)shows the same kind of structure as a raw NPMI heatmap at a late layer, and Fig\.[7](https://arxiv.org/html/2608.07814#A2.F7)reports how the dense co\-activation cliques shift with network depth\.

![Refer to caption](https://arxiv.org/html/2608.07814v1/figs/L47_npmi.png)Figure 6:Union\-mean NPMI co\-activation matrix at Qwen3\-30B\-A3B layer 47 \(over 13 datasets\), experts reordered by community: co\-firing experts form clear diagonal blocks even though their weights are near\-orthogonal\.![Refer to caption](https://arxiv.org/html/2608.07814v1/figs/coact_cliques.png)Figure 7:Per\-layer counts of dense expert cliques by size \(Qwen3\-30B\-A3B\)\. Both the number and the size of co\-activation cliques vary with depth, so a uniform per\-layer compression budget is suboptimal\.