ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

# ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
Source: [https://arxiv.org/html/2608.24938](https://arxiv.org/html/2608.24938)
Yifei LiuJunyi ChenSiqi FanChaoran FengMinghao LiLiujie ZhangWeihang Cheng\\correspondingLi Yuan\\corresponding

###### Abstract

Mixture\-of\-Experts \(MoE\) models scale capacity for strong quality while keeping per\-token compute bounded through sparse expert activation\. Yet low\-latency MoE serving is increasingly challenging, because it spans two inference phases with fundamentally different bottlenecks: prefill is dominated by token\-wise expert computation, whereas decode is constrained by memory traffic from the batch\-wise activated expert set\. However, existing training\-free acceleration methods optimize only a single resource proxy—either the experts each token executes or the experts a batch activates—and, either discard the excluded experts’ contribution or leave it only implicitly approximated\. In this paper, we propose ExFold, a unified training\-free expert\-folding framework for jointly accelerating MoE prefill and decode\. ExFold casts both prefill and decode as one budgeted output\-approximation problem: execute only a phase\-specific constrained expert set while projecting the contribution of budget\-excluded experts onto retained experts using calibrated scalar projectors\. Motivated by the observation that many expert outputs are directionally aligned but differ in magnitude, ExFold calibrates a pairwise scalar\-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts\. Under this view, prefill acceleration becomes token\-level Top\-KKfolding, and decode acceleration becomes batch\-level expert\-pool folding\. The two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared folding mechanism\. We implement ExFold as a plug\-and\-play plugin in vLLM, with a lightweight expert\-folding CUDA kernel, delivering up to 1\.41×\\timesTTFT and 2\.45×\\timesTPOT speedups while retaining about 99% of the original average quality\. Code of ExFold can be seen inhttps://github\.com/Time\-Rune/ExFold\-MoE\.

## Introduction

Figure 1:MoE prefill and decode have different bottlenecks\.Figure 2:Quality–speed trade\-offs for ExFold prefill and decode acceleration\.Mixture\-of\-Experts \(MoE\) scales model parameters by orders of magnitude while keeping the compute budget bounded by activating only a sparse subset of parameters per token\([8](https://arxiv.org/html/2608.24938#bib.bib11);[16](https://arxiv.org/html/2608.24938#bib.bib12)\)\. This property has made MoE the mainstream design of choice for many recent large models seeking higher capability\([16](https://arxiv.org/html/2608.24938#bib.bib12);[26](https://arxiv.org/html/2608.24938#bib.bib14);[32](https://arxiv.org/html/2608.24938#bib.bib15)\)\. At the same time, it places stringent demands on low\-latency serving\([4](https://arxiv.org/html/2608.24938#bib.bib31);[3](https://arxiv.org/html/2608.24938#bib.bib32);[7](https://arxiv.org/html/2608.24938#bib.bib33)\)in production: both the prefill latency \(time\-to\-first\-token, TTFT\) and the decode latency \(time\-per\-output\-token, TPOT\) matter, and the two exhibit distinct computational characteristics\. As shown in Fig\.[1](https://arxiv.org/html/2608.24938#Sx1.F1), prefill is dominated by token\-level expert computation, whereas decode is dominated by batch\-level expert memory traffic\.

Existing training\-free MoE acceleration methods fall into two families, according to which of these two budgets they optimize\.*Token\-wise expert sparsification*reduces the number of experts each token executes, including Dynamic\-MoE\([11](https://arxiv.org/html/2608.24938#bib.bib1)\), MoDES\([12](https://arxiv.org/html/2608.24938#bib.bib9)\), NAEE\([22](https://arxiv.org/html/2608.24938#bib.bib2)\), and MC\-MoE\([20](https://arxiv.org/html/2608.24938#bib.bib5)\)\.*Expert\-set consolidation*instead shrinks the active expert set, through static pruning and merging such as REAP\([18](https://arxiv.org/html/2608.24938#bib.bib3)\), HC\-SMoE\([2](https://arxiv.org/html/2608.24938#bib.bib6)\), and Sub\-MoE\([19](https://arxiv.org/html/2608.24938#bib.bib8)\), and dynamic batch\-level restriction such as Lynx\([9](https://arxiv.org/html/2608.24938#bib.bib10)\)and SERE\([31](https://arxiv.org/html/2608.24938#bib.bib4)\)\.

Yet both families frame acceleration as a resource\-reduction problem:they answer only which experts to execute, and leave the harder question unaddressed—what becomes of the experts they cut\. Their treatment of the excluded contribution is, at best, incidental\. Pruning and skipping discard it outright\([22](https://arxiv.org/html/2608.24938#bib.bib2)\)\. Static merging bakes it into a single, permanently compressed model that can no longer adapt to the per\-token or per\-batch budget it actually faces at inference\([20](https://arxiv.org/html/2608.24938#bib.bib5)\)\. Similarity\-based re\-routing swaps an excluded expert for a nearby one, but never calibrates how far the substitute’s output strays from the contribution it replaces\([31](https://arxiv.org/html/2608.24938#bib.bib4)\)\. Lost, frozen, or only implicitly approximated, the excluded expert mass is never explicitly reconstructed—so the approximation error compounds as the budget is tightest and these methods are pushed hardest\.

We introduce*Expert Folding*, which projects every budget\-excluded expert contribution onto an executed expert instead of discarding it, and build ExFold around it\. This design follows two empirical observations \(Figure[3](https://arxiv.org/html/2608.24938#Sx2.F3)\): many source experts have at least one target with directionally aligned outputs, while output magnitudes differ substantially across experts\. The former makes expert substitution possible; the latter explains why direct re\-routing is insufficient and motivates a directed scalar projector that corrects the source–target scale mismatch\. Concretely, a single frozen forward pass over unlabeled text calibrates two per\-layer matrices—a*scalar\-projector matrix*and a*projection\-loss matrix*: at inference, the loss matrix routes each excluded expert to its minimum\-loss target, and the scalar matrix folds each excluded contribution into that target’s router weight\.

Since folding is defined at the level of individual excluded experts, it applies unchanged in both phases\. Prefill selectsKpreK\_\{\\mathrm\{pre\}\}dominant experts per token, and decode selectsDDdominant experts per batch—both reuse the same scalar matrix, loss matrix, and folding operator to recover whatever falls outside the retained set\. This is the central design of ExFold: the two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared mechanism\.

We implement ExFold as a plug\-and\-play plugin in vLLM, with a lightweight expert\-folding CUDA kernel, delivering up to 1\.41×\\timesTTFT and 2\.45×\\timesTPOT speedups while retaining about 99% of the original average quality\. Our contributions are as follows:

- •Phase\-aware formulation\.We characterize the distinct expert budgets of prefill and decode and unify both as a single output\-approximation problem under phase\-specific selection constraints\.
- •Expert Folding\.We propose a training\-free folding mechanism that projects budget\-excluded expert contributions onto retained experts via a directional projector and a reconstruction\-loss matrix, whose scalar form is absorbed directly into the router weight\.
- •Evaluation\.We validate near\-lossless prefill, decode, and joint acceleration across multiple MoE architectures on real vLLM serving workloads\.

## Background and Motivation

### MoE Inference

![Refer to caption](https://arxiv.org/html/2608.24938v1/sim.png)Figure 3:\(a\-c\) MoE expert outputs can be aligned across experts and layers, and \(d\) their L2 norms vary substantially \.#### Sparse MoE layers\.

A sparse MoE layer replaces the dense FFN of a Transformer block withNNgated\-FFN experts and a router\([8](https://arxiv.org/html/2608.24938#bib.bib11);[16](https://arxiv.org/html/2608.24938#bib.bib12)\)\. For a hidden statexxof dimensiondd, experteecomputes

Ee​\(x\)=Wdown\(e\)​\(ϕ⁡\(Wgate\(e\)​x\)⊙Wup\(e\)​x\),E\_\{e\}\(x\)=W^\{\(e\)\}\_\{\\mathrm\{down\}\}\\left\(\\phi\\\!\\left\(W^\{\(e\)\}\_\{\\mathrm\{gate\}\}x\\right\)\\odot W^\{\(e\)\}\_\{\\mathrm\{up\}\}x\\right\),\(1\)whereϕ⁡\(⋅\)\\phi\(\\cdot\)is typically SiLU and⊙\\odotdenotes element\-wise multiplication\. The router produces logitsr⁡\(x\)=Wr​xr\(x\)=W\_\{r\}xoverNNrouted experts, selects the Top\-KKsetSK​\(x\)S\_\{K\}\(x\), and normalizes the gate weights within it:

αe​\(x\)=exp⁡\(re​\(x\)\)∑j∈SK​\(x\)exp⁡\(rj​\(x\)\),e∈SK​\(x\)\.\\alpha\_\{e\}\(x\)=\\frac\{\\exp\(r\_\{e\}\(x\)\)\}\{\\sum\_\{j\\in S\_\{K\}\(x\)\}\\exp\(r\_\{j\}\(x\)\)\},\\quad e\\in S\_\{K\}\(x\)\.\(2\)The MoE output is then

MoE⁡\(x\)=∑e∈SK​\(x\)αe​\(x\)​Ee​\(x\),\\mathrm\{MoE\}\(x\)=\\sum\_\{e\\in S\_\{K\}\(x\)\}\\alpha\_\{e\}\(x\)E\_\{e\}\(x\),\(3\)possibly with additional shared experts\. This computation makes the FFN path sparse per token, but it also introduces dynamic routing, expert dispatch, and expert\-output aggregation\.

#### Phase\-specific budgets\.

Although the same MoE layer \(Eq\.[3](https://arxiv.org/html/2608.24938#Sx2.E3)\) runs in both phases, its bottleneck shifts with the shape of the token batch \(Figure[1](https://arxiv.org/html/2608.24938#Sx1.F1)\)\. Prefill processes a prompt ofTTtokens at once, and each token independently activates its own supportSK​\(x\)S\_\{K\}\(x\), so the layer performs𝒪⁡\(T​K\)\\mathcal\{O\}\(TK\)expert\-FFN evaluations\. WhenTTis large, every loaded expert is reused across many tokens, so weight\-loading is amortized and the phase is compute\-bound on token\-wise expert FLOPs; reducingKKdirectly cuts this cost\. Decode advances each ofBBbatched requests by a single token per step, issuing onlyBBtokens, yet their supports⋃xSK​\(x\)\\bigcup\_\{x\}S\_\{K\}\(x\)typically cover most of theNNexperts\. The layer must therefore load this batch\-wise expert union to serve very few tokens each, yielding low arithmetic intensity and a memory\-bound phase whose cost tracks the number of distinct activated experts rather than FLOPs\. Efficient serving is thus governed by two different budgets: experts per token in prefill, and experts per batch in decode\.

### MoE Acceleration Methods

Depending on which budget they optimize, existing training\-free MoE acceleration methods fall into two families: token\-wise expert sparsification and expert\-set consolidation\.

#### Token\-wise expert sparsification\.

This family reduces the experts each token executes\. Dynamic\-MoE keeps per token the smallest expert set with cumulative router probability abovepp, so easier tokens use fewer experts\([11](https://arxiv.org/html/2608.24938#bib.bib1)\)\. MoDES skips experts per token by a calibration\-estimated importance\([12](https://arxiv.org/html/2608.24938#bib.bib9)\), NAEE prunes low\-importance executions from router or activation statistics\([22](https://arxiv.org/html/2608.24938#bib.bib2)\), and MC\-MoE merges co\-activated experts identified from calibration\([20](https://arxiv.org/html/2608.24938#bib.bib5)\)\. These methods relieve the prefill computation budget, but leave the decode\-time expert union uncontrolled\.

#### Expert\-set consolidation\.

This family shrinks the active expert set\. Static methods score every expert from calibration signals and permanently reduce the global pool: REAP prunes the least important experts by router weight and activation norm\([18](https://arxiv.org/html/2608.24938#bib.bib3)\), while HC\-SMoE\([2](https://arxiv.org/html/2608.24938#bib.bib6)\), REAM\([15](https://arxiv.org/html/2608.24938#bib.bib7)\), and Sub\-MoE\([19](https://arxiv.org/html/2608.24938#bib.bib8)\)merge similar experts\. Dynamic methods restrict the expert union within a decoding batch: Lynx re\-routes secondary requests to active experts\([9](https://arxiv.org/html/2608.24938#bib.bib10)\), and SERE calibrates pairwise expert similarity offline and substitutes each excluded expert with a similar retained one at decode\([31](https://arxiv.org/html/2608.24938#bib.bib4)\)\. These methods relieve decode\-side traffic, but their static transformations do not match the per\-token prefill budget\.

#### Gap\.

Both families decide*which*experts to execute, but neither preserves the contribution of the experts they exclude: pruning discards it, static merging bakes it into a permanently compressed model, and similarity re\-routing changes an expert’s destination without calibrating how far the substitute strays\. As the budget tightens, this error grows within each phase and compounds across prefill and decode, which are approximated against two different targets rather than one shared objective\.

Key Question *How can we design a unified framework to accelerate the prefill and decode of MoE models with minimal loss?*

## Design Insight

The gap makes recovery look hard from both sides—prefill and decode pull toward conflicting budgets, and faithfully restoring each excluded expert seems to require either recomputation or blind substitution—yet two observations dissolve both difficulties\.

#### The excluded contributions share one recovery target\.

Prefill and decode are constrained by different budgets—experts per token versus experts per batch—yet whichever experts a budget excludes, the residual it leaves behind has the identical formαe​\(x\)​Ee​\(x\)\\alpha\_\{e\}\(x\)E\_\{e\}\(x\)in the MoE output \(Eq\.[3](https://arxiv.org/html/2608.24938#Sx2.E3)\)\. The two phases therefore pose a single problem—reconstruct the excluded contributions from the retained experts—and differ only in the constraint that selects the retained set, not in what must be recovered\.

#### Expert redundancy is magnitude\-separable\.

Figure[3](https://arxiv.org/html/2608.24938#Sx2.F3)shows that the redundancy across experts is concentrated in one cheaply correctable degree of freedom—output scale\. In a two\-component PCA projection \(a\), raw expert outputs form a diffuse cloud, but normalizing each to unit scale collapses them onto a thin shared axis: once scale is removed, experts point in nearly the same direction\. Pairwise similarity confirms this—normalization lifts the average expert\-to\-expert similarity from0\.3350\.335to0\.5290\.529at layer 5 and from0\.2510\.251to0\.4710\.471at layer 17 \(b\)—and the same gap persists across all4848layers \(c\), so the alignment is structural rather than a few\-layer artifact\. Yet the discarded scale is large: raw output norms span over3×3\\timeswithin a layer \(max/min=3\.04×\\max/\\min=3\.04\\timesand3\.27×3\.27\\timesfor the two shown, d\)\. The mismatch between an excluded and a retained expert is therefore almost purely radial—a compatible direction is already available, only its magnitude is off\. A single per\-pair scalar that rescales the retained expert—not a reconstructed vector or a blind substitution—thus bounds how far the substitute strays and makes faithful recovery cheap in principle\. Full\-layer visualization is provided in Appendix D\.

Together these give the motivating question an affirmative answer: one recovery target, reached by a cheap per\-pair scalar correction and shared across both phases, with each phase changing only how the retained set is selected\. ExFold turns this into a concrete mechanism in the next section\.

![Refer to caption](https://arxiv.org/html/2608.24938v1/Main.png)Figure 4:Overview ofExFold: training\-free projector calibration and unified expert folding\.

## Method

### Unified Expert Folding

ExFold treats MoE acceleration as constrained output approximation rather than phase\-specific expert dropping\. Letxix\_\{i\}denote theii\-th input token andSK​\(xi\)S\_\{K\}\(x\_\{i\}\)its original Top\-KKsupport\. We extend the router weight byαe​\(xi\)=0\\alpha\_\{e\}\(x\_\{i\}\)=0fore∉SK​\(xi\)e\\notin S\_\{K\}\(x\_\{i\}\)\. Under phaseϕ∈\{pre,dec\}\\phi\\in\\\{\\mathrm\{pre\},\\mathrm\{dec\}\\\},Bϕ​\(xi\)B\_\{\\phi\}\(x\_\{i\}\)denotes the retained target set andOϕ​\(xi\)=SK​\(xi\)∖Bϕ​\(xi\)O\_\{\\phi\}\(x\_\{i\}\)=S\_\{K\}\(x\_\{i\}\)\\setminus B\_\{\\phi\}\(x\_\{i\}\)the omitted source experts\. We useEsE\_\{s\}for an omitted source expert andEtE\_\{t\}for its retained target, witht=πϕ​\(s,xi\)∈Bϕ​\(xi\)t=\\pi\_\{\\phi\}\(s,x\_\{i\}\)\\in B\_\{\\phi\}\(x\_\{i\}\)\.

Letss→t⋆s\_\{s\\to t\}^\{\\star\}be a directed scalar projector such thatEs​\(xi\)≈ss→t⋆​Et​\(xi\)E\_\{s\}\(x\_\{i\}\)\\approx s\_\{s\\to t\}^\{\\star\}E\_\{t\}\(x\_\{i\}\), and writets=πϕ​\(s,xi\)t\_\{s\}=\\pi\_\{\\phi\}\(s,x\_\{i\}\)for the target assigned to sourcess\. The approximated MoE output separates the retained contributions from the projected contributions of omitted experts:

y^ϕ​\(xi\)\\displaystyle\\widehat\{y\}\_\{\\phi\}\(x\_\{i\}\)=∑t∈SK​\(xi\)∩Bϕ​\(xi\)αt​\(xi\)​Et​\(xi\)⏟retained\\displaystyle=\\underbrace\{\\sum\_\{t\\in S\_\{K\}\(x\_\{i\}\)\\cap B\_\{\\phi\}\(x\_\{i\}\)\}\\alpha\_\{t\}\(x\_\{i\}\)E\_\{t\}\(x\_\{i\}\)\}\_\{\\text\{retained\}\}\(4\)\+∑s∈Oϕ​\(xi\)αs​\(xi\)​ss→ts⋆​Ets​\(xi\)⏟folded\.\\displaystyle\+\\underbrace\{\\sum\_\{s\\in O\_\{\\phi\}\(x\_\{i\}\)\}\\alpha\_\{s\}\(x\_\{i\}\)s\_\{s\\to t\_\{s\}\}^\{\\star\}E\_\{t\_\{s\}\}\(x\_\{i\}\)\}\_\{\\text\{folded\}\}\.Thus, omitted experts are not executed, while their calibrated contributions remain in the MoE output\. Routes folded into the same target are coalesced before expert computation, so each retained target is evaluated only once\. Unlike static expert merging, folding changes neither expert parameters nor the original router\([20](https://arxiv.org/html/2608.24938#bib.bib5);[2](https://arxiv.org/html/2608.24938#bib.bib6)\)\.

For the tokens𝒳ϕ\\mathcal\{X\}\_\{\\phi\}processed together in one layer, both phases share the objective

min⁡∑xi∈𝒳ϕBϕ,πϕ,𝐒⁡‖MoE⁡\(xi\)−y^ϕ​\(xi\)‖22s\.t\.cϕ​\(Bϕ\)≤τϕ,\\min\_\{B\_\{\\phi\},\\pi\_\{\\phi\},\\mathbf\{S\}\}\\sum\_\{x\_\{i\}\\in\\mathcal\{X\}\_\{\\phi\}\}\\\|\\mathrm\{MoE\}\(x\_\{i\}\)\-\\widehat\{y\}\_\{\\phi\}\(x\_\{i\}\)\\\|\_\{2\}^\{2\}\\quad\\mathrm\{s\.t\.\}\\quad c\_\{\\phi\}\(B\_\{\\phi\}\)\\leq\\tau\_\{\\phi\},\(5\)where𝐒=\[ss→t⋆\]\\mathbf\{S\}=\[s\_\{s\\to t\}^\{\\star\}\]is the scalar\-projector table\. The prefill cost counts retained experts per token, whereas the decode cost counts distinct experts retained for a batch\. We next calibrate𝐒\\mathbf\{S\}once and then solve the two budget constraints with the same folding rule\.

### Training\-Free Projector Calibration

Figure[4](https://arxiv.org/html/2608.24938#Sx3.F4)summarizes the offline calibration\. For each ordered, co\-routed source\-target pair\(s,t\)\(s,t\), we collect their outputs onmmcalibration tokens:

𝐮i=Es\(xi\),𝐯i=Et\(xi\),i=1,…,m\.\\mathbf\{u\}\_\{i\}=E\_\{s\}\(x\_\{i\}\),\\qquad\\mathbf\{v\}\_\{i\}=E\_\{t\}\(x\_\{i\}\),\\qquad i=1,\\ldots,m\.\(6\)Here𝐮i,𝐯i∈ℝd\\mathbf\{u\}\_\{i\},\\mathbf\{v\}\_\{i\}\\in\\mathbb\{R\}^\{d\}, and each token is weighted bywi=‖𝐮i‖2w\_\{i\}=\\\|\\mathbf\{u\}\_\{i\}\\\|\_\{2\}in the main method\. Given a projector family𝒫\\mathcal\{P\}, calibration solves the weighted output\-reconstruction problem

𝐏s→t⋆=arg⁡min⁡∑i=1m𝐏∈𝒫⁡wi​‖𝐮i−𝐯i​𝐏‖22\+λ​Ω​\(𝐏\),\\mathbf\{P\}\_\{s\\to t\}^\{\\star\}=\\arg\\min\_\{\\mathbf\{P\}\\in\\mathcal\{P\}\}\\sum\_\{i=1\}^\{m\}w\_\{i\}\\\|\\mathbf\{u\}\_\{i\}\-\\mathbf\{v\}\_\{i\}\\mathbf\{P\}\\\|\_\{2\}^\{2\}\+\\lambda\\Omega\(\\mathbf\{P\}\),\(7\)whereΩ\\Omegaregularizes the parameters of the selected projector family\. For the scalar parameterization𝐏=s​𝐈\\mathbf\{P\}=s\\mathbf\{I\}, the solution is

ss→t⋆=∑i=1mwi​⟨𝐯i,𝐮i⟩∑i=1mwi​‖𝐯i‖22\+λ\.s\_\{s\\to t\}^\{\\star\}=\\frac\{\\sum\_\{i=1\}^\{m\}w\_\{i\}\\langle\\mathbf\{v\}\_\{i\},\\mathbf\{u\}\_\{i\}\\rangle\}\{\\sum\_\{i=1\}^\{m\}w\_\{i\}\\\|\\mathbf\{v\}\_\{i\}\\\|\_\{2\}^\{2\}\+\\lambda\}\.\(8\)
Besides the scalar form, we evaluate diagonal and scalar\-plus\-low\-rank projectors in Table[3](https://arxiv.org/html/2608.24938#Sx5.T3)\. The deployed method uses the scalar solution because it can be folded into router weights without an online vector transform\. We specify these alternatives, parameterization, and closed\-form calibration in Appendix B\.

Finally, scalar transfer error is stored as

ℓs→t=∑i=1mwi​‖𝐮i−ss→t⋆​𝐯i‖22∑i=1mwi​‖𝐮i‖22\.\\ell\_\{s\\to t\}=\\frac\{\\sum\_\{i=1\}^\{m\}w\_\{i\}\\\|\\mathbf\{u\}\_\{i\}\-s\_\{s\\to t\}^\{\\star\}\\mathbf\{v\}\_\{i\}\\\|\_\{2\}^\{2\}\}\{\\sum\_\{i=1\}^\{m\}w\_\{i\}\\\|\\mathbf\{u\}\_\{i\}\\\|\_\{2\}^\{2\}\}\.\(9\)Calibration therefore returns one directed scalar table𝐒\(l\)=\[ss→t⋆\]\\mathbf\{S\}^\{\(l\)\}=\[s\_\{s\\to t\}^\{\\star\}\]and loss table𝐋\(l\)=\[ℓs→t\]\\mathbf\{L\}^\{\(l\)\}=\[\\ell\_\{s\\to t\}\]per MoE layer, without labels, gradients, or model updates\. Detailed settings, matrix, and collections are provided in Appendices A and D\.

MethodPrefillDecodeMATH500AIME24IFEvalIFBenchGPQALCB \(P/A\.@8\)Eval\+MMLUproAvg\.BaselineOriginal TopK=8100%100%97\.4064\.4883\.7329\.3163\.1368\.26 / 56\.8977\.4468\.5769\.04Prefill\-Only AccelerationPrefill TopK=675%100%95\.8064\.7983\.1826\.2764\.1469\.46 / 57\.3475\.6166\.3368\.20Prefill TopK=450%100%96\.0066\.5676\.0029\.0260\.6166\.47 / 57\.6373\.1765\.2866\.64MC\-MoE\-P450%100%96\.0064\.5879\.8526\.4845\.4568\.26 / 57\.1177\.4465\.4465\.44MoDES\-P450%100%95\.4067\.0882\.2630\.2643\.0669\.46 / 57\.7178\.6666\.5366\.59ExFold\-P450%100%97\.0066\.4684\.0029\.5362\.6370\.66/57\.8676\.8366\.9469\.26Decode\-Only AccelerationREAP\-D64100%50\.0%94\.8065\.7371\.1631\.2935\.3567\.07 / 52\.9975\.0051\.6161\.50REAP\-D32100%25\.0%70\.0023\.0233\.8324\.7411\.628\.00 / 2\.509\.765\.0422\.25SERE\-K4 \(ρ=0\.0\)100%S=4∗S\{=\}4^\{\\ast\}94\.0057\.0883\.5529\.1655\.5665\.27 / 50\.9764\.6363\.9364\.15SERE\-K2 \(ρ=0\.1\)100%S=2∗S\{=\}2^\{\\ast\}88\.2049\.1773\.7523\.2050\.0016\.77 / 5\.2442\.0760\.2850\.65ExFold\-D64100%50\.0%97\.0065\.3183\.9228\.1263\.6471\.26/58\.7674\.3966\.3668\.75ExFold\-D32100%25\.0%96\.8067\.2984\.4728\.0363\.6470\.06 / 57\.1975\.0066\.4568\.97Prefill & Decode AccelerationAll TopK=675%75\.0%95\.6065\.3181\.5227\.6659\.0968\.86 / 57\.1167\.6864\.6066\.29All TopK=450%50\.0%93\.6056\.8874\.3125\.3756\.3158\.08 / 42\.0726\.8357\.9756\.17MC\-MoE K=450%50\.0%93\.4057\.0874\.8625\.2039\.5257\.49 / 42\.5152\.4458\.4457\.30MoDES K=450%50\.0%96\.2063\.7581\.5228\.2942\.9370\.66 / 56\.5976\.2266\.1665\.72ExFold P4\+D6450%50\.0%97\.0067\.2982\.4428\.6062\.1271\.26 / 57\.2674\.3964\.8968\.50ExFold P4\+D3250%25\.0%96\.4065\.8379\.4828\.1458\.0869\.46 / 56\.2974\.3965\.0067\.10

Table 1:Qwen3\-30B\-A3B quality\.Boldmarks the best result per setting\. P4 denotes 4 experts per token in prefill, Dmmmeans a decode pool with sizemm, andS∗S^\{\\ast\}means SERE’s dynamic set\. P/A\.@8 means the score of Pass@8 and Avg@8\.
### Phase\-Specific Selection and Unified Folding

At inference time, prefill and decode use phase\-specific retained\-expert selectors but share the same transfer criterion and folding operator, parameterized by the calibrated scalar table𝐒\(l\)\\mathbf\{S\}^\{\(l\)\}and loss table𝐋\(l\)\\mathbf\{L\}^\{\(l\)\}\. Below, we omit the layer index, denote the router weight bywi,e=αe​\(xi\)w\_\{i,e\}=\\alpha\_\{e\}\(x\_\{i\}\), and useheh\_\{e\}for the cached output\-norm estimate of expertee\.

#### 1\. Select phase\-specific primary experts\.

As illustrated in Figure[4](https://arxiv.org/html/2608.24938#Sx3.F4), prefill selectsKpreK\_\{\\mathrm\{pre\}\}primary experts independently for each token, whereas decode selects a shared set of at mostDDexperts for the current batch𝒳q\\mathcal\{X\}\_\{q\}:

Bpre​\(xi\)\\displaystyle B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)=TopKe∈SK​\(xi\)⁡\(wi,e​he,Kpre\),\\displaystyle=\\operatorname\{TopK\}\_\{e\\in S\_\{K\}\(x\_\{i\}\)\}\\bigl\(w\_\{i,e\}h\_\{e\},K\_\{\\mathrm\{pre\}\}\\bigr\),\(10\)Bdec​\(𝒳q\)\\displaystyle B\_\{\\mathrm\{dec\}\}\(\\mathcal\{X\}\_\{q\}\)=TopKe∈∪iSK\(xi\)\(∑xi∈𝒳qwi,ehe,D\)\.\\displaystyle=\\operatorname\{TopK\}\_\{e\\in\\cup\_\{i\}S\_\{K\}\(x\_\{i\}\)\}\\left\(\\sum\_\{x\_\{i\}\\in\\mathcal\{X\}\_\{q\}\}w\_\{i,e\}h\_\{e\},D\\right\)\.\(11\)The scorewi,e​hew\_\{i,e\}h\_\{e\}estimates the magnitude of each routed contribution\. Prefill applies it locally to reduce token\-level computation; decode aggregates it across tokens to reduce the batch\-level expert union\.

#### 2\. Choose minimum\-loss transfers\.

For either phase, each omitted source expertEsE\_\{s\}selects the retained target with the smallest calibrated reconstruction loss:

πϕ​\(s\)=arg⁡mint∈Bϕ⁡ℓs→t,\\pi\_\{\\phi\}\(s\)=\\arg\\min\_\{t\\in B\_\{\\phi\}\}\\ell\_\{s\\to t\},\(12\)whereBϕB\_\{\\phi\}is the token\-level setBpre​\(xi\)B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)or the batch\-level setBdec​\(𝒳q\)B\_\{\\mathrm\{dec\}\}\(\\mathcal\{X\}\_\{q\}\)\. The two phases therefore share the same loss lookup and differ only in the candidate target set\.

#### 3\. Fold router metadata\.

LetSs→t=ss→t⋆S\_\{s\\to t\}=s\_\{s\\to t\}^\{\\star\}denote the scalar\-table lookup\. In prefill, omitted routes mapped to the same target are coalesced by updating the target weight:

w~i,t=wi,t\+∑s∈SK​\(xi\)∖Bpre​\(xi\)πpre​\(s\)=twi,s​Ss→t,t∈Bpre​\(xi\)\.\\widetilde\{w\}\_\{i,t\}=w\_\{i,t\}\+\\sum\_\{\\begin\{subarray\}\{c\}s\\in S\_\{K\}\(x\_\{i\}\)\\setminus B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)\\\\ \\pi\_\{\\mathrm\{pre\}\}\(s\)=t\\end\{subarray\}\}w\_\{i,s\}S\_\{s\\to t\},\\quad t\\in B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)\.\(13\)The fused MoE kernel consequently executes onlyKpreK\_\{\\mathrm\{pre\}\}routes\. In decode, each omitted route is instead remapped in place:

\(Es,wi,s\)⟼\(Eπdec​\(s\),wi,s​Ss→πdec​\(s\)\)\.\(E\_\{s\},w\_\{i,s\}\)\\longmapsto\\left\(E\_\{\\pi\_\{\\mathrm\{dec\}\}\(s\)\},w\_\{i,s\}S\_\{s\\to\\pi\_\{\\mathrm\{dec\}\}\(s\)\}\\right\)\.\(14\)Retained routes remain unchanged, while every remapped expert belongs toBdec​\(𝒳q\)B\_\{\\mathrm\{dec\}\}\(\\mathcal\{X\}\_\{q\}\); hence the batch touches at mostDDdistinct experts without changing the regular Top\-KKrouting layout\. Both transformations modify only router metadata before the existing fused MoE kernel\.

## Experiments

Table 2:GLM\-4\.5\-Air quality\.Boldfor the best per setting\.\(a\)Prefill acceleration\.

\(b\)Decode acceleration

\(c\)Offline acceleration

Figure 5:Online TTFT/TPOT speedups and offline serving throughput\.Table 3:Projector ablation under Top\-4 prefill\. Overheads use FP32 state and are relative to Prefill TopK=4\. HereΔ​K=4\\Delta K=4; R8/R16 incur2​Δ​K​R=64/1282\\Delta KR=64/128operations per hidden dimension\.L,H,CFFNL,H,C\_\{\\mathrm\{FFN\}\}: layers, hidden size, and one expert\-call time\.### Experimental Setup

#### Models\.

We evaluate Qwen3\-30B\-A3B as the primary model and GLM\-4\.5\-Air, DeepSeek\-V2\-Lite, DeepSeek\-V4\-Flash, Qwen3\.5\-35B\-A3B \(in Appendix C\) for cross\-architecture generalization\.

#### Benchmarks\.

We evaluate mathematical reasoning with MATH500 and AIME24, code generation with LiveCodeBenchV5 and HumanEval\+, instruction following with IFEval and IFBench, and knowledge reasoning with GPQA\-Diamond and MMLU\-Pro\([10](https://arxiv.org/html/2608.24938#bib.bib20);[14](https://arxiv.org/html/2608.24938#bib.bib24);[21](https://arxiv.org/html/2608.24938#bib.bib28);[33](https://arxiv.org/html/2608.24938#bib.bib25);[25](https://arxiv.org/html/2608.24938#bib.bib26);[28](https://arxiv.org/html/2608.24938#bib.bib27);[30](https://arxiv.org/html/2608.24938#bib.bib29)\)\. For efficiency, we report TTFT under prefill\-dominated serving, TPOT under generation\-dominated serving, and offline throughput in vLLM\([17](https://arxiv.org/html/2608.24938#bib.bib13)\)\. Metric and workload details are deferred to Appendix A\.

#### Baselines\.

For prefill, we compare with direct Top\-KKreduction, MC\-MoE, and MoDES under matched token\-wise expert budgets\. For decode, we compare with static expert pruning \(REAP\) and dynamic expert skipping \(SERE\)\. We report prefill\-only, decode\-only, and joint acceleration to distinguish phase\-specific quality loss from errors accumulated across both phases\([11](https://arxiv.org/html/2608.24938#bib.bib1);[18](https://arxiv.org/html/2608.24938#bib.bib3);[31](https://arxiv.org/html/2608.24938#bib.bib4)\)\.

Figure 6:Per\-token MoE output reconstruction under expert reduction\. Error denotes‖y^−y‖2\\\|\\widehat\{y\}\-y\\\|\_\{2\}\.
#### Hyperparameters\.

We calibrate ExFold on 32 unlabeled sequences of at most 4096 tokens, use source\-output\-norm weighting, and set the ridge coefficient toλ=10−3\\lambda=10^\{\-3\}\.P​X\\mathrm\{P\}XdenotesXXretained experts per prefill token, whileD​X\\mathrm\{D\}Xdenotes at mostXXactive experts per decode batch\. We implement ExFold with a custom Triton operator in vLLM and run efficiency experiments in BF16 on NVIDIA H800 GPUs\([29](https://arxiv.org/html/2608.24938#bib.bib30)\); complete calibration, hardware, and serving configurations are provided in Appendix A\.

### Main Quality Results

Tables[1](https://arxiv.org/html/2608.24938#Sx4.T1)and[2](https://arxiv.org/html/2608.24938#Sx5.T2)compare ExFold with phase\-specific baselines under matched execution budgets\. We focus on whether quality is preserved, rather than treating a lower expert count alone as an improvement\.

#### Prefill\-only acceleration\.

At the P4 budget, ExFold preserves the original Qwen3 average and outperforms all compute\-matched baselines \(69\.26 versus 66\.64 for Direct Top\-4\)\. The gains are most pronounced on instruction following and code generation, where hard dropping loses important routed contributions\. Thus, folding can halve prefill expert computation without the quality loss of direct sparsification\.

#### Decode\-only acceleration\.

The advantage of ExFold widens as the batch\-level expert budget becomes tighter\. At D32, it remains within 0\.07 points of the original model; SERE\-K2 recovers part of the quality lost by static pruning but remains substantially lower\. This trend shows that recovering omitted expert contributions becomes increasingly important when the active expert pool is aggressively constrained\.

#### Joint prefill and decode acceleration\.

Applying Direct Top\-4 to both phases compounds approximation error and reduces the average to 56\.17, while ExFold P4\+D64 retains 68\.50\. ExFold also remains stronger than MoDES when both phases are accelerated\. The shared projector therefore avoids stacking two independent dropping errors and supports both phase\-specific budgets under one approximation objective\.

#### Cross\-architecture generalization\.

On GLM\-4\.5\-Air, ExFold retains 97\.2% and 98\.4% of the original average quality under prefill\-only P4 and decode\-only D64, respectively\. When both phases are compressed, P4\+D32 still retains 96\.7%\. These results confirm that ExFold preserves quality beyond the primary model\. Additional DeepSeek\-V2\-Lite and Qwen3\.5 results can be seen in Appendix C\.

Table 4:Calibration\-corpus ablation for ExFold\-P4\.Table 5:Retained\-expert selection on Qwen3\-30B\-A3B\.

### Efficiency Evaluation

#### Online prefill latency\.

Figure[5](https://arxiv.org/html/2608.24938#Sx5.F5)\(a\) shows that reducing token\-level expert computation translates directly into lower TTFT\. ExFold reaches a1\.41×1\.41\\timesspeedup at 8 QPS and closely tracks compute\-matched Top\-4 methods, indicating little overhead from scalar folding\. Unlike direct reduction, it realizes this compute benefit while preserving model quality\.

#### Online decode latency\.

Figure[5](https://arxiv.org/html/2608.24938#Sx5.F5)\(b\) reveals a different scaling trend in decode: token\-wise Top\-4 and Top\-6 lose their benefit as QPS increases because the batch activates a broader expert union\. By bounding this union, ExFold reaches a2\.45×2\.45\\timesTPOT speedup at 8 QPS and sustains about2\.4×2\.4\\timesthrough 12 QPS\. This confirms that reducing batch\-level weight traffic, rather than token\-wise FLOPs alone, is essential for efficient decode\.

#### Offline serving throughput\.

Figure[5](https://arxiv.org/html/2608.24938#Sx5.F5)\(c\) shows that the online gains transfer to throughput\-oriented serving: ExFold improves offline throughput by1\.20×1\.20\\timesand exceeds Direct Top\-4\. This result also confirms that the Triton folding operator adds little runtime overhead under large batches\.

### Scaling to DeepSeek\-V4\-Flash

DeepSeek\-V4\-Flash is a 284B\-parameter MoE with 256 routed experts and Top\-6 routing\([6](https://arxiv.org/html/2608.24938#bib.bib17)\)\. It provides a substantially larger expert space than the models above and therefore tests whether folding remains effective when both the number of candidate experts and the batch\-level expert union grow\. We use P3 to retain three routed experts per prefill token and D128/D64 to cap the decode expert pool at 50%/25% of the routed experts\. All quality rows use the same full\-suite evaluator protocol; Appendix[F](https://arxiv.org/html/2608.24938#A6)reports the task sizes and sampling rules\.

Table 6:DeepSeek\-V4\-Flash quality\.Boldmarks the best result per setting\. P3 denotes 3 experts per token in prefill, and Dmmmeans a decode pool with sizemm\.Table 7:DeepSeek\-V4\-Flash speedups on H800:\(a\)8K TTFT,\(b\)256\-token TPOT, and\(c\)offline throughput\.Table[7](https://arxiv.org/html/2608.24938#Sx5.T7)shows that direct all\-stage K=3 reduction retains only 88\.17% of the original average, with the largest losses on AIME25/26\. ExFold P3\+D64 instead retains 97\.68%, a 9\.51\-point retention gain under the same three\-expert prefill budget, while P3\+D128 retains 99\.32%\. Decode\-only D64 and D128 remain within 0\.2 average points of Original\. These results support a quality frontier rather than one universal budget: D128 is the near\-lossless setting, whereas D64 trades 1\.64 additional quality\-retention points for a tighter expert pool\.

Figure[7](https://arxiv.org/html/2608.24938#Sx5.T7)follows the same online/offline structure as Figure[5](https://arxiv.org/html/2608.24938#Sx5.F5)\. P3 improves mean 8K TTFT across QPS 1–8, reaching 1\.32×\\timesat QPS 8\. In decode, D64 stabilizes near 1\.15×\\timesTPOT speedup once the batch is saturated, while D128 remains the quality\-first operating point\. Under the saturated throughput workload, D64 raises output throughput from 5\.22k to 6\.72k tokens/s \(1\.29×\\times\), ahead of Direct K=3, REAP\-D128, and D128\. The code release exposes both budgets as serving arguments and includes the calibrated matrix used by these runs\.

### Ablations and Analysis

#### Which projector parameterization is practical?

Table[3](https://arxiv.org/html/2608.24938#Sx5.T3)compares projector families under the same Top\-4 budget\. Pairwise scalars outperform global and layer\-wise scaling, while diagonal and low\-rank projectors require substantially more state and unfused vector transforms\. Although the diagonal projector lowers perplexity, the pairwise scalar performs better on six of seven downstream metrics; low\-rank variants further add 2\.44–4\.87 GiB of state without consistent gains\.Pairwise scalars give the best trade\-off\.In appendix B, we detail each implementation\.

#### Does ExFold require task\-specific calibration?

Table[4](https://arxiv.org/html/2608.24938#Sx5.T4)varies only the calibration domain\. Mixed calibration achieves the best average \(73\.00\), but all domains remain within 1\.9 points and task\-matched data is not consistently optimal\. This stability suggests that calibration captures expert geometry rather than benchmark\-specific behavior\.No task\-specific calibration is required\.

#### How should ExFold select retained experts?

Table[5](https://arxiv.org/html/2608.24938#Sx5.T5)compares router scoreSiS\_\{i\}, output magnitudeHiH\_\{i\}, and their product under fixed budgets\. Router score omits output scale, while magnitude alone omits routing relevance; their product approximates contribution magnitude and yields the best average in both phases\.Rank retained experts bySi×HiS\_\{i\}\\times H\_\{i\}\.

#### Why does expert folding preserve quality?

Figure[6](https://arxiv.org/html/2608.24938#Sx5.F6)visualizes the routed expert contributions and resulting MoE output for a representative token\. Direct Top\-4 and MoDES omit routed vectors and produce large reconstruction errors of 1\.83 and 0\.54, respectively\. REAP reduces the global expert pool and lowers the error to 0\.16 by selecting Top\-8 within the retained pool, but it still executes eight experts per token and therefore does not reduce expert FLOPs\([12](https://arxiv.org/html/2608.24938#bib.bib9);[18](https://arxiv.org/html/2608.24938#bib.bib3)\)\. ExFold instead transfers omitted contributions to directionally aligned retained experts and folds their calibrated scales into the gate weights, achieving the smallest error of 0\.07 with only four executed experts\.Folding recovers omitted contributions\.Additional token\-level cases are shown in Appendix E\.

## Conclusion

We presented ExFold, a training\-free framework for jointly accelerating MoE prefill and decode\. It combines phase\-specific retained\-set selectors with a shared directed projector that folds excluded contributions into router metadata\. Across MoE architectures, ExFold improves TTFT, TPOT, and serving throughput while preserving more quality than expert dropping\. Because folding changes only router metadata, it remains complementary to kernel, scheduling, and parallelism optimizations\. These results establish output recovery as a practical basis for unified MoE inference acceleration\.

## References

- Caoet al\.\(2026\)M\. Cao, K\. Chen, H\. Duan, Y\. Fang, Z\. Fei, T\. Gao, Ge Jiaye, M\. Li, H\. Liu, J\. Liu, Y\. Liu,et al\.OpenCompass: A Universal Evaluation Platform for Large Language Models\.arXiv preprint arXiv:2605\.19276\.Cited by:[Appendix A](https://arxiv.org/html/2608.24938#A1.SS0.SSS0.Px4.p1.1)\.
- Chenet al\.\(2025a\)I\. Chen, H\. Liu, W\. Sun, C\. Chao, Y\. Hsu, and C\. LeeRetraining\-Free Merging of Sparse Mixture\-of\-Experts via Hierarchical Clustering\.InInternational Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Expert\-set consolidation\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px2.p1.1),[Unified Expert Folding](https://arxiv.org/html/2608.24938#Sx4.SSx1.p2.3)\.
- Chenet al\.\(2025b\)J\. Chen, S\. Bai, Z\. Wang, S\. Wu, C\. Du, H\. Yang, R\. Gong, S\. Liu, F\. Wu, and G\. ChenPre3\{\}^\{3\}: enabling deterministic pushdown automata for faster structured llm generation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11253–11267\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p1.1)\.
- Chenet al\.\(2026\)J\. Chen, C\. Du, R\. Liu, S\. Yao, D\. Yan, J\. Liao, S\. Liu, F\. Wu, and G\. ChenTokenFlow: responsive llm text streaming serving under request burst via preemptive scheduling\.InProceedings of the 21st European Conference on Computer Systems,pp\. 497–513\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p1.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-AIDeepSeek\-V2: A Strong, Economical, and Efficient Mixture\-of\-Experts Language Model\.arXiv preprint arXiv:2405\.04434\.Cited by:[Table 8](https://arxiv.org/html/2608.24938#A1.T8.1.1.4.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-V4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[Appendix F](https://arxiv.org/html/2608.24938#A6.SS0.SSS0.Px1.p1.1),[Scaling to DeepSeek\-V4\-Flash](https://arxiv.org/html/2608.24938#Sx5.SSx4.p1.1)\.
- Duet al\.\(2026\)C\. Du, J\. Chen, H\. Tang, K\. Liu, T\. Lan, L\. Qu, C\. Niu, S\. Liu, G\. Chen, and F\. WuC2\{\}^\{2\}kv: compressed and composable kv cache reuse for efficient llm inference\.arXiv preprint arXiv:2607\.17715\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\.Journal of Machine Learning Research\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p1.1),[Sparse MoE layers\.](https://arxiv.org/html/2608.24938#Sx2.SSx1.SSS0.Px1.p1.1)\.
- Guptaet al\.\(2024\)V\. Gupta, J\. H\. Ju, K\. Sinha, A\. Gavrilovska, and A\. P\. IyerLynx: enabling efficient moe inference through dynamic batch\-aware expert selection\.arXiv preprint arXiv:2411\.08982\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Expert\-set consolidation\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px2.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring Mathematical Problem Solving with the MATH Dataset\.InAdvances in Neural Information Processing Systems,Cited by:[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.2.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Huanget al\.\(2024\)Q\. Huang, Z\. An, N\. Zhuang, M\. Tao, C\. Zhang, Y\. Jin, K\. Xu, K\. Xu, L\. Chen, S\. Huang, and Y\. FengHarder Tasks Need More Experts: Dynamic Routing in MoE Models\.arXiv preprint arXiv:2403\.07652\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Token\-wise expert sparsification\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px1.p1.1),[Baselines\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px3.p1.1)\.
- Huanget al\.\(2025\)Y\. Huang, Z\. Wang, Z\. Yuan, Y\. Ding, R\. Gong, J\. Guo, X\. Liu, and J\. ZhangMoDES: Accelerating Mixture\-of\-Experts Multimodal Large Language Models via Dynamic Expert Skipping\.arXiv preprint arXiv:2511\.15690\.Cited by:[Appendix E](https://arxiv.org/html/2608.24938#A5.p1.1),[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Token\-wise expert sparsification\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px1.p1.1),[Why does expert folding preserve quality?](https://arxiv.org/html/2608.24938#Sx5.SSx5.SSS0.Px4.p1.1)\.
- Hugging Face H4 Team \(2024\)Hugging Face H4 TeamMATH\-500\.Note:https://huggingface\.co/datasets/HuggingFaceH4/MATH\-500Benchmark datasetCited by:[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.2.1)\.
- Jainet al\.\(2024\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code\.arXiv preprint arXiv:2403\.07974\.Cited by:[Appendix A](https://arxiv.org/html/2608.24938#A1.SS0.SSS0.Px4.p1.1),[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.7.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Jhaet al\.\(2026\)S\. Jha, M\. Hashemzadeh, A\. Saheb Pasand, A\. Parviz, M\. Lee, and B\. KnyazevREAM: Merging Improves Pruning of Experts in LLMs\.arXiv preprint arXiv:2604\.04356\.Cited by:[Expert\-set consolidation\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px2.p1.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang F\. Bressandet al\.Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p1.1),[Sparse MoE layers\.](https://arxiv.org/html/2608.24938#Sx2.SSx1.SSS0.Px1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient Memory Management for Large Language Model Serving with PagedAttention\.InACM Symposium on Operating Systems Principles,Cited by:[Appendix A](https://arxiv.org/html/2608.24938#A1.SS0.SSS0.Px5.p1.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Lasbyet al\.\(2025\)M\. Lasby, I\. Lazarevich, N\. Sinnadurai, S\. Lie, Y\. Ioannou, and V\. ThangarasaREAP the Experts: Why Pruning Prevails for One\-Shot MoE Compression\.arXiv preprint arXiv:2510\.13999\.Cited by:[Appendix E](https://arxiv.org/html/2608.24938#A5.p1.1),[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Expert\-set consolidation\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px2.p1.1),[Baselines\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px3.p1.1),[Why does expert folding preserve quality?](https://arxiv.org/html/2608.24938#Sx5.SSx5.SSS0.Px4.p1.1)\.
- Liet al\.\(2025\)L\. Li, Q\. Zhu, J\. Wang, W\. Li, H\. Gu, S\. Han, and Y\. GuoSub\-MoE: Efficient Mixture\-of\-Expert LLMs Compression via Subspace Expert Merging\.arXiv preprint arXiv:2506\.23266\.Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Expert\-set consolidation\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)P\. Li, Z\. Zhang, P\. Yadav, Y\. Sung, Y\. Cheng, M\. Bansal, and T\. ChenMerge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy\.InThe Twelfth International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Introduction](https://arxiv.org/html/2608.24938#Sx1.p3.1),[Token\-wise expert sparsification\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px1.p1.1),[Unified Expert Folding](https://arxiv.org/html/2608.24938#Sx4.SSx1.p2.3)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix A](https://arxiv.org/html/2608.24938#A1.SS0.SSS0.Px4.p1.1),[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.8.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Luet al\.\(2024\)X\. Lu, Q\. Liu, Y\. Xu, A\. Zhou, S\. Huang, B\. Zhang, J\. Yan, and H\. LiNot All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture\-of\-Experts Large Language Models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Introduction](https://arxiv.org/html/2608.24938#Sx1.p3.1),[Token\-wise expert sparsification\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px1.p1.1)\.
- Math\-AI Team \(2024\)Math\-AI TeamAmerican Invitational Mathematics Examination \(AIME\) 2024\.Note:https://huggingface\.co/datasets/math\-ai/aime24Benchmark datasetCited by:[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.3.1)\.
- NVIDIA \(2025\)NVIDIANVIDIA Nemotron Post\-Training Dataset v1\.Note:https://huggingface\.co/datasets/nvidia/Nemotron\-Post\-Training\-Dataset\-v1Dataset cardCited by:[Appendix A](https://arxiv.org/html/2608.24938#A1.SS0.SSS0.Px3.p1.1)\.
- Pyatkinet al\.\(2025\)V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. HajishirziGeneralizing Verifiable Instruction Following\.arXiv preprint arXiv:2507\.02833\.Cited by:[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.5.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3\-30B\-A3B\.Note:https://huggingface\.co/Qwen/Qwen3\-30B\-A3BModel cardCited by:[Table 8](https://arxiv.org/html/2608.24938#A1.T8.1.1.2.1),[Introduction](https://arxiv.org/html/2608.24938#Sx1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5\-35B\-A3B\.Note:https://huggingface\.co/Qwen/Qwen3\.5\-35B\-A3BModel cardCited by:[Table 8](https://arxiv.org/html/2608.24938#A1.T8.1.1.5.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: A Graduate\-Level Google\-Proof Q&A Benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.6.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Tilletet al\.\(2019\)P\. Tillet, H\. T\. Kung, and D\. CoxTriton: An Intermediate Language and Compiler for Tiled Neural Network Computations\.InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages,Cited by:[Appendix A](https://arxiv.org/html/2608.24938#A1.SS0.SSS0.Px5.p1.1),[Hyperparameters\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px4.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-Pro: A More Robust and Challenging Multi\-Task Language Understanding Benchmark\.arXiv preprint arXiv:2406\.01574\.Cited by:[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.9.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Wuet al\.\(2026\)J\. Wu, J\. Cheng, F\. Lv, D\. Ou, and L\. YuanSERE: Similarity\-based Expert Re\-routing for Efficient Batch Decoding in MoE Models\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.24938#Sx1.p2.1),[Introduction](https://arxiv.org/html/2608.24938#Sx1.p3.1),[Expert\-set consolidation\.](https://arxiv.org/html/2608.24938#Sx2.SSx2.SSS0.Px2.p1.1),[Baselines\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px3.p1.1)\.
- Z\.ai \(2025\)Z\.aiGLM\-4\.5\-Air\.Note:https://huggingface\.co/zai\-org/GLM\-4\.5\-AirModel cardCited by:[Table 8](https://arxiv.org/html/2608.24938#A1.T8.1.1.3.1),[Introduction](https://arxiv.org/html/2608.24938#Sx1.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-Following Evaluation for Large Language Models\.arXiv preprint arXiv:2311\.07911\.Cited by:[Table 9](https://arxiv.org/html/2608.24938#A1.T9.1.1.4.1),[Benchmarks\.](https://arxiv.org/html/2608.24938#Sx5.SSx1.SSS0.Px2.p1.1)\.

## Appendix AReproducibility Details

#### Model architectures\.

Table[8](https://arxiv.org/html/2608.24938#A1.T8)summarizes the evaluated released checkpoints\. We report the architecture exposed by each checkpoint configuration; shared experts are executed in addition to the routed Top\-KKexperts\. “Expert FFN” is the hidden width of each expert feed\-forward network\.

Table 8:Architectures of the evaluated MoE checkpoints\.
#### Benchmarks and metrics\.

Table[9](https://arxiv.org/html/2608.24938#A1.T9)lists the evaluated benchmarks\. “Cases” is the number of problems in the evaluated benchmark snapshot, and “Samples” is the number of generations per problem\. The GLM\-4\.5\-Air MMLU\-Pro comparison uses a fixed 490\-case subset containing the first 35 examples from each of 14 categories\. For IFBench, we evaluate the 294\-case OpenCompass snapshot and average prompt\- and instruction\-level accuracy under strict and loose checking\. For LiveCodeBench v5, we use the same fixed 167\-problem subset for every method and generate eight solutions per problem\. The Qwen3\-30B\-A3B evaluation uses 32 samples for each AIME 2024 problem, whereas the GLM\-4\.5\-Air evaluation uses eight\. LiveCodeBench reports pass@8 and mean correctness over the eight samples; pass@8 is its primary score in aggregate comparisons\.

Table 9:Benchmarks, evaluation sizes, and reported metrics\.
#### Calibration protocol\.

For the primary Qwen3 configuration, we calibrate once on 32 unlabeled sequences of at most 4,096 tokens and keep the resulting projector fixed across downstream tasks\. DeepSeek\-V2\-Lite uses the same calibration size and token limit; Qwen3\.5 aggregates four disjoint eight\-sequence calibration shards\. GLM\-4\.5\-Air uses a separate 16\-sequence capture with at most 1,024 tokens per sequence\. The 32\-sequence pool is drawn from the chat, code, and science, technology, engineering, and mathematics \(STEM\) portions of the public NVIDIA Nemotron Post\-Training Dataset\([24](https://arxiv.org/html/2608.24938#bib.bib23)\)\. All calibrations accumulate only forward activations, without task labels, gradients, or evaluator feedback\. For each ordered co\-routed expert pair, we accumulate the sufficient statistics in Eq\.[15](https://arxiv.org/html/2608.24938#A2.E15)with source\-output\-norm weighting andλ=10−3\\lambda=10^\{\-3\}\. Scalar coefficients are clipped to\[−4,4\]\[\-4,4\], self\-pair coefficients are fixed to one, and unobserved pairs receive loss103010^\{30\}so that an observed target is preferred whenever one exists\.

#### Quality evaluation\.

Generation is orchestrated by an OpenCompass\-compatible pipeline\([1](https://arxiv.org/html/2608.24938#bib.bib19)\)\. IFEval disables thinking and uses prompt\-level strict scoring, while IFBench uses the four\-way composite defined above\. LiveCodeBench and HumanEval\+ are scored by executing generated programs against their benchmark test suites with the LiveCodeBench and EvalPlus evaluators\([14](https://arxiv.org/html/2608.24938#bib.bib24);[21](https://arxiv.org/html/2608.24938#bib.bib28)\)\. Within each model–benchmark comparison, prompts, decoding parameters, sample counts, and scoring code are fixed\. Repeated generations are scored individually before aggregation\. All reported scores are percentages, and “Avg\.” is the arithmetic mean of the displayed primary metrics\.

#### System and serving configuration\.

Efficiency experiments run on eight\-GPU NVIDIA H800 80GB servers\. The primary H800 runtime uses Ubuntu 24\.04\.2 LTS, NVIDIA driver 550\.163\.01, vLLM 0\.10\.2\([17](https://arxiv.org/html/2608.24938#bib.bib13)\), PyTorch 2\.8\.0, CUDA 12\.8, and Triton 3\.4\.0\([29](https://arxiv.org/html/2608.24938#bib.bib30)\)\. Prefill serving uses tensor parallelism four, sweeps 1–8 queries per second \(QPS\) with 8,192\-token prompts and one generated token, and reports mean time to first token \(TTFT\)\. The matched decode comparison uses vLLM 0\.11\.0 and tensor parallelism one; it sweeps 2–12 QPS over 512 requests with one input token and 256 generated tokens and reports mean time per output token \(TPOT\)\. Offline serving uses tensor parallelism eight and reports completed\-request throughput for 512 requests with 2,048\-token inputs and 512 generated tokens\. ExFold speedups are normalized to an Original run with the same hardware, engine configuration, requests, and decoding parameters\. DeepSeek\-V2\-Lite and Qwen3\.5 quality generation use bfloat16 \(BF16\) and tensor parallelism one on H800 or H20 GPUs; GLM\-4\.5\-Air uses tensor parallelism four on H800 GPUs\.

## Appendix BProjector Parameterizations and Implementation

LetLL,EE, andHHdenote the number of MoE layers, routed experts per layer, and hidden dimensions\. For a calibration tokenxix\_\{i\}that co\-routes a source expertEsE\_\{s\}and target expertEtE\_\{t\}, we write𝐮i=Es​\(xi\)\\mathbf\{u\}\_\{i\}=E\_\{s\}\(x\_\{i\}\)and𝐯i=Et​\(xi\)\\mathbf\{v\}\_\{i\}=E\_\{t\}\(x\_\{i\}\);ll,ss, andttindex the layer, source expert, and target expert\. For the projector\-family ablation, every family uses the same fixed calibration sequences, source\-output\-norm weighting, and prefill Top\-4 budget\. Each sharing pattern computes its own reconstruction losses for target assignment\. The diagonal and low\-rank alternatives execute through a custom unfused reconstruction path\. Table[10](https://arxiv.org/html/2608.24938#A2.T10)counts transformation parameters and omits the per\-pair assignment\-loss table\. LetP≤L​E2P\\leq LE^\{2\}denote the number of materialized directed source–target pairs\.

#### Scalar sharing patterns\.

Let𝒢\\mathcal\{G\}denote a group of calibration tuples\(l,s,t,i\)\(l,s,t,i\)\. A scalar shared by that group is fitted by

s𝒢⋆=∑\(l,s,t,i\)∈𝒢wi​𝐮i⊤​𝐯i∑\(l,s,t,i\)∈𝒢wi​‖𝐯i‖22\+λ\.s\_\{\\mathcal\{G\}\}^\{\\star\}=\\frac\{\\sum\_\{\(l,s,t,i\)\\in\\mathcal\{G\}\}w\_\{i\}\\,\\mathbf\{u\}\_\{i\}^\{\\top\}\\mathbf\{v\}\_\{i\}\}\{\\sum\_\{\(l,s,t,i\)\\in\\mathcal\{G\}\}w\_\{i\}\\,\\\|\\mathbf\{v\}\_\{i\}\\\|\_\{2\}^\{2\}\+\\lambda\}\.\(15\)Herewi=‖𝐮i‖2w\_\{i\}=\\\|\\mathbf\{u\}\_\{i\}\\\|\_\{2\}is the source\-output\-norm weight used by the calibration implementation\. The global variant uses one group for the entire model, the layer\-wise variant uses one group per MoE layer, and the pairwise variant uses one group for each directed source–target pair in each layer\. Only the pairwise form preserves directed expert\-level differences while remaining a scalar metadata update\. For target assignment, we use the normalized reconstruction loss

ℓs→t=∑iwi​‖𝐮i−ss→t⋆​𝐯i‖22∑iwi​‖𝐮i‖22\+ϵ,\\ell\_\{s\\to t\}=\\frac\{\\sum\_\{i\}w\_\{i\}\\\|\\mathbf\{u\}\_\{i\}\-s\_\{s\\to t\}^\{\\star\}\\mathbf\{v\}\_\{i\}\\\|\_\{2\}^\{2\}\}\{\\sum\_\{i\}w\_\{i\}\\\|\\mathbf\{u\}\_\{i\}\\\|\_\{2\}^\{2\}\+\\epsilon\},\(16\)whereϵ\>0\\epsilon\>0prevents division by zero\. Lower loss indicates thatEtE\_\{t\}better reconstructs the contribution ofEsE\_\{s\}\.

#### Diagonal projector\.

The diagonal alternative fits one coefficient per hidden dimension\. Its element\-wise weighted ridge solution is

𝐝s→t⋆=∑iwi​\(𝐯i⊙𝐮i\)∑iwi​\(𝐯i⊙𝐯i\)\+λ​𝟏,\\mathbf\{d\}\_\{s\\to t\}^\{\\star\}=\\frac\{\\sum\_\{i\}w\_\{i\}\(\\mathbf\{v\}\_\{i\}\\odot\\mathbf\{u\}\_\{i\}\)\}\{\\sum\_\{i\}w\_\{i\}\(\\mathbf\{v\}\_\{i\}\\odot\\mathbf\{v\}\_\{i\}\)\+\\lambda\\mathbf\{1\}\},\(17\)where the division is element\-wise\. The resultingDiag⁡\(𝐝s→t⋆\)\\operatorname\{Diag\}\(\\mathbf\{d\}\_\{s\\to t\}^\{\\star\}\)cannot be absorbed into a router weight and must act on the hidden vector online\.

#### Scalar\-plus\-low\-rank projector\.

Let𝐔,𝐕∈ℝm×H\\mathbf\{U\},\\mathbf\{V\}\\in\\mathbb\{R\}^\{m\\times H\}stack source and target outputs, and𝐖=Diag⁡\(w1,…,wm\)\\mathbf\{W\}=\\operatorname\{Diag\}\(w\_\{1\},\\ldots,w\_\{m\}\)\. We fit a layer\-wise orthonormal basis𝐐\(l\)∈ℝH×R\\mathbf\{Q\}^\{\(l\)\}\\in\\mathbb\{R\}^\{H\\times R\}to centered scalar\-residual samples by randomized truncated singular value decomposition \(SVD\) and parameterize

𝐌s→t=ss→t⋆​𝐈\+𝐐\(l\)​𝐂s→t,𝐂s→t∈ℝR×H\.\\mathbf\{M\}\_\{s\\to t\}=s\_\{s\\to t\}^\{\\star\}\\mathbf\{I\}\+\\mathbf\{Q\}^\{\(l\)\}\\mathbf\{C\}\_\{s\\to t\},\\qquad\\mathbf\{C\}\_\{s\\to t\}\\in\\mathbb\{R\}^\{R\\times H\}\.\(18\)Defining𝐙=𝐖1/2​𝐕𝐐\(l\)\\mathbf\{Z\}=\\mathbf\{W\}^\{1/2\}\\mathbf\{V\}\\mathbf\{Q\}^\{\(l\)\},𝐑s→t=𝐖1/2​\(𝐔−ss→t⋆​𝐕\)\\mathbf\{R\}\_\{s\\to t\}=\\mathbf\{W\}^\{1/2\}\(\\mathbf\{U\}\-s\_\{s\\to t\}^\{\\star\}\\mathbf\{V\}\), and𝐆=𝐙⊤​𝐙\\mathbf\{G\}=\\mathbf\{Z\}^\{\\top\}\\mathbf\{Z\}, weighted ridge regression gives

𝐂s→t⋆=\(𝐆\+λ​𝐈\)−1​𝐙⊤​𝐑s→t\.\\mathbf\{C\}\_\{s\\to t\}^\{\\star\}=\(\\mathbf\{G\}\+\\lambda\\mathbf\{I\}\)^\{\-1\}\\mathbf\{Z\}^\{\\top\}\\mathbf\{R\}\_\{s\\to t\}\.\(19\)The evaluated artifact collects a rank\-32 basis and retains at most 1,024 directed pairs per layer ranked by accumulated source\-weighted source energy\. We evaluate its firstR∈\{8,16\}R\\in\\\{8,16\\\}directions; pairs outside the materialized support use only the scalar term\. The basis and pair\-specific correction require an online vector transform and substantial state, preventing reuse of the standard fused MoE path\.

Table 10:Implementation differences among the evaluated projector families\.
### Phase\-Specific Routing and Folding

All retained sets and projector lookups below are defined separately for each MoE layer; we omit the layer index for clarity\. Let the original routes for tokenxix\_\{i\}beSK​\(xi\)=\(ei,1,…,ei,K\)S\_\{K\}\(x\_\{i\}\)=\(e\_\{i,1\},\\ldots,e\_\{i,K\}\), ordered by router score, and letwi,ew\_\{i,e\}be the routed weight of experteeandheh\_\{e\}its cached output\-norm estimate;wi,e=0w\_\{i,e\}=0whene∉SK​\(xi\)e\\notin S\_\{K\}\(x\_\{i\}\)\. LetKpreK\_\{\\mathrm\{pre\}\}be the per\-token prefill budget, let𝒳q\\mathcal\{X\}\_\{q\}be the tokens processed in decode stepqq, letB=\|𝒳q\|B=\|\\mathcal\{X\}\_\{q\}\|, and letDDbe the decode expert\-pool budget\. The prefill path ranks the token’s routed experts by estimated contribution magnitude and retainsKpreK\_\{\\mathrm\{pre\}\}of them:

Bpre​\(xi\)=TopKe∈SK​\(xi\)⁡\(wi,e​he,Kpre\)\.B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)=\\operatorname\{TopK\}\_\{e\\in S\_\{K\}\(x\_\{i\}\)\}\\left\(w\_\{i,e\}h\_\{e\},K\_\{\\mathrm\{pre\}\}\\right\)\.\(20\)Decode supports two retained\-pool selectors that share the same folding operator\. The dynamic path used for the cross\-model results and online decode measurement ranks the union of routed experts by its aggregate contribution:

Bdecdyn\(𝒳q\)=TopKe∈∪iSK\(xi\)\(∑xi∈𝒳qwi,ehe,D\)\.B\_\{\\mathrm\{dec\}\}^\{\\mathrm\{dyn\}\}\(\\mathcal\{X\}\_\{q\}\)=\\operatorname\{TopK\}\_\{e\\in\\cup\_\{i\}S\_\{K\}\(x\_\{i\}\)\}\\left\(\\sum\_\{x\_\{i\}\\in\\mathcal\{X\}\_\{q\}\}w\_\{i,e\}h\_\{e\},D\\right\)\.\(21\)An additional static implementation removes this batch reduction from the per\-step decode path\. It forms a pool once from the calibration\-time norms,

Bdecstatic=TopKe∈\{1,…,E\}⁡\(he,D\),B\_\{\\mathrm\{dec\}\}^\{\\mathrm\{static\}\}=\\operatorname\{TopK\}\_\{e\\in\\\{1,\\ldots,E\\\}\}\(h\_\{e\},D\),\(22\)and precomputes one target and scalar for every possible source expert\. Given any retained setBϕB\_\{\\phi\}, an omitted source expert selects the target with minimum calibrated projection loss:

π⁡\(s∣Bϕ\)=arg⁡mint∈Bϕ⁡ℓs→t,\\pi\(s\\mid B\_\{\\phi\}\)=\\arg\\min\_\{t\\in B\_\{\\phi\}\}\\ell\_\{s\\to t\},\(23\)whereBϕB\_\{\\phi\}is the prefill set in Eq\.[20](https://arxiv.org/html/2608.24938#A2.E20)or either decode pool in Eqs\.[21](https://arxiv.org/html/2608.24938#A2.E21)–[22](https://arxiv.org/html/2608.24938#A2.E22)\.

#### Prefill folding\.

The prefill selector places the retained routes from Eq\.[20](https://arxiv.org/html/2608.24938#A2.E20)first, after which the Triton operator coalesces omitted routes assigned to the same retained target:

w~i,t=wi,t\+∑s∈SK​\(xi\)∖Bpre​\(xi\)π⁡\(s∣Bpre​\(xi\)\)=twi,s​ss→t⋆,t∈Bpre​\(xi\)\.\\widetilde\{w\}\_\{i,t\}=w\_\{i,t\}\+\\sum\_\{\\begin\{subarray\}\{c\}s\\in S\_\{K\}\(x\_\{i\}\)\\setminus B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)\\\\ \\pi\(s\\mid B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)\)=t\\end\{subarray\}\}w\_\{i,s\}s\_\{s\\to t\}^\{\\star\},\\quad t\\in B\_\{\\mathrm\{pre\}\}\(x\_\{i\}\)\.\(24\)The operator does not materialize omitted expert outputs\. Retained expert identifiers remain unchanged, so the fused MoE kernel executes onlyKpreK\_\{\\mathrm\{pre\}\}experts per token\.

#### Decode remapping\.

Decode first obtains a dynamic or static pool\. Each routed expert outside that pool selects its target using Eq\.[23](https://arxiv.org/html/2608.24938#A2.E23)and is remapped in place:

\(Es,wi,s\)⟼\(Eπ⁡\(s∣Bdec\),wi,s​ss→π⁡\(s∣Bdec\)⋆\)\.\(E\_\{s\},w\_\{i,s\}\)\\longmapsto\\left\(E\_\{\\pi\(s\\mid B\_\{\\mathrm\{dec\}\}\)\},w\_\{i,s\}s\_\{s\\to\\pi\(s\\mid B\_\{\\mathrm\{dec\}\}\)\}^\{\\star\}\\right\)\.\(25\)The lookup and remapping occur before dispatch, so every routed identifier presented to the fused kernel belongs to a pool of at mostDDexperts\.

#### Decode kernels\.

HereBdecB\_\{\\mathrm\{dec\}\}is the dynamic or static pool selected above\. The evaluated dynamic path scans theB​KBKrouted slots to build Eq\.[21](https://arxiv.org/html/2608.24938#A2.E21), then searchesDDcandidate targets for each omitted route\. The static implementation instead cachesπ⁡\(s∣Bdec\)\\pi\(s\\mid B\_\{\\mathrm\{dec\}\}\)andss→π⁡\(s∣Bdec\)⋆s\_\{s\\to\\pi\(s\\mid B\_\{\\mathrm\{dec\}\}\)\}^\{\\star\}for allEEsources\. Listing[1](https://arxiv.org/html/2608.24938#listing1)is the resulting per\-route kernel: one target and scalar lookup per routed slot, orO⁡\(B​K\)O\(BK\)work, while preserving the original\[B,K\]\[B,K\]layout\.

@triton\.jit

def\_static\_projector\_remap\_kernel\(

weights,ids,target\_ids,target\_scale,

out\_weights,out\_ids,B,

K:tl\.constexpr,BLOCK\_B:tl\.constexpr\):

batch=tl\.program\_id\(0\)\*BLOCK\_B\\

\+tl\.arange\(0,BLOCK\_B\)

valid=batch<B

forkintl\.static\_range\(K\):

w=tl\.load\(weights\+batch\*K\+k,

mask=valid\)

src=tl\.load\(ids\+batch\*K\+k,

mask=valid\)

dst=tl\.load\(target\_ids\+src,

mask=valid,other=0\)

scale=tl\.load\(target\_scale\+src,

mask=valid,other=1\.0\)

tl\.store\(out\_weights\+batch\*K\+k,

w\*scale,mask=valid\)

tl\.store\(out\_ids\+batch\*K\+k,

dst,mask=valid\)

Listing 1Abbreviated static ExFold remapping kernel\.

## Appendix CCross\-Model Quality Results

Table[11](https://arxiv.org/html/2608.24938#A3.T11)reports DeepSeek\-V2\-Lite\-Chat under joint compression\. Table[12](https://arxiv.org/html/2608.24938#A3.T12)separates prefill\-only, decode\-only, and joint folding on Qwen3\.5\-35B\-A3B\. In the method labels, P4 retains four routed experts per prefill token; D32 and D64 cap each decode batch’s active expert pool at 32 and 64 experts; a combined label applies both constraints\. For Qwen3\.5 P4 and P4\+D64, directed pairs with calibrated loss above 0\.99 are excluded from target assignment; after prefill folding, the retained weights are rescaled per token to preserve the original routed\-weight sum\. This safeguard is not enabled for D64\. All rows use complete benchmark coverage under the evaluation protocol described above\. The “Eval\+” column reports HumanEval\+ pass@1\.

Table 11:DeepSeek\-V2\-Lite\-Chat quality; bold marks the column best\.Table 12:Qwen3\.5\-35B\-A3B quality; bold marks the column best\.
## Appendix DExpert Geometry and Projector Structure

### Full\-Layer Expert Geometry

We visualize expert\-output geometry for every MoE layer\. For Qwen3 output matrices𝐄s\\mathbf\{E\}\_\{s\}and𝐄t\\mathbf\{E\}\_\{t\}collected on common inputs, raw similarity is1−‖𝐄s−𝐄t‖F/dmax1\-\\\|\\mathbf\{E\}\_\{s\}\-\\mathbf\{E\}\_\{t\}\\\|\_\{F\}/d\_\{\\max\}, wheredmaxd\_\{\\max\}is the largest raw pairwise distance in that layer\. For this diagnostic, the aligned view fits the closed\-form scalar on the same captured outputs, averages the two directed similarities, and reuses the samedmaxd\_\{\\max\}\. GLM reconstructs the corresponding raw and aligned distances from source\-norm\-weighted co\-routing statistics and averages the two directions using their observation counts\. Thus, each aligned distance is no larger than its raw counterpart; these analysis\-only fits are separate from the fixed deployment projectors visualized below\. The magnitude bar charts show every layer–expert position, use an independent y\-axis in each layer, and highlight the minimum\- and maximum\-magnitude experts\. Unavailable observations are gray; the isolated GLM layer\-45 outlier is hatched and excluded from that panel’s axis scaling\. Across observed directed off\-diagonal pairs, scalar alignment raises the mean similarity from 0\.449 to 0\.613 for Qwen3\. Across observed symmetric GLM entries, the corresponding mean rises from 0\.899 to 0\.932 for GLM\-4\.5\-Air\.

Figure 7:Expert\-output magnitudes across all Qwen3 MoE layers\.Figure 8:GLM\-4\.5\-Air expert\-output magnitudes \(continued\); hatching marks one isolated outlier\.#### Qwen3\-30B\-A3B\.

Figures[7](https://arxiv.org/html/2608.24938#A4.F7),[9](https://arxiv.org/html/2608.24938#A4.F9), and[10](https://arxiv.org/html/2608.24938#A4.F10)show all 48 layers and 128 routed experts\. The matrices exhibit layer\-dependent structure, and the norm view confirms expert\-wise magnitude variation\.

![Refer to caption](https://arxiv.org/html/2608.24938v1/qwen3_raw_similarity_all_layers.png)Figure 9:Raw expert\-output similarity across all Qwen3 MoE layers\.![Refer to caption](https://arxiv.org/html/2608.24938v1/qwen3_scalar_aligned_similarity_all_layers.png)Figure 10:Scalar\-aligned similarity for observed Qwen3 expert pairs\.
#### GLM\-4\.5\-Air\.

Figures[8](https://arxiv.org/html/2608.24938#A4.F8),[11](https://arxiv.org/html/2608.24938#A4.F11), and[12](https://arxiv.org/html/2608.24938#A4.F12)provide the corresponding 45\-layer diagnostics\. GLM statistics are collected for co\-routed expert pairs; gray cells denote pairs that were never jointly observed\. The matrices again show nonuniform pair structure and output\-magnitude differences\.

![Refer to caption](https://arxiv.org/html/2608.24938v1/glm45air_raw_similarity_all_layers.png)Figure 11:Raw co\-routed expert similarity across all GLM\-4\.5\-Air MoE layers\.![Refer to caption](https://arxiv.org/html/2608.24938v1/glm45air_scalar_aligned_similarity_all_layers.png)Figure 12:Scalar\-aligned co\-routed expert similarity across all GLM\-4\.5\-Air MoE layers\.### Layer\-Wise Projector Structure

Figures[13](https://arxiv.org/html/2608.24938#A4.F13)and[14](https://arxiv.org/html/2608.24938#A4.F14)visualize dense common\-input projector geometry for Qwen3\. Every expert processes the same hidden\-state matrixXX; rows are source experts, columns are candidate targets, and self\-pairs are suppressed\. Scalar colors are clipped symmetrically at the 99th percentile of\|ss→t⋆\|\|s\_\{s\\to t\}^\{\\star\}\|\. For a layerll, Figure[14](https://arxiv.org/html/2608.24938#A4.F14)reports the directed normalized Frobenius lossℓ~s→t=‖Es​\(X\)−ss→t⋆​Et​\(X\)‖F/dmax\(l\)=1−Sscalar​\(s,t\)\\widetilde\{\\ell\}\_\{s\\to t\}=\\\|E\_\{s\}\(X\)\-s\_\{s\\to t\}^\{\\star\}E\_\{t\}\(X\)\\\|\_\{F\}/d\_\{\\max\}^\{\(l\)\}=1\-S\_\{\\mathrm\{scalar\}\}\(s,t\), wheredmax\(l\)d\_\{\\max\}^\{\(l\)\}is the maximum raw pair distance in that layer\. Its median over directed off\-diagonal pairs is 0\.407, and 79\.9% of pairs are below 0\.5\. This common\-input diagnostic avoids conflating missing pair support with projection quality; lower values indicate more compatible transfers\.

![Refer to caption](https://arxiv.org/html/2608.24938v1/qwen3_scalar_projectors_all_layers.png)Figure 13:Common\-input scalar\-projector matrices across all Qwen3 MoE layers\.![Refer to caption](https://arxiv.org/html/2608.24938v1/qwen3_projection_loss_all_layers.png)Figure 14:Common\-input scalar\-aligned Frobenius losses across all Qwen3 MoE layers\.## Appendix EToken\-Level Reconstructions

Figure[15](https://arxiv.org/html/2608.24938#A5.F15)provides proxy\-space illustrations for tokens from mathematics, scientific reasoning, instruction following, and code generation\. The columns compare the original Top\-8 routes with Direct Top\-4, MoDES token\-wise selection\([12](https://arxiv.org/html/2608.24938#bib.bib9)\), REAP using Top\-8 within a static 64\-expert pool\([18](https://arxiv.org/html/2608.24938#bib.bib3)\), and ExFold Top\-4\. In this comparison, MoDES applies its calibrated layer\-wise threshold to retain a token\-dependent subset of at most four leading routes\. Expert directions come from two\-dimensional classical multidimensional scaling \(MDS\) of the all\-expert cosine\-distance matrix, and arrow lengths equal router weight times the calibrated expert\-output norm; segment labels1,…,81,\\ldots,8denote route\-rank positions rather than global expert identifiers, and primed labels denote REAP routes\. ExFold omits per\-segment labels for clarity\. Colored arrows show individual expert contributions in a strict head\-to\-tail chain: every arrow starts at the endpoint of the preceding contribution, and the chain endpoint is the corresponding aggregate\. Background arrows ending in diamonds show the original aggregate𝐲\\mathbf\{y\}and each method’s approximate aggregate𝐲^\\widehat\{\\mathbf\{y\}\}; the latter is dashed\. The displayed proxy distance is their relative Euclidean distance in this two\-dimensional space\. Within each dataset, we inspect at most eight evaluation examples and 96 deterministically spaced tokens per example\. We select an illustrative case whose Direct\-Top\-4 proxy distance is at least 0\.15, for which ExFold reduces that distance by at least 15%, and for which ExFold has the smallest proxy distance among the displayed approximations\. To span model depth, the four datasets target layers 0, 16, 32, and 47, respectively; each panel uses the nearest qualifying layer and the first case in deterministic scan order\. These diagrams illustrate routing geometry; their reported distances are proxy\-space quantities and are not hidden\-state reconstruction errors\. They are qualitative examples rather than aggregate evaluation evidence\.

Figure 15:Token\-level proxy reconstructions\. Expert contributions are accumulated strictly head\-to\-tail; each chain terminates at𝐲\\mathbf\{y\}or𝐲^\\widehat\{\\mathbf\{y\}\}\.

## Appendix FDeepSeek\-V4\-Flash Extension Details

#### Model and quality runtime\.

DeepSeek\-V4\-Flash contains 284B total and 13B activated parameters, 43 transformer layers, 256 routed experts plus one shared expert, and Top\-6 routing\([6](https://arxiv.org/html/2608.24938#bib.bib17)\)\. Quality evaluation uses tensor parallelism eight,max\_num\_seqs=32, and CUDA Graph\. Every method is evaluated with the same prompts, chat formatting, decoding rules, and scorers\. The full\-suite collector is required to have no missing tasks or infrastructure failures before it is admitted to Table[7](https://arxiv.org/html/2608.24938#Sx5.T7)\. All averages and retention ratios are computed from underlying unrounded task scores\.

#### Calibration data and disclosure\.

The released DeepSeek matrix is calibrated from 64 unlabeled inputs: 56 general instruction, code, and mathematics inputs from Tulu\-3, plus eight benchmark inputs \(four IFEval and four IFBench\) without labels, reference answers, or evaluator feedback\. At most 64 observer tokens are collected per sequence, with maximum input length 4,096\. This is*transductive, benchmark\-aware calibration*, not a zero\-contact held\-out evaluation\. Calibration uses no benchmark answers and changes no model weights, but the input exposure must be preserved when reporting the results\.

The DeepSeek artifact uses common\-input expert outputs, source\-output\-norm weighting, and unbounded least\-squares scalars\. Unlike the Qwen configuration, it does not clip scalar coefficients\.

#### Confidence\-aware P3 safeguard\.

DeepSeek\-V4\-Flash exhibits a wider range of router scores and expert\-output magnitudes than the primary Qwen model\. For an omitted sourcessand retained targettt, we therefore compute a source\-relative residual

ℓs→trel\\displaystyle\\ell\_\{s\\rightarrow t\}^\{\\mathrm\{rel\}\}=∑iwi​‖𝐮i−as→t​𝐯i‖22∑iwi​‖𝐮i‖22\+ϵ,\\displaystyle=\\frac\{\\sum\_\{i\}w\_\{i\}\\\|\\mathbf\{u\}\_\{i\}\-a\_\{s\\rightarrow t\}\\mathbf\{v\}\_\{i\}\\\|\_\{2\}^\{2\}\}\{\\sum\_\{i\}w\_\{i\}\\\|\\mathbf\{u\}\_\{i\}\\\|\_\{2\}^\{2\}\+\\epsilon\},\(26\)cs\\displaystyle c\_\{s\}=\[1−mint⁡ℓs→trel\]01\.\\displaystyle=\\left\[1\-\\min\_\{t\}\\ell\_\{s\\rightarrow t\}^\{\\mathrm\{rel\}\}\\right\]\_\{0\}^\{1\}\.The scalar\-transfer partws​cs​as→tw\_\{s\}c\_\{s\}a\_\{s\\rightarrow t\}is added to the minimum\-loss target\. The remaining weightws​\(1−cs\)w\_\{s\}\(1\-c\_\{s\}\)falls back to the retained Top\-3 routes in proportion to their original router weights\. This continuous safeguard does not execute additional experts: P3 still invokes exactly three routed experts\. It only avoids forcing a poorly calibrated pair to absorb the entire omitted contribution\.

#### Speed protocol and operating\-point boundary\.

All DeepSeek speed measurements use one H800 server with model weights, code, data, and outputs on local storage\. The vLLM runtime uses tensor parallelism four and CUDA Graph\. Online load curves usemax\_num\_seqs=32and the same request schedule for every method\. Prefill uses 8,192 input tokens and one output token at QPS 1, 2, 4, 6, and 8\. Decode uses one input token and 256 output tokens at QPS 2, 4, 8, 12, 16, 24, and 32\. The offline\-throughput panel usesmax\_num\_seqs=128, QPS 64, concurrency 128, 32 warmups, and 1,024 measured requests\.

The TTFT peak at QPS 4 includes queueing amplification and is therefore a serving\-level speedup rather than a pure\-kernel claim\. Under the matchedmax\_num\_seqs=32decode limit, D64 stabilizes near 1\.15×\\timesTPOT at QPS 8–32, while D128 remains near parity\. The saturatedmax\_num\_seqs=128run yields a 1\.286×\\timesoutput\-throughput gain for D64\. Because quality is measured withmax\_num\_seqs=32, the throughput bar is reported as a separate saturation boundary rather than the same quality–speed operating point\.

#### Open\-source reproduction\.

The repository athttps://github\.com/Time\-Rune/ExFold\-MoEcontains the Qwen3 and DeepSeek\-V4\-Flash runtime patches, CUDA/Triton kernels, final calibration matrices, correctness tests, and one\-command quality and speed launchers\. Model weights and benchmark datasets remain under their original licenses and are downloaded from their providers\.