ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

# Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Source: [https://arxiv.org/html/2609.06072](https://arxiv.org/html/2609.06072)
Sehyun YunJoonha ParkTaesik GongAffiliation:Ulsan National Institute of Science and Technology \(UNIST\)Affiliation:Ulsan, Republic of KoreaEmail:[\{ahin,nawhji,joonhapark,taesik\.gong\}@unist\.ac\.kr](mailto:)

###### Abstract

Parameter\-efficient fine\-tuning \(PEFT\) of mixture\-of\-experts \(MoE\) models commonly attaches a separate low\-rank adapter to each expert\. This expert\-wise design fragments adaptation in three ways: capacity is split across narrow low\-rank updates, gradient supervision becomes sparse and imbalanced under sparse routing, and execution is decomposed into many small GEMMs\. We find that such expert\-wise separation is often unnecessary, as subsets of LoRA adapters become functionally similar during fine\-tuning, revealing redundancy among expert\-specific adapters\. Based on this redundancy, we proposeACE\(Adapter Consolidation across Experts\), which groups redundant experts and replaces their expert\-specific adapters with group\-shared higher\-rank LoRA modules under the same PEFT budget\. ACE further introduces grouped adapter execution, which consolidates fragmented expert\-wise adapter computations into fewer, larger group\-level GEMMs\. Across evaluations covering 12 datasets and four MoE backbones, ACE achieves the highest observed mean accuracy among the parameter\-matched PEFT methods on the three backbones with complete baseline coverage, while providing1\.31×1\.31\\timesto1\.48×1\.48\\timeswall\-clock training speedup over expert\-wise LoRA without increasing peak memory\. Our code is available at[https://github\.com/UbiquitousAILab/ACE](https://github.com/UbiquitousAILab/ACE)\.

## 1Introduction

Mixture\-of\-Experts \(MoE\) models scale model capacity by activating only a small subset of experts for each input, allowing them to increase parameters without a proportional increase in per\-token computation\([Shazeer et al\., 2017](https://arxiv.org/html/2609.06072#bib.bib3);[Fedus et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib4);[Du et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib6)\)\. This efficiency has made MoE architectures increasingly popular in large language models\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib5);[OpenAI et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib7);[Dai et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib8)\)\. However, adapting MoE models to downstream tasks remains challenging\. Although sparse routing reduces per\-token computation, MoE models still contain a very large number of parameters across many experts, making full fine\-tuning substantially more expensive and complex than in comparable dense models\([Dai et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib8);[Kim et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib32);[Aminabadi et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib33)\)\. These challenges motivate parameter\-efficient fine\-tuning \(PEFT\) methods such as LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib1)\)\.

LoRA was originally developed for dense architectures, where it is commonly applied by attaching a separate low\-rank adapter to each target module\. A direct extension to the expert\-side modules of MoE models therefore attaches independent adapters to each expert\([Hu et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib1);[Meng et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib29)\)\. Although natural, this expert\-wise design induces fragmentation when combined with sparse routing, along three dimensions: capacity, supervision, and execution\. First, under a fixed trainable parameter budget, allocating separate adapters to many experts limits the rank available to each expert\. Second, each adapter receives gradients only from tokens routed to its expert, making gradient supervision sparse and imbalanced across experts\. Third, adapter execution is decomposed into many small matrix multiplication \(GEMM\) operations, incurring repeated launch overhead and poor hardware utilization\.

![Refer to caption](https://arxiv.org/html/2609.06072v1/moe_details.png)Figure 1:Overview ofACE\.Left: Standard expert\-wise LoRA in MoE uses a separate rank\-rradapter for each expert, fragmenting adaptation capacity, yielding sparse updates, and causing many small computations\.Right:ACEgroups experts with similar adapter\-induced representations and replaces their separate adapters with group\-shared higher\-rank adapters under the same trainable parameter budget\. Each shared adapter is updated by all experts in its group and computed once per group, improving capacity, supervision density, and execution efficiency\.This fragmentation would be unavoidable if each expert required a distinct task\-specific adapter\. However, we find that expert\-specific LoRA adapters are often redundant: within the same MoE layer, they form functionally similar groups during fine\-tuning\. Based on this observation, we proposeACE\(AdapterConsolidation acrossExperts\), a PEFT method for MoE models that consolidates functionally similar expert adapters into group\-shared higher\-rank LoRA modules\. Within each MoE layer,ACEmeasures adapter similarity on task\-relevant inputs, groups similar experts together and replaces each group of expert\-specific adapters with a shared higher\-rank adapter under the same trainable parameter budget\. This reallocation pools fragmented adaptation capacity and allows each shared adapter to receive gradients whenever any expert in its group is active\. For execution,ACEperforms grouped adapter execution, which consolidates fragmented expert\-wise adapter computations into fewer, larger group\-level operations, reducing small\-GEMM overhead\. Figure[1](https://arxiv.org/html/2609.06072#S1.F1)provides an overview ofACEand its key components: similarity\-based grouping, rank consolidation, shared updates and grouped adapter execution\.

Across evaluations covering 12 datasets and four MoE backbones,ACEconsistently improves adaptation quality and training efficiency under matched parameter budgets\. For example, on OLMoE,ACEimproves the average accuracy across eight commonsense benchmarks from 74\.11% to 75\.61% \(\+1\.50 points\) over expert\-wise LoRA, while providing 1\.31×\\times–1\.48×\\timeswall\-clock training speedups across the evaluated backbones without increasing peak memory\. These results indicate that similarity\-guided adapter consolidation improves both adaptation quality and execution efficiency under the same trainable parameter budget\.

## 2Related Work

##### Parameter\-Efficient Fine\-Tuning\.

Parameter\-efficient fine\-tuning \(PEFT\) adapts large pretrained models by training only a small set of additional parameters while keeping the backbone frozen\([Houlsby et al\., 2019](https://arxiv.org/html/2609.06072#bib.bib23);[Li and Liang, 2021](https://arxiv.org/html/2609.06072#bib.bib24);[Lester et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib25);[Karimi Mahabadi et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib26);[Li et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib31)\)\. Among PEFT methods, low\-rank adaptation \(LoRA\) has become a standard approach because it parameterizes weight updates efficiently without changing the backbone architecture\([Hu et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib1);[Dettmers et al\., 2023](https://arxiv.org/html/2609.06072#bib.bib19)\)\. Subsequent work improves the use of fixed trainable parameter budgets through rank allocation or alternative low\-rank parameterizations\([Zhang et al\., 2023](https://arxiv.org/html/2609.06072#bib.bib27);[Liu et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib2);[Meng et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib29)\)\. However, these methods are designed for dense architectures, where each adapted module receives gradients from all inputs\. This assumption breaks down in MoE layers, where adapters are distributed across sparsely and unevenly activated experts\. Our work addresses this MoE\-specific structural mismatch by studying how a fixed LoRA budget should be organized under such routing\.

##### Fine\-Tuning of MoE Models\.

Mixture\-of\-Experts \(MoE\) architectures scale model capacity by activating only a subset of expert networks for each input\([Shazeer et al\., 2017](https://arxiv.org/html/2609.06072#bib.bib3);[Fedus et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib4);[Du et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib6)\)\. However, the large number of expert parameters increases the cost of fine\-tuning for downstream adaptation\. To reduce this cost, prior work has explored selective or routed adaptation strategies for MoE models, primarily focusing on which experts or modules should be tuned\. ESFT\([Wang et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib22)\)tunes the experts most relevant to a target task\. PERFT\([Liu et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib20)\)proposes PERFT\-R, which applies MoE\-style routing to LoRA adapters for MoE fine\-tuning\. EPnG\([Lee et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib47)\)reallocates LoRA capacity across experts based on routing importance\. In contrast,ACEkeeps the pretrained experts and router intact and addresses expert\-wise LoRA fragmentation by consolidating functionally similar adapters into shared group\-level modules\.

## 3Methodology

### 3\.1Problem: Fragmentation in MoE LoRA

We first identify a structural mismatch between expert\-wise LoRA and sparse MoE routing\. Consider an MoE layer withEEexperts, where the router activates a sparse expert subsetℛ⁡\(x\)\\mathcal\{R\}\(x\)of sizek≪Ek\\ll Efor an inputxx\. A direct LoRA extension assigns each experteean independent low\-rank updateΔ​We\\Delta W\_\{e\}, yielding an adapted expert weightW~e=We\+Δ​We\\widetilde\{W\}\_\{e\}=W\_\{e\}\+\\Delta W\_\{e\}, whereΔ​We=Be​Ae\\Delta W\_\{e\}=B\_\{e\}A\_\{e\}\. Letpe​\(x\)p\_\{e\}\(x\)denote the router probability assigned to experteefor inputxx\. The resulting layer output is

y⁡\(x\)=∑e∈ℛ⁡\(x\)pe​\(x\)​\(We​x\+Δ​We​x\)\.y\(x\)=\\sum\_\{e\\in\\mathcal\{R\}\(x\)\}p\_\{e\}\(x\)\(W\_\{e\}x\+\\Delta W\_\{e\}x\)\.Although simple, this expert\-wise design fragments adaptation along three coupled dimensions\.

#### 3\.1\.1Capacity Fragmentation

Under a fixed layer\-level adapter budgetℬ\\mathcal\{B\}, expert\-wise LoRA divides parameters across experts\. For expertee, a rank\-rer\_\{e\}LoRA adapter contains\(din\+dout\)​re\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\)r\_\{e\}trainable parameters, so the layer\-level budget isℬ=∑e=1E\(din\+dout\)​re\\mathcal\{B\}=\\sum\_\{e=1\}^\{E\}\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\)r\_\{e\}\. If this budget is allocated uniformly, each expert receives

re=ℬE⁡\(din\+dout\)\.r\_\{e\}=\\frac\{\\mathcal\{B\}\}\{E\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\)\}\.Thus, increasing the number of experts linearly reduces the rank available to each adapter\. The budget is split intoEEnarrow low\-rank updates rather than forming stronger task\-specific corrections\.

#### 3\.1\.2Gradient Fragmentation

Sparse routing also fragments gradient supervision\. For a training batch𝒳=\{xn\}n=1N\\mathcal\{X\}=\\\{x\_\{n\}\\\}\_\{n=1\}^\{N\}, the adapter of experteereceives gradients only from tokens routed to that expert:

∇Δ​Weℒ∝∑xn∈𝒳𝟏\[e∈ℛ\(xn\)\]pe\(xn\)∇y⁡\(xn\)ℒxn⊤\.\\nabla\_\{\\Delta W\_\{e\}\}\\mathcal\{L\}\\propto\\sum\_\{x\_\{n\}\\in\\mathcal\{X\}\}\\mathbf\{1\}\[e\\in\\mathcal\{R\}\(x\_\{n\}\)\]p\_\{e\}\(x\_\{n\}\)\\,\\nabla\_\{y\(x\_\{n\}\)\}\\mathcal\{L\}\\,x\_\{n\}^\{\\top\}\.The indicator𝟏\[e∈ℛ\(xn\)\]\\mathbf\{1\}\[e\\in\\mathcal\{R\}\(x\_\{n\}\)\]zeroes out contributions from all non\-routed tokens\. If routing is balanced, each adapter is updated by only ak/Ek/Efraction of the batch on average\. In practice, routing is often imbalanced\([Shazeer et al\., 2017](https://arxiv.org/html/2609.06072#bib.bib3);[Dai et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib8)\), so some adapters receive even fewer updates\. Expert\-wise LoRA therefore splits not only parameters, but also the learning signal, yielding sparse and imbalanced adapter updates\.

#### 3\.1\.3Execution Fragmentation

Finally, expert\-wise LoRA fragments execution\. Each active adapter requires two small GEMMs,

he=Ae​x,Δ​We​x=Be​he\.h\_\{e\}=A\_\{e\}x,\\qquad\\Delta W\_\{e\}x=B\_\{e\}h\_\{e\}\.Across a batch, these operations are invoked separately for many active experts and routed token subsets\. This produces numerous small adapter\-side GEMMs rather than fewer large matrix multiplications\. Such small GEMMs are inefficient on GPUs because they incur repeated launch overhead and poor utilization\([NVIDIA Corporation, 2026](https://arxiv.org/html/2609.06072#bib.bib40)\)\. As a result, even a small LoRA budget can introduce non\-negligible runtime overhead when replicated across many experts\.

These three effects arise from a common design choice in expert\-wise LoRA, where each expert owns an isolated adapter while each input activates only a small subset of experts\. This raises a central question:*do expert adapters truly need to remain separate?*

### 3\.2Motivation: Functional Grouping of Expert LoRA Adapters

The expert\-wise separation described above is often unnecessary in practice\. During fine\-tuning, LoRA adapters within the same MoE layer frequently form functionally similar groups\. Adapters within these groups produce similar adapter\-induced representations on shared task\-relevant inputs\. We quantify this behavior by computing pairwise similarity between adapter outputs on a shared probe set; details are given in Section[3\.3](https://arxiv.org/html/2609.06072#S3.SS3)\.

![Refer to caption](https://arxiv.org/html/2609.06072v1/sim_matrix1.png)Figure 2:Functional similarity among expert LoRA adapters\. Each heatmap shows pairwise CKA similarity within one MoE layer, with x\- and y\-axes denoting expert indices\. Values closer to 1 indicate more similar adapter\-induced representations\. High\-similarity blocks reveal groups of experts with similar LoRA corrections\. Panels correspond to layers 0, 4, and 14 of the gate projection in OLMoE on HellaSwag\. Full details are in Appendix[F](https://arxiv.org/html/2609.06072#A6)\.Figure[2](https://arxiv.org/html/2609.06072#S3.F2)shows high\-similarity blocks, indicating that some adapters learn similar task\-specific corrections while others remain distinct\. This motivatesACEto consolidate functionally similar adapters into shared higher\-rank modules instead of preserving many low\-rank expert\-specific adapters\.ACEuses this functional grouping structure to pool fragmented capacity via rank consolidation \(Section[3\.3](https://arxiv.org/html/2609.06072#S3.SS3)\), densify gradient supervision via shared adapter updates \(Section[3\.4](https://arxiv.org/html/2609.06072#S3.SS4)\), and reduce small\-GEMM fragmentation via grouped adapter execution \(Section[3\.5](https://arxiv.org/html/2609.06072#S3.SS5)\)\.

### 3\.3Rank Consolidation

ACEfirst identifies functionally similar expert adapters within each MoE layer and consolidates their fragmented ranks under the same trainable parameter budget\. After a short warm\-up ofTinitT\_\{\\mathrm\{init\}\}steps, we collect a probe set of hidden states𝒳probe=\{xn\}n=1N\\mathcal\{X\}\_\{\\mathrm\{probe\}\}=\\\{x\_\{n\}\\\}\_\{n=1\}^\{N\}, wherexn∈ℝdinx\_\{n\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\}\. The same probe inputs are fed to all expert adapters in the layer, regardless of their original routing decisions, so that similarity reflects adapter behavior rather than routing frequency\.

For expertee, we form the adapter\-induced representation matrix

Ze=\[\(Δ​We​x1\)⊤⋯\(Δ​We​xN\)⊤\]∈ℝN×dout\.Z\_\{e\}=\\begin\{bmatrix\}\(\\Delta W\_\{e\}x\_\{1\}\)^\{\\top\}\\\\ \\cdots\\\\ \(\\Delta W\_\{e\}x\_\{N\}\)^\{\\top\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{N\\times d\_\{\\mathrm\{out\}\}\}\.We mean\-center each representation matrix over the probe samples asZ¯e=H​Ze\\bar\{Z\}\_\{e\}=HZ\_\{e\}, whereH=IN−1N​𝟏𝟏⊤H=I\_\{N\}\-\\frac\{1\}\{N\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\. For expertsiiandjj, we compute their pairwise similarity using linear CKA\([Kornblith et al\., 2019](https://arxiv.org/html/2609.06072#bib.bib10)\):

si​j=‖Z¯i⊤​Z¯j‖F2‖Z¯i⊤​Z¯i‖F​‖Z¯j⊤​Z¯j‖F\.s\_\{ij\}=\\frac\{\\\|\\bar\{Z\}\_\{i\}^\{\\top\}\\bar\{Z\}\_\{j\}\\\|\_\{F\}^\{2\}\}\{\\\|\\bar\{Z\}\_\{i\}^\{\\top\}\\bar\{Z\}\_\{i\}\\\|\_\{F\}\\,\\\|\\bar\{Z\}\_\{j\}^\{\\top\}\\bar\{Z\}\_\{j\}\\\|\_\{F\}\}\.We use CKA becauseACEgroups adapters by their functional behavior rather than by raw parameter similarity\. CKA is a standard representation\-similarity measure\([Jiang et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib41)\)and has been used in compression settings to assess representational preservation across model variants\([Pons et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib21)\)\. Its scale invariance is useful for MoE adapters, whose output magnitudes can differ under sparse and imbalanced routing\.

Using these scores,ACEgroups experts with high mutual similarity within each MoE layer\. In our experiments, we instantiate this step by constructing a weighted expert\-similarity graph and applying Louvain community detection\([Blondel et al\., 2008](https://arxiv.org/html/2609.06072#bib.bib18)\), which avoids manually choosing a fixed similarity threshold and thus reduces sensitivity to layer\-wise variation in similarity scale\. This yields expert groups\{𝒢g\}g=1G\\\{\\mathcal\{G\}\_\{g\}\\\}\_\{g=1\}^\{G\}\.

For each group𝒢g\\mathcal\{G\}\_\{g\},ACEreplaces the separate expert\-specific adapters\{Δ​We\}e∈𝒢g\\\{\\Delta W\_\{e\}\\\}\_\{e\\in\\mathcal\{G\}\_\{g\}\}with a single group\-shared adapter

Δ​Wgshare=Bg​Ag,rg=∑e∈𝒢gre,\\Delta W\_\{g\}^\{\\mathrm\{share\}\}=B\_\{g\}A\_\{g\},\\qquad r\_\{g\}=\\sum\_\{e\\in\\mathcal\{G\}\_\{g\}\}r\_\{e\},whereAg∈ℝrg×dinA\_\{g\}\\in\\mathbb\{R\}^\{r\_\{g\}\\times d\_\{\\mathrm\{in\}\}\}andBg∈ℝdout×rgB\_\{g\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{g\}\}\. Each experte∈𝒢ge\\in\\mathcal\{G\}\_\{g\}then uses

W~e=We\+Δ​Wgshare\.\\widetilde\{W\}\_\{e\}=W\_\{e\}\+\\Delta W\_\{g\}^\{\\mathrm\{share\}\}\.Because experts within the same layer share input and output dimensions, this consolidation preserves the total number of trainable adapter parameters:

∑g=1G\(dout\+din\)​rg=∑e=1E\(dout\+din\)​re\.\\sum\_\{g=1\}^\{G\}\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)r\_\{g\}=\\sum\_\{e=1\}^\{E\}\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)r\_\{e\}\.Thus,ACEdoes not increase the trainable parameter budget; it reallocates the same fragmented expert\-wise ranks into fewer, higher\-rank group\-level adapters\.

To reduce discontinuity at the consolidation step, we initialize each shared adapter from a similarity\-central anchor within its group and use standard LoRA initialization for the remaining rank dimensions; details are provided in Appendix[E](https://arxiv.org/html/2609.06072#A5)\.

### 3\.4Shared Adapter Updates

Rank consolidation also changes how adapters receive supervision\. After consolidation, each routed expert uses the shared adapter of its group\. Letg⁡\(e\)g\(e\)denote the group assignment of expertee\. The MoE output becomes

y⁡\(x\)=∑e∈ℛ⁡\(x\)pe​\(x\)​\(We​x\+Δ​Wg⁡\(e\)share​x\)\.y\(x\)=\\sum\_\{e\\in\\mathcal\{R\}\(x\)\}p\_\{e\}\(x\)\\bigl\(W\_\{e\}x\+\\Delta W\_\{g\(e\)\}^\{\\mathrm\{share\}\}x\\bigr\)\.
Under expert\-wise LoRA, the adapter of experteeis updated only wheneeis routed\. InACE, the shared adapter for groupggis updated whenever any expert in𝒢g\\mathcal\{G\}\_\{g\}is routed\. This aggregates update opportunities at the group level, providing denser and more balanced supervision despite sparse routing\.

### 3\.5Grouped Adapter Execution

Rank consolidation improves adaptation capacity, but its computational benefit depends on how shared adapters are executed\. A naïve implementation would share adapter parameters within each group while still following the expert\-wise LoRA execution pattern, evaluating the same shared adapter separately for each routed expert\.

ACEavoids this by exploiting a structural property of MoE layers\. For shared\-input expert projections, such as gate and up projections, routed experts process the same input activation in parallel\. Thus, when multiple routed experts use the same group\-shared adapter, the corresponding adapter output can be computed once and reused across those experts\.

Formally, letℛ⁡\(x\)\\mathcal\{R\}\(x\)denote the set of experts selected by the router for inputxx, and let𝒜⁡\(x\)=\{g⁡\(e\)∣e∈ℛ⁡\(x\)\}\\mathcal\{A\}\(x\)=\\\{g\(e\)\\mid e\\in\\mathcal\{R\}\(x\)\\\}be the set of active adapter groups\. For each active groupg∈𝒜⁡\(x\)g\\in\\mathcal\{A\}\(x\),ACEevaluates

cg​\(x\)=Δ​Wgshare​xc\_\{g\}\(x\)=\\Delta W\_\{g\}^\{\\mathrm\{share\}\}xonce and reusescg​\(x\)c\_\{g\}\(x\)for every routed experte∈ℛ⁡\(x\)e\\in\\mathcal\{R\}\(x\)satisfyingg⁡\(e\)=gg\(e\)=g\.

Since a LoRA update consists of two GEMMs,h=Ag​xh=A\_\{g\}xandcg​\(x\)=Bg​hc\_\{g\}\(x\)=B\_\{g\}h, this group\-level evaluation replaces repeated expert\-wise adapter calls with fewer, larger matrix multiplications\. As a result,ACEreduces small\-GEMM overhead and adapter\-side execution fragmentation\.

## 4Experiments

We evaluateACEon reasoning and generation tasks using MoE\-based language models\. Across all experiments, we compare methods under matched trainable parameter budgets and use the same optimization setup and evaluation protocol within each setting\. Unless otherwise specified, models are trained with AdamW, weight decay0\.10\.1\. ForACE, we useTinit=150T\_\{\\mathrm\{init\}\}=150, a probe set ofN=10,000N=10\{,\}000tokens, and Louvain resolutionγ=0\.8\\gamma=0\.8\. All experiments are conducted on NVIDIA H200 GPUs\. Detailed model configurations, hyperparameters, and compute infrastructure are provided in Appendix[A](https://arxiv.org/html/2609.06072#A1)and Appendix[B](https://arxiv.org/html/2609.06072#A2)\.

### 4\.1Commonsense Reasoning

We evaluateACEon eight commonsense reasoning benchmarks: HellaSwag \(HS\)\([Zellers et al\., 2019](https://arxiv.org/html/2609.06072#bib.bib12)\), WinoGrande \(WG\)\([Sakaguchi et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib13)\), ARC\-Challenge \(ARC\-c\) and ARC\-Easy \(ARC\-e\)\([Clark et al\., 2018](https://arxiv.org/html/2609.06072#bib.bib14)\), BoolQ\([Clark et al\., 2019](https://arxiv.org/html/2609.06072#bib.bib15)\), OpenBookQA \(OBQA\)\([Mihaylov et al\., 2018](https://arxiv.org/html/2609.06072#bib.bib16)\), Physical IQA \(PIQA\)\([Bisk et al\., 2020](https://arxiv.org/html/2609.06072#bib.bib17)\), and Social IQA \(SIQA\)\([Sap et al\., 2019](https://arxiv.org/html/2609.06072#bib.bib34)\)\. We report accuracy for all tasks\. We conduct experiments on three MoE\-based language models: OLMoE\-1B\-7B\([Muennighoff et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib9)\), Qwen1\.5\-MoE\-A2\.7B\([Qwen, 2024](https://arxiv.org/html/2609.06072#bib.bib11)\), and Moonlight\([Liu et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib42)\)\. For this setting, models are fine\-tuned on Commonsense\-170K and evaluated on each benchmark separately\([Hu et al\., 2023](https://arxiv.org/html/2609.06072#bib.bib30)\)\.

We compareACEwith representative PEFT baselines, including LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib1)\), AdaLoRA\([Zhang et al\., 2023](https://arxiv.org/html/2609.06072#bib.bib27)\), DoRA\([Liu et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib2)\), and PiSSA\([Meng et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib29)\), as well as two MoE\-aware PEFT baselines, PERFT\-R\([Liu et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib20)\)and EPnG\([Lee et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib47)\)\. For fair comparison, all methods adapt the same expert projections and use comparable trainable parameter budgets within each backbone\.

Table[1](https://arxiv.org/html/2609.06072#S4.T1)reports the commonsense reasoning results\. Across the evaluated MoE backbones,ACEachieves the best average performance under matched trainable parameter budgets\. The gains over dense\-Transformer PEFT baselines suggest that methods designed for dense Transformer modules do not fully account for the sparse\-routing structure of MoE layers, which can limit how effectively their trainable adapter parameters are utilized\. These results show that adapter consolidation provides an effective way to use limited adaptation capacity in MoE models\. Results on Kimi\-Linear\-48B\-A3B\([Team et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib43)\), the largest backbone evaluated, show the same trend; results are in Appendix[J](https://arxiv.org/html/2609.06072#A10)\.

ModelMethod\# Params \(%\)↓\\downarrowHS↑\\uparrowWG↑\\uparrowARC\-c↑\\uparrowARC\-e↑\\uparrowBoolQ↑\\uparrowOBQA↑\\uparrowPIQA↑\\uparrowSIQA↑\\uparrowAvg\.↑\\uparrowOLMoE\(1B/7B\)LoRA0\.2790\.82±0\.23\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.23\}70\.08±0\.34\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.34\}69\.42±1\.00\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.00\}85\.61±0\.09\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.09\}66\.71±1\.12\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.12\}74\.60±0\.92\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.92\}83\.10±0\.22\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.22\}52\.51±0\.90\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.90\}74\.11±0\.39\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}DoRA0\.3390\.98±0\.27\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.27\}70\.25±0\.34\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.34\}69\.46±0\.31\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.31\}85\.98±0\.14\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.14\}66\.96±0\.71\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.71\}74\.93±1\.21\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.21\}83\.31±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}52\.35±0\.89\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.89\}74\.28±0\.23\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.23\}AdaLoRA0\.54→\\rightarrow0\.2790\.68±0\.14\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.14\}68\.14±0\.25\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.25\}68\.51±0\.23\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.23\}84\.69±0\.84\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.84\}65\.52±0\.39\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}73\.67±0\.12\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.12\}83\.08±0\.47\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.47\}51\.31±0\.61\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.61\}73\.20±0\.11\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.11\}PiSSA0\.2790\.98±0\.19\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.19\}70\.46±0\.25\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.25\}69\.96±0\.44\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.44\}86\.38±0\.27\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.27\}68\.71±0\.34\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.34\}75\.27±0\.50\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.50\}83\.44±0\.55\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.55\}52\.83±0\.53\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.53\}74\.75±0\.16\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.16\}PERFT\-R0\.2787\.36±0\.89\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.89\}71\.46±2\.43\\mkern 1\.0mu\{\\scriptstyle\\pm 2\.43\}66\.38±1\.13\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.13\}82\.76±0\.28\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.28\}64\.04±4\.73\\mkern 1\.0mu\{\\scriptstyle\\pm 4\.73\}73\.87±0\.23\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.23\}80\.05±0\.85\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.85\}51\.62±0\.10\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.10\}72\.19±0\.49\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.49\}EPnG0\.2790\.82±0\.34\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.34\}68\.95±0\.55\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.55\}68\.83±0\.60\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.60\}85\.75±0\.60\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.60\}66\.96±0\.52\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.52\}74\.47±1\.40\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.40\}83\.15±0\.39\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}52\.27±0\.21\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.21\}73\.90±0\.33\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.33\}ACE\(ours\)0\.2792\.24±0\.03\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.03\}72\.85±0\.51\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.51\}70\.88±0\.18\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.18\}86\.21±0\.34\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.34\}68\.04±1\.99\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.99\}77\.33±1\.10\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.10\}83\.93±0\.14\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.14\}53\.41±0\.44\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.44\}75\.61±0\.28\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.28\}Qwen\(3B/14B\)LoRA0\.2193\.23±0\.24\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.24\}77\.66±0\.21\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.21\}79\.44±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}90\.63±0\.07\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.07\}70\.84±0\.39\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}82\.67±0\.58\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.58\}86\.80±0\.35\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.35\}56\.72±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}79\.75±0\.03\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.03\}DoRA0\.2693\.25±0\.25\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.25\}78\.08±0\.25\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.25\}79\.64±0\.47\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.47\}90\.71±0\.12\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.12\}70\.58±0\.51\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.51\}82\.80±1\.11\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.11\}86\.73±0\.20\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.20\}56\.64±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}79\.80±0\.19\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.19\}AdaLoRA0\.42→\\rightarrow0\.2193\.21±0\.07\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.07\}74\.45±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}78\.89±0\.13\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.13\}90\.42±0\.11\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.11\}69\.49±1\.10\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.10\}82\.07±0\.50\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.50\}86\.29±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}55\.63±0\.67\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.67\}78\.81±0\.22\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.22\}PiSSA0\.2193\.28±0\.18\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.18\}79\.19±0\.67\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.67\}78\.79±0\.54\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.54\}90\.69±0\.44\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.44\}71\.70±0\.11\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.11\}83\.60±0\.53\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.53\}86\.58±0\.22\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.22\}56\.05±0\.26\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.26\}79\.98±0\.11\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.11\}PERFT\-R0\.2192\.02±0\.32\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.32\}73\.35±0\.73\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.73\}78\.38±0\.3\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.3\}90\.95±0\.19\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.19\}69\.45±0\.88\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.88\}80\.93±0\.42\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.42\}85\.95±0\.36\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.36\}53\.91±0\.77\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.77\}78\.12±0\.20\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.20\}EPnG0\.2193\.50±0\.08\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.08\}76\.53±0\.48\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.48\}79\.78±0\.39\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}90\.67±0\.16\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.16\}69\.46±1\.23\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.23\}83\.40±0\.20\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.20\}86\.38±0\.08\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.08\}56\.64±0\.26\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.26\}79\.54±0\.09\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.09\}ACE\(ours\)0\.2193\.81±0\.17\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.17\}79\.19±0\.56\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.56\}79\.50±0\.35\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.35\}90\.66±0\.22\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.22\}71\.49±0\.47\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.47\}83\.87±1\.15\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.15\}86\.67±0\.62\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.62\}56\.99±0\.55\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.55\}80\.27±0\.18\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.18\}Moonlight\(3B/16B\)LoRA0\.2292\.23±0\.07\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.07\}74\.59±0\.72\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.72\}84\.67±0\.27\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.27\}94\.28±0\.11\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.11\}67\.17±0\.94\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.94\}82\.00±0\.72\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.72\}86\.92±0\.36\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.36\}52\.47±0\.92\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.92\}79\.29±0\.32\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.32\}DoRA0\.2792\.15±0\.14\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.14\}74\.67±0\.36\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.36\}84\.93±0\.47\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.47\}94\.35±0\.18\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.18\}67\.59±0\.26\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.26\}82\.07±0\.83\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.83\}86\.98±0\.55\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.55\}52\.29±0\.50\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.50\}79\.38±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}AdaLoRA0\.43→\\rightarrow0\.2291\.63±0\.15\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.15\}72\.64±0\.45\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.45\}84\.96±0\.10\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.10\}94\.08±0\.32\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.32\}67\.52±0\.42\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.42\}81\.16±0\.32\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.32\}86\.74±0\.08\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.08\}51\.93±0\.55\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.55\}78\.83±0\.18\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.18\}PiSSA0\.2291\.38±0\.12\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.12\}72\.45±0\.16\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.16\}84\.67±0\.18\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.18\}93\.93±0\.15\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.15\}62\.53±2\.38\\mkern 1\.0mu\{\\scriptstyle\\pm 2\.38\}80\.80±0\.53\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.53\}86\.83±0\.17\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.17\}52\.25±0\.71\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.71\}78\.11±0\.23\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.23\}PERFT\-R0\.2290\.9±0\.13\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.13\}72\.53±0\.75\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.75\}84\.13±0\.39\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}94\.05±0\.13\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.13\}65\.92±1\.4\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.4\}81\.27±0\.9\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.9\}86\.71±0\.39\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}51\.40±0\.47\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.47\}78\.36±0\.51\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.51\}EPnG0\.2292\.95±0\.05\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.05\}77\.03±0\.57\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.57\}84\.78±0\.44\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.44\}94\.50±0\.38\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.38\}67\.56±0\.35\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.35\}83\.53±0\.64\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.64\}87\.69±0\.59\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.59\}52\.97±0\.75\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.75\}80\.13±0\.24\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.24\}ACE\(ours\)0\.2293\.49±0\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.30\}79\.95±0\.34\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.34\}84\.30±0\.09\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.09\}94\.75±0\.21\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.21\}69\.31±0\.82\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.82\}84\.93±1\.01\\mkern 1\.0mu\{\\scriptstyle\\pm 1\.01\}87\.71±0\.38\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.38\}54\.93±0\.26\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.26\}81\.17±0\.19\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.19\}

Table 1:Commonsense reasoning results across MoE backbones\. Model entries show active/total backbone parameters\. The \# Params column denotes the percentage of trainable adapter parameters\. We report accuracy for each benchmark; the Avg\. column reports the mean benchmark accuracy averaged over three runs, with standard deviation across runs\. The best result for each model and metric is shown in bold\.ModelPeak Mem\. \(GB\)Train Time \(hr\)SpeedupLoRAACELoRAACEOLMoE98\.2198\.182\.471\.891\.31×\\timesQwen89\.9089\.846\.184\.191\.48×\\timesMoonlight48\.8048\.8023\.7516\.771\.42×\\timesKimi118\.22118\.2287\.3960\.411\.45×\\timesTable 2:Training efficiency over three epochs\. We compareACEwith expert\-wise LoRA under the same batch size and training setup\. Speedup is computed as LoRA time divided byACEtime\.
### 4\.2Mathematical Reasoning, Code Generation, and Legal Understanding

To test whether the gains ofACEextend beyond commonsense reasoning, we evaluate four domain\-specific benchmarks on OLMoE\-1B\-7B and Qwen1\.5\-MoE\-A2\.7B\. We fine\-tune on MetaMathQA\([Yu et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib44)\)for GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib36)\)and MATH\([Hendrycks et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib35)\), on CodeFeedback\-Filtered\-Instruction\([Zheng et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib45)\)for HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib37)\), and use CaseHOLD\([Zheng et al\., 2021](https://arxiv.org/html/2609.06072#bib.bib46)\)for legal understanding\. Training and evaluation details are in Appendix[B\.2](https://arxiv.org/html/2609.06072#A2.SS2)\.

Table[3](https://arxiv.org/html/2609.06072#S4.T3)shows thatACEachieves the best average on both backbones and leads on most individual benchmarks, indicating that adapter consolidation generalizes across specialized domains\.

### 4\.3Computational and Memory Efficiency

We evaluate computational efficiency against expert\-wise LoRA, the execution pattern directly targeted byACE\. All methods are compared under the same effective batch size and training setup within each backbone\. As shown in Table[2](https://arxiv.org/html/2609.06072#S4.T2),ACEreduces wall\-clock training time across all evaluated MoE backbones, with speedups ranging from 1\.31×\\timeson OLMoE to 1\.48×\\timeson Qwen, 1\.42×\\timeson Moonlight, and 1\.45×\\timeson Kimi\. The improvement comes without additional memory cost: peak memory remains unchanged compared with LoRA\.

These results indicate that grouped adapter execution effectively reduces the small\-GEMM overhead introduced by expert\-wise LoRA\. Instead of repeatedly evaluating separate adapter computations for each routed expert,ACEevaluates shared adapters at the group level, yielding practical training\-time gains while preserving the same trainable parameter budget\. Additional runtime and memory results for other PEFT baselines are provided in Appendix[N](https://arxiv.org/html/2609.06072#A14), and a per\-stage breakdown of the consolidation pipeline in Appendix[O](https://arxiv.org/html/2609.06072#A15)\.

OLMoEQwenMethod\# Params \(%\)↓\\downarrowGSM8K↑\\uparrowMATH↑\\uparrowHEval↑\\uparrowCHold↑\\uparrowAvg\.↑\\uparrow\# Params \(%\)↓\\downarrowGSM8K↑\\uparrowMATH↑\\uparrowHEval↑\\uparrowCHold↑\\uparrowAvg\.↑\\uparrowLoRA1\.0865\.7325\.9530\.4978\.5350\.180\.8666\.3436\.3177\.4472\.1163\.05AdaLoRA2\.16→\\rightarrow1\.0865\.2024\.6826\.2275\.6547\.941\.72→\\rightarrow0\.8666\.6438\.3379\.2769\.0663\.33DoRA1\.1466\.1125\.7828\.0578\.6449\.650\.9167\.1037\.4177\.4472\.2663\.55PiSSA1\.0865\.8824\.6832\.3279\.8350\.680\.8666\.1939\.3475\.6174\.9364\.02ACE\(ours\)1\.0866\.4926\.9631\.1082\.5751\.780\.8667\.4837\.4979\.2775\.2464\.87

Table 3:Results on mathematical reasoning, code generation, and legal understanding across MoE backbones\. We report accuracy for GSM8K, MATH, and CaseHOLD \(CHold\), and pass@10 for HumanEval \(HEval\)\. The Avg\. reports the mean across the four benchmarks\. The best result for each model and metric is shown in bold\.

## 5Further Analysis

### 5\.1DoesACEMitigate MoE LoRA Fragmentations?

We evaluate whetherACEmitigates the three fragmentations from Section[3\.1](https://arxiv.org/html/2609.06072#S3.SS1)using effective rank \(capacity\), gradient statistics \(supervision\), and CUDA profiling \(execution\)\.

##### Capacity fragmentation\.

We measure the*effective rank*of each learned adapter updateΔ​W=B​A\\Delta W=BA, computed as the exponential of the entropy of its normalized singular values\([Roy and Vetterli, 2007](https://arxiv.org/html/2609.06072#bib.bib28)\)\. This metric captures how much of the nominal low\-rank capacity is effectively used by the learned update\.

Figure 3:Distribution of effective ranks for LoRA\-r=2r\{=\}2, LoRA\-r=64r\{=\}64, andACE\. The parameter counts denote the total number of trainable parameters for each method\.Figure[3](https://arxiv.org/html/2609.06072#S5.F3)compares the effective\-rank distributions of LoRA\-r=2r\{=\}2, LoRA\-r=64r\{=\}64, andACEwith base rankr=2r\{=\}2\. LoRA\-r=2r\{=\}2is strongly bottlenecked \(mean effective rank 1\.77\), whereas LoRA\-r=64r\{=\}64shifts the distribution to a mean of 46\.8 at the cost of roughly32×32\\timesmore trainable parameters\. Under the same budget as LoRA\-r=2r\{=\}2,ACEshifts the distribution close to LoRA\-r=64r\{=\}64, indicating that pooling expert\-wise low\-rank budgets recovers effective adaptation capacity without increasing trainable parameters\.

##### Gradient\-supervision fragmentation\.

Figure 4:Gradient\-supervision fragmentation for up\-projection LoRA adapters across three MoE backbones\.ACEreduces both gradient\-norm imbalance and the zero\-gradient ratio relative to expert\-wise LoRA\.Under sparse routing, each expert\-wise LoRA receives gradients only when its expert is selected, producing uneven and often near\-zero updates\. To quantify this, we measure two statistics over per\-expert gradient norms within each MoE layer: \(i\) the coefficient of variation \(CV\), capturing imbalance across experts, and \(ii\) the fraction of adapter rows with entirely zero gradients, capturing the prevalence of unsupervised parameters\. Both are averaged over 50 training steps\. Additional measurement details and gate\-projection results are provided in Appendix[M](https://arxiv.org/html/2609.06072#A13)\. Figure[4](https://arxiv.org/html/2609.06072#S5.F4)shows thatACEconsistently reduces both metrics across Qwen, OLMoE, and Moonlight backbones\. For instance, on OLMoEup\_proj,ACEreduces the CV from 1\.449 to 0\.531 and the zero\-gradient ratio from 3\.18% to 1\.17%\. These results indicate that shared\-group updates densify supervision by updating each consolidated adapter whenever any expert in its group is routed\.

##### Execution fragmentation\.

MetricLoRAACEMatrix multiplication calls106\.0K62\.2K \(−41\.3%\-41\.3\\%\)Matrix multiplication CUDA time838ms558ms \(−33\.4%\-33\.4\\%\)CUDA kernel events482\.9K389\.0K \(−19\.4%\-19\.4\\%\)CUDA kernel time10\.36s8\.14s \(−21\.4%\-21\.4\\%\)Wall\-clock time20\.24s17\.68s \(−12\.6%\-12\.6\\%\)

Table 4:CUDA profiling prefill usingtorch\.profiler\. Matrix multiplication rows correspond to PyTorchaten::mmoperations in the profiled region\. Kernel\-level rows include all CUDA operations in the same region\. Values in parentheses indicate the relative reduction compared with LoRA\.For execution fragmentation, we examine whether grouping adapters reduces the many small matrix multiplications induced by expert\-wise LoRA\. We profile prefill on Qwen1\.5\-MoE\-A2\.7B, measure matrix multiplication calls, their CUDA execution time, and overall CUDA kernel activity\. Table[4](https://arxiv.org/html/2609.06072#S5.T4)shows thatACEconsistently reduces these profiler\-level indicators, including matrix multiplication calls, CUDA kernel events, and CUDA execution time\. The reduction also translates into lower wall\-clock prefill time, although the wall\-clock gain is smaller because the profiled region includes non\-adapter computation, CPU\-side overhead, and framework overhead\. These results indicate thatACEreduces execution fragmentation in practice\.

### 5\.2CKA Similarity Reflects Functional Redundancy

Figure 5:Adapter swapping analysis\. Swapping high\-CKA adapters causes much smaller NLL degradation than swapping low\-CKA or random adapters, indicating that CKA similarity reflects functional interchangeability\.ACEgroups experts whose adapter representations are similar under CKA\. To justify this criterion, we test whether high\-CKA similarity implies that two adapters make similar task\-specific corrections\. We use adapter swapping as a direct test: if two adapters are functionally redundant, replacing one with the other without retraining should cause only a small change in model behavior\.

For each MoE layer, we choose an anchor gate\-projection adapter, rank the remaining adapters by CKA similarity, and replace either the top\-ρ\\rho, bottom\-ρ\\rho, or random\-ρ\\rhofraction with the anchor, whereρ∈\{0\.05,0\.10,0\.15,0\.20\}\\rho\\in\\\{0\.05,0\.10,0\.15,0\.20\\\}\. We then measure the change in negative log\-likelihood,Δ​NLL\\Delta\\mathrm\{NLL\}, without retraining\.

Figure[5](https://arxiv.org/html/2609.06072#S5.F5)shows that high\-similarity swaps cause substantially smaller NLL degradation than low\-similarity or random swaps across selection ratios\. This indicates that CKA similarity captures functional interchangeability, meaning that CKA\-based grouping can consolidate adapters while minimizing behavioral disruption\. Up\-projection results are provided in Appendix[I\.1](https://arxiv.org/html/2609.06072#A9.SS1)\.

We further observe positive CKA gaps across all four evaluated backbones on commonsense reasoning \(0\.298–0\.438\), as well as across mathematical reasoning, code generation, and legal understanding on OLMoE and Qwen\. This indicates that the groupable functional structure exploited byACEgeneralizes across both model architectures and task domains\. Details are provided in Appendix[I\.2](https://arxiv.org/html/2609.06072#A9.SS2)\.

MethodSim\.GroupRankCons\.Grp\.Exec\.Avg\.↑\\uparrowTime↓\\downarrowMem\.↓\\downarrowLoRA×\\times×\\times×\\times74\.112\.4798\.21w/o Sim\. GroupRandom×\\times✓✓74\.72\(−0\.89\)\(\-0\.89\)1\.8998\.18Full Merge×\\times✓✓73\.57\(−2\.04\)\(\-2\.04\)2\.03\(\+0\.14\)\(\+0\.14\)97\.61w/o Rank Cons\.✓×\\times✓72\.63\(−2\.98\)\(\-2\.98\)1\.8496\.54w/o Grp\. Exec\.✓✓×\\times75\.612\.83\(\+0\.94\)\(\+0\.94\)98\.18ACE✓✓✓75\.611\.8998\.18

Table 5:Component ablation ofACE\.*Sim\. Group*denotes CKA\-based functional grouping;*Rank cons\.*pools per\-expert ranks into a higher\-rank group adapter;*Grp\. exec\.*evaluates each shared adapter once per active group\. Under*w/o Sim\. Group*,*Random*groups experts randomly, and*Full Merge*consolidates all experts into one group\. Time in hours, memory in GB\.
### 5\.3Ablation Study

Table[5](https://arxiv.org/html/2609.06072#S5.T5)ablates the main components ofACEon OLMoE\.Full Mergeperforms worse than LoRA, showing that excessive consolidation can remove useful expert specialization, while Random Group removes the CKA\-based alignment used byACE\. Among the variants that preserve the trainable\-parameter budget,Full Mergehas the lowest peak memory \(97\.61 GB\) since grouped execution caches one activation per group, confirming that consolidation reduces activation memory alongside compute\.No Rank Cons\.isolates the effect of sharing without pooling expert\-wise ranks\.No Grp\. Exec\.keeps the same consolidated adapters asACEbut disables grouped execution; accuracy is unchanged, while runtime increases from 1\.89 to 2\.83 hours\.

These results show that the components play distinct roles: similarity grouping selects which adapters to consolidate, rank consolidation increases effective capacity, and grouped execution provides the runtime gain\.

### 5\.4Expert Adaptation Is a Primary Target in MoE PEFT

OLMoEQwenAdaptation Scope\# ParamsAvg\.↑\\uparrow\# ParamsAvg\.↑\\uparrowAttention only \(Q, K, V\)0\.3671\.46±0\.2071\.46\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.20\}0\.2679\.72±0\.2079\.72\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.20\}MoE only \(LoRA\)0\.2774\.11±0\.3974\.11\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.39\}0\.2179\.75±0\.0379\.75\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.03\}MoE only \(ACE\)0\.2775\.61±0\.28\\mathbf\{75\.61\}\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.28\}0\.2180\.27±0\.18\\mathbf\{80\.27\}\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.18\}Attn \+ MoE \(LoRA\)0\.6375\.12±0\.3875\.12\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.38\}0\.4780\.46±0\.1380\.46\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.13\}Attn \+ MoE \(ACE\)0\.6375\.99±0\.16\\mathbf\{75\.99\}\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.16\}0\.4780\.59±0\.12\\mathbf\{80\.59\}\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.12\}

Table 6:Effect of adaptation scope on OLMoE and Qwen, averaged over eight commonsense reasoning benchmarks and three runs\. The \# Params columns report the percentage of trainable parameters\.Finally, we examine whether expert\-side adaptation is necessary, or whether one can avoid expert\-wise fragmentation by adapting only attention modules\. To this end, Table[6](https://arxiv.org/html/2609.06072#S5.T6)compares attention\-only, MoE\-only, and combined attention\-plus\-MoE adaptation on OLMoE and Qwen\.

Across both backbones, MoE\-side adaptation provides a stronger performance–parameter trade\-off than attention\-only adaptation\. In particular, MoE\-onlyACEoutperforms attention\-only adaptation while using fewer trainable parameters on both OLMoE and Qwen\. This trend is consistent with recent findings that MLP\-side adaptation can be more effective than attention\-side adaptation in Transformer fine\-tuning\([Hayou et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib38);[Ward et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib39)\)\.

When attention and MoE adapters are trained together,ACEagain improves over the LoRA counterpart\. This suggests that adapter consolidation is complementary to attention\-side adaptation rather than merely replacing it\. Overall, these results support our focus on expert\-side LoRA fragmentation as a central issue in MoE PEFT\.

## 6Conclusion

We introducedACE, a PEFT method that addresses the three fragmentations of expert\-wise LoRA in MoE models: capacity, gradient supervision, and execution\. By consolidating functionally aligned expert adapters into group\-shared higher\-rank modules with grouped execution,ACEachieves a stronger quality\-efficiency trade\-off than parameter\-matched PEFT baselines across multiple MoE backbones and benchmarks\. More broadly, our results suggest that sharing adaptation across experts can be beneficial compared with maintaining fully separate expert\-wise adapters in MoE fine\-tuning\.

## Limitations

Our work focuses on improving the efficiency of PEFT for MoE models\. While the method itself does not introduce new model capabilities, it facilitates more efficient adaptation of large language models to downstream tasks\. Two limitations are worth noting\. First, grouped adapter execution requires routed experts to share the same per\-expert input, so it applies to up and gate projections but not to the down projection\. Extending the consolidation pipeline to such components is not straightforward and remains for future work\. Second,ACEassumes that expert adapters contain functional redundancy\. When a task demands strong expert specialization, excessive consolidation can erase useful distinctions, as shown by the Full Merge ablation in Table[5](https://arxiv.org/html/2609.06072#S5.T5), which underperforms expert\-wise LoRA \(73\.57 vs\. 74\.11\)\.ACEaddresses this through similarity\-guided grouping rather than uniform merging, but the appropriate grouping granularity \(e\.g\., the resolution parameterγ\\gamma\) may still need tuning when expert specialization is critical\.

## Acknowledgments

This work was supported by grants from the Institute for Information & Communications Technology Planning & Evaluation \(IITP\), funded by the Korea government \(MSIT\): the Artificial Intelligence Graduate School Support program \(RS\-2020\-II201336, UNIST\), the AI Star Fellowship Program \(RS\-2025\-25442824, Ulsan National Institute of Science and Technology\), and the Leading Generative AI Human Resources Development program \(IITP\-2026\-RS\-2024\-00360227\)\. This work was also supported by the National Research Foundation of Korea \(NRF\) grant funded by the Korea government \(MSIT\) \(RS\-2025\-00553241\)\. In addition, this research was supported by the “Advanced GPU Utilization Support Program” funded by the Government of the Republic of Korea \(Ministry of Science and ICT\)\.

We also thank the UAI Lab members, especially Jeonghoon, for their valuable feedback and support in improving the overall quality of this paper\.

## References

- Aminabadiet al\.\(2022\)R\. Y\. Aminabadi, S\. Rajbhandari, A\. A\. Awan, C\. Li, D\. Li, E\. Zheng, O\. Ruwase, S\. Smith, M\. Zhang, J\. Rasley, and Y\. HeDeepSpeed\-inference: enabling efficient inference of transformer models at unprecedented scale\.InProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis,SC ’22\.External Links:ISBN 9784665454445Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.InThirty\-Fourth AAAI Conference on Artificial Intelligence,Cited by:[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Blondelet al\.\(2008\)V\. D\. Blondel, J\. Guillaume, R\. Lambiotte, and E\. LefebvreFast unfolding of communities in large networks\.Journal of statistical mechanics: theory and experiment2008\(10\),pp\. P10008\.Cited by:[§3\.3](https://arxiv.org/html/2609.06072#S3.SS3.p3.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374Cited by:[§4\.2](https://arxiv.org/html/2609.06072#S4.SS2.p1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Link](https://aclanthology.org/N19-1300/),[Document](https://dx.doi.org/10.18653/v1/N19-1300)Cited by:[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.2](https://arxiv.org/html/2609.06072#S4.SS2.p1.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\.X\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu, Z\. Xie, Y\.K\. Li, P\. Huang, F\. Luo, C\. Ruan, Z\. Sui, and W\. LiangDeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 1280–1297\.External Links:[Link](https://aclanthology.org/2024.acl-long.70/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.70)Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1),[§3\.1\.2](https://arxiv.org/html/2609.06072#S3.SS1.SSS2.p1.2)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQLoRA: efficient finetuning of quantized llms\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 10088–10115\.External Links:[Document](https://dx.doi.org/10.52202/075280-0441),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/1feb87871436031bdc0f2beaa62a049b-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1)\.
- Duet al\.\(2022\)N\. Du, Y\. Huang, A\. M\. Dai, S\. Tong, D\. Lepikhin, Y\. Xu, M\. Krikun, Y\. Zhou, A\. W\. Yu, O\. Firat, B\. Zoph, L\. Fedus, M\. P\. Bosma, Z\. Zhou, T\. Wang, E\. Wang, K\. Webster, M\. Pellat, K\. Robinson, K\. Meier\-Hellstern, T\. Duke, L\. Dixon, K\. Zhang, Q\. Le, Y\. Wu, Z\. Chen, and C\. CuiGLaM: efficient scaling of language models with mixture\-of\-experts\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 5547–5569\.External Links:[Link](https://proceedings.mlr.press/v162/du22c.html)Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.External Links:[Link](http://jmlr.org/papers/v23/21-0998.html)Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1)\.
- Hayouet al\.\(2026\)S\. Hayou, N\. Ghosh, and B\. YuPLop: precise loRA placement for efficient finetuning of large models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3lGkVgNZ5a)Cited by:[§5\.4](https://arxiv.org/html/2609.06072#S5.SS4.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.NeurIPS\.Cited by:[§4\.2](https://arxiv.org/html/2609.06072#S4.SS2.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[1st item](https://arxiv.org/html/2609.06072#A2.I1.i1.p1.1),[§1](https://arxiv.org/html/2609.06072#S1.p1.1),[§1](https://arxiv.org/html/2609.06072#S1.p2.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p2.1)\.
- Huet al\.\(2023\)Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. LeeLLM\-adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5254–5276\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.319/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.319)Cited by:[§B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1.p1.1),[§B\.3](https://arxiv.org/html/2609.06072#A2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Jianget al\.\(2025\)J\. Jiang, J\. Zhou, and Z\. ZhuTracing representation progression: analyzing and enhancing layer\-wise similarity\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 1118–1143\.Cited by:[§3\.3](https://arxiv.org/html/2609.06072#S3.SS3.p2.3)\.
- Karimi Mahabadiet al\.\(2021\)R\. Karimi Mahabadi, S\. Ruder, M\. Dehghani, and J\. HendersonParameter\-efficient multi\-task fine\-tuning for transformers via shared hypernetworks\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 565–576\.External Links:[Link](https://aclanthology.org/2021.acl-long.47/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.47)Cited by:[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1)\.
- Kimet al\.\(2021\)Y\. J\. Kim, A\. A\. Awan, A\. Muzio, A\. F\. C\. Salinas, L\. Lu, A\. Hendy, S\. Rajbhandari, Y\. He, and H\. H\. AwadallaScalable and efficient moe training for multitask multilingual models\.arXiv preprint arXiv:2109\.10465\.Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1)\.
- Kornblithet al\.\(2019\)S\. Kornblith, M\. Norouzi, H\. Lee, and G\. HintonSimilarity of neural network representations revisited\.InInternational conference on machine learning,pp\. 3519–3529\.Cited by:[§C\.3](https://arxiv.org/html/2609.06072#A3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2609.06072#S3.SS3.p2.2)\.
- Leeet al\.\(2026\)A\. Lee, S\. Yun, and T\. GongEPnG: adaptive expert prune\-and\-grow for parameter\-efficient moe fine\-tuning\.InProceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services Workshops,pp\. 93–98\.Cited by:[6th item](https://arxiv.org/html/2609.06072#A2.I1.i6.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p2.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 3045–3059\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.243/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.243)Cited by:[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)D\. Li, Y\. Ma, N\. Wang, Z\. Ye, Z\. Cheng, Y\. Tang, Y\. Zhang, L\. Duan, J\. Zuo, C\. Yang, and M\. TangMixLoRA: enhancing large language models fine\-tuning with lora\-based mixture of experts\.External Links:2404\.15159,[Link](https://arxiv.org/abs/2404.15159)Cited by:[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 4582–4597\.External Links:[Link](https://aclanthology.org/2021.acl-long.353/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.353)Cited by:[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan, Y\. Chen, H\. Zheng, Y\. Liu, S\. Liu, B\. Yin, W\. He, H\. Zhu, Y\. Wang, J\. Wang, M\. Dong, Z\. Zhang, Y\. Kang, H\. Zhang, X\. Xu, Y\. Zhang, Y\. Wu, X\. Zhou, and Z\. YangMuon is scalable for llm training\.External Links:2502\.16982,[Link](https://arxiv.org/abs/2502.16982)Cited by:[Appendix A](https://arxiv.org/html/2609.06072#A1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Liuet al\.\(2024\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDora: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[2nd item](https://arxiv.org/html/2609.06072#A2.I1.i2.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p2.1)\.
- Liuet al\.\(2026\)Y\. Liu, Y\. Ma, Y\. Lu, S\. Chen, Z\. Ding, and V\. TrespParameter\-efficient routed fine\-tuning: mixture\-of\-experts demands mixture of adaptation modules\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 4439–4457\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.232/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.232),ISBN 979\-8\-89176\-386\-9Cited by:[5th item](https://arxiv.org/html/2609.06072#A2.I1.i5.p1.1),[§B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1.p1.1),[§B\.1\.3](https://arxiv.org/html/2609.06072#A2.SS1.SSS3.Px1.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p2.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPissa: principal singular values and singular vectors adaptation of large language models\.Advances in Neural Information Processing Systems37,pp\. 121038–121072\.Cited by:[4th item](https://arxiv.org/html/2609.06072#A2.I1.i4.p1.1),[§1](https://arxiv.org/html/2609.06072#S1.p2.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p2.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2381–2391\.External Links:[Link](https://aclanthology.org/D18-1260/),[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Muennighoffet al\.\(2025\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, E\. P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, N\. A\. Smith, P\. W\. Koh, A\. Singh, and H\. HajishirziOLMoe: open mixture\-of\-experts language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xXTkbTBmqq)Cited by:[Appendix A](https://arxiv.org/html/2609.06072#A1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Narayanet al\.\(2018\)S\. Narayan, S\. B\. Cohen, and M\. LapataDon’t give me the details, just the summary\! topic\-aware convolutional neural networks for extreme summarization\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 1797–1807\.External Links:[Link](https://aclanthology.org/D18-1206/),[Document](https://dx.doi.org/10.18653/v1/D18-1206)Cited by:[Appendix H](https://arxiv.org/html/2609.06072#A8.p1.1)\.
- NVIDIA Corporation \(2026\)NVIDIA CorporationcuBLAS library documentation\.NVIDIA Corporation\.Note:cuBLAS 13\.2 documentation\. Accessed: 2026\-05\-19External Links:[Link](https://docs.nvidia.com/cuda/cublas/)Cited by:[§3\.1\.3](https://arxiv.org/html/2609.06072#S3.SS1.SSS3.p1.2)\.
- OpenAIet al\.\(2024\)OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, J\. Belgum, I\. Bello, J\. Berdine, G\. Bernadett\-Shapiro, C\. Berner, L\. Bogdonoff, O\. Boiko, M\. Boyd, A\. Brakman, G\. Brockman, T\. Brooks, M\. Brundage, K\. Button, T\. Cai, R\. Campbell, A\. Cann, B\. Carey, C\. Carlson, R\. Carmichael, B\. Chan, C\. Chang, F\. Chantzis, D\. Chen, S\. Chen, R\. Chen, J\. Chen, M\. Chen, B\. Chess, C\. Cho, C\. Chu, H\. W\. Chung, D\. Cummings, J\. Currier, Y\. Dai, C\. Decareaux, T\. Degry, N\. Deutsch, D\. Deville, A\. Dhar, D\. Dohan, S\. Dowling, S\. Dunning, A\. Ecoffet, A\. Eleti, T\. Eloundou, D\. Farhi, L\. Fedus, N\. Felix, S\. P\. Fishman, J\. Forte, I\. Fulford, L\. Gao, E\. Georges, C\. Gibson, V\. Goel, T\. Gogineni, G\. Goh, R\. Gontijo\-Lopes, J\. Gordon, M\. Grafstein, S\. Gray, R\. Greene, J\. Gross, S\. S\. Gu, Y\. Guo, C\. Hallacy, J\. Han, J\. Harris, Y\. He, M\. Heaton, J\. Heidecke, C\. Hesse, A\. Hickey, W\. Hickey, P\. Hoeschele, B\. Houghton, K\. Hsu, S\. Hu, X\. Hu, J\. Huizinga, S\. Jain, S\. Jain, J\. Jang, A\. Jiang, R\. Jiang, H\. Jin, D\. Jin, S\. Jomoto, B\. Jonn, H\. Jun, T\. Kaftan, Ł\. Kaiser, A\. Kamali, I\. Kanitscheider, N\. S\. Keskar, T\. Khan, L\. Kilpatrick, J\. W\. Kim, C\. Kim, Y\. Kim, J\. H\. Kirchner, J\. Kiros, M\. Knight, D\. Kokotajlo, Ł\. Kondraciuk, A\. Kondrich, A\. Konstantinidis, K\. Kosic, G\. Krueger, V\. Kuo, M\. Lampe, I\. Lan, T\. Lee, J\. Leike, J\. Leung, D\. Levy, C\. M\. Li, R\. Lim, M\. Lin, S\. Lin, M\. Litwin, T\. Lopez, R\. Lowe, P\. Lue, A\. Makanju, K\. Malfacini, S\. Manning, T\. Markov, Y\. Markovski, B\. Martin, K\. Mayer, A\. Mayne, B\. McGrew, S\. M\. McKinney, C\. McLeavey, P\. McMillan, J\. McNeil, D\. Medina, A\. Mehta, J\. Menick, L\. Metz, A\. Mishchenko, P\. Mishkin, V\. Monaco, E\. Morikawa, D\. Mossing, T\. Mu, M\. Murati, O\. Murk, D\. Mély, A\. Nair, R\. Nakano, R\. Nayak, A\. Neelakantan, R\. Ngo, H\. Noh, L\. Ouyang, C\. O’Keefe, J\. Pachocki, A\. Paino, J\. Palermo, A\. Pantuliano, G\. Parascandolo, J\. Parish, E\. Parparita, A\. Passos, M\. Pavlov, A\. Peng, A\. Perelman, F\. de Avila Belbute Peres, M\. Petrov, H\. P\. de Oliveira Pinto, Michael, Pokorny, M\. Pokrass, V\. H\. Pong, T\. Powell, A\. Power, B\. Power, E\. Proehl, R\. Puri, A\. Radford, J\. Rae, A\. Ramesh, C\. Raymond, F\. Real, K\. Rimbach, C\. Ross, B\. Rotsted, H\. Roussez, N\. Ryder, M\. Saltarelli, T\. Sanders, S\. Santurkar, G\. Sastry, H\. Schmidt, D\. Schnurr, J\. Schulman, D\. Selsam, K\. Sheppard, T\. Sherbakov, J\. Shieh, S\. Shoker, P\. Shyam, S\. Sidor, E\. Sigler, M\. Simens, J\. Sitkin, K\. Slama, I\. Sohl, B\. Sokolowsky, Y\. Song, N\. Staudacher, F\. P\. Such, N\. Summers, I\. Sutskever, J\. Tang, N\. Tezak, M\. B\. Thompson, P\. Tillet, A\. Tootoonchian, E\. Tseng, P\. Tuggle, N\. Turley, J\. Tworek, J\. F\. C\. Uribe, A\. Vallone, A\. Vijayvergiya, C\. Voss, C\. Wainwright, J\. J\. Wang, A\. Wang, B\. Wang, J\. Ward, J\. Wei, C\. Weinmann, A\. Welihinda, P\. Welinder, J\. Weng, L\. Weng, M\. Wiethoff, D\. Willner, C\. Winter, S\. Wolrich, H\. Wong, L\. Workman, S\. Wu, J\. Wu, M\. Wu, K\. Xiao, T\. Xu, S\. Yoo, K\. Yu, Q\. Yuan, W\. Zaremba, R\. Zellers, C\. Zhang, M\. Zhang, S\. Zhao, T\. Zheng, J\. Zhuang, W\. Zhuk, and B\. ZophGPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1)\.
- Ponset al\.\(2024\)I\. Pons, B\. Yamamoto, A\. H\. Reali Costa, and A\. JordaoEffective layer pruning through similarity metric perspective\.InInternational Conference on Pattern Recognition,pp\. 423–438\.Cited by:[§3\.3](https://arxiv.org/html/2609.06072#S3.SS3.p2.3)\.
- Qwen \(2024\)QwenQwen1\.5\-moe: matching 7b model performance with 1/3 activated parameters\.Note:[https://qwenlm\.github\.io/blog/qwen\-moe/](https://qwenlm.github.io/blog/qwen-moe/)Accessed: 2025\-09\-23Cited by:[Appendix A](https://arxiv.org/html/2609.06072#A1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Roy and Vetterli \(2007\)O\. Roy and M\. VetterliThe effective rank: a measure of effective dimensionality\.In2007 15th European signal processing conference,pp\. 606–610\.Cited by:[§5\.1](https://arxiv.org/html/2609.06072#S5.SS1.SSS0.Px1.p1.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.Commun\. ACM64\(9\),pp\. 99–106\.External Links:ISSN 0001\-0782,[Link](https://doi.org/10.1145/3474381),[Document](https://dx.doi.org/10.1145/3474381)Cited by:[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. ChoiSocial IQa: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 4463–4473\.External Links:[Link](https://aclanthology.org/D19-1454/),[Document](https://dx.doi.org/10.18653/v1/D19-1454)Cited by:[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§1](https://arxiv.org/html/2609.06072#S1.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px2.p1.1),[§3\.1\.2](https://arxiv.org/html/2609.06072#S3.SS1.SSS2.p1.2)\.
- Teamet al\.\(2025\)K\. Team, Y\. Zhang, Z\. Lin, X\. Yao, J\. Hu, F\. Meng, C\. Liu, X\. Men, S\. Yang, Z\. Li, W\. Li, E\. Lu, W\. Liu, Y\. Chen, W\. Xu, L\. Yu, Y\. Wang, Y\. Fan, L\. Zhong, E\. Yuan, D\. Zhang, Y\. Zhang, T\. Y\. Liu, H\. Wang, S\. Fang, W\. He, S\. Liu, Y\. Li, J\. Su, J\. Qiu, B\. Pang, J\. Yan, Z\. Jiang, W\. Huang, B\. Yin, J\. You, C\. Wei, Z\. Wang, C\. Hong, Y\. Chen, G\. Chen, Y\. Wang, H\. Zheng, F\. Wang, Y\. Liu, M\. Dong, Z\. Zhang, S\. Pan, W\. Wu, Y\. Wu, L\. Guan, J\. Tao, G\. Fu, X\. Xu, Y\. Wang, G\. Lai, Y\. Wu, X\. Zhou, Z\. Yang, and Y\. DuKimi linear: an expressive, efficient attention architecture\.External Links:2510\.26692,[Link](https://arxiv.org/abs/2510.26692)Cited by:[Appendix A](https://arxiv.org/html/2609.06072#A1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p3.1)\.
- Wanget al\.\(2024\)Z\. Wang, D\. Chen, D\. Dai, R\. Xu, Z\. Li, and Y\. WuLet the expert stick to his last: expert\-specialized fine\-tuning for sparse architectural large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 784–801\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.46/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.46)Cited by:[§B\.1\.3](https://arxiv.org/html/2609.06072#A2.SS1.SSS3.Px2.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px2.p1.1)\.
- Wardet al\.\(2025\)J\. Ward, P\. Riechers, and A\. ShaiRank\-1 loras encode interpretable reasoning signals\.arXiv preprint arXiv:2511\.06739\.Cited by:[§5\.4](https://arxiv.org/html/2609.06072#S5.SS4.p2.1)\.
- Xuet al\.\(2024\)R\. Xu, W\. Shi, Y\. Yu, Y\. Zhuang, Y\. Zhu, M\. D\. Wang, J\. C\. Ho, C\. Zhang, and C\. YangBMRetriever: tuning large language models as better biomedical text retrievers\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 22234–22254\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1241/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1241)Cited by:[§B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1.p1.1)\.
- Yuet al\.\(2024\)L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetamath: bootstrap your own mathematical questions for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 45040–45061\.Cited by:[§4\.2](https://arxiv.org/html/2609.06072#S4.SS2.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/),[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p1.1)\.
- Zhanget al\.\(2023\)Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. ZhaoAdaptive budget allocation for parameter\-efficient fine\-tuning\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=lq62uWRJjiY)Cited by:[3rd item](https://arxiv.org/html/2609.06072#A2.I1.i3.p1.1),[§B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1.p1.1),[§2](https://arxiv.org/html/2609.06072#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.06072#S4.SS1.p2.1)\.
- Zhaoet al\.\(2024\)W\. Zhao, S\. Wang, Y\. Hu, Y\. Zhao, B\. Qin, X\. Zhang, Q\. Yang, D\. Xu, and W\. CheSAPT: a shared attention framework for parameter\-efficient continual learning of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11641–11661\.External Links:[Link](https://aclanthology.org/2024.acl-long.625/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.625)Cited by:[§B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1.p1.1)\.
- Zhenget al\.\(2021\)L\. Zheng, N\. Guha, B\. R\. Anderson, P\. Henderson, and D\. E\. HoWhen does pretraining help? assessing self\-supervised learning for law and the casehold dataset of 53,000\+ legal holdings\.InProceedings of the eighteenth international conference on artificial intelligence and law,pp\. 159–168\.Cited by:[§4\.2](https://arxiv.org/html/2609.06072#S4.SS2.p1.1)\.
- Zhenget al\.\(2024\)T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. YueOpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12834–12859\.External Links:[Link](https://aclanthology.org/2024.findings-acl.762/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.762)Cited by:[§4\.2](https://arxiv.org/html/2609.06072#S4.SS2.p1.1)\.

## Appendix AModel Configurations

We summarize the key hyperparameters of the MoE models used in our experiments, OLMoE\-1B\-7B\([Muennighoff et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib9)\)and Qwen1\.5\-MoE\-A2\.7B\([Qwen, 2024](https://arxiv.org/html/2609.06072#bib.bib11)\), Moonlight\-16B\-A3B\([Liu et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib42)\), and Kimi\-Linear\-48B\-A3B\([Team et al\., 2025](https://arxiv.org/html/2609.06072#bib.bib43)\), in Table[7](https://arxiv.org/html/2609.06072#A1.T7)\. This provides a clear overview of the architectural differences between the models\.

HyperparameterOLMoE\-1B\-7BQwen1\.5\-MoE\-A2\.7BMoonlight\-16B\-A3BKimi\-Linear\-48B\-A3BTotal Parameters6\.9B14\.3B16B48BActive Parameters1\.3B2\.7B3B3BLayers \(LL\)16242727Hidden Dimension \(dmodeld\_\{\\text\{model\}\}\)2048204820482304Number of Experts \(NN\)6460 \+ 1 \(shared\)64 \+ 2 \(shared\)256 \+ 1 \(shared\)Top\-K Routing \(kk\)8468Expert FFN Intermediate Size1024140814081024

Table 7:A summary of the MoE model configurations used in our experiments\.
## Appendix BComputing Infrastructure

All experiments were conducted on a workstation running Ubuntu 22\.04\.5 LTS with CUDA 12\.8\. Unless otherwise specified, each experiment was conducted using NVIDIA H200 GPUs with 141 GiB of memory\.

For efficiency measurements, wall\-clock time is reported as the actual end\-to\-end elapsed training time, and peak GPU memory is measured usingtorch\.cuda\.max\_memory\_allocated\(\)\.

### B\.1Configuration for Table[1](https://arxiv.org/html/2609.06072#S4.T1)and Table[2](https://arxiv.org/html/2609.06072#S4.T2)

#### B\.1\.1Hyperparameters

Unless otherwise specified, all methods were trained under the same optimization setup across the four MoE backbones used in Table[1](https://arxiv.org/html/2609.06072#S4.T1)and[17](https://arxiv.org/html/2609.06072#A10.T17): OLMoE\-1B\-7B\-0125, Qwen1\.5\-MoE\-A2\.7B, Moonlight\-16B\-A3B, and Kimi\-Linear\-48B\-A3B\. We use the AdamW optimizer with weight decay 0\.1 and a constant learning\-rate schedule\. The maximum sequence length was set to 256, and the effective batch size was 64\. For Table[1](https://arxiv.org/html/2609.06072#S4.T1), we report results over three random seeds \(0, 1, and 2\)\. For all low\-rank adaptation methods, the rank was fixed tor=2r=2\. We trained all models for 3 epochs, following[Liu et al\. \(2026\)](https://arxiv.org/html/2609.06072#bib.bib20);[Hu et al\. \(2023\)](https://arxiv.org/html/2609.06072#bib.bib30)\. For LoRA methods, the LoRA scaling factor was set to 16 and the LoRA dropout was set to 0\.05\. Following prior LoRA\-based fine\-tuning work that uses smaller learning rates for larger backbones\([Zhao et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib49);[Xu et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib50)\), we use1×10−51\\times 10^\{\-5\}for ACE, LoRA, DoRA, and PiSSA on OLMoE and Qwen,5×10−65\\times 10^\{\-6\}on Moonlight, and2×10−62\\times 10^\{\-6\}on Kimi\-Linear\. Our learning rate for OLMoE is also consistent with prior MoE PEFT work[Liu et al\. \(2026\)](https://arxiv.org/html/2609.06072#bib.bib20)\. AdaLoRA is treated separately, as it required a larger learning rate than the shared setting used for the other methods\([Zhang et al\., 2023](https://arxiv.org/html/2609.06072#bib.bib27)\)\. We therefore use7×10−57\\times 10^\{\-5\}on OLMoE and Qwen,4×10−54\\times 10^\{\-5\}on Moonlight, and1×10−51\\times 10^\{\-5\}on Kimi\-Linear for AdaLoRA\.

ForACE, in addition to the shared training setup above, we set theTi​n​i​t=T\_\{init\}=150 and the Louvain resolution parameterγ\\gammato 0\.8\. We applyACEto the up and gate projections; the down projection retains standard expert\-wise LoRA, since its input is already expert\-specific and does not admit grouped adapter execution \(see Section[3\.5](https://arxiv.org/html/2609.06072#S3.SS5)for details\)\.

#### B\.1\.2Baselines

- •LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.06072#bib.bib1)\): A parameter\-efficient fine\-tuning method that learns low\-rank update matrices while keeping the original model weights frozen\.
- •DoRA\([Liu et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib2)\): A LoRA\-based variant that separates weight updates into directional and magnitude components to improve adaptation quality\.
- •AdaLoRA\([Zhang et al\., 2023](https://arxiv.org/html/2609.06072#bib.bib27)\): A dynamic extension of LoRA that redistributes the adaptation rank during training according to the estimated importance of different parameters\.
- •PiSSA\([Meng et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib29)\): A low\-rank adaptation method that initializes adapters using principal singular components of pretrained weights, leading to more effective and stable optimization\.
- •PERFT\-R\([Liu et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib20)\): A MoE\-aware PEFT method that introduces separately routed LoRA adapter experts, applying MoE\-style routing on top of the adapter modules\.
- •EPnG\([Lee et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib47)\): A MoE\-aware PEFT method that reallocates LoRA capacity across experts according to their routing importance, assigning larger adaptation capacity to more important experts\.

#### B\.1\.3Notes on MoE\-Specific Baselines

We provide additional context on how MoE\-specific PEFT methods are positioned in our comparison\.

##### PERFT\-R\.

PERFT\-R\([Liu et al\., 2026](https://arxiv.org/html/2609.06072#bib.bib20)\)is a MoE\-aware PEFT method that introduces MoE\-style routing over LoRA adapters through separately routed adapter experts\. We include PERFT\-R as a MoE\-specific PEFT baseline because it explicitly incorporates MoE structure into adapter tuning, unlike generic PEFT methods adapted from dense Transformers\. We note that our reported numbers may differ from those in the original PERFT\-R paper, as we evaluate all methods under a unified setup that matches the expert\-wise LoRA configuration used for the other baselines in this work \(e\.g\., adapted projections, parameter budget, and optimization hyperparameters\), rather than the configuration adopted in the original PERFT\-R evaluation\. This setup is chosen to ensure a fair comparison across all PEFT baselines under matched conditions\.

##### ESFT\.

ESFT\([Wang et al\., 2024](https://arxiv.org/html/2609.06072#bib.bib22)\)selects task\-relevant experts and directly fine\-tunes their parameters\. It is therefore a selective expert tuning method rather than a LoRA\-style adapter\-based PEFT method\. Since its trainable parameter count depends on the number of selected experts and tuned layers, ESFT is difficult to match to the adapter\-based budget used by ACE and the LoRA baselines\. For a fair comparison, we restrict the main quantitative evaluation to methods under the same LoRA\-based PEFT setting, and discuss ESFT as related MoE fine\-tuning work in Section[2](https://arxiv.org/html/2609.06072#S2)\.

#### B\.1\.4Seed\-wise Consistency

To further characterize run\-to\-run variability in the main results, we compareACEwith the strongest fully evaluated non\-ACEbaseline for each backbone in Table[1](https://arxiv.org/html/2609.06072#S4.T1)\.

BackboneMethodSeed 0Seed 1Seed 2OLMoEPiSSA74\.5874\.9074\.78ACE75\.7075\.8475\.30QwenPiSSA80\.1179\.9679\.89ACE80\.3280\.4280\.07MoonlightEPnG79\.8980\.1380\.36ACE80\.9881\.3581\.19Table 8:Seed\-wise average accuracy against the strongest fully evaluated non\-ACEbaseline for each backbone\.Across all three random seeds,ACEconsistently achieves higher average accuracy than the corresponding strongest baseline on each backbone\. In total,ACEis higher in all nine matched backbone–seed comparisons\. These results indicate that the improvements reported in Table[1](https://arxiv.org/html/2609.06072#S4.T1)are consistent across random seeds\.

### B\.2Configuration for Table[3](https://arxiv.org/html/2609.06072#S4.T3)

For mathematical reasoning and code generation, we use the same optimization setup as in Section[B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1), except that the maximum sequence length was set to 1024, the effective batch size was 16, the rank was set to 8, and all models were trained for 1 epoch\. For code datasets, we report pass@10 using a sampling temperature of 0\.7; for math datasets, we report exact\-match accuracy\. For legal understanding, we follow the optimization setup of Section[B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1)with rank 8 and report accuracy\.

MethodTinitT\_\{\\text\{init\}\}HSWGARC\-cARC\-eBoolQOBQAPIQASIQAAvg\.LoRA–90\.8270\.0869\.4285\.6166\.7174\.6083\.1052\.5174\.11ACE5092\.1472\.4571\.3386\.6269\.3677\.4082\.3753\.3875\.63ACE15092\.2773\.3270\.8286\.4569\.1476\.6084\.0652\.9775\.70ACE50092\.5270\.8869\.6286\.4168\.7876\.4083\.7353\.5375\.23

Table 9:Effect of warm\-up duration onACE\. We varyTinitT\_\{\\text\{init\}\}while keeping all other settings fixed, and include LoRA as a baseline\.ACEshows limited sensitivity to the exact warm\-up duration and outperforms LoRA at everyTinitT\_\{\\text\{init\}\}, withTinit=150T\_\{\\text\{init\}\}=150giving the best average performance in this sweep\. Results are from one random seed\.
### B\.3Dataset Statistics

We use four training datasets comprising 764,455 source examples: Commonsense170K \(170,420\), MetaMathQA \(395,000\), CodeFeedback\-Filtered\-Instruction \(156,526\), and CaseHOLD \(42,509\)\. For the commonsense reasoning results in Table[1](https://arxiv.org/html/2609.06072#S4.T1), following[Hu et al\., 2023](https://arxiv.org/html/2609.06072#bib.bib30), Commonsense170K combines the training splits of eight commonsense tasks\. After tokenization and sequence packing, each training set is randomly split into 98% training and 2% validation subsets\.

For evaluation, we use 30,403 examples across 12 benchmarks: HellaSwag \(10,042\), WinoGrande \(1,267\), ARC\-Challenge \(1,172\), ARC\-Easy \(2,376\), BoolQ \(3,270\), OpenBookQA \(500\), PIQA \(1,838\), Social IQA \(1,954\), GSM8K \(1,319\), MATH \(1,187\), HumanEval \(164\), and CaseHOLD \(5,314\)\.

### B\.4Configuration for Table[5](https://arxiv.org/html/2609.06072#S5.T5)

The ablation study in Table[5](https://arxiv.org/html/2609.06072#S5.T5)is conducted on OLMoE\-1B\-7B\-0125 using the same training setup and hyperparameters as Table[1](https://arxiv.org/html/2609.06072#S4.T1)\(Appendix[B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1)\)\. All compared methods share the same optimization configuration\. The trainable parameter budget is matched for LoRA, Random, Full Merge, w/o Grp\. Exec\., andACE, while w/o Rank Cons\. is reported separately because disabling rank consolidation changes the number of trainable adapter parameters\.

### B\.5Licenses and Terms of Use

All model backbones and datasets used in this work are publicly available, and we cite their original sources throughout the paper\. OLMoE\-1B\-7B\-0125 is released under Apache\-2\.0, Qwen1\.5\-MoE\-A2\.7B under the Tongyi Qianwen License, and Moonlight\-16B\-A3B and Kimi\-Linear\-48B\-A3B under the MIT License\. For the datasets, we use only their publicly released versions for research evaluation and follow the applicable licenses and terms of use specified by their original providers\. Some datasets do not provide unambiguous standalone license information in their original releases; for these resources, we follow the usage conditions provided by the corresponding authors or benchmark distributions\.

## Appendix CSensitivity and Robustness to Hyperparameters

### C\.1Stability Analysis for Fixed Grouping and Warm\-up Duration

##### Local grouping stability\.

To justify using a fixed grouping after warm\-up, we examine the trajectory of expert grouping over training across the model\. Every 15 optimization steps, we compute the expert similarity matrices and apply the same grouping procedure as in the main method, yielding a grouping structure at steptt\.

For each step, we convert the partition into the set of expert pairs assigned to the same group,

𝒮\(t\)=\{\(i,j\)∣1≤i<j≤E,Π\(t\)\(i\)=Π\(t\)\(j\)\}\.\\mathcal\{S\}^\{\(t\)\}=\\\{\(i,j\)\\mid 1\\leq i<j\\leq E,\\ \\Pi^\{\(t\)\}\(i\)=\\Pi^\{\(t\)\}\(j\)\\\}\.We then compare the grouping at stepttwith the grouping from the previous measurement stept−Δt\-\\Deltausing Jaccard similarity:

Jlocal​\(t\)=\|𝒮\(t\)∩𝒮\(t−Δ\)\|\|𝒮\(t\)∪𝒮\(t−Δ\)\|\.J\_\{\\mathrm\{local\}\}\(t\)=\\frac\{\|\\mathcal\{S\}^\{\(t\)\}\\cap\\mathcal\{S\}^\{\(t\-\\Delta\)\}\|\}\{\|\\mathcal\{S\}^\{\(t\)\}\\cup\\mathcal\{S\}^\{\(t\-\\Delta\)\}\|\}\.Here,Δ=15\\Delta=15corresponds to the interval between two consecutive grouping measurements\.

Figure 6:Local stability of expert grouping over training in the all\-layer view\. Every 15 steps, we recompute the grouping and compare it with the grouping from the previous measurement step using Jaccard similarity\. The left and right panels correspond to training on the HellaSwag and WinoGrande datasets, respectively\. In both settings, the local similarity increases quickly and then plateaus, indicating that the grouping structure has stabilized\.Figure[6](https://arxiv.org/html/2609.06072#A3.F6)shows that the grouping changes rapidly early in training and then enters a stable regime, where consecutive grouping estimates become highly similar\. This supports our design choice of using a single grouping after warm\-up, since the grouping structure does not continue to change substantially once it enters the stable regime\. We chooseTinit=150T\_\{\\text\{init\}\}=150as a conservative default: it lies in the stable regime while still leaving most of training for shared\-adapter optimization\.

##### Effect on routing statistics\.

ACE keeps the pretrained router frozen and applies the fixed group assignment only after expert selection\. To examine whether fixed grouping alters routing behavior, we compare aggregate routing statistics between ACE and expert\-wise LoRA throughout training\.

StepLoad Entropy↑\\uparrowAux\. Loss↓\\downarrowLoRA / ACELoRA / ACE5000\.9613 / 0\.96051\.1168 / 1\.121410000\.9614 / 0\.96161\.1161 / 1\.1162Final0\.9619 / 0\.96201\.1142 / 1\.1133Table 10:Routing statistics for LoRA and ACE during training\.Across training, the maximum absolute differences between ACE and LoRA are only 0\.0008 in load entropy and 0\.0046 in auxiliary loss\. These results indicate that fixed group assignments have little effect on the measured aggregate routing statistics in this setting\.

##### CKA consistency with later training\.

To assess whether the similarity structure identified during warm\-up remains representative later in training, we compare intermediate CKA matrices with those at the final checkpoint\. For this post\-hoc diagnostic, we recompute centered linear CKA at each checkpoint using the same fixed 10,000\-token held\-out validation probe, which is distinct from the training examples used for consolidation \(Appendix[K](https://arxiv.org/html/2609.06072#A11)\)\. This allows us to compare checkpoints under a common, unseen input set\. For each of the 16 MoE layers and two projection types \(gate\_projandup\_proj\), we vectorize the strict upper triangle of the64×6464\\times 64CKA matrix, corresponding to 2,016 unique expert pairs, and compute its Pearson and Spearman correlations with the corresponding vector at the final checkpoint \(step 2,805\)\. We report the unweighted mean across the resulting 32 layer–projection combinations\. The Pearson/Spearman correlations increase from 0\.577/0\.587 at step 150 to 0\.771/0\.772 at step 500 and 0\.901/0\.896 at step 1000, indicating that the structure identified at our default consolidation step already shows moderate alignment with the final structure and becomes progressively more consistent during training\.

However, stronger CKA agreement does not necessarily lead to better downstream performance: consolidation at step 500 yields a slightly lower average accuracy than at step 150 \(75\.23 vs\. 75\.70; Table[9](https://arxiv.org/html/2609.06072#A2.T9)\)\. These results suggest thatTinitT\_\{\\mathrm\{init\}\}should balance structural reliability with the remaining optimization time after consolidation, rather than simply maximizing agreement with the final checkpoint\.

##### Sensitivity to warm\-up duration\.

Table[9](https://arxiv.org/html/2609.06072#A2.T9)shows that performance is stable acrossTinitT\_\{\\text\{init\}\}, with a maximum gap of only 0\.47 points between configurations\. This robustness stems from an inherent trade\-off: a largerTinitT\_\{\\text\{init\}\}yields more stable grouping estimates, while a smallerTinitT\_\{\\text\{init\}\}lets the shared adapters benefit from group\-level updates for longer\. These two competing effects offset each other, makingACErelatively insensitive to the exact choice ofTinitT\_\{\\text\{init\}\}\. Notably,ACEoutperforms LoRA \(74\.11\) across allTinitT\_\{\\text\{init\}\}settings, confirming robustness to this hyperparameter\. We adoptTinit=150T\_\{\\text\{init\}\}=150as the default, which yields the best average score in this sweep\.

##### Practical guidance\.

The warm\-up durationTinitT\_\{\\mathrm\{init\}\}controls a trade\-off between grouping reliability and remaining optimization time after consolidation\. Consolidating too early can use unstable adapter representations, whereas consolidating too late leaves limited time for shared adapters to benefit from group\-level updates\. Our sensitivity results show thatACEis robust within a reasonable range, soTinitT\_\{\\mathrm\{init\}\}does not need to be finely tuned\. In practice, we recommend consolidating after the steepest initial loss decrease, when training begins to enter a more gradual descent, but well before the majority of training is complete\.

### C\.2Sensitivity to the Louvain Resolution Parameter

We analyze how the Louvain resolution parameterγ\\gammaaffects performance and efficiency on OLMoE\-1B\-7B\-0125 across the eight commonsense reasoning benchmarks of Section[4\.1](https://arxiv.org/html/2609.06072#S4.SS1)\. A largerγ\\gammaproduces more, smaller groups \(approaching one adapter per expert\), while a smallerγ\\gammaproduces fewer, larger groups \(approaching full consolidation\)\.

MethodGroup CountAvg\.↑\\uparrowEval Time \(s\)↓\\downarrowLoRA–74\.1120\.82ACE\(γ=3\.2\\gamma=3\.2\)24274\.5919\.76ACE\(γ=1\.6\\gamma=1\.6\)21575\.0417\.67ACE\(γ=1\.2\\gamma=1\.2\)14175\.2117\.24ACE\(γ=0\.8\\gamma=0\.8, default\)9775\.6117\.22ACE\(γ=0\.4\\gamma=0\.4\)6775\.7516\.97ACE\(γ=0\.1\\gamma=0\.1\)4675\.8216\.85ACE\(full merge,γ→0\\gamma\\\!\\to\\\!0\)–73\.5716\.41

Table 11:Effect of the Louvain resolution parameterγ\\gammaon OLMoE\. Group Count is the total number of consolidated adapter groups across all MoE layers\. Avg\. is the eight\-benchmark commonsense reasoning average\. Eval time is the wall\-clock duration of one full evaluation pass on a single H200\.Table[11](https://arxiv.org/html/2609.06072#A3.T11)shows three consistent trends\. First,ACE*robustly*outperforms LoRA across all evaluated group counts \(46\-242\), with average accuracy varying by only 1\.23 points, soγ\\gammadoes not require careful tuning\. Second, over\-aggressive consolidation hurts performance: fully merging all experts \(γ→0\\gamma\\\!\\to\\\!0\) drops below LoRA \(73\.57 vs\. 74\.11\), indicating that some degree of expert specialization must be preserved\. Third, evaluation time decreases monotonically asγ\\gammashrinks \(fewer, larger groups\), achieving up to a 19\.1% reduction over LoRA atγ=0\.1\\gamma=0\.1\. We adoptγ=0\.8\\gamma=0\.8as the default because it offers a robust accuracy\-efficiency trade\-off \(\+1\.50 over LoRA,−17\.3%\-17\.3\\%eval time\) and shows the most consistent behavior across models and datasets in our experiments\.

At the defaultγ=0\.8\\gamma=0\.8,ACEforms 97 multi\-expert consolidated groups together with 42 singleton experts that remain unmerged\. Among the consolidated groups, the median group size is 23 experts and the largest contains 43 experts\. This placesACEbetween expert\-wise LoRA and uniform full consolidation\.

### C\.3Necessity of Centering in CKA Calculation

Standard CKA mean\-centers representation matrices over the probe set before computing similarity\([Kornblith et al\., 2019](https://arxiv.org/html/2609.06072#bib.bib10)\)\. Since expert activations in ACE already reside in a shared feature space, centering is not strictly required from a coordinate\-alignment perspective\. We use centered CKA simply for consistency with the standard formulation\. To verify that this choice does not affect our results, we compare centered and uncentered CKA at the grouping step on OLMoE\.

MetricCentered CKAUncentered CKARaw MSE Gap↑\\uparrow\-2\.417e\-6\-3\.911e\-6Louvain Q↑\\uparrow0\.3950\.374Downstream Average Accuracy↑\\uparrow75\.7075\.44

Table 12:Comparison between centered and uncentered CKA on OLMoE commonsense reasoning\. Both variants use the same ACE pipeline and differ only in whether the representation matrices are mean\-centered before computing CKA similarity\. Results are from one random seed\.The two variants yield broadly comparable grouping quality and downstream performance, confirming that the centering choice does not critically affectACE\.

### C\.4Sensitivity to Probe\-Set Size

To examine whether the expert grouping discovered byACEis sensitive to the probe\-set size, we compare group assignments obtained with smaller probe sets against the default 10k\-token reference in the OLMoE commonsense setting\.

Probe TokensJaccardPair\-F1ARI5120\.9230\.9530\.9221,0000\.9460\.9690\.9482,0000\.9580\.9760\.9605,0000\.9890\.9940\.99110,0001\.0001\.0001\.000Table 13:Similarity of group assignments obtained with different probe\-set sizes to the default 10k\-token reference on OLMoE\.Even 512 probe tokens recover a highly similar grouping \(ARI=0\.922=0\.922\), while 5k tokens produce nearly identical assignments to the 10k\-token reference\. These results indicate thatACEis robust to substantial reductions in probe\-set size\.

## Appendix DRank Variation and Optimization Dynamics

\(a\)Average accuracy on GSM8K and MATH across ranks\.\(b\)Final training loss across ranks\.
Figure 7:Rank sensitivity analysis of ACE and LoRA\. ACE maintains stronger average performance across different adapter ranks and shows lower final training loss, indicating robustness to the choice of adaptation capacity\.To examine whether the gains of ACE depend on the adapter rank used in the main experiments, we compare ACE and LoRA acrossr∈\{1,2,4,8,16,32,64\}r\\in\\\{1,2,4,8,16,32,64\\\}\. Figure[7](https://arxiv.org/html/2609.06072#A4.F7)\(a\) reports the average accuracy over GSM8K and MATH, and Figure[7](https://arxiv.org/html/2609.06072#A4.F7)\(b\) reports the final training loss at each rank\.

As shown in Figure[7](https://arxiv.org/html/2609.06072#A4.F7)\(a\), ACE maintains higher average accuracy than LoRA across most rank settings\. Importantly, this trend holds not only at the main experimental setting, but also when the adapter capacity is made smaller or larger\. This indicates that the improvement of ACE is not tied to a particular rank choice, but is consistently observed across different adaptation\-capacity regimes\.

Figure[7](https://arxiv.org/html/2609.06072#A4.F7)\(b\) provides optimization\-side evidence for the same trend\. ACE obtains lower final training loss than LoRA across ranks, suggesting that the accuracy improvement is accompanied by a more favorable training objective rather than appearing only in downstream evaluation\. Taken together, the rank\-wise accuracy and final\-loss results show that ACE remains robust to the choice of adapter rank and continues to provide stable gains when the available adaptation capacity changes\.

## Appendix EAnalysis of Group\-Anchor Strategies

To minimize functional discontinuity during consolidation,ACEinitializes each shared adapter from a similarity\-central anchor within the group while assigning the remaining rank dimensions using standard LoRA initialization\. We further compare three strategies for constructing this group anchor\. Avg\.Δ​W\\Delta Wforms a representative update by averaging the expert\-specific full updates within a group and then refitting the result to the low\-rank LoRA form via SVD\. MaxΔ​W\\Delta Wselects the expert with the largest update magnitude as the anchor\. CKA\-center selects the expert that is most central with respect to the within\-group CKA similarities, thereby matching the similarity\-based initialization principle used in our methodology\.

MethodHellaSwagWinoGrandeOBQAAverageAvg\.Δ​W\\Delta W92\.5267\.1776\.6078\.76MaxΔ​W\\Delta W92\.4668\.2776\.4079\.04CKA\-center92\.2768\.9876\.6079\.28

Table 14:Analysis of group\-anchor strategies for shared\-adapter initialization\.As shown in Table[14](https://arxiv.org/html/2609.06072#A5.T14), the three strategies yield broadly comparable results, but CKA\-center provides the strongest overall performance\. It is also the most efficient choice\. Avg\.Δ​W\\Delta Wintroduces the highest overhead because it requires constructing an averaged full update and performing an additional SVD refitting step\. MaxΔ​W\\Delta Walso incurs extra computation due to the additional update\-magnitude scoring required for anchor selection\. In contrast, CKA\-center directly reuses the similarity statistics already computed during grouping and therefore adds essentially no extra cost\. Since it is both the most effective and the most efficient option, we adopt CKA\-center as the default group\-anchor strategy\.

ModelMethod\# Params \(%\)HellaSwagWinoGrandeARC\-cARC\-eBoolQOBQAPIQAAvg\.OLMoELoRA0\.2787\.5864\.0953\.9267\.8573\.5564\.4070\.1368\.79AdaLoRA0\.54→\\rightarrow0\.2786\.0662\.2755\.6373\.9971\.8362\.6064\.6468\.15DoRA0\.3387\.4163\.0652\.1369\.9975\.9060\.8074\.9769\.18ACE \(Ours\)0\.2789\.0871\.1155\.4675\.2978\.9064\.6078\.5173\.28QwenLoRA0\.2191\.8277\.0373\.9887\.8483\.7078\.6084\.7782\.53AdaLoRA0\.42→\\rightarrow0\.2190\.1965\.9075\.2688\.3073\.2460\.6082\.4876\.57DoRA0\.2691\.6775\.3772\.1087\.5483\.8877\.2084\.1181\.70ACE \(Ours\)0\.2192\.6177\.1974\.4088\.0184\.1679\.8084\.9883\.02

Table 15:Accuracy \(%\) on commonsense reasoning benchmarks\. All methods use the same adapter rank on MoE experts, resulting in nearly identical trainable parameter budgets\.
## Appendix FDetails for Figure[2](https://arxiv.org/html/2609.06072#S3.F2)

Figure[2](https://arxiv.org/html/2609.06072#S3.F2)presents the pairwise CKA similarity matrix of OLMoE\-1B\-7B\-0125, computed using the gate projection outputs from the 0\-th, 4\-th and 14\-th layers\. The similarity is measured for all pairs of experts, resulting in a full pairwise similarity matrix\. The CKA scores used for the visualization are computed from the HellaSwag dataset\. Values closer to 1 indicate higher similarity, while values closer to 0 indicate lower similarity\.

## Appendix GResults with Dataset\-Specific Training

In addition to the main experiments, we evaluate our method in a dataset\-specific training setting, where a separate model is trained for each dataset, to analyze its behavior on individual benchmarks\. We use two MoE backbones, OLMoE\-1B\-7B\-0125 and Qwen1\.5\-MoE\-A2\.7B, and report results on HellaSwag, WinoGrande, ARC\-Challenge, ARC\-Easy, BoolQ, OpenBookQA, and PIQA\.

We compare our approach with LoRA, AdaLoRA, and DoRA\. For a fair comparison, all methods apply adapters to the expert modules and use the same rank configuration within each setting, so that the number of trainable parameters remains nearly identical across methods\. Unless otherwise noted, we use the same training setup and hyperparameters as in Appendix[B\.1\.1](https://arxiv.org/html/2609.06072#A2.SS1.SSS1), except that each model is trained for 1,500 steps\.

As shown in Table[15](https://arxiv.org/html/2609.06072#A5.T15),ACEachieves stronger performance than the competing methods on most benchmarks and consistently outperforms the baselines across several datasets\.

## Appendix HFree\-Form Generation on XSum

To evaluate generation quality beyond accuracy and pass@10, we evaluate PEFT methods on the XSum[Narayan et al\. \(2018\)](https://arxiv.org/html/2609.06072#bib.bib48)summarization benchmark using OLMoE\. All methods are compared under matched trainable\-parameter budgets\.

MethodROUGE\-1↑\\uparrowROUGE\-2↑\\uparrowROUGE\-L↑\\uparrowLoRA32\.9813\.7726\.27DoRA33\.1613\.9026\.45AdaLoRA30\.0111\.8224\.17PiSSA30\.4812\.2624\.72PERFT\-R33\.5714\.3126\.90ACE33\.8214\.3727\.01Table 16:XSum summarization results on OLMoE under matched trainable\-parameter budgets\.ACEachieves the highest score on all three ROUGE metrics, indicating that its gains extend to free\-form text generation\.

## Appendix IAdditional Functional Redundancy Analyses

### I\.1Expert Swap on Up Projection

Figure 8:Effect of adapter swapping based on expert similarity applied on up projection\. Replacing adapters with high similarity experts causes only minor increases in NLL, while dissimilar or random experts lead to substantially larger degradation\.We conduct this analysis on OLMoE\-1B\-7B\-0125 withΔ​NLL\\Delta\\mathrm\{NLL\}evaluated on HellaSwag\. Figure[8](https://arxiv.org/html/2609.06072#A9.F8)reveals a strong dependence on LoRA similarity\. Replacing LoRA with highly similar ones results in only minor increases in Negative Log Likelihood \(NLL\) across all selection ratios\. For instance, whenρ=0\.05\\rho=0\.05, the NLL rises by approximately0\.00340\.0034relative to the baseline\. In contrast, swapping with low\-similarity adapters leads to substantially larger degradation, increasing the NLL by0\.01870\.0187, while random swaps produce a moderate increase of about0\.00510\.0051\.

This trend persists as the replacement ratio increases\. Atρ=0\.20\\rho=0\.20, high\-similarity swaps increase the NLL by only0\.01090\.0109, whereas low\-similarity swaps cause a much larger rise of0\.03960\.0396\. Random swaps again fall between these two extremes, increasing the NLL by0\.02350\.0235\.

Taken together, these results indicate that CKA similarity reliably reflects functional interchangeability between expert adapters\. Experts with highly aligned adapter responses can therefore substitute for one another with minimal performance degradation, supporting the design principle ofACEthat adaptation capacity can be consolidated across redundant experts\.

### I\.2Functional Redundancy across Models and Tasks

Across commonsense reasoning, the CKA gaps are 0\.299, 0\.298, 0\.438, and 0\.344 for OLMoE, Qwen, Moonlight, and Kimi, respectively\. We additionally observe positive gaps on mathematical reasoning \(0\.259/0\.215\), code generation \(0\.251/0\.208\), and legal understanding \(0\.323/0\.286\) for OLMoE/Qwen, respectively\.

To verify that this separation is not specific to the data used for group construction, we construct groups on one data split and evaluate their CKA separation on an independent split\. In the OLMoE commonsense setting, the discovered groups retain a CKA gap of 0\.293 on the held\-out split, whereas random groupings yield a near\-zero gap of−0\.0006±0\.0034\-0\.0006\\pm 0\.0034\.

## Appendix JExtended Backbone Comparison Including Kimi

To examine whetherACE’s gains over expert\-wise LoRA generalize to a larger MoE backbone, we extend the comparison to Kimi\-Linear\-48B\-A3B\. Table[17](https://arxiv.org/html/2609.06072#A10.T17)comparesACEwith LoRA across all four MoE backbones evaluated in this work\. For each backbone we report the eight\-benchmark average and per\-task accuracies, with results for OLMoE, Qwen1\.5\-MoE, and Moonlight from Table[1](https://arxiv.org/html/2609.06072#S4.T1)\.ACEconsistently improves over LoRA on every backbone, including Kimi \(84\.18 vs\. 83\.66\), indicating that the gains of adapter consolidation extend to a larger backbone with substantially more experts\.

ModelMethod\# Params \(%\)↓\\downarrowHS↑\\uparrowWG↑\\uparrowARC\-c↑\\uparrowARC\-e↑\\uparrowBoolQ↑\\uparrowOBQA↑\\uparrowPIQA↑\\uparrowSIQA↑\\uparrowAvg\.↑\\uparrowOLMoE\(1B/7B\)LoRA0\.2790\.8270\.0869\.4285\.6166\.7174\.6083\.1052\.5174\.11ACE0\.2792\.2472\.8570\.8886\.2168\.0477\.3383\.9353\.4175\.61Qwen\(3B/14B\)LoRA0\.2193\.2377\.6679\.4490\.6370\.8482\.6786\.8056\.7279\.75ACE0\.2193\.8179\.1979\.5090\.6671\.4983\.8786\.6756\.9980\.27Moonlight\(3B/16B\)LoRA0\.2292\.2374\.5984\.6794\.2867\.1782\.0086\.9252\.4779\.29ACE0\.2293\.4979\.9584\.3094\.7569\.3184\.9387\.7154\.9381\.17Kimi\(3B/48B\)LoRA0\.2794\.3379\.2491\.0497\.3575\.1487\.4090\.5954\.1583\.66ACE0\.2794\.8281\.1490\.3697\.4376\.3088\.0090\.9754\.4584\.18

Table 17:Commonsense reasoning comparison between LoRA andACEacross four MoE backbones\. Results for OLMoE, Qwen1\.5\-MoE, and Moonlight are from Table[1](https://arxiv.org/html/2609.06072#S4.T1); Kimi\-Linear\-48B\-A3B is added here\.ACEachieves higher average accuracy than LoRA on every backbone, including the largest one\. The best result per backbone\-row pair is in bold\.
## Appendix KDetails for Louvain Method

Group\.HSWGARC\-cARC\-eBoolQOBQAPIQASIQAAvg\.Louvain92\.2472\.8570\.8886\.2168\.0477\.3383\.9353\.4175\.61Thresh\.92\.1771\.7771\.0586\.6667\.9477\.0784\.2153\.1575\.50

Table 18:Comparison between Louvain\-based grouping and fixed\-threshold grouping on OLMoE commonsense reasoning\. “Thresh\.” denotes fixed\-threshold grouping witht=0\.9t=0\.9\. Both variants use the same ACE pipeline and differ only in how CKA\-based expert groups are formed\.### K\.1Grouping Implementation Details

##### Probe set construction\.

We construct the probe setXprobeX\_\{\\mathrm\{probe\}\}exclusively from the fine\-tuning training split and do not use validation or test examples for expert grouping\. After the warm\-up phase, We collectN=10,000N=10\{,\}000non\-padding token activations from randomly ordered packed training sequences\. The same probe set is used for all experts within a layer, so the resulting CKA scores reflect adapter\-induced representations under identical inputs rather than differences in routing frequency\. We keep the probe set fixed across ACE variants and do not tune it on downstream validation performance\.

##### Layer\-wise graph construction\.

We perform expert grouping independently for each MoE layer\. LetS\(ℓ\)∈ℝEℓ×EℓS^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{E\_\{\\ell\}\\times E\_\{\\ell\}\}denote the CKA similarity matrix among theEℓE\_\{\\ell\}experts in layerℓ\\ell\. We first symmetrizeS\(ℓ\)S^\{\(\\ell\)\}and set its diagonal entries to zero to remove self\-similarity\. We then construct a non\-negative weighted adjacency matrixW\(ℓ\)W^\{\(\\ell\)\}as

Wi​j\(ℓ\)=max⁡\(0,Si​j\(ℓ\)−b\(ℓ\)\),W^\{\(\\ell\)\}\_\{ij\}=\\max\\\!\\bigl\(0,\\,S^\{\(\\ell\)\}\_\{ij\}\-b^\{\(\\ell\)\}\\bigr\),whereb\(ℓ\)b^\{\(\\ell\)\}is the median of the off\-diagonal entries ofS\(ℓ\)S^\{\(\\ell\)\}\. This transformation suppresses weak pairwise similarities and retains only expert relations that exceed the typical similarity level within layerℓ\\ell\. The resulting graph is therefore a layer\-specific weighted graph whose nodes correspond to experts and whose edge weights encode CKA similarity above the layer\-specific baseline\.

### K\.2Layer\-wise Grouping with the Louvain Method

Given the preprocessed graphW\(ℓ\)W^\{\(\\ell\)\}, we apply the Louvain method independently to each MoE layer\. We optimize the generalized weighted modularity

Q=12​m∑i,j\(Wi​j−γki​kj2​m\)𝟏\[zi=zj\],Q=\\frac\{1\}\{2m\}\\sum\_\{i,j\}\\left\(W\_\{ij\}\-\\gamma\\frac\{k\_\{i\}k\_\{j\}\}\{2m\}\\right\)\\mathbf\{1\}\[z\_\{i\}=z\_\{j\}\],whereki=∑jWi​jk\_\{i\}=\\sum\_\{j\}W\_\{ij\}is the weighted degree of expertii,2​m=∑i,jWi​j2m=\\sum\_\{i,j\}W\_\{ij\},ziz\_\{i\}denotes the group assignment of expertii, andγ\\gammais the resolution parameter\. In all experiments, we setγ=0\.8\\gamma=0\.8\.

Starting from singleton groups, the Louvain method repeatedly considers moving an expertiito one of its neighboring groups\. After removingiifrom its current group, the modularity gain of assigning it to a candidate group𝒞\\mathcal\{C\}is proportional to

Δ​Q​\(i→𝒞\)∝ki,in​\(𝒞\)−γ​ki​tot​\(𝒞\)2​m,\\Delta Q\(i\\rightarrow\\mathcal\{C\}\)\\propto k\_\{i,\\mathrm\{in\}\}\(\\mathcal\{C\}\)\-\\gamma\\frac\{k\_\{i\}\\,\\mathrm\{tot\}\(\\mathcal\{C\}\)\}\{2m\},whereki,in​\(𝒞\)k\_\{i,\\mathrm\{in\}\}\(\\mathcal\{C\}\)denotes the total edge weight from expertiito group𝒞\\mathcal\{C\}, andtot⁡\(𝒞\)\\mathrm\{tot\}\(\\mathcal\{C\}\)is the sum of weighted degrees of the experts currently assigned to𝒞\\mathcal\{C\}\. We therefore greedily move each expert to the neighboring group that yields the largest positive increase in modularity; if no positive gain is available, the expert remains in its current group\. Once no further positive local move is possible, the resulting groups are collapsed into super\-nodes and the same procedure is repeated on the coarsened graph\. This yields a set of expert groups\{𝒢g\(ℓ\)\}g=1Gℓ\\\{\\mathcal\{G\}^\{\(\\ell\)\}\_\{g\}\\\}\_\{g=1\}^\{G\_\{\\ell\}\}for each layerℓ\\ell\.

### K\.3Comparison with Fixed\-Threshold Grouping

The default ACE implementation forms expert groups by applying Louvain community detection to the CKA similarity graph\. To examine whether ACE depends critically on Louvain grouping, we compare it with a fixed\-threshold variant that uses the same CKA similarity matrix but replaces only the grouping rule\.

For each layerℓ\\elland projection typepp, letS\(ℓ,p\)∈ℝE×ES^\{\(\\ell,p\)\}\\in\\mathbb\{R\}^\{E\\times E\}denote the CKA similarity matrix acrossEEexperts\. Fixed\-threshold grouping constructs an expert graph using a thresholdtt:

ℰt\(ℓ,p\)=\{\(i,j\)∣i≠j,Si​j\(ℓ,p\)≥t\}\.\\mathcal\{E\}^\{\(\\ell,p\)\}\_\{t\}=\\\{\(i,j\)\\mid i\\neq j,\\ S^\{\(\\ell,p\)\}\_\{ij\}\\geq t\\\}\.
The expert groups are then defined as the connected components of𝒢t\(ℓ,p\)=\(𝒱,ℰt\(ℓ,p\)\)\\mathcal\{G\}^\{\(\\ell,p\)\}\_\{t\}=\(\\mathcal\{V\},\\mathcal\{E\}^\{\(\\ell,p\)\}\_\{t\}\), where𝒱\\mathcal\{V\}is the set of experts in layerℓ\\ell\. We sett=0\.9t=0\.9\. We use the same training setup as in Table[1](https://arxiv.org/html/2609.06072#S4.T1)and keep all other components of ACE unchanged, including rank consolidation, shared adapter updates, grouped adapter execution, and optimization hyperparameters\.

Table[18](https://arxiv.org/html/2609.06072#A11.T18)shows that fixed\-threshold grouping achieves performance close to Louvain grouping on OLMoE commonsense reasoning, with an average score of 75\.50 compared to 75\.61 for Louvain\. This indicates that ACE does not depend critically on a specific grouping algorithm, as long as functionally similar experts are consolidated\. However, Louvain slightly outperforms the fixed\-threshold variant and avoids manually selecting a global threshold\. We therefore use Louvain grouping as the default strategy\.

## Appendix LImplementation and Profiling of Grouped Adapter Execution

### L\.1Implementation Details

Section[3\.5](https://arxiv.org/html/2609.06072#S3.SS5)presents grouped execution for a representative adapted linear transformation and omits projection indices for clarity\. In practice, however, LoRA can be attached independently to the up, gate, and down projections of each expert, and the grouped execution argument applies only when the shared\-input property holds\.

For expertee, the feed\-forward network \(FFN\) can be written as

ue=\(Weup\+Δ​Weup\)​x,u\_\{e\}=\(W\_\{e\}^\{\\mathrm\{up\}\}\+\\Delta W\_\{e\}^\{\\mathrm\{up\}\}\)x,ve=\(Wegate\+Δ​Wegate\)​x,v\_\{e\}=\(W\_\{e\}^\{\\mathrm\{gate\}\}\+\\Delta W\_\{e\}^\{\\mathrm\{gate\}\}\)x,he=ϕ⁡\(ve\)⊙ue,h\_\{e\}=\\phi\(v\_\{e\}\)\\odot u\_\{e\},ye=\(Wedown\+Δ​Wedown\)​he,y\_\{e\}=\(W\_\{e\}^\{\\mathrm\{down\}\}\+\\Delta W\_\{e\}^\{\\mathrm\{down\}\}\)h\_\{e\},wherexxis the routed token representation andϕ⁡\(⋅\)\\phi\(\\cdot\)denotes the gating nonlinearity\. This gated\-FFN \(SwiGLU\-style\) form is the architecture used by all MoE backbones in our experiments\. The key distinction is that the up and gate projections both receive the same pre\-expert inputxx, whereas the down projection receivesheh\_\{e\}, which is already expert\-specific because it depends on the outputs of the expert’s own up/gate branches\.

This distinction determines where grouped execution is effective\. After consolidation, the shared adapter corrections for the up and gate projections can be computed once per active group,

cgup​\(x\)=Δ​Wgup,share​x,c\_\{g\}^\{\\mathrm\{up\}\}\(x\)=\\Delta W\_\{g\}^\{\\mathrm\{up,share\}\}x,cggate​\(x\)=Δ​Wggate,share​x,c\_\{g\}^\{\\mathrm\{gate\}\}\(x\)=\\Delta W\_\{g\}^\{\\mathrm\{gate,share\}\}x,using grouped GEMM, and then scattered to all routed experts assigned to that group\. The same reuse is not available for the down projection\. Even when the routed token is the same, the intermediate stateheh\_\{e\}differs across experts, so evaluating a down\-side adapter still requires expert\-specific inputs\. Recovering the same acceleration for the down projection would require additional adapter merging or weight materialization, which introduces nontrivial overhead\. For this reason, our accelerated grouped adapter execution is implemented only for the up and gate projections\.

### L\.2Fine\-Grained Runtime Profiling

To identify the source of the runtime improvement from grouped adapter execution, we profile expert\-wise LoRA andACEon OLMoE under matched input batches, routing, trainable\-parameter budgets, and optimizer settings\. Results are mean±\\pmstandard deviation over five runs, each with 20 warm\-up and 100 measured steps\.

ComponentLoRA \(ms\)ACE\(ms\)ReductionTotal step3010\.383010\.38±33\.86\\mkern 1\.0mu\{\\scriptstyle\\pm 33\.86\}1688\.131688\.13±9\.31\\mkern 1\.0mu\{\\scriptstyle\\pm 9\.31\}43\.92%Forward1351\.951351\.95±17\.20\\mkern 1\.0mu\{\\scriptstyle\\pm 17\.20\}845\.43845\.43±6\.49\\mkern 1\.0mu\{\\scriptstyle\\pm 6\.49\}37\.47%Adapter forward517\.48517\.48±3\.14\\mkern 1\.0mu\{\\scriptstyle\\pm 3\.14\}56\.0956\.09±0\.28\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.28\}89\.16%Backward1619\.531619\.53±21\.18\\mkern 1\.0mu\{\\scriptstyle\\pm 21\.18\}809\.91809\.91±3\.30\\mkern 1\.0mu\{\\scriptstyle\\pm 3\.30\}49\.99%Optimizer24\.3524\.35±0\.66\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.66\}21\.8821\.88±0\.25\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.25\}–Router linear0\.3300\.330±0\.001\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.001\}0\.3300\.330±0\.001\\mkern 1\.0mu\{\\scriptstyle\\pm 0\.001\}–

Table 19:Fine\-grained runtime profiling of expert\-wise LoRA andACEwith grouped adapter execution on OLMoE\. Adapter forward is a subrange of the forward pass\.Grouped execution reduces post\-consolidation step time by 43\.92%, with the largest reduction occurring in adapter forward execution \(89\.16%\)\. The adapter\-forward reduction accounts for approximately 91% of the measured forward\-time reduction, confirming that the efficiency gain is concentrated in the fragmented adapter computation targeted byACE\.

These synchronized profiling measurements are intended for component\-level attribution and are therefore not directly comparable to the end\-to\-end wall\-clock measurements in Table[2](https://arxiv.org/html/2609.06072#S4.T2), which additionally include data\-pipeline and framework overhead\.

## Appendix MGradient Supervision Analysis Details

We provide additional details for the gradient\-supervision fragmentation analysis in Section[5\.1](https://arxiv.org/html/2609.06072#S5.SS1)\. We conduct this analysis under the commonsense fine\-tuning setting, using Commonsense\-170K as the training data\. For each backbone and method, we measure LoRA\-adapter gradients over 50 matched training steps, using the post\-consolidation phase forACE\. We compute the statistics separately for each MoE layer and then average across layers\.

For each expert adapter, we first compute the gradient norm of its LoRA parameters\. We then measure two quantities\. The first is the coefficient of variation \(CV\) of per\-expert gradient norms within the same MoE layer,

CV=Stde\(∥∇ΔWe∥\)Meane\(∥∇ΔWe∥\),\\mathrm\{CV\}=\\frac\{\\mathrm\{Std\}\_\{e\}\\left\(\\\|\\nabla\\Delta W\_\{e\}\\\|\\right\)\}\{\\mathrm\{Mean\}\_\{e\}\\left\(\\\|\\nabla\\Delta W\_\{e\}\\\|\\right\)\},which captures how unevenly gradient signal is distributed across experts\. The second is the zero\-gradient ratio, defined as the fraction of LoRA adapter rows whose gradients are exactly zero,

ZeroGrad=\#​\{adapter rows with zero gradient\}\#​\{adapter rows\}\.\\mathrm\{ZeroGrad\}=\\frac\{\\\#\\\{\\text\{adapter rows with zero gradient\}\\\}\}\{\\\#\\\{\\text\{adapter rows\}\\\}\}\.A lower CV indicates more balanced supervision across experts, and a lower zero\-gradient ratio indicates that fewer adapter parameters are left unsupervised at each step\.

Figure[9](https://arxiv.org/html/2609.06072#A13.F9)reports the same analysis for gate\-projection adapters\. Consistent with the up\-projection results in Figure[4](https://arxiv.org/html/2609.06072#S5.F4),ACEreduces both gradient imbalance and zero\-gradient ratios across MoE backbones\. This supports the conclusion that shared\-group updates densify supervision not only for up projections but also for gate projections\.

Figure 9:Gradient\-supervision fragmentation for gate\-projection LoRA adapters across three MoE backbones\.ACEreduces both gradient\-norm imbalance and the zero\-gradient ratio relative to expert\-wise LoRA\.
## Appendix NAdditional Training Efficiency Results

Table[20](https://arxiv.org/html/2609.06072#A14.T20)reports method\-level training time and peak memory across MoE backbones\. All methods are evaluated under the same effective batch size within each backbone\. When a method requires a smaller per\-device batch size due to memory constraints, gradient accumulation is adjusted to keep the effective batch size fixed\. We report wall\-clock training time over 3 epochs and peak GPU memory\. Means and standard deviations are computed over available seeds\(0, 1, 2\)\.

ModelMethodTrain Time \(hr\)Peak Memory \(GB\)MoonlightLoRA23\.75±\\pm0\.6048\.80±\\pm0\.00AdaLoRA43\.84±\\pm2\.0550\.07±\\pm0\.62DoRA41\.10±\\pm0\.9580\.88±\\pm0\.02PiSSA23\.64±\\pm0\.3348\.80±\\pm0\.00ACE\(ours\)16\.77±\\pm0\.2348\.80±\\pm0\.00QwenLoRA6\.18±\\pm0\.0589\.90±\\pm0\.01AdaLoRA10\.90±\\pm0\.3490\.58±\\pm0\.00DoRA18\.56±\\pm0\.0696\.57±\\pm0\.01PiSSA6\.25±\\pm0\.0989\.84±\\pm0\.01ACE\(ours\)4\.19±\\pm0\.0289\.84±\\pm0\.01OLMoELoRA2\.47±\\pm0\.0398\.21±\\pm0\.01AdaLoRA3\.90±\\pm0\.0598\.54±\\pm0\.00DoRA7\.36±\\pm0\.4585\.10±\\pm0\.00PiSSA2\.40±\\pm0\.0398\.22±\\pm0\.00ACE\(ours\)1\.89±\\pm0\.0298\.18±\\pm0\.02Kimi\-LinearLoRA87\.39±\\pm0\.80118\.22±\\pm0\.00ACE\(ours\)60\.41±\\pm0\.38118\.22±\\pm0\.00

Table 20:Additional training efficiency results across PEFT methods and MoE backbones\. Training time is reported as wall\-clock hours over 3 epochs, and peak memory is reported in GB\. Means and standard deviations are computed over three seeds \(0, 1, and 2\)\. All methods use the same effective batch size within each backbone; methods with higher memory requirements may use smaller per\-device batches with additional gradient accumulation\. For Kimi\-Linear, we report the available LoRA comparison\.ModelEELayersTotal ExpertsWarm\-up Overhead \(s\)CKA \(s\)Louvain \(s\)Consolidation \(s\)Total Overhead \(% of ACE\)ACE Speedup over LoRAOLMoE\-1B\-7B64161,0240\.00123\.900\.120\.370\.035 hr \(1\.83%\)1\.31×\\timesQwen1\.5\-MoE\-A2\.7B60241,4400\.00275\.170\.150\.940\.077 hr \(1\.83%\)1\.48×\\timesMoonlight\-16B\-A3B64271,7280\.00328\.890\.190\.560\.092 hr \(0\.55%\)1\.42×\\timesKimi\-Linear\-48B\-A3B256276,9120\.002393\.110\.793\.190\.666 hr \(1\.10%\)1\.45×\\times

Table 21:Per\-stage breakdown of theACEconsolidation pipeline\. The three stages \(pairwise CKA computation, Louvain community detection, and adapter consolidation\) are reported in seconds\. Total overhead is the sum of the three stages relative to total ACE training time\. The warm\-up phase itself adds no extra cost, as it is identical to standard LoRA training\. All models are trained for 3 epochs, following the configuration of Table[1](https://arxiv.org/html/2609.06072#S4.T1)\. Shared experts, which are not included in the ACE grouping procedure, are excluded from the expert counts\.
## Appendix OOverhead Analysis of the Consolidation Pipeline

The consolidation pipeline ofACEintroduces a one\-time preprocessing cost that consists of three stages: \(i\) CKA computation across all expert pairs within each MoE layer, \(ii\) Louvain community detection on the resulting similarity graph, and \(iii\) adapter consolidation \(CKA\-center anchor selection, shared high\-rank adapter initialization, and optimizer\-state reallocation\)\. Note that the warm\-up phase itself contributes*no*additional cost: duringTinitT\_\{\\text\{init\}\}steps,ACEsimply trains expert\-wise LoRA adapters identically to the LoRA baseline, so the warm\-up time is already accounted for in the total training time and is not counted as overhead\. The dominant scaling term is the pairwise CKA computation, which has𝒪⁡\(E2\)\\mathcal\{O\}\(E^\{2\}\)complexity per layer, whereEEis the number of experts\. We report a per\-stage breakdown across all four MoE backbones to characterize how this overhead grows withEEand how much it costs relative to total training time\.

Table[21](https://arxiv.org/html/2609.06072#A14.T21)shows three observations\. First, the consolidation pipeline accounts for at most 1\.83% of total ACE training time across all evaluated backbones, despite the𝒪⁡\(E2\)\\mathcal\{O\}\(E^\{2\}\)scaling of pairwise CKA\. Second, CKA computation dominates the overhead, consistently accounting for the overwhelming majority of the one\-time cost across all backbones, while Louvain and adapter consolidation remain almost flat and contribute a negligible fraction even at the largest scale\. The cost of CKA computation is controllable via the size of the probe set used to compute the similarity matrices\. A smaller probe set proportionally reduces CKA runtime, reducing overhead on larger MoE backbones\. Third, the overhead is more than offset by per\-step efficiency gains from grouped adapter execution:ACEachieves at least a1\.31×1\.31\\timesend\-to\-end speedup over LoRA on every backbone\.

## Appendix POptimization Dynamics

Figure 10:Training loss curves on OLMoE, Qwen, Moonlight with Commonsense\.Figure 11:Training loss curves on OLMoE, Qwen with MetaMathQA\.Figure 12:Training loss curves on OLMoE, Qwen with CodeFeedback\.Figure 13:Final training loss of LoRA andACEacross datasets and MoE backbones\. Lower values indicate better convergence\.As shown in Figure[10](https://arxiv.org/html/2609.06072#A16.F10),[11](https://arxiv.org/html/2609.06072#A16.F11), and[12](https://arxiv.org/html/2609.06072#A16.F12), after the merge step,ACEdemonstrates a steeper decrease in training loss compared to LoRA, suggesting that consolidating functionally similar adapters leads to more effective optimization dynamics\. This improvement disappears immediately after merging due to initialization overhead\. After stabilization, the improvement appears and persists throughout training\. Consistently, Figure[13](https://arxiv.org/html/2609.06072#A16.F13)shows thatACEreaches lower final training loss than LoRA across all evaluated dataset–backbone pairs, indicating that the post\-merge optimization benefit carries through to convergence\. For analysis, we report training loss curves after applying one\-dimensional Gaussian smoothing to the plotted loss values\. The Gaussian smoothing scale was set separately for each dataset:σ\\sigma= 1\.5 for MetaMathQA,σ\\sigma= 2\.0 for CodeFeedback, andσ\\sigma= 1\.0 for Commonsense Reasoning\.

## Appendix QLLM Usage

We used an LLM to refine the writing, improve grammatical accuracy, and assist with code organization and cleanup\.