从专家到子专家：MoE LLMs的细粒度参数高效微调

# From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
Source: [https://arxiv.org/html/2609.25655](https://arxiv.org/html/2609.25655)
Chang Liu11footnotemark:1Affiliation:Alibaba GroupAffiliation:Southeast UniversityYao LiuAffiliation:Alibaba GroupYue WuAffiliation:Alibaba GroupJieping YeAffiliation:Alibaba Group

###### Abstract

As large language models \(LLMs\) scale rapidly, dense full\-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectures such as Mixture\-of\-Experts \(MoE\) models\. This shift raises a key question for parameter\-efficient fine\-tuning \(PEFT\): at what granularity should parameters be selected and updated? Existing PEFT methods such as LoRA operate on predefined weight matrices, while expert\-level sparse tuning methods update entire selected experts\. However, we observe that activated experts are internally sparse, with only a small fraction of intermediate channels strongly responding to downstream tasks, indicating that expert\-level adaptation is still too coarse\. We proposeNSFT\(*Neural Sub\-expert Fine\-Tuning*\), a fine\-grained PEFT framework that refines MoE adaptation from experts to sub\-experts\. NSFT decomposes each expert along the intermediate dimension into structured channel groups and selects task\-relevant sub\-experts by combining routing importance with intra\-expert activation saliency\. To optimize sparse partial updates, NSFT further introduces learning\-rate scaling and dynamic gradient scaling to compensate for the reduced effective update magnitude\. Experiments on OLMoE and Ling\-mini\-2\.0 across challenging domain\-specific tasks and general benchmarks show that NSFT consistently outperforms representative PEFT and expert\-level sparse tuning baselines, while using substantially fewer trainable parameters and preserving competitive general capability\. These results suggest that sub\-expert\-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs\. Our code is available at[https://github\.com/aheadformore/NSFT](https://github.com/aheadformore/NSFT)\.

## 1Introduction

As large language models \(LLMs\) scale into the hundreds of billions—and increasingly, trillions—of parameters, the conventional paradigm of dense, full\-parameter training faces mounting computational and memory bottlenecks\([Bai et al\., 2023](https://arxiv.org/html/2609.25655#bib.bib6);[Grattafiori et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib9)\)\. This pressure has catalyzed a fundamental shift from dense parameterization toward sparse, modular architectures\([Jiang et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib16);[Yang et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib3);[Liu et al\., 2024a](https://arxiv.org/html/2609.25655#bib.bib4);[Team et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib10)\)\. Along with this architectural transition, parameter\-efficient adaptation raises a crucial question:*at what granularity should model parameters be selected and updated for effective downstream learning?*The choice of adaptation granularity not only affects parameter efficiency and training cost, but also relates to broader questions about modularity in Transformers\([Geva et al\., 2021](https://arxiv.org/html/2609.25655#bib.bib7)\), sparsity in knowledge representation\([Hu et al\., 2022](https://arxiv.org/html/2609.25655#bib.bib14);[Liu et al\., 2024b](https://arxiv.org/html/2609.25655#bib.bib15)\), and the intrinsic redundancy of model parameters\([Frankle and Carbin, 2018](https://arxiv.org/html/2609.25655#bib.bib8);[Cheng et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib18)\)\.

Historically, adaptation granularity has largely followed architectural topology\. In dense LLMs, parameter\-efficient fine\-tuning \(PEFT\) methods such as LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.25655#bib.bib14)\)typically operate on linear weight matrices, approximating task\-specific updates with low\-rank components\. However, such matrix\-level adaptation may spread gradients across broadly coupled neurons, making it difficult to precisely target the functional components most relevant to downstream tasks\. Mixture\-of\-Experts \(MoE\) architectures introduce routed computation paths\([Fedus et al\., 2022](https://arxiv.org/html/2609.25655#bib.bib13)\), enabling a coarser form of sparsity at the expert level\. Recent methods such as ESFT\([Wang et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib2)\)therefore update only selected experts for modular adaptation\. Nevertheless, expert\-level selection remains too coarse\. As shown in Figure[1](https://arxiv.org/html/2609.25655#S1.F1)\(left\), channel activations inside activated experts are highly concentrated near zero, indicating that only a small fraction of internal channels are strongly involved in the target task \(refer to the Appendix[E](https://arxiv.org/html/2609.25655#A5)for more details\)\. Updating the whole expert therefore introduces substantial redundancy and limits the parameter efficiency of expert\-level tuning\.

To address this limitation, we proposeNSFT\(*Neural Sub\-expert Fine\-Tuning*\), a fine\-grained PEFT method that moves sparse adaptation below the expert level\. As illustrated in Figure[2](https://arxiv.org/html/2609.25655#S1.F2), NSFT decomposes each expert along its intermediate dimension into structured sub\-experts and selects the most task\-relevant ones for fine\-tuning\. Inspired by MoNE\([Cheng et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib18)\), our method treats sub\-experts as compact functional units rather than updating entire experts\. By focusing updates on responsive internal channel groups, NSFT constructs sparse and precise adaptation pathways while avoiding redundant updates to weakly activated expert parameters\.

![Refer to caption](https://arxiv.org/html/2609.25655v1/figures/intro.png)Figure 1:Left: channel\-level gate\-score distributions of shallow and deep OLMoE experts on MMedC, showing strong intra\-expert sparsity\. The dashed line marks the top\-50% cumulative activation threshold\. Right: PEFT performance on MMedC under different trainable\-parameter budgets, where NSFT achieves a better accuracy–parameter trade\-off than ESFT and LoRA variants\.However, fine\-grained sub\-expert tuning also introduces a new optimization challenge\. Since only a small subset of channels inside each expert is updated, the effective update magnitude becomes much smaller than that of full\-expert or expert\-level tuning\. If the original optimization settings are used directly, the selected sub\-experts may be under\-updated, limiting adaptation effectiveness\. To address this issue, we further introduce an adaptive training strategy with two complementary components: learning\-rate scaling, which compensates for the reduced global update step under sparse partial updates, and gradient scaling, which redistributes update strength among selected sub\-experts according to their relative importance\.

We evaluate NSFT on two MoE backbones,OLMoE\([Muennighoff et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib24)\)andLing\-mini\-2\.0\([inclusionAI, 2025](https://arxiv.org/html/2609.25655#bib.bib37)\), across a wide range of challenging domain\-specific adaptation tasks, including multilingual medicine\([Qiu et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib19)\), scientific information extraction\([Wadden et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib20)\), domain\-specific RAG\([Liu et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib21)\), mathematical reasoning\([Tian et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib23)\), code generation\([Olmo et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib22)\), and table question answering\([Wu et al\., 2025a](https://arxiv.org/html/2609.25655#bib.bib32)\)\. These tasks cover diverse cognitive demands, such as factual recall, logical reasoning, instruction following, and structured output synthesis, providing a comprehensive testbed for evaluating parameter\-efficient adaptation\.

Our results show that conventional PEFT and sparse tuning methods still face clear efficiency–performance limitations\. As shown in Figure[1](https://arxiv.org/html/2609.25655#S1.F1)\(right\), NSFT achieves a better accuracy–parameter trade\-off on MMedC than ESFT and LoRA variants, reaching higher accuracy with substantially fewer trainable parameters\. In particular, NSFT saves up to66\.9%trainable parameters compared with LoRA\-QKVO\+MLP under comparable or stronger performance, and requires fewer parameters than ESFT to reach the same accuracy level\. Across OLMoE and Ling\-mini, NSFT consistently improves domain adaptation over ESFT and LoRA while preserving competitive general capability, demonstrating that fine\-grained sub\-expert selection provides a more effective use of the trainable parameter budget\.

Our contributions are threefold:

- •Conceptual: We revisit the adaptation granularity of MoE models and show that expert\-level sparse tuning remains too coarse for domain\-specific fine\-tuning\. By shifting the update unit from entire experts to sub\-experts, we provide a finer\-grained perspective on how task\-relevant knowledge is localized and adapted within MoE architectures\.
- •Methodological: We design and implement the Sub\-experts framework, enabling fine\-grained, structured sparsity through principled decomposition of expert modules, thereby advancing the frontier of efficient adaptation in MoE models\.
- •Empirical: We demonstrate across diverse real\-world domains and two MoE backbones that NSFT consistently outperforms representative PEFT and expert\-level sparse tuning baselines, while requiring substantially fewer trainable parameters\. These results highlight the effectiveness of fine\-grained sub\-expert adaptation as a more parameter\-efficient and performance\-competitive approach for customizing large\-scale MoE LLMs\.

![Refer to caption](https://arxiv.org/html/2609.25655v1/figures/overview.png)Figure 2:Illustration of sub\-expert selection\. Routed experts are first identified, and channel\-wise activations are computed within each expert\. With group sizeG=4G=4, consecutive channels are partitioned into sub\-expert groups, whereE​1E1\-g​1g1denotes the first group of Expert 1\. All groups are then scored and ranked, and the groups selected under the thresholdτ\\tauare used as trainable sub\-experts for fine\-tuning\.
## 2Related Works

Adaptation Granularity in Dense LLMs\.The pursuit of efficient LLM adaptation has shifted attention from full\-parameter updates to the choice of trainable granularity within model architectures\. In dense models, existing methods mainly define update units from three perspectives\. First,externally added modules, such as Adapters\([Houlsby et al\., 2019](https://arxiv.org/html/2609.25655#bib.bib25);[Wang et al\., 2022](https://arxiv.org/html/2609.25655#bib.bib26)\)and Soft Prompting\([Li and Liang, 2021](https://arxiv.org/html/2609.25655#bib.bib27);[Zhang et al\., 2023](https://arxiv.org/html/2609.25655#bib.bib28)\), introduce learnable auxiliary components while keeping the backbone frozen\. Second, reparameterization\-based methods operate onindividual linear layers; LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.25655#bib.bib14)\), DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2609.25655#bib.bib15)\), and their successors\([Wu et al\., 2025b](https://arxiv.org/html/2609.25655#bib.bib17);[Che et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib29)\)approximate weight updates with low\-rank decompositions\. Third, selective fine\-tuning methods directly update subsets of existing parameters\([Xu et al\., 2021](https://arxiv.org/html/2609.25655#bib.bib30);[Hui et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib11);[He et al\., 2023](https://arxiv.org/html/2609.25655#bib.bib31)\)\. Despite their differences, most dense\-model PEFT methods remain closely tied to linear transformations, treating Transformers more as collections of tunable operators than as modular functional units\.

Expert\-Level Adaptation in Sparse Architectures\.MoE architectures introduce structural sparsity through routing and expert modules, naturally enabling coarser\-grained adaptation at the expert level\([Liu et al\., 2024a](https://arxiv.org/html/2609.25655#bib.bib4);[Team et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib10);[Dai et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib5)\)\. In this setting, theexpertbecomes an intuitive update unit\. Expert\-Specialized Fine\-Tuning \(ESFT\)\([Wang et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib2)\)updates only experts that strongly respond to target data\. DES\-MoE\([Li et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib1)\)further adjusts expert selection through a multi\-stage process for multi\-domain adaptation, while DR\-LoRA\([Deng et al\., 2026](https://arxiv.org/html/2609.25655#bib.bib12)\)uses non\-uniform expert activation to assign different LoRA ranks within experts\. These studies show that expert\-level adaptation can improve efficiency for sparse MoE models, but they still treat each selected expert as an indivisible unit\.

From Full Experts to Sub\-Expert Units\.Recent evidence suggests that experts themselves remain internally redundant\. Mixture of Neuron Experts \(MoNE\)\([Cheng et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib18)\)shows that experts can be decomposed into neuron\-level grains, where activating only high\-response neurons can preserve inference performance\. However, directly tuning at the neuron level can be overly fragmented, leading to unstable optimization and weak update capacity\. We therefore introduce*sub\-experts*as fine\-grained but structured update units\. By grouping neurons into cohesive sub\-expert units, our method retains the flexibility of neuron\-level selection while avoiding the instability of isolated\-neuron updates, providing a more robust paradigm for parameter\-efficient adaptation of MoE LLMs\.

## 3Method

### 3\.1Preliminaries

##### Mixture\-of\-Experts Architecture\.

We first revisit the standard MoE architecture\. Letx∈ℝdmodelx\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}denote the input hidden state to layerll\. An MoE layer containsNEN\_\{E\}experts\{El,e\}e=1NE\\\{E\_\{l,e\}\\\}\_\{e=1\}^\{N\_\{E\}\}and a router that assigns routing weights to experts\. For a GLU\-style expert, we write

El,e​\(x\)=𝐖downl,e​\(SiLU⁡\(gl,e​\(x\)\)⊙𝐖upl,e​x\),E\_\{l,e\}\(x\)=\\mathbf\{W\}\_\{\\mathrm\{down\}\}^\{l,e\}\\left\(\\mathrm\{SiLU\}\\\!\\left\(g\_\{l,e\}\(x\)\\right\)\\odot\\mathbf\{W\}\_\{\\mathrm\{up\}\}^\{l,e\}x\\right\),\(1\)wheregl,e​\(x\)=𝐖gatel,e​xg\_\{l,e\}\(x\)=\\mathbf\{W\}\_\{\\mathrm\{gate\}\}^\{l,e\}x\. Here,𝐖gatel,e,𝐖upl,e∈ℝdexpert×dmodel\\mathbf\{W\}\_\{\\mathrm\{gate\}\}^\{l,e\},\\mathbf\{W\}\_\{\\mathrm\{up\}\}^\{l,e\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{expert\}\}\\times d\_\{\\mathrm\{model\}\}\}and𝐖downl,e∈ℝdmodel×dexpert\\mathbf\{W\}\_\{\\mathrm\{down\}\}^\{l,e\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\\times d\_\{\\mathrm\{expert\}\}\}\.

The router produces expert weights𝐩l​\(x\)∈ℝNE\\mathbf\{p\}\_\{l\}\(x\)\\in\\mathbb\{R\}^\{N\_\{E\}\}and selects the top\-KKexperts\. Letℰl​\(x\)\\mathcal\{E\}\_\{l\}\(x\)denote the selected expert set\. The MoE output is then

MoEl​\(x\)=∑e∈ℰl​\(x\)pl,e​\(x\)​El,e​\(x\)\.\\mathrm\{MoE\}\_\{l\}\(x\)=\\sum\_\{e\\in\\mathcal\{E\}\_\{l\}\(x\)\}p\_\{l,e\}\(x\)E\_\{l,e\}\(x\)\.\(2\)

##### Neuron Expert Decomposition\.

Recent studies such as MoNE\([Cheng et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib18)\)show that each MoE expert can be decomposed into finer\-grained*neuron experts*\. For a GLU\-style expert, the output of experteein layerllcan be expressed asEl,e​\(x\)=∑m=1dexpertal,e,m​\(x\)​𝐀l,e,m​xE\_\{l,e\}\(x\)=\\sum\_\{m=1\}^\{d\_\{\\mathrm\{expert\}\}\}a\_\{l,e,m\}\(x\)\\mathbf\{A\}\_\{l,e,m\}x, whereal,e,m​\(x\)=SiLU​\(gl,e​\(x\)\)ma\_\{l,e,m\}\(x\)=\\mathrm\{SiLU\}\\\!\\left\(g\_\{l,e\}\(x\)\\right\)\_\{m\}is the activation of themm\-th intermediate channel, and𝐀l,e,m=𝐖downl,e\[:,m\]𝐖upl,e\[m,:\]\\mathbf\{A\}\_\{l,e,m\}=\\mathbf\{W\}\_\{\\mathrm\{down\}\}^\{l,e\}\[:,m\]\\mathbf\{W\}\_\{\\mathrm\{up\}\}^\{l,e\}\[m,:\]is the rank\-one transformation induced by this channel\. This formulation reveals that an expert is a superposition of neuron\-wise rank\-one transformations, motivating us to select and update task\-relevant neuron groups as sub\-experts rather than treating the entire expert as an indivisible unit\.

### 3\.2Fine\-grained Sub\-expert Selection

Existing expert\-level sparse adaptation methods, such as ESFT, improve parameter efficiency by updating only a small subset of experts during supervised fine\-tuning\. However, they still treat each selected expert as an indivisible unit\. In practice, the parameter space inside an activated expert remains highly sparse: only a small portion of its intermediate channels are consistently responsive to the downstream task, while many others contribute marginally\. As a result, even after expert selection, updating the full expert may still introduce substantial redundancy\.

This motivates us to refine the granularity of sparse adaptation from the*expert level*to the*sub\-expert level*\. Instead of treating each expert as a monolithic MLP block, we decompose it along the intermediate dimension into multiple finer\-grained sub\-experts\. Such a decomposition is natural for MoE feed\-forward blocks, where each expert is implemented as a gated MLP and its intermediate channels already exhibit highly non\-uniform activation patterns\. As shown in Figure[2](https://arxiv.org/html/2609.25655#S1.F2), we organize consecutive intermediate channels into fixed\-size groups, where each group containsGGchannels and serves as a candidate sub\-expert\. This grouping provides a structured adaptation unit that is finer than a full expert but less fragmented than individual neurons\. Compared with expert\-level selection, this finer\-grained formulation further removes weakly activated channels inside selected experts and yields a more faithful approximation to the effective task\-specific computation path\.

Formally, for layerlland expertee, let𝒯l,e\\mathcal\{T\}\_\{l,e\}denote the set of tokens routed to this expert, and letm∈\{1,…,dexpert\}m\\in\\\{1,\\dots,d\_\{\\mathrm\{expert\}\}\\\}index the intermediate channels of the expert\. We first accumulate the channel\-wise gate responses over all routed tokens:

Ml,e,m=∑x∈𝒯l,e\|SiLU​\(gl,e​\(x\)\)m\|\.M\_\{l,e,m\}=\\sum\_\{x\\in\\mathcal\{T\}\_\{l,e\}\}\\left\|\\mathrm\{SiLU\}\\\!\\left\(g\_\{l,e\}\(x\)\\right\)\_\{m\}\\right\|\.\(3\)To characterize the relative importance distribution inside each expert, we normalize the accumulated responses asM^l,e,m=Ml,e,m∑m′Ml,e,m′\\hat\{M\}\_\{l,e,m\}=\\frac\{M\_\{l,e,m\}\}\{\\sum\_\{m^\{\\prime\}\}M\_\{l,e,m^\{\\prime\}\}\}\. In addition, letNl,eN\_\{l,e\}denote the number of times experteeis selected by the router in layerll\. We define the expert\-level routing importance asRl,e=Nl,e∑e′Nl,e′R\_\{l,e\}=\\frac\{N\_\{l,e\}\}\{\\sum\_\{e^\{\\prime\}\}N\_\{l,e^\{\\prime\}\}\}\.

Based on both expert\-level activity and intra\-expert channel saliency, we define the channel\-level sub\-expert importance score as:

Sl,e,m=Rl,e​M^l,e,m\.S\_\{l,e,m\}=R\_\{l,e\}\\hat\{M\}\_\{l,e,m\}\.\(4\)
To obtain structured sub\-experts, we further aggregate channel scores within each group\. Let𝒢l,e,j\\mathcal\{G\}\_\{l,e,j\}denote thejj\-th group ofGGconsecutive channels in experteeof layerll\. Its group\-level importance score is computed asSl,e,jG=∑m∈𝒢l,e,jSl,e,mS\_\{l,e,j\}^\{G\}=\\sum\_\{m\\in\\mathcal\{G\}\_\{l,e,j\}\}S\_\{l,e,m\}\. WhenG=1G=1, the selection reduces to individual channel selection; largerGGvalues produce more structured but coarser sub\-expert units\.

This score jointly captures whether an expert is globally important for the current task and whether a specific sub\-expert group is locally important within that expert\. After obtainingSl,e,jGS\_\{l,e,j\}^\{G\}, we perform layer\-wise selection by flattening all sub\-expert groups in the same layer and ranking them in descending order\. For layerll, we select the smallest subset whose cumulative score exceeds a thresholdτ\\tau:∑i=1Kl\(SlG\)\(i\)≥τ\\sum\_\{i=1\}^\{K\_\{l\}\}\\left\(S\_\{l\}^\{G\}\\right\)^\{\(i\)\}\\geq\\tau, where\{\(SlG\)\(i\)\}\\\{\(S\_\{l\}^\{G\}\)^\{\(i\)\}\\\}denotes the sorted group\-level sub\-expert scores in layerll\. The selected subset is then used as the sub\-experts for downstream tuning\. In this way, our method preserves the most task\-relevant internal channel groups of activated experts while filtering out low\-contribution parameters\.

### 3\.3Adaptive Training for Sub\-expert Fine\-tuning

After identifying the selected sub\-experts via Section[3\.2](https://arxiv.org/html/2609.25655#S3.SS2), fine\-tuning only a small subset of channels within each expert creates two optimization issues\. For experteein layerll, let𝒜l,e\\mathcal\{A\}\_\{l,e\}denote the set of selected intermediate channels induced by the selected sub\-expert groups, and letKl,e=\|𝒜l,e\|K\_\{l,e\}=\|\\mathcal\{A\}\_\{l,e\}\|be the number of selected channels\. First, since onlyKl,eK\_\{l,e\}out ofdexpertd\_\{\\mathrm\{expert\}\}channels are updated, the effective update magnitude becomes much smaller than in full\-expert tuning, leading to*step\-size underestimation*\. Second, even among the selected channels, their task relevance is heterogeneous, while standard optimization would treat them uniformly\. We address these two issues with learning\-rate scaling and gradient scaling, respectively\.

##### Learning\-rate scaling\.

We first compensate for the loss of global update magnitude caused by partial updates\. We scale the base learning rate according to the inverse active ratio:

ηl,e=ηbase×min⁡\(dexpertKl,e,αmax\),\\eta\_\{l,e\}=\\eta\_\{\\mathrm\{base\}\}\\times\\min\\\!\\left\(\\frac\{d\_\{\\mathrm\{expert\}\}\}\{K\_\{l,e\}\},\\ \\alpha\_\{\\max\}\\right\),\(5\)whereαmax\\alpha\_\{\\max\}is a clipping threshold for stability\. Intuitively, when only a fractionKl,e/dexpertK\_\{l,e\}/d\_\{\\mathrm\{expert\}\}of channels is trainable, each selected channel should receive a proportionally larger update so that the overall optimization progress remains comparable to full\-expert tuning\.

##### Gradient scaling\.

Learning\-rate scaling restores the*global*update magnitude, but it does not distinguish the relative importance of different selected channels\. We therefore further modulate the gradient inside each selected expert by applying a channel\-group\-level scaling mask during the backward pass\. Let𝒢l,e\\mathcal\{G\}\_\{l,e\}denote the set of selected channel groups\. For each parameter tensor, we define a mask𝐦l,e\\mathbf\{m\}\_\{l,e\}whose entries are zero on frozen channels and equal to a group\-specific scaling coefficientσg\\sigma\_\{g\}on channels belonging to groupg∈𝒢l,eg\\in\\mathcal\{G\}\_\{l,e\}\. The resulting masked gradient is𝐠~θ=∇θℒ⊙𝐦l,e\\tilde\{\\mathbf\{g\}\}\_\{\\theta\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\\odot\\mathbf\{m\}\_\{l,e\}\.

The scaling coefficientσg\\sigma\_\{g\}is determined by the relative importance of each selected group\. We first compute the captured importance mass of the selected channels:ρl,e=∑m∈𝒜l,eM^l,e,m\\rho\_\{l,e\}=\\sum\_\{m\\in\\mathcal\{A\}\_\{l,e\}\}\\hat\{M\}\_\{l,e,m\}\. For each selected groupgg, we define its activation energy asgenergy\(g\)=∑m∈gM^l,e,mg^\{\(g\)\}\_\{\\mathrm\{energy\}\}=\\sum\_\{m\\in g\}\\hat\{M\}\_\{l,e,m\}, and compute the mean group energy:g¯energy=ρl,e/\|𝒢l,e\|\\bar\{g\}\_\{\\mathrm\{energy\}\}=\\rho\_\{l,e\}/\|\\mathcal\{G\}\_\{l,e\}\|\. The group\-specific scaling coefficient is then given by

σg=clip⁡\(\(genergy\(g\)g¯energy\)γ,1\.0,σmax\)\.\\sigma\_\{g\}=\\mathrm\{clip\}\\\!\\left\(\\left\(\\frac\{g^\{\(g\)\}\_\{\\mathrm\{energy\}\}\}\{\\bar\{g\}\_\{\\mathrm\{energy\}\}\}\\right\)^\{\\gamma\},\\ 1\.0,\\ \\sigma\_\{\\max\}\\right\)\.\(6\)This design amplifies gradients for groups whose activation energy is higher than the average selected group, while avoiding excessive scaling through clipping\. In this way, gradient scaling redistributes update strength among selected sub\-experts according to their relative task relevance\.

##### Entropy\-adaptive modulation\.

To avoid overly sharp or overly flat gradient allocation, we further control the contrast among groups with an entropy\-adaptive exponent:

γ=0\.5\+0\.5⋅H⁡\(𝒢l,e\)Hmax,H\(𝒢l,e\)=−∑gp^glogp^g,\\gamma=0\.5\+0\.5\\cdot\\frac\{H\(\\mathcal\{G\}\_\{l,e\}\)\}\{H\_\{\\max\}\},\\quad H\(\\mathcal\{G\}\_\{l,e\}\)=\-\\sum\_\{g\}\\hat\{p\}\_\{g\}\\log\\hat\{p\}\_\{g\},\(7\)where

p^g=genergy\(g\)ρl,e,Hmax=log⁡\|𝒢l,e\|\.\\hat\{p\}\_\{g\}=\\frac\{g^\{\(g\)\}\_\{\\mathrm\{energy\}\}\}\{\\rho\_\{l,e\}\},\\quad H\_\{\\max\}=\\log\|\\mathcal\{G\}\_\{l,e\}\|\.\(8\)When the group energy distribution is uniform,γ\\gammaapproaches11, preserving inter\-group contrast\. When it is highly concentrated,γ\\gammamoves closer to0\.50\.5, reducing extreme disparities and improving training stability\.

##### Static and dynamic gradient scaling\.

A straightforward implementation is to compute the scaling coefficients once from pre\-collected importance statistics and keep them fixed throughout fine\-tuning\. However, such static scaling can be partially absorbed by Adam\-style adaptive optimization\. Adam maintains the first\-order momentmtm\_\{t\}and second\-order momentvtv\_\{t\}of gradients, and updates parameters using the normalized directionmt/vtm\_\{t\}/\\sqrt\{v\_\{t\}\}\. Therefore, when gradients are multiplied by a fixed coefficient, bothmtm\_\{t\}andvtv\_\{t\}are scaled accordingly, making the normalized update relatively insensitive to this constant factor\. As a result, the intended effect of static gradient scaling may be weakened during training\.

To address this issue, we further introduce dynamic gradient scaling\. Instead of using a fixed scaling mask, we update channel activation statistics online and periodically refresh the scaling coefficients with EMA smoothing\. This allows the gradient allocation to track the evolving importance of selected sub\-experts while avoiding overly noisy updates\. In this way, dynamic scaling acts as an adaptive refinement of static scaling rather than a separate equivalent variant\.

Overall, learning\-rate scaling and gradient scaling play complementary roles: the former restores the global step size under sparse updates, while the latter allocates gradient strength among selected channels according to their relative task relevance, with dynamic scaling further improving this allocation during training\.

Table 1:In\-domain fine\-tuning results on OLMoE under thresholdτ=0\.2\\tau=0\.2\. Each domain column reports the result after fine\-tuning on the corresponding domain dataset, whileTrain\.denotes the percentage of trainable parameters\. Avg\. and General Avg\. denote the average in\-domain performance and the average general\-benchmark performance, respectively\.

## 4Experiments

### 4\.1Implementation Details

##### Fine\-tuning datasets\.

We evaluate our method on multiple domain\-specific supervised fine\-tuning datasets covering five application domains: mathematics, medicine, scientific reasoning, retrieval\-augmented question answering, and code generation\. Specifically, they includeMATH,MMedC\([Qiu et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib19)\),SciRIFF\([Wadden et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib20)\),RAGQA, andCode\. For RAGQA, we include both thePubMedQA\([Jin et al\., 2019](https://arxiv.org/html/2609.25655#bib.bib42)\)andHealth\([Kotonya and Toni, 2020](https://arxiv.org/html/2609.25655#bib.bib43)\)subsets\. The reported MATH score is the average overMATH500\([Hendrycks et al\., 2021](https://arxiv.org/html/2609.25655#bib.bib36)\)andGSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2609.25655#bib.bib35)\), while the reported Code score is the average overHumanEval\([Chen et al\., 2021](https://arxiv.org/html/2609.25655#bib.bib33)\)andMBPP\([Austin et al\., 2021](https://arxiv.org/html/2609.25655#bib.bib34)\)\. These datasets are chosen to cover diverse expert activation patterns and task\-specific adaptation requirements\.

##### Evaluation benchmarks\.

To evaluate general capability after fine\-tuning, we use four general\-purpose benchmarks:GPQA\([Rein et al\., 2023](https://arxiv.org/html/2609.25655#bib.bib41)\),MMLU\-Redux\([Gema et al\., 2025](https://arxiv.org/html/2609.25655#bib.bib38)\),C\-Eval\([Huang et al\., 2023](https://arxiv.org/html/2609.25655#bib.bib39)\), andIFEval\([Zhou et al\., 2023](https://arxiv.org/html/2609.25655#bib.bib40)\)\. These benchmarks evaluate scientific reasoning, broad knowledge understanding, Chinese knowledge and reasoning, and instruction following, respectively\. In all tables,General Avg\.denotes the average score over these four benchmarks\.

##### Baselines\.

We compare our method against three categories of baselines\.Base: the original pretrained model without any downstream fine\-tuning\.Full FT: full fine\-tuning of all model parameters, serving as an upper\-capacity baseline\.LoRA\-r16andLoRA\-r32: parameter\-efficient fine\-tuning with LoRA of rankr=16r=16andr=32r=32, respectively\.ESFT: expert\-level sparse fine\-tuning, which updates only a subset of experts during adaptation\. These baselines allow us to compare our method against dense adaptation, parameter\-efficient tuning, and expert\-level sparse tuning under a unified setting\.

We conduct experiments on two representative MoE language models of different scales:OLMoE\-7B\([Muennighoff et al\., 2024](https://arxiv.org/html/2609.25655#bib.bib24)\)andLing\-mini\-2\.0\-16B\([inclusionAI, 2025](https://arxiv.org/html/2609.25655#bib.bib37)\)\. Using two models of substantially different sizes allows us to verify that the proposed method is not tied to a specific parameter scale and can generalize across different MoE architectures\. We provide more details regarding the experiment in the Appendix[A](https://arxiv.org/html/2609.25655#A1)\.

Table 2:TableQA results on Ling\-mini\-2\.0\. The table reports subtask performance, the average TableQA result \(PoT Avg\.\), and the average general\-benchmark performance after fine\-tuning\.![Refer to caption](https://arxiv.org/html/2609.25655v1/figures/threshold_group.png)Figure 3:Threshold and group\-size ablations\. Left: performance under different selection thresholds on MMedC and SciRiFF\. The NSFT results correspond to NSFT\-SG, which uses learning\-rate scaling and static gradient scaling\. Right: effect of group size on in\-domain \(MMedC\) and zero\-shot general performance\.![Refer to caption](https://arxiv.org/html/2609.25655v1/figures/scaling_improvement.png)Figure 4:Ablation of gradient scaling \(\+SG\) and learning\-rate scaling \(\+SLR\) on MMedC multilingual subtasks using OLMoE\. Bars show gains over the NSFT baseline, i\.e\., the variant with only sub\-expert selection and no adaptive training strategy\.

### 4\.2Main Results

We evaluate whether NSFT offers a better balance among domain adaptation, parameter efficiency, and general capability preservation than PEFT and expert\-level sparse tuning baselines\. Results are reported on both OLMoE and Ling\-mini across domain\-specific and general benchmarks\. Unless otherwise specified, NSFT refers to the full method with selection thresholdτ=0\.2\\tau=0\.2, learning\-rate scaling, and dynamic gradient scaling\.

##### NSFT significantly improves in\-domain adaptation on OLMoE while preserving competitive general capability\.

Table[1](https://arxiv.org/html/2609.25655#S3.T1)summarizes the main results on OLMoE across multiple domain\-specific fine\-tuning datasets\. NSFT\-OLMoE achieves the best in\-domain performance among efficient adaptation baselines, with an average score of 46\.62, substantially outperforming ESFT\-OLMoE and both LoRA variants\. Full FT obtains a higher in\-domain average of 49\.91, but its general\-benchmark average drops sharply to 35\.98\. In contrast, NSFT\-OLMoE maintains a general average of 45\.96, nearly matching ESFT\-OLMoE’s 46\.24, while requiring only5\.67%average trainable parameters\. These results show that sub\-expert\-level adaptation effectively strengthens domain\-specific learning without the severe generalization loss caused by full fine\-tuning\.

##### On TableQA, NSFT achieves a markedly better efficiency–performance trade\-off than ESFT and LoRA\.

As shown in Table[2](https://arxiv.org/html/2609.25655#S4.T2), NSFT\-Ling consistently outperforms ESFT\-Ling on fine\-grained TableQA subtasks\. It achieves a PoT average of34\.44, substantially higher than ESFT\-Ling’s 17\.68, while using only 2\.86% trainable parameters\. Compared with LoRA, NSFT\-Ling is also much more parameter\-efficient and surpasses the strongest LoRA variant, LoRA\-r64\. Moreover, it preserves stronger general capability, reaching a general average of 66\.18, which is higher than all LoRA variants and clearly above Full FT\. These results show that fine\-grained sub\-expert adaptation provides a better efficiency–performance trade\-off for TableQA fine\-tuning\. For more detailed results and discussions on this part, please refer to the Appendix[B](https://arxiv.org/html/2609.25655#A2)\.

These results show that both ESFT and LoRA are suboptimal for domain\-specific MoE adaptation\. ESFT is too coarse because it updates entire experts, while LoRA applies generic low\-rank updates without explicitly targeting task\-relevant expert components\. In contrast, NSFT selects responsive sub\-experts inside activated experts, enabling more precise parameter updates and a better efficiency–generalization trade\-off\.

### 4\.3Ablation Analysis

To investigate the key factors behind NSFT, we conduct ablations on both sub\-expert selection and adaptive optimization\. We analyze the sensitivity to the selection threshold, the effect of group sizeGG, the contribution of learning\-rate and gradient scaling, and the difference between static and dynamic scaling\. These studies show how fine\-grained selection and adaptive training jointly contribute to the effectiveness and stability of NSFT\. Additional ablation results, including detailed threshold robustness, group granularity and dynamic scaling analyses, are provided in Appendix[C](https://arxiv.org/html/2609.25655#A3)[D](https://arxiv.org/html/2609.25655#A4)[F](https://arxiv.org/html/2609.25655#A6)\.

Table 3:Static vs\. dynamic gradient scaling using OLMoE\. NSFT\-SG uses a fixed scaling mask, while NSFT\-DSG\_iter1 and NSFT\-DSG\_iter5 update the dynamic scaling mask every one and five training steps, respectively\. Results are reported on MMedC, SciRiFF, and RAG\-Instruct, including in\-domain and general\-benchmark averages\.##### Threshold Robustness\.

Figure[3](https://arxiv.org/html/2609.25655#S4.F3)\(left\) compares ESFT and NSFT under different selection thresholds on MMedC and SciRiFF\-4k\. NSFT consistently outperforms ESFT across thresholds, showing that sub\-expert\-level selection is more robust and effective than expert\-level sparse tuning under different sparsity budgets\. This advantage mainly comes from allocating updates to task\-relevant internal channels rather than entire experts\. For a fair comparison with ESFT and to balance performance with training cost, we useτ=0\.2\\tau=0\.2for both ESFT and NSFT in the main experiments, following the original ESFT setting\.

##### Choosing the Sub\-expert Group Granularity\.

In our implementation, each expert is partitioned into contiguous channel groups along the intermediate dimension, and each group serves as a candidate sub\-expert\. Figure[3](https://arxiv.org/html/2609.25655#S4.F3)\(right\) shows the effect of group size on MMedC multilingual and zero\-shot general performance\. AsGGincreases from 1 to 16, the zero\-shot general average improves steadily, while the in\-domain MMedC performance remains stable\. The best overall balance is achieved atG=16G=16, where the general performance peaks and the in\-domain score stays among the highest\. This indicates a trade\-off between flexibility and expressiveness\. Overly small groups fragment the trainable subspace and reduce effective update capacity, while overly large groups introduce redundant channels and weaken fine\-grained selection\. A moderate group size, especiallyG=16G=16, provides the best balance\.

##### Effect of Gradient and Learning\-rate Scaling\.

Figure[4](https://arxiv.org/html/2609.25655#S4.F4)compares the effect of gradient scaling \(\+SG\) and learning\-rate scaling \(\+SLR\) on multilingual MMedC tasks\. We observe that \+SLR consistently improves performance on most languages and yields clear gains on both weighted and unweighted averages, indicating that compensating for the reduced effective step size is critical under partial updates\. \+SG also improves several languages and the weighted average, although its effect is less stable\. This is because, when only a subset of sub\-experts is updated, the effective update magnitude becomes significantly smaller; learning\-rate scaling directly compensates for this effect at the optimizer level, while gradient scaling further enhances the selected sparse updates\. These results confirm the effectiveness of our adaptive scaling strategy\.

##### Static vs\. Dynamic Scaling\.

To evaluate the effect of update frequency, we compare static gradient scaling with dynamic gradient scaling updated every step or every five steps, as shown in Table[3](https://arxiv.org/html/2609.25655#S4.T3)\. Dynamic scaling generally improves in\-domain adaptation over static scaling, showing that refreshing the scaling mask during training better captures the evolving importance of selected sub\-experts\. The best update interval is task\-dependent: iter1 performs best on MMedC, while iter5 achieves stronger results on SciRiFF and better general retention on RAG\-Instruct\. This suggests that dynamic scaling improves over static scaling, but its update frequency should balance rapid adaptation with noise suppression\.

## 5Conclusion

We present NSFT, a fine\-grained PEFT framework for MoE LLMs that refines sparse adaptation from whole experts to sub\-experts\. By selecting task\-relevant internal channel groups inside activated experts, NSFT enables more precise updates than expert\-level sparse tuning\. To make such partial updates effective, we further introduce learning\-rate scaling and dynamic gradient scaling to compensate for the reduced effective update magnitude\. Experiments on OLMoE\-7B and Ling\-mini\-2\.0\-16B show that NSFT consistently outperforms representative PEFT and expert\-level sparse tuning baselines, while using substantially fewer trainable parameters and preserving competitive general capability\. These results indicate that expert\-level sparsity remains too coarse, and that exploiting fine\-grained sub\-expert structure is a promising direction for efficient adaptation of MoE LLMs\.

## References

- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Baiet al\.\(2023\)J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1)\.
- Cheet al\.\(2025\)C\. Che, Z\. Wang, P\. Yang, Q\. Wang, H\. Ma, and Z\. ShiLoRA in lora: towards parameter\-efficient architecture expansion for continual visual instruction tuning\.arXiv preprint arXiv:2508\.06202\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Chenget al\.\(2025\)R\. Cheng, Y\. Guan, Y\. Ding, Q\. Hu, Y\. Wei, C\. Yuan, Y\. Shen, W\. Chen, and Y\. GongMixture of neuron experts\.arXiv preprint arXiv:2510\.05781\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1),[§1](https://arxiv.org/html/2609.25655#S1.p3.1),[§2](https://arxiv.org/html/2609.25655#S2.p3.1),[§3\.1](https://arxiv.org/html/2609.25655#S3.SS1.SSS0.Px2.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu,et al\.Deepseekmoe: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1280–1297\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p2.1)\.
- Denget al\.\(2026\)G\. Deng, B\. Li, R\. Chen, H\. Wang, L\. Wen, and L\. SongDR\-lora: dynamic rank lora for mixture\-of\-experts adaptation\.arXiv preprint arXiv:2601\.04823\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p2.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p2.1)\.
- Frankle and Carbin \(2018\)J\. Frankle and M\. CarbinThe lottery ticket hypothesis: finding sparse, trainable neural networks\.arXiv preprint arXiv:1803\.03635\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1)\.
- Gemaet al\.\(2025\)A\. P\. Gema, J\. O\. J\. Leang, G\. Hong, A\. Devoto, A\. C\. M\. Mancino, R\. Saxena, X\. He, Y\. Zhao, X\. Du, M\. R\. G\. Madani,et al\.Are we done with mmlu?\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5069–5096\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px2.p1.1)\.
- Gevaet al\.\(2021\)M\. Geva, R\. Schuster, J\. Berant, and O\. LevyTransformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 5484–5495\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1)\.
- Heet al\.\(2023\)H\. He, J\. Cai, J\. Zhang, D\. Tao, and B\. ZhuangSensitivity\-aware visual parameter\-efficient fine\-tuning\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11825–11835\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1),[§1](https://arxiv.org/html/2609.25655#S1.p2.1),[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Huanget al\.\(2023\)Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, Y\. Fu,et al\.C\-eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.Advances in neural information processing systems36,pp\. 62991–63010\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px2.p1.1)\.
- Huiet al\.\(2025\)T\. Hui, Z\. Zhang, S\. Wang, W\. Xu, Y\. Sun, and H\. WuHft: half fine\-tuning for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12791–12819\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- inclusionAI \(2025\)inclusionAILing\-mini\-2\.0\.Note:[https://huggingface\.co/inclusionAI/Ling\-mini\-2\.0](https://huggingface.co/inclusionAI/Ling-mini-2.0)Hugging Face model repositoryCited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1),[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px3.p2.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1)\.
- Jinet al\.\(2019\)Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. LuPubmedqa: a dataset for biomedical research question answering\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 2567–2577\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Kotonya and Toni \(2020\)N\. Kotonya and F\. ToniExplainable automated fact\-checking for public health claims\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7740–7754\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)J\. Li, B\. Wang, X\. Zhou, and X\. HuDynamic expert specialization: towards catastrophic forgetting\-free multi\-domain moe adaptation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 18489–18504\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p2.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 4582–4597\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Liuet al\.\(2024a\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1),[§2](https://arxiv.org/html/2609.25655#S2.p2.1)\.
- Liuet al\.\(2024b\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDora: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1),[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Liuet al\.\(2025\)W\. Liu, J\. Chen, K\. Ji, L\. Zhou, W\. Chen, and B\. WangRag\-instruct: boosting llms with diverse retrieval\-augmented instructions\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 3865–3888\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1)\.
- Muennighoffet al\.\(2024\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, P\. Walsh, O\. Tafjord, N\. Lambert,et al\.Olmoe: open mixture\-of\-experts language models\.arXiv preprint arXiv:2409\.02060\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1),[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px3.p2.1)\.
- Olmoet al\.\(2025\)T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison,et al\.Olmo 3\.arXiv preprint arXiv:2512\.13961\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1)\.
- Qiuet al\.\(2024\)P\. Qiu, C\. Wu, X\. Zhang, W\. Lin, H\. Wang, Y\. Zhang, Y\. Wang, and W\. XieTowards building multilingual language model for medicine\.Nature Communications15\(1\),pp\. 8384\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1),[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px2.p1.1)\.
- Teamet al\.\(2025\)K\. Team, Y\. Bai, Y\. Bao, G\. Chen, J\. Chen, N\. Chen, R\. Chen, Y\. Chen, Y\. Chen, Y\. Chen,et al\.Kimi k2: open agentic intelligence\.arXiv preprint arXiv:2507\.20534\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1),[§2](https://arxiv.org/html/2609.25655#S2.p2.1)\.
- Tianet al\.\(2025\)X\. Tian, Y\. Ji, H\. Wang, S\. Chen, S\. Zhao, Y\. Peng, H\. Zhao, and X\. LiNot all correct answers are equal: why your distillation source matters\.External Links:2505\.14464,[Link](https://arxiv.org/abs/2505.14464)Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1)\.
- Waddenet al\.\(2025\)D\. Wadden, K\. Shi, J\. Morrison, A\. Li, A\. Naik, S\. Singh, N\. Barzilay, K\. Lo, T\. Hope, L\. Soldaini,et al\.Sciriff: a resource to enhance language model instruction\-following over scientific literature\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 6083–6120\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1),[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2022\)Y\. Wang, S\. Mukherjee, X\. Liu, J\. Gao, A\. H\. Awadallah, and J\. GaoAdamix: mixture\-of\-adapter for parameter\-efficient tuning of large language models\.arXiv preprint arXiv:2205\.124101\(2\),pp\. 4\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Wanget al\.\(2024\)Z\. Wang, D\. Chen, D\. Dai, R\. Xu, Z\. Li, and Y\. WuLet the expert stick to his last: expert\-specialized fine\-tuning for sparse architectural large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 784–801\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p2.1),[§2](https://arxiv.org/html/2609.25655#S2.p2.1)\.
- Wuet al\.\(2025a\)X\. Wu, J\. Yang, L\. Chai, G\. Zhang, J\. Liu, X\. Du, D\. Liang, D\. Shu, X\. Cheng, T\. Sun,et al\.Tablebench: a comprehensive and complex benchmark for table question answering\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 25497–25506\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p5.1)\.
- Wuet al\.\(2025b\)Y\. Wu, H\. Piao, L\. Huang, R\. Wang, W\. Li, H\. Pfister, D\. Meng, K\. Ma, and Y\. WeiS\-lora: scalable low\-rank adaptation for class incremental learning\.arXiv e\-prints,pp\. arXiv–2501\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Xuet al\.\(2021\)R\. Xu, F\. Luo, Z\. Zhang, C\. Tan, B\. Chang, S\. Huang, and F\. HuangRaise a child in large language model: towards effective and generalizable fine\-tuning\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 9514–9528\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2609.25655#S1.p1.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, C\. Tan, H\. Xu, C\. Wang, J\. Huang, and S\. HuangTowards adaptive prefix tuning for parameter\-efficient language model fine\-tuning\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1239–1248\.Cited by:[§2](https://arxiv.org/html/2609.25655#S2.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§4\.1](https://arxiv.org/html/2609.25655#S4.SS1.SSS0.Px2.p1.1)\.

## Appendix AExperimental Setup and Dataset Details

### A\.1Training and Evaluation Setup

We conduct experiments on two Mixture\-of\-Experts \(MoE\) models: Ling\-mini\-2\.0 and OLMoE, across seven downstream tasks including TableQA, SciRIFF, Math, Code, RAGQA, MMedC, and Multilingual\. All methods are trained for 2 epochs with a global batch size of 8 and a per\-device batch size of 1 on 8 GPUs \(NVIDIA A100 80G\)\. We use the AdamW optimizer withβ1\\beta\_\{1\}= 0\.9,β2\\beta\_\{2\}= 0\.95, andϵ\\epsilon= 1e\-8\. The peak learning rate is set to 3e\-5 with a polynomial \(linear\) decay schedule to a minimum learning rate of 3e\-6, preceded by a warmup phase covering 2% of total training steps\. The maximum sequence length is 8,192 tokens, and we employ a bin\-packing strategy to concatenate multiple samples into a single sequence for efficient training\. All experiments are conducted in BFloat16 precision with FlashAttention\-2 and gradient checkpointing enabled\. The random seed is fixed at 1,234 for reproducibility\.

We set the learning\-rate clipping thresholdαm​a​x\\alpha\_\{max\}= 5\.0 and the gradient\-scaling upper boundσm​a​x\\sigma\_\{max\}= 5\.0 in all experiments, which prevents the inverse\-active\-ratio scaling in Eq\. \(5\) and the group\-level gradient coefficient in Eq\. \(6\) from producing excessively large updates and thus stabilizes training\.

We compare four fine\-tuning methods under this unified setting\. Full SFT updates all model parameters using DeepSpeed ZeRO Stage 3 with a weight decay of 0\.1\. ESFT selects task\-relevant experts via a gate score threshold of 0\.2 and fine\-tunes only those experts, using DeepSpeed ZeRO Stage 1 with a weight decay of 0\.1\. LoRA applies low\-rank adaptation with rank 32,α\\alpha= 64, and dropout 0\.05, targeting the MLP layers \(gate\_proj, up\_proj, down\_proj\), trained with DeepSpeed ZeRO Stage 1 and a weight decay of 0\.1\. NSFT performs fine\-grained sub\-expert updates with dynamic gradient scaling guided by pre\-computed gate energy distributions\. NSFT uses DeepSpeed ZeRO Stage 1 with weight decay set to 0 to prevent frozen parameters from drifting, and employs MoE\-aware learning rate scaling with an EMA decay factor of 0\.9 and a scale update interval of 5 steps\. By default, NSFT uses a group size of 16 and a gate score threshold of 0\.2; we also ablate on group sizes of 1, 2, 4, 8, 32 and thresholds of 0\.2, 0\.3, 0\.4\.

### A\.2Datasets

#### A\.2\.1Dataset Details

We evaluate our method across seven diverse and challenging downstream tasks spanning distinct domains:

- •TableQA \(TableInstruct\), a table understanding benchmark requiring structured reasoning over tabular data;
- •SciRIFF, a scientific instruction\-following dataset demanding domain\-specific reasoning across scientific literature;
- •Math \(GSM8k and MATH500\), evaluated on two complementary mathematical reasoning benchmarks and reported as their average score;
- •Code \(HumanEval and MBPP\), evaluated on two widely\-used code generation benchmarks and reported as their average score;
- •RAG \(PubMedQA and Health\_Claims\), a retrieval\-augmented QA task evaluated on two benchmarks and reported as their average, requiring the model to synthesize answers from long retrieved contexts in the biomedical domain\.

All datasets are formatted in the standard chat\-based message format and pre\-tokenized offline with a maximum sequence length of 8,192 tokens\. We adopt a bin\-packing strategy to concatenate multiple samples into packed sequences for efficient training\.

#### A\.2\.2On the choice of evaluation datasets and benchmarks

Our benchmark suite is deliberately chosen to be complex and challenging — spanning structured reasoning \(TableQA\), domain\-specific scientific and medical knowledge \(SciRIFF, MMedC\), long\-context retrieval comprehension \(RAGQA\), multi\-step mathematical and code reasoning \(Math, Code\), and cross\-lingual generalization \(Multilingual\)\. These tasks demand deep adaptation of model knowledge and are far more likely to expose the limitations of parameter\-efficient approaches when compared to full fine\-tuning\. By evaluating under such demanding conditions, we aim to provide a more rigorous and convincing assessment of our proposed NSFT method, demonstrating that it can maintain competitive performance even in scenarios where conventional PEFT methods typically fall short\.

## Appendix BAdditional Results on TableQA

Table 4:Detailed TableQA results on Ling\-mini\-2\.0\. We compare Full FT, LoRA with different ranks and target modules, ESFT under different selection thresholds, and NSFT under different thresholds and group sizes\. The table reports performance on four TableQA subtasks, the PoT average, and the average performance on general benchmarks\. Bold numbers indicate the best result within each column\.Table[4](https://arxiv.org/html/2609.25655#A2.T4)provides a more detailed comparison on TableQA using Ling\-mini\-2\.0\. We include LoRA variants with different ranks and target modules, ESFT under different selection thresholds, and NSFT under different thresholds and group sizes\. These results further validate the advantage of fine\-grained sub\-expert adaptation\.

##### NSFT achieves the strongest efficiency–performance trade\-off on TableQA\.

Compared with LoRA and ESFT, NSFT obtains competitive or stronger TableQA performance while using substantially fewer trainable parameters\. In particular, NSFT\-G4\-th0\.2 reaches a PoT average of 34\.44 with only 2\.86% trainable parameters, outperforming all ESFT variants and approaching the strongest LoRA variant, LoRA\-128 MLP\+QKVO, which obtains a PoT average of 35\.35 but requires 29\.69% trainable parameters\. This indicates that updating fine\-grained sub\-experts can recover most of the task\-specific adaptation ability of much larger PEFT configurations with nearly an order of magnitude fewer trainable parameters\.

##### NSFT is consistently stronger than expert\-level sparse tuning\.

Across different thresholds, NSFT clearly outperforms ESFT on TableQA\. The best ESFT variant reaches a PoT average of 21\.65, whereas multiple NSFT variants exceed this score by a large margin\. For example, NSFT\-G16\-th0\.2, NSFT\-G8\-th0\.2, NSFT\-G4\-th0\.2, NSFT\-G2\-th0\.2, and NSFT\-G1\-th0\.2 all outperform ESFT\-th0\.3\. This confirms that expert\-level selection is too coarse for TableQA: updating whole experts still includes many task\-irrelevant internal channels, while sub\-expert selection can allocate the update budget to more relevant parts inside experts\.

##### NSFT better preserves general capability than full fine\-tuning and large LoRA variants\.

Although Full FT obtains the highest PoT average, it significantly reduces the general average from 66\.45 to 62\.59\. Large LoRA variants also tend to underperform the base model on general benchmarks\. In contrast, NSFT maintains competitive general capability, with several variants achieving general averages around 66\.0–66\.5 while substantially improving TableQA performance\. This shows that fine\-grained sub\-expert tuning provides a better balance between domain specialization and general capability retention\.

##### Discussion on ESFT degradation\.

Table[4](https://arxiv.org/html/2609.25655#A2.T4)shows that ESFT underperforms the Ling\-mini\-2\.0 base model on TableQA, while LoRA variants that only update MoE\-related modules also provide limited gains without a much larger trainable budget\. A likely reason is that Ling\-mini itself adopts a finer expert partitioning than OLMoE, using 256 experts instead of 64\. This architectural design indicates that Ling\-mini distributes its computation across more fine\-grained expert units\. Accordingly, adaptation should also be performed at a finer granularity: updating entire selected experts may still be too coarse, while sub\-expert\-level updates can better match the model’s internal organization\.

This highlights the need to align adaptation granularity with model granularity\. NSFT further decomposes experts into structured sub\-expert groups and updates only the most responsive ones, providing a finer update unit that better matches Ling\-mini’s architecture\. This explains why NSFT achieves stronger TableQA performance with fewer trainable parameters and suggests that fine\-grained MoE backbones may benefit more from sub\-expert\-level adaptation\.

## Appendix CRobustness across Selection Thresholds

Table 5:Performance comparison under different selection thresholds on MMedC\. We compare ESFT and NSFT variants under different selection thresholdsτ\\tau\. Best results under each threshold are bolded\. The NSFT results correspond to NSFT\-SG, which uses learning\-rate scaling and static gradient scaling\.Table 6:Performance comparison under different selection thresholds on SciRiFF under 4k / 8k settings\. Each cell reports results under 4k and 8k training lengths, respectively\. Best results under each threshold and setting are bolded\. The NSFT results correspond to NSFT\-SG, which uses learning\-rate scaling and static gradient scaling\.##### Discussion\.

Tables[5](https://arxiv.org/html/2609.25655#A3.T5)and[6](https://arxiv.org/html/2609.25655#A3.T6)report detailed results under different selection thresholds\. NSFT consistently outperforms ESFT across both MMedC and SciRiFF, showing that sub\-expert\-level selection is more robust than expert\-level sparse tuning under different sparsity budgets\. Performance generally improves asτ\\tauincreases, since more task\-relevant units are retained\. These results suggest that the optimal granularity depends on the target domain, but the advantage of fine\-grained sub\-expert selection remains consistent\.

## Appendix DGroup Size Sensitivity across MoE Backbones

Table 7:Relationship between group size, activated expert coverage, and TableQA PoT performance on Ling\-mini\-2\.0\.##### The optimal group size is backbone\-dependent\.

Although our main experiments on OLMoE suggest that a moderate group size such asG=16G=16provides a strong balance between flexibility and stability, the results on Ling\-mini\-2\.0 show that the optimal group size can shift across MoE backbones\. As shown in Table[7](https://arxiv.org/html/2609.25655#A4.T7), when applying NSFT to TableQA on Ling\-mini\-2\.0, the best PoT performance is achieved atG=4G=4, reaching 34\.44\. This differs from the trend observed on OLMoE, where larger groups such asG=16G=16are often more favorable\. This indicates that group size should not be treated as a universal constant, but rather as a backbone\- and task\-dependent hyperparameter\.

##### Smaller groups provide broader expert coverage on Ling\-mini\-2\.0\.

Table[7](https://arxiv.org/html/2609.25655#A4.T7)reveals a clear relationship between group size, expert coverage, and task performance\. AsGGdecreases from 32 to 1, the number of involved experts increases substantially, from 230 to 1024\. This suggests that smaller groups allow NSFT to select sub\-experts from a much broader set of experts, thereby covering more diverse computation paths\. For Ling\-mini\-2\.0 on TableQA, this broader coverage appears to be important, likely because TableQA requires heterogeneous capabilities such as instruction understanding, fact checking, numerical reasoning, and data analysis\. Therefore, a smaller group size can better capture the distributed expert patterns needed by this task\.

##### Connection to expert\-level tuning\.

It is also worth noting that when the group size becomes equal to the expert intermediate dimension, i\.e\.,G=1024G=1024in Ling\-mini\-2\.0, each expert contains only one group\. In this extreme case, sub\-expert selection degenerates into expert\-level selection, which is essentially equivalent to ESFT\. As shown in Table[7](https://arxiv.org/html/2609.25655#A4.T7), this setting involves 173 experts and achieves a TableQA PoT score of 17\.68, much lower than the best sub\-expert setting withG=4G=4\. This result further confirms that expert\-level granularity is too coarse: updating an entire expert introduces substantial redundant parameters, while decomposing experts into finer\-grained sub\-experts enables more precise and effective adaptation\.

These results provide a practical guideline for choosing the group size\. When task\-relevant computation is distributed across many experts, smaller groups such asG=4G=4orG=8G=8offer broader expert coverage and greater selection flexibility\. When useful activations are concentrated within fewer experts, moderately larger groups such asG=16G=16can be more stable and efficient\. We therefore treatGGas a lightweight hyperparameter and select it with a small validation sweep, using expert coverage and intra\-expert activation ratio as diagnostic signals to avoid both overly fragmented and overly coarse updates\.

## Appendix EAdditional Analysis for Intra\-expert Sparsity in ESFT

![Refer to caption](https://arxiv.org/html/2609.25655v1/figures/subexperts_activation.png)Figure 5:Intra\-expert activation sparsity across domains\. Each subplot shows the normalized channel\-wise gate\-score distribution of a representative OLMoE expert\. The dashed orange line marks the top\-50% cumulative activation threshold\. The strong concentration near zero across domains indicates that only a small subset of channels is highly activated inside selected experts\.![Refer to caption](https://arxiv.org/html/2609.25655v1/figures/layer_wise_subexperts_activation.png)Figure 6:Channel\-wise gate\-score distributions for randomly sampled experts from shallow, middle, and deep layers on MMedC, with two experts sampled from each depth range\. The dashed orange line marks the threshold covering the top 50% cumulative activation mass\.### E\.1Intra\-expert activation sparsity across domains

Figure[5](https://arxiv.org/html/2609.25655#A5.F5)visualizes the channel\-wise gate\-score distributions of representative OLMoE experts from different layers and domains\. Across all datasets, the distributions are highly concentrated near zero, while only a small fraction of channels receive relatively large activation scores\. The dashed orange line marks the threshold covering the top 50% cumulative activation mass, and it consistently lies in a low\-value region\. This indicates that even within activated or selected experts, the effective activation mass is dominated by a small subset of internal channels\. Such a pattern is observed across diverse domains, including medical, scientific, retrieval, mathematical, code, and table reasoning tasks, suggesting that intra\-expert sparsity is a general property rather than a domain\-specific artifact\. These findings further support our motivation for sub\-expert selection: updating an entire selected expert, as in expert\-level sparse tuning, inevitably includes many weakly activated channels, whereas fine\-grained sub\-expert selection can focus the trainable budget on the most responsive internal components\.

### E\.2Consistent sparsity across experts

To further verify that intra\-expert sparsity is not limited to a few special cases, Figure[6](https://arxiv.org/html/2609.25655#A5.F6)visualizes the distributions of normalized gate scores for randomly sampled experts from shallow, middle, and deep layers\. Specifically, we randomly select two experts from each of the shallow, middle, and deep layers\. Despite differences in layer depth and expert identity, all sampled experts exhibit a highly similar pattern: most channels are concentrated in the near\-zero region, while only a small fraction of channels form a long tail with relatively larger activations\. The dashed line marking the threshold that covers the top 50% cumulative activation mass also lies in a very small\-value region across all cases\.

These results confirm that strong intra\-expert sparsity is a stable and widespread phenomenon, rather than an artifact of a particular layer or expert\. This consistency further motivates our sub\-expert selection strategy, which allocates updates to the most responsive internal channel groups instead of updating the full expert uniformly\.

## Appendix FAdditional Results on Static and Dynamic Gradient Scaling

In the main text, we report the averaged results over three datasets to summarize the overall effect of static and dynamic gradient scaling\. Here, we provide detailed results on each dataset for a more fine\-grained comparison\.

Tables[8](https://arxiv.org/html/2609.25655#A6.T8)and[9](https://arxiv.org/html/2609.25655#A6.T9)provide additional comparisons between static gradient scaling and dynamic gradient scaling on MMedC and SciRiFF\. Overall, dynamic scaling generally improves in\-domain adaptation over the static variant, but the best update frequency varies across tasks\. On MMedC, updating the scaling mask every training step achieves the best weighted average and general\-benchmark average, while updating every five steps gives the best unweighted average\. On SciRiFF, the five\-step update performs better, achieving the highest weighted and unweighted validation averages under both 4096 and 8192 settings\. These results suggest that dynamic scaling can better track the evolving importance of selected sub\-experts, while a less frequent update schedule can be beneficial when online statistics are noisy\.

Table 8:Static vs\. dynamic gradient scaling on MMedC multilingual subtasks using OLMoE\.DSGdenotes dynamic gradient scaling, andDSG\-EMAapplies EMA smoothing\. The table reports language\-wise results, weighted/unweighted averages, and general\-benchmark average\.Table 9:Static vs\. dynamic gradient scaling on SciRiFF subtasks using OLMoE\.
## Appendix GLow\-Budget and Budget\-Matched Comparisons

##### Lower\-budget regime\.

Table[10](https://arxiv.org/html/2609.25655#A7.T10)extends the comparison between ESFT and NSFT to selection thresholds below the defaultτ=0\.2\\tau=0\.2\. Atτ=0\.05\\tau=0\.05, the coarse selection unit of ESFT causes nearly every layer to retain only one complete expert, which represents a practical lower bound on its parameter budget\. In contrast, NSFT can continue reducing the budget by selecting groups within experts\. NSFT consistently obtains higher MMedC performance while updating fewer parameters across all three thresholds, demonstrating a stronger accuracy–parameter trade\-off in the low\-budget regime\.

Table 10:Low\-budget comparison between ESFT and NSFT on MMedC\. Train\. denotes the percentage of trainable parameters\.
##### Closely matched parameter budgets\.

Because ESFT selects complete experts whereas NSFT selects groups within experts, their trainable\-parameter ratios cannot generally be matched exactly using the same threshold\. We therefore adjust the NSFT threshold toτ=0\.22\\tau=0\.22to obtain budgets close to those of ESFT atτ=0\.20\\tau=0\.20\. As shown in Table[11](https://arxiv.org/html/2609.25655#A7.T11), NSFT remains stronger on both SciRIFF and MMedC under closely matched budgets\. This result rules out a larger trainable\-parameter budget as the source of NSFT’s improvement\.

Table 11:Comparison under closely matched trainable\-parameter budgets\. The NSFT threshold is adjusted to match the ESFT budget as closely as permitted by their different selection granularities\.
##### Weight\-decay control\.

The main experiments follow the recommended settings of each baseline: ESFT uses weight decay0\.10\.1, whereas NSFT uses weight decay00to avoid decay\-induced updates to masked parameters\. As a control, setting the ESFT weight decay to00yields an MMedC average of 44\.01, compared with 44\.31 under its original weight decay of0\.10\.1\. The result is slightly lower rather than higher, indicating that the advantage of NSFT is not caused by the different weight\-decay settings\.

## Appendix HAlternative Sub\-Expert Grouping Strategies

Our default method forms structured groups from consecutive channels\. This choice does not assume that individual channels have explicit human\-interpretable semantics; rather, it provides a simple and low\-cost way to construct stable update units in the learned parameter layout\. To test whether the result depends on this choice, we compare it with three alternatives while keeping the remaining training settings fixed: \(1\) random grouping, which randomly assigns channels to groups of size 16; \(2\) activation\-clustering grouping, which clusters channels according to the cosine similarity of their normalized up\-projection responses over tokens; and \(3\) individual\-channel selection \(G=1G=1\), which performs top\-KKnon\-contiguous selection without structured grouping\.

Table[12](https://arxiv.org/html/2609.25655#A8.T12)shows that random and activation\-clustering groups mitigate the general\-capability degradation of individual\-channel selection, but remain weaker than consecutive grouping\. The default strategy therefore provides the best overall balance between in\-domain adaptation, general capability, simplicity, and grouping overhead\.

Table 12:Comparison of alternative sub\-expert grouping strategies on MMedC\. W\-Avg\. denotes the weighted average over the multilingual subtasks\.
## Appendix IContributions of Selection and Adaptive Optimization

To isolate the effect of fine\-grained selection, we begin with NSFT\-G16\-Select, which uses only sub\-expert selection atτ=0\.2\\tau=0\.2and removes all adaptive optimization strategies\. We then progressively add static and dynamic gradient scaling\. The static variant includes learning\-rate scaling and a fixed gradient\-scaling mask, whereas the dynamic variant periodically refreshes the mask using updated activation statistics\.

As shown in Table[13](https://arxiv.org/html/2609.25655#A9.T13), selection alone already outperforms ESFT on both MMedC and SciRIFF\-8k, confirming that the primary gain comes from refining the update unit from experts to sub\-experts\. Static and dynamic scaling then provide further, complementary improvements, with dynamic scaling giving the largest gain on SciRIFF\-8k\.

Table 13:Step\-by\-step ablation separating sub\-expert selection from adaptive optimization\.
## Appendix JEffect of the Entropy\-Adaptive Exponent

The entropy\-adaptive exponentγ\\gammasmooths group\-wise gradient scaling: when group energies are relatively uniform,γ\\gammaapproaches 1 and preserves their contrast; when the distribution is concentrated,γ\\gammamoves toward 0\.5 and suppresses extreme scaling coefficients\. We evaluate its effect using theG=32G=32model with static gradient scaling on MMedC\. Table[14](https://arxiv.org/html/2609.25655#A10.T14)shows that replacing the adaptive exponent with a fixed exponent of 1\.0 reduces the MMedC average from 45\.06 to 44\.87\. Although the gain is modest, it consistently supports adaptive smoothing over the raw energy\-ratio scaling rule\.

Table 14:Ablation of the entropy\-adaptive exponent usingG=32G=32and static gradient scaling on MMedC\.
## Appendix KTraining Efficiency and Dynamic\-Scaling Overhead

Table[15](https://arxiv.org/html/2609.25655#A11.T15)reports wall\-clock training time, peak GPU memory, and MMedC performance\. Wall\-clock time is measured using eight NVIDIA A100 GPUs, while peak per\-GPU memory is measured using four A100 GPUs under the same MMedC setting\. NSFT\-selection has a training time close to ESFT and already achieves higher accuracy, showing that sub\-expert selection itself introduces little additional time overhead\. All NSFT variants also retain memory consumption close to ESFT and substantially below Full FT\.

The main additional cost of full NSFT comes from collecting updated activation statistics and refreshing the scaling mask\. Removing dynamic updates reduces training time from 35m 29s to 20m 38s without increasing memory usage\. Dynamic scaling should therefore be viewed as an optional performance enhancement: users may choose selection\-only, static\-scaling, or full dynamic\-scaling NSFT according to their compute budget\. More frequent updates track rapidly changing sub\-expert importance more closely, whereas less frequent updates can be more stable when the activation statistics are noisy or heterogeneous\.

Table 15:Training cost and performance on MMedC\. Wall\-clock time and peak per\-GPU memory are measured with eight and four A100 GPUs, respectively\.
## Appendix LLimitations

Although NSFT achieves strong results across representative MoE backbones and diverse downstream tasks, several aspects deserve further exploration\. First, we evaluate NSFT on two competitive MoE models, OLMoE\-7B and Ling\-mini\-2\.0\-16B, which already cover different scales and architectural characteristics\. The consistent improvements across these models suggest that sub\-expert\-level adaptation is broadly effective\. Future work may further examine NSFT on larger and more heterogeneous MoE LLMs to better characterize its scaling behavior\.

Second, NSFT uses the group sizeGGand selection thresholdτ\\tauto control the granularity and budget of sub\-expert selection\. Our experiments show stable gains across multiple choices of these hyperparameters, and the default setting works well in practice\. Still, for new domains or substantially different MoE architectures, a lightweight validation sweep may help identify the best efficiency–performance trade\-off\.

## Appendix MBroader Impacts

This work improves the parameter efficiency of adapting MoE LLMs to downstream domains, which can reduce computational cost, memory usage, and energy consumption compared with full\-parameter fine\-tuning\. This may make domain\-specific model adaptation more accessible to users with limited hardware resources\.

However, efficient adaptation may also lower the barrier to customizing powerful models for harmful or misleading purposes\. Moreover, domain\-specific fine\-tuning can amplify biases, errors, or unsafe behaviors present in the training data\. We therefore recommend careful data curation, safety evaluation, and deployment monitoring, especially for high\-stakes applications\.