XPERT: Expert Knowledge Transfer for Effective Training of Language Models

# XPERT: Expert Knowledge Transfer for Effective Training of Language Models
Source: [https://arxiv.org/html/2605.08842](https://arxiv.org/html/2605.08842)
###### Abstract

Mixture\-of\-Experts \(MoE\) language models organize knowledge into explicitly routed expert modules, making expert\-level representations traceable and analyzable\. By analyzing expert activation patterns in MoE large language models \(LLMs\), we find that a subset of experts is consistently activated across diverse knowledge domains\. These common experts encode cross\-domain, generalizable knowledge that is closely related to model generalization, naturally raising the question of how such identifiable expert knowledge can be practically reused\. Motivated by this observation, we proposeXPERT, a framework that extracts, consolidates, and reuses expert knowledge from pre\-trained MoE LLMs to support more effective training of language models across different model scales\. XPERT identifies cross\-domain experts via inference\-only analysis, refines their representations through tensor decomposition, and adapts the extracted knowledge to reuse in downstream models\. Experiments on language understanding and dialogue generation benchmarks show that models benefiting from reused expert knowledge achieve consistently stronger performance and faster convergence compared to strong baselines\. These results highlight MoE LLMs as structured and reusable knowledge sources, and demonstrate the value of expert\-level knowledge reuse for improving model training\.

Machine Learning, ICML

## 1Introduction

With the rapid advancement of large language models \(LLMs\)\(Wiggins and Tejani,[2022](https://arxiv.org/html/2605.08842#bib.bib1); Chowdheryet al\.,[2023](https://arxiv.org/html/2605.08842#bib.bib2); Achiamet al\.,[2023](https://arxiv.org/html/2605.08842#bib.bib3); Grattafioriet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib5); Zhonget al\.,[2025](https://arxiv.org/html/2605.08842#bib.bib20)\), many models have demonstrated strong performance across a wide range of tasks and domains\(Zhanget al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib19); Sunet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib18); Yiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib21)\)\. While early LLMs were predominantly based on dense architectures\(Touvronet al\.,[2023](https://arxiv.org/html/2605.08842#bib.bib6); Mannet al\.,[2020](https://arxiv.org/html/2605.08842#bib.bib7); Baiet al\.,[2023](https://arxiv.org/html/2605.08842#bib.bib8)\), recent research has increasingly shifted towards sparse Mixture\-of\-Experts \(MoE\) models\(Shazeeret al\.,[2017a](https://arxiv.org/html/2605.08842#bib.bib9); Daiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib44); Muennighoffet al\.,[2025](https://arxiv.org/html/2605.08842#bib.bib43); Teamet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib64); Liuet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib10)\), which scale model capacity by activating only a small subset of specialized experts per input\. This conditional computation paradigm enables MoE models to achieve strong performance while reducing training and inference costs\.

Beyond computational efficiency, a key property of MoE architectures is that knowledge is explicitly modularized at the level of experts\. Through routing mechanisms, each expert is selectively activated by specific inputs, making the relationship between expert representations and task characteristics both explicit and traceable\. Motivated by this property, we analyze expert activation patterns in MoE LLMs using data from different knowledge domains\.

We observe that while many experts are domain\-specific, a subset of experts remains consistently active across diverse domains\. As shown in Figure[1](https://arxiv.org/html/2605.08842#S1.F1), certain experts \(e\.g\., experts 8, 17, and 30 in OLMoE\-7B\) exhibit high activation across multiple domains, indicating that they encode shared cross\-domain knowledge\. We refer to such experts as*common experts*\. The existence of common experts suggests a close connection between expert\-level representations and model generalization\. This intuition is further supported by recent MoE designs such as DeepSeekMoE\(Daiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib44)\), which explicitly introduce shared experts activated across all inputs, highlighting the importance of general\-purpose expert knowledge\. These observations naturally raise a key question:*what practical use can be made of expert knowledge that is both identifiable and closely tied to generalization?*

In the vision domain, prior work has shown that generalizable knowledge extracted from pre\-trained convolutional neural networks or vision Transformers can be reused to improve training efficiency and stability across tasks and model configurations\(Wanget al\.,[2023a](https://arxiv.org/html/2605.08842#bib.bib40); Shiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib42)\)\. Inspired by this line of work, we explore whether expert knowledge in MoE\-based LLMs can be similarly extracted and transferred to improve the efficiency and effectiveness of model training\. We proposeXPERT, a framework that extracts cross\-domain common expert knowledge from MoE LLMs and reuses it to support more effective training of models across different scales\.

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/layer15_expert_freq.png)Figure 1:Experts activation frequencies of layer 15 in OLMoE\-7B across different domains\. Additional examples are provided in Appendix[A](https://arxiv.org/html/2605.08842#A1)\.![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/lct.png)Figure 2:The framework of XPERT\.ZZis the tensor formed by stacking the selected experts parameter matrices, and\(𝒢,U\)\(\\mathcal\{G\},U\)represents the refined knowledge corresponding to a specific parameter matrix\. After parameter\-scale adaptation in Step 3, the extracted expert knowledge is used to initialize the FFN layers of language models with different scales\.Under the XPERT framework, expert knowledge is identified through a purely inference\-based analysis\. As illustrated in Step 1 of Figure[2](https://arxiv.org/html/2605.08842#S1.F2), we perform forward passes on data drawn from multiple knowledge domains and record expert activation patterns\. Experts that are consistently activated across domains are selected, as such activation patterns indicate cross\-domain universal knowledge closely tied to model generalization\. To transform the selected experts into a compact and reusable form, XPERT further introduces a knowledge consolidation step\. As shown in Step 2 of Figure[2](https://arxiv.org/html/2605.08842#S1.F2), experts from each transformer block are aggregated into high\-order tensors, upon which tensor decomposition is applied to extract shared components that capture domain\-agnostic expert knowledge\.

A practical challenge arises when transferring the extracted knowledge to target models whose parameter dimensions differ from those of the source MoE LLMs\. To address this mismatch, we propose a parameter\-scale adaptation method that adjusts the extracted tensor representations to align with the dimensionality of the target model\. As illustrated in Step 3 of Figure[2](https://arxiv.org/html/2605.08842#S1.F2), the reconstructed representations are adapted into Feed\-Forward Network \(FFN\) parameter matrices that match the target model’s hidden and intermediate dimensions, and are used to initialize its FFN layers prior to training\. All remaining parameters are initialized using standard random initialization\.

We evaluate XPERT using OLMoE\-7B\(Muennighoffet al\.,[2025](https://arxiv.org/html/2605.08842#bib.bib43)\)and DeepSeekMoE\-16B\(Daiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib44)\)as source MoE LLMs on supervised fine\-tuning \(SFT\) benchmarks spanning language understanding and dialogue generation across multiple domains\. Across all settings, XPERT extracts a compact subset of expert\-derived representations \(approximately 1\.25% of the parameters in source LLMs\)\. Notably, expert selection, knowledge consolidation, and parameter\-scale adaptation are entirely training\-free\. When used as an initialization prior, XPERT\-initialized models consistently outperform strong baselines, including training from scratch and knowledge distillation, while converging faster and requiring up to 5×\\timesless pre\-training data\. Overall, these results demonstrate that expert knowledge in MoE LLMs can be systematically reused as an effective and generalizable initialization prior\. XPERT highlights the potential of MoE LLMs as structured knowledge sources beyond their original deployment\. Our contributions can be summarized as follows:

- •We systematically study the reuse of expert knowledge in MoE LLMs, and show that a subset of experts consistently encodes cross\-domain universal knowledge closely tied to model generalization\.
- •We propose XPERT, a framework that extracts, consolidates, and adapts expert knowledge from pre\-trained MoE LLMs to support more effective model training, enabling parameter\-level reuse in models of diverse dimensions and sizes\.
- •Through extensive experiments across multiple model scales and diverse downstream tasks, we demonstrate that XPERT\-initialized models exhibit improved training dynamics and stronger downstream performance compared to other baselines\.

## 2Related Work

#### Mixture\-of\-Experts Language Model

MoE language models have been widely studied as an efficient architecture for scaling model capacity through conditional computation, where only a subset of experts is activated for each input\(Shazeeret al\.,[2017b](https://arxiv.org/html/2605.08842#bib.bib61)\)\. Subsequent work has explored large\-scale MoE pre\-training and deployment, focusing on efficiency, scalability, and expert specialization, including Switch Transformer\(Feduset al\.,[2022](https://arxiv.org/html/2605.08842#bib.bib62)\), GShard\(Lepikhinet al\.,[2020](https://arxiv.org/html/2605.08842#bib.bib63)\), and more recent MoE\-based LLMs such as DeepSeekMoE\(Daiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib44)\), OLMoE\(Muennighoffet al\.,[2025](https://arxiv.org/html/2605.08842#bib.bib43)\), and Gemini\(Teamet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib64)\)\. Several studies further analyze expert routing behaviors and activation patterns to understand specialization and load balancing\(Feduset al\.,[2022](https://arxiv.org/html/2605.08842#bib.bib62); Huanget al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib60)\)\. However, existing work primarily examines experts in the context of routing efficiency and task specialization, and does not investigate whether expert parameters encode reusable knowledge that can be systematically extracted and transferred to new models\.

#### Reusable Knowledge in Models

Prior work has explored whether knowledge embedded in large models can be explicitly identified and reused\. Some studies estimate parameter or component importance through gradient sensitivity\(Molchanovet al\.,[2019](https://arxiv.org/html/2605.08842#bib.bib72)\), loss\-change tracking\(Frankle and Carbin,[2018](https://arxiv.org/html/2605.08842#bib.bib73)\), or data valuation methods\(Xuet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib77)\)\. While effective at highlighting influential parameters, these approaches typically reveal patterns that are either highly dispersed or aggregated at coarse granularities, limiting their suitability as reusable knowledge units\.

Other work focuses on parameter\-efficient adaptation, such as adapters and low\-rank updates\(Houlsbyet al\.,[2019](https://arxiv.org/html/2605.08842#bib.bib74); Huet al\.,[2022](https://arxiv.org/html/2605.08842#bib.bib75); Li and Liang,[2021](https://arxiv.org/html/2605.08842#bib.bib76)\), which improve training efficiency but do not explicitly isolate transferable, task\-agnostic knowledge representations\. Overall, existing approaches offer limited insight into how generalizable knowledge can be explicitly localized and reused across models\. Whether LLMs contain structured, reusable knowledge components remains an open question, particularly for MoE architectures, where knowledge is explicitly modularized at the expert level\.

#### Model Initialization

Model initialization strongly influences optimization dynamics and generalization in deep neural networks\. While classical schemes such as Xavier and Kaiming focus on training stability\(Glorot and Bengio,[2010](https://arxiv.org/html/2605.08842#bib.bib65); Heet al\.,[2015](https://arxiv.org/html/2605.08842#bib.bib66)\), modern language models often rely on pretrained initialization to reuse representations learned from large\-scale data, leading to faster convergence and improved performance\(Radfordet al\.,[2019](https://arxiv.org/html/2605.08842#bib.bib68); Devlinet al\.,[2019](https://arxiv.org/html/2605.08842#bib.bib69)\)\. Prior studies further show that initialization biases which parameter substructures are activated during training and can steer optimization toward better\-generalizing solutions\(Frankle and Carbin,[2018](https://arxiv.org/html/2605.08842#bib.bib73); Mishkin and Matas,[2015](https://arxiv.org/html/2605.08842#bib.bib67)\)\. XPERT builds on this insight by reusing expert knowledge from MoE LLMs at initialization, providing a structured inductive bias that improves training effectiveness\.

## 3Expert Knowledge Transfer Framework

We presentXPERT, a framework that reuses expert knowledge from pre\-trained MoE LLMs to initialize new models for improving training effectiveness\. XPERT consists of three stages: cross\-domain expert selection \(Section[3\.2](https://arxiv.org/html/2605.08842#S3.SS2)\), expert knowledge consolidation via tensor decomposition \(Section[3\.3](https://arxiv.org/html/2605.08842#S3.SS3)\), and parameter\-scale adaptation for initialization \(Section[3\.4](https://arxiv.org/html/2605.08842#S3.SS4)\)\.

### 3\.1Preliminary

The MoE architecture consists of multiple expertsEEand a routerG​\(x\)G\(x\)\. The input data are processed through the router, which selects a subset of experts for computation\. In an MoE layer withNNexperts, each tokenxxis assigned a subset of experts for computation through the router:

y\\displaystyle y=∑i=1NGi​\(x\)​Ei​\(x\),\\displaystyle=\\sum^\{N\}\_\{i=1\}G\_\{i\}\(x\)E\_\{i\}\(x\),Gi​\(x\)\\displaystyle G\_\{i\}\(x\)=\{gi,if​gi∈T​o​p​K​\(\{gj\|1≤j≤N\},K\),0,otherwise,\\displaystyle=\\left\\\{\\begin\{array\}\[\]\{ll\}g\_\{i\},\\quad\\text\{if \}g\_\{i\}\\in TopK\(\\\{g\_\{j\}\\;\|\\;1\\leq j\\leq N\\\},\\;K\),\\\\ 0,\\quad\\text\{otherwise\},\\end\{array\}\\right\.whereEiE\_\{i\}represents theii\-th expert, which is implemented as the FFN layer in a Transformer model\. The termgi=softmaxi​\(W​x\)g\_\{i\}=\\mathrm\{softmax\}\_\{i\}\(Wx\)denotes the softmax value obtained for theii\-th expert after the gating mechanism processes the inputxx\.WWrepresents the weight parameters of the gating function\.

In some recent MoE LLMs\(Daiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib44); Liuet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib10)\), shared experts have been introduced to handle common knowledge in different tasks\. The calculation of an MoE architecture with shared experts is formulated asy=∑i=1NGi​\(x\)​Ei​\(x\)\+∑i=1SEis​\(x\)y=\\sum^\{N\}\_\{i=1\}G\_\{i\}\(x\)E\_\{i\}\(x\)\+\\sum^\{S\}\_\{i=1\}E\_\{i\}^\{s\}\(x\), whereEsE^\{s\}represents the shared experts,SSis the number of the shared experts\.

### 3\.2Cross\-domain Experts Selection

As discussed in the Introduction, we aim to identify experts that encode cross\-domain knowledge closely tied to model generalization\. Such experts are expected to be consistently activated across inputs from multiple domains, indicating that they capture knowledge shared beyond a single task or domain\. Accordingly, the first step of XPERT is to identify common experts with strong cross\-domain activation patterns\. In certain MoE architectures, such as DeepSeekMoE, shared experts are explicitly introduced and activated across all inputs\. We directly treat these shared experts as common experts\. For more general MoE settings where shared experts are not predefined, we introduce an automatic selection criterion based on expert activation statistics\.

As shown in Step 1 of Figure[2](https://arxiv.org/html/2605.08842#S1.F2), we begin by utilizingMMdifferent domain\-specific datasetsD1,D2,…,DMD\_\{1\},D\_\{2\},\\dots,D\_\{M\}, each corresponding to a different domain\. We define the activation probability of the expertiion the datasetDmD\_\{m\}asPi,mP\_\{i,m\}\.Pi,mP\_\{i,m\}reflects the usage probability of theii\-th expert on domainmm, that is, the proportion of the number of times the expertiiis activated to the total number of activation of all experts\.

To identify experts that are active across multiple domains, we compute the average activation level of expertiiacross all datasets:

Ai=1M​∑m=1MPi,m\.\\displaystyle A\_\{i\}=\\frac\{1\}\{M\}\\sum^\{M\}\_\{m=1\}P\_\{i,m\}\.\(1\)
In the above function, a higherAiA\_\{i\}indicates that expertiiis frequently activated across multiple domains, suggesting that it encapsulates a greater amount of shared knowledge among domains\. Furthermore, to avoid selecting experts that exhibit high activation in only a few domains while remaining inactive in others, we also consider the consistency score, which measures the balance of an expert’s activation across different domains\. A lower consistency score of expertiiimplies that the activation of expertiiis relatively balanced across all the domains, rather than being highly responsive to only a few domains\. The consistency score of the expertiiis:

Ci=1M​∑m=1M\|Pi,m−Ai\|\.\\displaystyle C\_\{i\}=\\frac\{1\}\{M\}\\sum^\{M\}\_\{m=1\}\\lvert P\_\{i,m\}\-A\_\{i\}\\rvert\.\(2\)
Finally, by combining the average activation level of experts across all datasets with their activation consistency, we identify the experts with cross\-domain common knowledge:ε=T​o​p​K​\(\{Ai−Ci\|1≤i≤N\},n\)\\varepsilon=TopK\(\\\{A\_\{i\}\-C\_\{i\}\|1\\leq i\\leq N\\\},\\;n\), wherennrepresents the number of experts with common knowledge to be selected\.ε\\varepsilonis the set of the selected experts with common knowledge\.

### 3\.3Generalizable Expert Knowledge Consolidation

While the experts we selected are good at handling common knowledge across domains, the shared generalizable knowledge they contain still requires further refinement\. To preserve higher\-purity shared knowledge from these experts, we need to further keep the common information between them and remove their differences, which helps create cleaner generalization knowledge\.

Specifically, we employ Tucker decomposition\(Tucker,[1966](https://arxiv.org/html/2605.08842#bib.bib55)\)to extract shared structures from the parameters of the selected experts, aiming to refine generalizable knowledge\. Tucker decomposition factorizes a high\-dimensional tensor into a core tensor and several factor matrices, retaining only partial information along each dimension\. The core tensor obtained through Tucker decomposition captures the relationships between the principal components across different dimensions of the original tensor\. By retaining only a small number of principal components, the impact of noise from individual experts is reduced, allowing the core tensor to effectively capture key commonalities among experts \(more detailed analysis is given in the Appendix[D](https://arxiv.org/html/2605.08842#A4)\)\.

First, we stack the expert matrices together to form a higher\-order tensorZ∈ℝdi×do×nZ\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{o\}\\times n\}\. Here,did\_\{i\}anddod\_\{o\}are the dimensions of the expert matrices from the LLM\. For example,did\_\{i\}could be the embedding dimension in the FFN layer, anddod\_\{o\}might be the intermediate size in the same FFN layer\.

As illustrated in Step 2 of Figure[2](https://arxiv.org/html/2605.08842#S1.F2), we employ the Tucker decomposition to the tensorZZ:

𝒢,U\\displaystyle\\mathcal\{G\},\\;U=T​u​c​k​e​r​\(Z,\(R1,R2,R3\)\),\\displaystyle=Tucker\(Z,\\;\(R\_\{1\},\\;R\_\{2\},\\;R\_\{3\}\)\),\(3\)where𝒢∈ℝR1×R2×R3\\mathcal\{G\}\\in\\mathbb\{R\}^\{R\_\{1\}\\times R\_\{2\}\\times R\_\{3\}\}represents the core tensor,U=\(U1,U2,U3\)U=\(U\_\{1\},\\;U\_\{2\},\\;U\_\{3\}\)andUiU\_\{i\}denotes the factor matrix along theii\-th dimension\. The valuesR1R\_\{1\},R2R\_\{2\}, andR3R\_\{3\}are hyperparameters that determine the rank of the principal components retained in each dimension\. The resulting core tensor and factor matrices together form a compact representation of generalization and common knowledge, refined from the selected experts\. In practice, this consolidated representation accounts for only a small fraction of the parameters of the source MoE model \(e\.g\., approximately1\.25%in OLMoE\-7B\), while preserving the dominant generalizable structures shared across experts\.

### 3\.4Parameter\-scale Adaptation and Initialization

After refining common expert knowledge through tensor decomposition, the final step of XPERT is to adapt the consolidated representation to the parameter dimensions of target models and use it for initialization\. This step ensures that expert knowledge extracted from a source MoE LLM can be flexibly reused across models with different dimensions and sizes\.

As illustrated in Step 3 of Figure[2](https://arxiv.org/html/2605.08842#S1.F2), the consolidation step produces a core tensor𝒢∈ℝR1×R2×R3\\mathcal\{G\}\\in\\mathbb\{R\}^\{R\_\{1\}\\times R\_\{2\}\\times R\_\{3\}\}and factor matricesU=\(U1,U2,U3\)U=\(U\_\{1\},U\_\{2\},U\_\{3\}\)\. Since the third mode corresponds to the expert dimension, we first aggregate expert\-specific components by averaging:U3¯=1/n​∑i=1nU3​\[:,i\]\\bar\{U\_\{3\}\}=1/n\\sum^\{n\}\_\{i=1\}U\_\{3\}\[:,i\], whereU¯3∈ℝR3×1\\bar\{U\}\_\{3\}\\in\\mathbb\{R\}^\{R\_\{3\}\\times 1\}\. We then reconstruct the consolidated FFN parameter matrix via mode\-wise multiplication:

Γ=𝒢×1U1×2U2×3U3¯,\\displaystyle\\Gamma=\\mathcal\{G\}\\;\\times\_\{1\}\\;U\_\{1\}\\;\\times\_\{2\}\\;U\_\{2\}\\;\\times\_\{3\}\\;\\bar\{U\_\{3\}\},\(4\)whereΓ∈ℝdi×do\\Gamma\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{o\}\}\. For each blocklland FFN projectionjj, the reconstructed matrixΓl,j\\Gamma\_\{l,j\}is then resized via parameter\-scale adaptation to match the target FFN dimensions and then used for FFN initialization\.

To operationalize this resizing step, we introduce a parameter\-scale adaptation mechanism that maps a reconstructed matrix to arbitrary target FFN dimensions while preserving its dominant structure\. A naive approach would directly truncate rows or columns when reducing dimensionality, or replicate and pad parameters when expanding dimensionality\. However, such operations ignore the internal organization of the parameter matrix and can severely disrupt the structural patterns and semantic information encoded in the weights, leading to substantial degradation in downstream performance\.

Instead, we design a structure\-preserving adaptation mechanism that aims to retain as much informative content as possible while flexibly adjusting matrix size\. Concretely, given a reconstructed matrixΓ∈ℝdi×do\\Gamma\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{o\}\}and target dimensions\(di′,do′\)\(d\_\{i\}^\{\\prime\},d\_\{o\}^\{\\prime\}\), we adapt it toM^∈ℝdi′×do′\\hat\{M\}\\in\\mathbb\{R\}^\{d\_\{i\}^\{\\prime\}\\times d\_\{o\}^\{\\prime\}\}in two stages: \(i\) low\-rank factorization with importance scoring to identify structurally salient components, and \(ii\) importance\-guided row and column resampling that contracts or expands the matrix while preserving its dominant subspace\. This design enables XPERT to reuse expert\-derived knowledge across models of different sizes without destroying the underlying parameter structure\.

#### Stage 1\. Factorization and scoring

We perform a truncated singular value decomposition \(SVD\) onΓ\\Gamma:Γ≈U​Σ​V⊤\\Gamma\\approx U\\Sigma V^\{\\top\}, whereU∈ℝdi×rU\\in\\mathbb\{R\}^\{d\_\{i\}\\times r\},Σ∈ℝr×r\\Sigma\\in\\mathbb\{R\}^\{r\\times r\},V∈ℝdo×rV\\in\\mathbb\{R\}^\{d\_\{o\}\\times r\}, andrris the number of retained principal components\. We compute importance scores for rows ofUUandVV:

si\(U\)=\|ui\|2,i=1,…,di;sj\(V\)=\|vj\|2,j=1,…,do,\\displaystyle s\_\{i\}^\{\(U\)\}=\|u\_\{i\}\|\_\{2\},\\;i=1,\\dots,d\_\{i\};\\quad s\_\{j\}^\{\(V\)\}=\|v\_\{j\}\|\_\{2\},\\;j=1,\\dots,d\_\{o\},whereuiu\_\{i\}denotes theii\-th row ofUUandvjv\_\{j\}denotes thejj\-th row ofVV\.

#### Stage 2\. Resampling to target size

For the row dimension, whendi′≤did\_\{i\}^\{\\prime\}\\leq d\_\{i\}, we select the indices of the top\-pprows by score:

ℐ=TopK​\(\{si\(U\)\},di′\),U′=U​\[ℐ,:\]∈ℝdi′×r\.\\displaystyle\\mathcal\{I\}=\\mathrm\{TopK\}\(\\\{s\_\{i\}^\{\(U\)\}\\\},d\_\{i\}^\{\\prime\}\),\\;\\;U^\{\\prime\}=U\[\\mathcal\{I\},:\]\\in\\mathbb\{R\}^\{d\_\{i\}^\{\\prime\}\\times r\}\.Whendi′\>did\_\{i\}^\{\\prime\}\>d\_\{i\}, we replicate the most important\(di′−di\)\(d\_\{i\}^\{\\prime\}\-d\_\{i\}\)rows and append them:

ℐ=TopK​\(\{si\(U\)\},di′−di\),U′=\[UU​\[ℐ,:\]\]∈ℝdi′×r\.\\displaystyle\\mathcal\{I\}=\\mathrm\{TopK\}\(\\\{s\_\{i\}^\{\(U\)\}\\\},d\_\{i\}^\{\\prime\}\-d\_\{i\}\),\\;\\;U^\{\\prime\}=\\left\[\\begin\{matrix\}U\\\\ U\[\\mathcal\{I\},:\]\\end\{matrix\}\\right\]\\in\\mathbb\{R\}^\{d\_\{i\}^\{\\prime\}\\times r\}\.The row dimension is handled analogously usingsj\(V\)\{s\_\{j\}^\{\(V\)\}\}to obtainV′∈ℝdo′×rV^\{\\prime\}\\in\\mathbb\{R\}^\{d\_\{o\}^\{\\prime\}\\times r\}\. Finally, we reconstruct the resized matrix asM^=U′​Σ​V′⁣⊤∈ℝdi′×do′\\hat\{M\}=U^\{\\prime\}\\Sigma V^\{\\prime\\top\}\\in\\mathbb\{R\}^\{d\_\{i\}^\{\\prime\}\\times d\_\{o\}^\{\\prime\}\}\.

The resulting matrixM^\\hat\{M\}is then used to initialize the corresponding FFN weight matrix of the target model, while all remaining parameters are initialized with standard random initialization\. This adaptation is entirely training\-free and enables XPERT to reuse consolidated expert knowledge across models with different FFN dimensions\.

Table 1:Results of models with different scales on language understanding benchmarks\. We evaluate models of different dimensions and depths\. All baselines are first pre\-trained on 5B tokens, and fine\-tuned for 3 epochs on these datasets\.\#Params\#SizeBaselineCommonsense & Reading ComprehensionLawMedicineAvg\.BoolQHellaswagWinoGrandePIQACaseHoldMedMCQA270Mdim=102416 LayersScratch71\.5625\.2149\.6451\.9680\.9332\.3551\.94Distillation72\.6626\.2849\.6452\.3982\.5434\.0752\.93XPERT\-OLMoE73\.2126\.9950\.7554\.4683\.1135\.0553\.93XPERT\-DeepSeek73\.2426\.5350\.6753\.4383\.1335\.1253\.69391Mdim=20488 LayersScratch70\.8326\.9549\.0950\.0078\.5032\.0951\.24Distillation71\.3226\.2149\.4150\.2280\.4832\.3051\.66XPERT\-OLMoE72\.6327\.2750\.5951\.8081\.5034\.1652\.99XPERT\-DeepSeek71\.5627\.1550\.3652\.7781\.1034\.5052\.91480Mdim=204812 LayersScratch71\.5626\.8048\.8651\.9080\.9732\.7852\.14Distillation71\.6226\.4049\.1750\.8781\.3532\.3951\.97XPERT\-OLMoE73\.7627\.4050\.3655\.2882\.9035\.0454\.12XPERT\-DeepSeek72\.3526\.4750\.1251\.2582\.1235\.5052\.97570Mdim=204816 LayersScratch70\.8626\.2948\.6251\.8980\.1633\.0451\.81Distillation71\.7726\.4748\.8652\.6881\.3533\.5252\.44XPERT\-OLMoE73\.9127\.5649\.9655\.4482\.9634\.7854\.10XPERT\-DeepSeek72\.1427\.4450\.7554\.8482\.5434\.4753\.70

## 4Experiments

Our experiments aim to evaluate whether expert knowledge extracted from MoE LLMs can be effectively reused to support more efficient and robust model training across tasks and domains\. We assess XPERT from two perspectives in datasets:task generality, covering both language understanding and dialogue generation, anddomain generality, spanning multiple vertical knowledge domains\. Across all experiments, we focus on comparing different initialization strategies under controlled training settings\.

### 4\.1Baselines and Datasets

Baselines\.XPERT\-initialized models are dense language models following the Llama architecture, instantiated with varying model dimensions and depths\. To ensure fair comparison, all baselines adopt the same model architecture and training configuration\. We compare XPERT with representative baselines that differ in how external knowledge is introduced into the model under an identical architecture and training pipeline\.Scratchtrains the target model from standard random initialization without any external knowledge transfer\.Distillation\(Guet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib50)\)pre\-trains the target model from random initialization using knowledge distillation from a source LLM \(OLMoE\-7B\)\. This baseline serves as a control to examine whether the benefits of XPERT can be attributed solely to transferring knowledge through teacher–student supervision, rather than through parameter\-level knowledge reuse\.XPERT\-OLMoEinitializes the target model using expert knowledge extracted from OLMoE\-7B\(Muennighoffet al\.,[2025](https://arxiv.org/html/2605.08842#bib.bib43)\)\.XPERT\-DeepSeekinitializes the target model using expert knowledge extracted from DeepSeekMoE\-16B\(Daiet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib44)\)\.

For fair comparison, models with the same scale have identical parameter counts and are pre\-trained on the same number of tokens\. After pre\-training, each model is fine\-tuned independently on individual SFT datasets, allowing evaluation on task\-specific performance and training dynamics\.

Datasets\.We evaluate XPERT on a diverse set of SFT tasks defined along two dimensions: task type and knowledge domain\. For language understanding, we adopt widely used benchmarks includingBoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2605.08842#bib.bib45)\),HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2605.08842#bib.bib46)\),PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2605.08842#bib.bib48)\),WinoGrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2605.08842#bib.bib49)\),CaseHold\(Zhenget al\.,[2021](https://arxiv.org/html/2605.08842#bib.bib57)\), andMedMCQA\(Palet al\.,[2022](https://arxiv.org/html/2605.08842#bib.bib58)\), which cover reasoning, commonsense, and domain\-specific understanding\. For dialogue generation, we follow the evaluation setup of MiniLLM\(Guet al\.,[2024](https://arxiv.org/html/2605.08842#bib.bib50)\)and evaluate onDolly111https://github\.com/databrickslabs/dolly/tree/master,S\-NI\(Wanget al\.,[2022](https://arxiv.org/html/2605.08842#bib.bib51)\),UnNI\(Honovichet al\.,[2023](https://arxiv.org/html/2605.08842#bib.bib52)\),SelfInst\(Wanget al\.,[2023b](https://arxiv.org/html/2605.08842#bib.bib53)\), andVicuna\(Chianget al\.,[2023](https://arxiv.org/html/2605.08842#bib.bib54)\)\.

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/histogram_dolly.png)

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/histogram_uinst.png)

Figure 3:Comparison of fine\-tuning performance between 16\-layer XPERT\-OLMoE and Scratch under varying pre\-training data budgets \(2B, 5B, and 10B tokens\)\.
### 4\.2Main Results

To comprehensively evaluate the effectiveness of XPERT in reusing expert knowledge for model training, we conduct experiments from three perspectives:downstream task performance,fine\-tuning convergence behavior, androbustness under different pre\-training budgets\.

Table 2:Results of models with different scales on dialogue generation benchmarks\. For evaluating generative capabilities of models, models \(pre\-trained on 5B tokens\) are first fine\-tuned on the Dolly dataset, and evaluated on the datasets shown in the table\.\#Params\#SizeBaselineDialogue GenerationAvg\.\(Rouge\-L\)DollyEvalS\-NIUnNISelfInstVicuna270Mdim=102416 LayersScratch21\.3114\.6719\.329\.4913\.7715\.73Distillation22\.4315\.0020\.509\.2613\.5216\.14XPERT\-OLMoE23\.6716\.5821\.558\.4414\.4616\.94XPERT\-DeepSeek23\.2215\.2021\.339\.5613\.2516\.51391Mdim=20488 LayersScratch22\.9415\.6918\.168\.5014\.2515\.91Distillation22\.7816\.6318\.148\.3413\.0115\.78XPERT\-OLMoE23\.1717\.3921\.479\.1815\.0517\.25XPERT\-DeepSeek23\.3517\.3420\.028\.7213\.9916\.68480Mdim=204812 LayersScratch22\.4816\.5720\.769\.4114\.2016\.68Distillation22\.9015\.7021\.479\.0114\.3916\.69XPERT\-OLMoE23\.3619\.4321\.839\.5714\.8317\.80XPERT\-DeepSeek23\.3818\.0120\.4810\.3013\.9217\.22570Mdim=204816 LayersScratch22\.8615\.6119\.678\.3113\.4215\.97Distillation22\.0916\.0620\.2710\.2014\.0916\.54XPERT\-OLMoE24\.1919\.8222\.6011\.3114\.5718\.50XPERT\-DeepSeek23\.4217\.9923\.2010\.4314\.3717\.88

Superior performance across downstream benchmarks\.Table[1](https://arxiv.org/html/2605.08842#S3.T1)and Table[2](https://arxiv.org/html/2605.08842#S4.T2)summarize the results on language understanding and dialogue generation benchmarks, respectively\. Across a broad range of tasks and domains, models trained with XPERT consistently outperform strong baselines, including Scratch and Distillation\. For instance, on the S\-NI generation task, the 16\-layer XPERT model \(570M\) improves over Scratch by4\.21and over Distillation by3\.76\.

Prior work shows that knowledge distillation degrades when there is a large capacity gap between teacher and student models\(Mirzadehet al\.,[2020](https://arxiv.org/html/2605.08842#bib.bib39); Gaoet al\.,[2020](https://arxiv.org/html/2605.08842#bib.bib15); Wang and Yoon,[2021](https://arxiv.org/html/2605.08842#bib.bib16)\)\. In our experiments, we use OLMoE\-7B as the teacher model for all distillation baselines to eliminate confounding factors from different source models with XPERT baselines\. Under the same teacher and training setup, XPERT achieves consistently better results than knowledge distillation\.

This suggests that the knowledge encoded in MoE LLMs is difficult to transfer effectively through supervision alone\. In contrast, XPERT reuses knowledge at the expert parameter level by explicitly identifying and consolidating expert representations closely related to generalization\. This enables more precise and stable knowledge transfer that better matches the capacity of the target model\. In addition, XPERT avoids repeated forward passes through the teacher model, resulting in substantially lower training cost compared to distillation\. Additional results are provided in Appendix[C](https://arxiv.org/html/2605.08842#A3)\.

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/RougeL.png)

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/Loss.png)

Figure 4:Comparison of Rouge\-L and loss curves between Scratch and XPERT\-initialized models on the DollyEval dataset\.Faster convergence during fine\-tuning\.To examine how expert knowledge reuse influences optimization behavior, we compare the fine\-tuning dynamics of XPERT\-OLMoE and Scratch\. Figure[4](https://arxiv.org/html/2605.08842#S4.F4)presents the Rouge\-L scores and training loss curves on the Dolly dataset for 16\-layer models \(570M\)\. Models trained with XPERT reach the same Rouge\-L performance approximately2\.4×\\timesearlier than Scratch and achieve comparable loss values about1\.7×\\timesfaster\. This accelerated convergence indicates that reusing expert\-level knowledge shapes a more favorable optimization landscape, allowing the model to exploit training signals more efficiently from the early stages of fine\-tuning\.

The faster convergence observed during downstream fine\-tuning indicates that the reused cross\-domain expert knowledge provides a strong and well\-aligned inductive bias, enabling models to adapt more efficiently to task\-specific supervision\. By initializing models with expert representations that already encode generalizable structure, XPERT reduces the amount of optimization needed to rediscover such knowledge during SFT, leading to faster convergence and improved training efficiency\.

Table 3:Ablation results for different modules in XPERT\. For each ablation variant, only the specified component is modified, while all other components follow the same configuration as the full XPERT pipeline\. WinoG\. denotes the Winogrande dataset\.MethodBoolQHellaswagPIQAWinoG\.Avg\.Experts Selection AblationTop\-1 expert71\.6826\.7152\.0748\.8649\.83Random Selection71\.8926\.9753\.3248\.4650\.16Knowledge Consolidation AblationAverage Aggregation72\.0226\.7351\.4148\.2249\.60CP Decomposition70\.7025\.8151\.5849\.7249\.45SVD70\.0926\.7252\.3449\.5749\.68Parameter\-scale Adaptation AblationTruncating71\.3524\.7352\.1249\.3349\.38XPERT73\.9127\.5655\.4450\.7551\.92

XPERT reduces the amount of pre\-training data needed to reach the same performance\.Figure[3](https://arxiv.org/html/2605.08842#S4.F3)reports fine\-tuning results on the UnNI dataset for 16\-layer 570M models pre\-trained with different amounts of data, while keeping the model scale and training pipeline fixed\. Across all settings, XPERT\-OLMoE consistently outperforms the Scratch baseline\. Notably, on the UnNI dataset, a Scratch model pre\-trained with 10B tokens fails to match the performance of XPERT\-OLMoE pre\-trained with only 2B tokens\. These results indicate that XPERT substantially lowers the amount of pre\-training data required to reach a given downstream performance under identical training pipelines\. Compared with Scratch, XPERT provides a stronger starting point that enables the model to benefit more effectively from the same training signal, leading to performance gains comparable to several\-fold increases in pre\-training data\. Additional results across tasks are provided in Appendix[E](https://arxiv.org/html/2605.08842#A5)\.

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/histgram_pretrain_data.png)Figure 5:Effect of domain diversity in expert selection on downstream SFT performance of XPERT\-initialized models\.Mixed\-Domainuses a diverse corpus spanning multiple knowledge domains, including \(but not limited to\) Wikipedia, GitHub, and arXiv\.
### 4\.3Ablation Studies

#### Component\-wise Ablation of XPERT

Table[3](https://arxiv.org/html/2605.08842#S4.T3)presents ablation results for the core components of XPERT:

- •Expert selection\.Top\-1 expertleads to clear performance degradation, indicating that reusing a single expert is insufficient to capture the shared structure underlying cross\-domain generalizable knowledge\.Random Selectionalso results in degraded performance, as it introduces domain\-specialized experts whose representations encode task\-specific patterns instead of cross\-domain universal structure\.
- •Knowledge consolidation\.Replacing Tucker decomposition with simpler or alternative strategies, includingAverage Aggregation,CP Decomposition, orSVD, consistently reduces performance\. Direct averaging retains expert\-specific variations and noise, while CP and SVD impose more restrictive factorization forms that fail to preserve the multi\-dimensional shared structure across experts and parameter dimensions, resulting in weaker initialization\.
- •Parameter\-scale adaptation\.Removing the proposed adaptation mechanism and directly truncating reconstructed parameters to match target dimensions \(Truncating\) also harms performance, confirming that naive resizing disrupts the structural integrity of expert representations\.

Overall, XPERT achieves the best results across all tasks, demonstrating that cross\-domain expert selection, knowledge consolidation, and parameter\-scale adaptation jointly play a critical role in effective expert knowledge reuse\.

#### Effect of Domain Choice for Expert Selection

We study how the choice of data domains influences expert selection by identifying experts using different types of domain\-specific inputs \(Figure[5](https://arxiv.org/html/2605.08842#S4.F5)\)\. The results indicate thatMixed\-Domain, which is the expert selection strategy adopted by XPERT, consistently leads to stronger downstream SFT performance than using data from any single domain\. Selecting experts based on heterogeneous data encourages the identification of experts that are active across domains, which are more likely to encode shared, domain\-agnostic knowledge relevant to generalization\. In contrast, experts selected from a single domain tend to over\-specialize, resulting in weaker transfer across downstream tasks\. These findings underscore the importance of domain diversity in expert selection for extracting reusable expert knowledge\.

#### Impact of the Number of Experts and Decomposition Rank

Table[4](https://arxiv.org/html/2605.08842#S4.T4)shows that the number of selected experts has a noticeable impact on downstream performance\. Selecting a moderate number of experts yields the best results: usingK=8K=8experts consistently outperforms smaller settings, while further increasing the number of experts toK=16K=16leads to degraded performance\. This non\-monotonic trend suggests that selecting too many experts may introduce domain\-specific or task\-dependent information, which dilutes the purity of the extracted, generalizable knowledge\.

The effect of the Tucker rank is comparatively less pronounced\. Moderate rank settings are sufficient to preserve task\-relevant representations, while overly large ranks provide limited additional benefit\. Overall, these results suggest that effective expert reuse benefits from selecting a moderate number of experts and using compact factorization\.

Table 4:Downstream performance under different numbers of selected common experts and Tucker ranks\.Top\-KSelected\#RankBoolQHellaswagPIQAWinoG\.Avg\.2\(256, 512, 2\)73\.3127\.3452\.0748\.7050\.36\(512, 1024, 2\)72\.0527\.4052\.9448\.8650\.318\(256, 512, 8\)73\.9127\.5655\.4449\.9651\.71\(512, 1024, 8\)73\.0027\.6653\.9052\.3351\.7216\(256, 512, 16\)71\.9028\.2353\.9749\.6450\.93\(512, 1024, 16\)73\.0628\.0752\.4750\.0450\.91

## 5Conclusion

In this work, we show that MoE LLMs contain experts that are consistently activated across domains, revealing cross\-domain, generalizable\-relevant expert knowledge\. Building on this observation, we propose XPERT, a training\-free framework that extracts and reuses such expert knowledge to support effective model training across different scales\. Experiments demonstrate that expert knowledge reuse improves downstream performance and accelerates convergence, highlighting the potential of MoE LLMs as structured and reusable knowledge sources\.

## Impact Statement

This paper presents work whose goal is to advance the understanding of expert knowledge reuse in Mixture\-of\-Experts language models and to improve training efficiency of downstream models with diverse scales\. As with other advances in language model training, this work may have broader societal implications related to the use of language technologies\. However, we do not identify any ethical concerns or societal impacts that are unique to the methods proposed in this paper beyond those already well established in prior research on large language models\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez, I\. Stoica, and E\. P\. Xing \(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality\.External Links:[Link](https://lmsys.org/blog/2023-03-30-vicuna/)Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.\(2023\)Palm: scaling language modeling with pathways\.Journal of Machine Learning Research24\(240\),pp\. 1–113\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of NAACL\-HLT,pp\. 2924–2936\.Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- D\. Dai, C\. Deng, C\. Zhao, R\. X\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu, Z\. Xie, Y\. K\. Li, P\. Huang, F\. Luo, C\. Ruan, Z\. Sui, and W\. Liang \(2024\)DeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.External Links:2401\.06066,[Link](https://arxiv.org/abs/2401.06066)Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1),[§1](https://arxiv.org/html/2605.08842#S1.p3.1),[§1](https://arxiv.org/html/2605.08842#S1.p7.1),[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2605.08842#S3.SS1.p2.3),[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Frankle and M\. Carbin \(2018\)The lottery ticket hypothesis: finding sparse, trainable neural networks\.arXiv preprint arXiv:1803\.03635\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Gao, Y\. Shen, Q\. Li, and C\. C\. Loy \(2020\)Residual knowledge distillation\.arXiv preprint arXiv:2002\.09168\.Cited by:[§4\.2](https://arxiv.org/html/2605.08842#S4.SS2.p3.1)\.
- X\. Glorot and Y\. Bengio \(2010\)Understanding the difficulty of training deep feedforward neural networks\.InProceedings of the thirteenth international conference on artificial intelligence and statistics,pp\. 249–256\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2015\)Delving deep into rectifiers: surpassing human\-level performance on imagenet classification\.InProceedings of the IEEE international conference on computer vision,pp\. 1026–1034\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px3.p1.1)\.
- O\. Honovich, T\. Scialom, O\. Levy, and T\. Schick \(2023\)Unnatural instructions: tuning language models with \(almost\) no human labor\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 14409–14428\.Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly \(2019\)Parameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px2.p2.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px2.p2.1)\.
- Q\. Huang, Z\. An, N\. Zhuang, M\. Tao, C\. Zhang, Y\. Jin, K\. Xu, L\. Chen, S\. Huang, and Y\. Feng \(2024\)Harder tasks need more experts: dynamic routing in moe models\.arXiv preprint arXiv:2403\.07652\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen \(2020\)Gshard: scaling giant models with conditional computation and automatic sharding\.arXiv preprint arXiv:2006\.16668\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px1.p1.1)\.
- X\. L\. Li and P\. Liang \(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.arXiv preprint arXiv:2101\.00190\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px2.p2.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1),[§3\.1](https://arxiv.org/html/2605.08842#S3.SS1.p2.3)\.
- B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal,et al\.\(2020\)Language models are few\-shot learners\.arXiv preprint arXiv:2005\.141651,pp\. 3\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- S\. I\. Mirzadeh, M\. Farajtabar, A\. Li, N\. Levine, A\. Matsukawa, and H\. Ghasemzadeh \(2020\)Improved knowledge distillation via teacher assistant\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 5191–5198\.Cited by:[§4\.2](https://arxiv.org/html/2605.08842#S4.SS2.p3.1)\.
- D\. Mishkin and J\. Matas \(2015\)All you need is a good init\.arXiv preprint arXiv:1511\.06422\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Molchanov, A\. Mallya, S\. Tyree, I\. Frosio, and J\. Kautz \(2019\)Importance estimation for neural network pruning\.External Links:1906\.10771,[Link](https://arxiv.org/abs/1906.10771)Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, N\. A\. Smith, P\. W\. Koh, A\. Singh, and H\. Hajishirzi \(2025\)OLMoE: open mixture\-of\-experts language models\.External Links:2409\.02060,[Link](https://arxiv.org/abs/2409.02060)Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1),[§1](https://arxiv.org/html/2605.08842#S1.p7.1),[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p1.1)\.
- A\. Pal, L\. K\. Umapathi, and M\. Sankarasubbu \(2022\)MedMCQA: a large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.InProceedings of the Conference on Health, Inference, and Learning,G\. Flores, G\. H\. Chen, T\. Pollard, J\. C\. Ho, and T\. Naumann \(Eds\.\),Proceedings of Machine Learning Research, Vol\.174,pp\. 248–260\.External Links:[Link](https://proceedings.mlr.press/v174/pal22a.html)Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- N\. Shazeer, \*\. Mirhoseini, \*\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017a\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017b\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Shi, S\. Xia, X\. Yang, H\. Chen, Z\. Kou, and X\. Geng \(2024\)Building variable\-sized models via learngene pool\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 14946–14954\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p4.1)\.
- H\. Sun, D\. Zhou, Y\. Li, S\. Lu, C\. Yi, Q\. Chen, Z\. Xu, W\. Luo, K\. Zhang, D\. Zhan,et al\.\(2024\)Parrot: multilingual visual instruction tuning\.CoRR\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- G\. Team, P\. Georgiev, V\. I\. Lei, R\. Burnell, L\. Bai, A\. Gulati, G\. Tanzer, D\. Vincent, Z\. Pan, S\. Wang,et al\.\(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1),[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- L\. R\. Tucker \(1966\)Some mathematical notes on three\-mode factor analysis\.Psychometrika31\(3\),pp\. 279–311\.Cited by:[§3\.3](https://arxiv.org/html/2605.08842#S3.SS3.p2.1)\.
- L\. Wang and K\. Yoon \(2021\)Knowledge distillation and student\-teacher learning for visual intelligence: a review and new outlooks\.IEEE transactions on pattern analysis and machine intelligence44\(6\),pp\. 3048–3068\.Cited by:[§4\.2](https://arxiv.org/html/2605.08842#S4.SS2.p3.1)\.
- Q\. Wang, X\. Yang, S\. Lin, J\. Wang, and X\. Geng \(2023a\)Learngene: inheriting condensed knowledge from the ancestry model to descendant models\.arXiv preprint arXiv:2305\.02279\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p4.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023b\)Self\-instruct: aligning language models with self\-generated instructions\.InACL,Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap,et al\.\(2022\)Super\-naturalinstructions: generalization via declarative instructions on 1600\+ nlp tasks\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 5085–5109\.Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- W\. F\. Wiggins and A\. S\. Tejani \(2022\)On the opportunities and risks of foundation models for natural language processing in radiology\.Radiology: Artificial Intelligence4\(4\),pp\. e220119\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- Z\. Xu, S\. Guo, Y\. Wang, R\. Chen, H\. Li, X\. Li, and H\. Jiang \(2024\)Code comment inconsistency detection based on confidence learning\.IEEE Transactions on Software Engineering50\(3\),pp\. 598–617\.Cited by:[§2](https://arxiv.org/html/2605.08842#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Yi, Y\. He, D\. Zhan, and H\. Ye \(2024\)Bridge the modality and capability gaps in vision\-language model selection\.Advances in Neural Information Processing Systems37,pp\. 34429–34452\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- Y\. Zhang, S\. Lu, Y\. Li, Y\. Ma, Q\. Chen, Z\. Xu, W\. Luo, K\. Zhang, D\. Zhan, and H\. Ye \(2024\)Wings: learning multimodal llms without text\-only forgetting\.Advances in Neural Information Processing Systems37,pp\. 31828–31853\.Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.
- L\. Zheng, N\. Guha, B\. R\. Anderson, P\. Henderson, and D\. E\. Ho \(2021\)When does pretraining help? assessing self\-supervised learning for law and the casehold dataset of 53,000\+ legal holdings\.InProceedings of the eighteenth international conference on artificial intelligence and law,pp\. 159–168\.Cited by:[§4\.1](https://arxiv.org/html/2605.08842#S4.SS1.p3.1)\.
- X\. Zhong, C\. Yi, and H\. Ye \(2025\)Efficient evaluation of large language models via collaborative filtering\.External Links:2504\.08781,[Link](https://arxiv.org/abs/2504.08781)Cited by:[§1](https://arxiv.org/html/2605.08842#S1.p1.1)\.

## Appendix AExperts Frequency in OLMoE\-7B

In the Introduction[1](https://arxiv.org/html/2605.08842#S1), we present the activation frequency of experts in the FFN layer of Layer 15 in the OLMoE\-7B model across different domains \(Github, Tulu, and Arxiv\)\. Here, we provide additional visualizations\. Figure[6](https://arxiv.org/html/2605.08842#A1.F6)shows the activation frequencies of experts in Layer 0 and Layer 7 of the OLMoE\-7B model across various domains\. It can be observed that different experts within the same layer exhibit similar activation trends across these domains\. For example, Expert 0, 21, and 40 in Layer 0, as well as Expert 2, 4, 17, 35, and 58 in Layer 7, all maintain high activation across multiple domains\. This phenomenon indicates that MoE architectures inherently suffer from load imbalance among experts\. Experts that are consistently activated across domains likely encode knowledge shared by multiple domains\. We consider these experts to contain common knowledge that can be applied to a variety of tasks\.

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/layer0_expert_freq.png)

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/layer7_expert_freq.png)

Figure 6:Expert activation frequencies of layer 0 \(left\) and layer 7 \(right\) in OLMoE\-7B across different domains\.
## Appendix BExperimental Details

We set the embedding dimension of the target models to 2048, with an FFN intermediate size of 1024\. To evaluate the effect of model scale, we experiment with models of 8, 12, and 16 transformer layers\. Since expert knowledge is extracted independently from each block of the source MoE LLM, the resulting representations are organized in a block\-wise manner\. Accordingly, models with different depths are initialized using expert\-derived parameters extracted from the corresponding number of blocks in the source model \(e\.g\., an 8\-layer model uses parameters from the first 8 blocks\)\. All baseline models are subsequently trained using identical pre\-training procedures and the same number of tokens to ensure fair comparison\. After pre\-training, the models are further fine\-tuned on the aforementioned SFT datasets for evaluation\.

For fine\-tuning on the BoolQ, Hellaswag, MMLU, PIQA, and WinoGrande language understanding datasets, all baselines are trained for only 3 epochs\. For evaluating generative capabilities of models, we follow the MiniLLM setting: models are first trained on the Dolly dataset for 100 epochs, and then evaluated on the DollyEval, S\-NI, UnNI, SelfInst, and VicunaEval datasets\.

Datasets and code\.We use the RedPajama\-V2 as our pre\-training data, and the fine\-tuning datasets for downstream tasks are presented in Section[4](https://arxiv.org/html/2605.08842#S4)\. As for the source code, we provide an anonymous link, check\.

Software Environment\.All experiments were conducted using PyTorch 2\.1 with CUDA version 12\.1 and 4×H100 GPUs\.

Model Configuration\.The LLMs are OLMoE\-7B and DeepSeekMoE\-16B, and the target models are Llama models \(dense\) with different sizes\.

Baseline Design Details\.In addition to the baselines based on XPERT \(XPERT\-OLMoEandXPERT\-DeepSeek\), we also includeScratchandDistillationas comparative baselines\. Specifically,Scratchrefers to a model that is randomly initialized, pre\-trained, and then fine\-tuned on downstream tasks\.Distillationdenotes a model that undergoes pre\-training with distillation from the LLM, followed by fine\-tuning on downstream tasks\. To ensure the fairness of our comparisons, all baselines are configured with identical model parameter sizes and are pre\-trained on datasets of the same scale\.

Training Hyperparameters\.The models are pre\-trained using AdamW with an initial learning rate of 4e\-4, batch size of 64, and a total of 20k\-40k pre\-training steps\. During fine\-tuning, we set the learning rate to 3e\-4, and batch size to 8\.

## Appendix CMore Results with Different Amount of Pre\-training Data

In this section, we present the fine\-tuning results of various baselines on downstream datasets after pre\-training with different amounts of data in Table[5](https://arxiv.org/html/2605.08842#A3.T5)and Table[6](https://arxiv.org/html/2605.08842#A3.T6)\. It can be observed that XPERT consistently outperforms other baselines across different tasks, demonstrating a significant advantage\.

Table 5:Results of models with different scales on model generation benchmarks \(2B tokens pre\-trained\)\.ModelBaselineDollyEvalS\-NIUnNISelfInstVicunaEval8 Layers391MScratch21\.9515\.4619\.338\.7113\.51Distillation21\.6715\.4717\.947\.5813\.24XPERT\-OLMoE22\.6416\.2818\.9310\.0913\.83XPERT\-DeepSeek22\.6616\.2820\.179\.3514\.6012 Layers480MScratch22\.3914\.3919\.288\.6513\.81Distillation21\.3114\.3321\.478\.8813\.91XPERT\-OLMoE23\.3616\.4419\.799\.4414\.01XPERT\-DeepSeek22\.7614\.3219\.069\.3814\.1616 Layers570MScratch22\.9716\.7419\.759\.1713\.35Distillation22\.9115\.8418\.858\.7713\.98XPERT\-OLMoE23\.8417\.5021\.979\.7114\.30XPERT\-DeepSeek23\.3116\.7120\.8910\.2613\.24Table 6:Results of models with different scales on model generation benchmarks \(4B tokens pre\-trained\)\.ModelBaselineDollyEvalS\-NIUnNISelfInstVicunaEval8 Layers391MScratch22\.0816\.5219\.828\.7313\.13Distillation21\.9015\.4718\.678\.3313\.14XPERT\-OLMoE22\.6617\.3920\.189\.5914\.33XPERT\-DeepSeek22\.7017\.0419\.748\.2913\.7712 Layers480MScratch22\.8416\.2519\.578\.9614\.87Distillation22\.7316\.6219\.489\.2014\.20XPERT\-OLMoE23\.4917\.9522\.089\.4815\.53XPERT\-DeepSeek23\.7617\.4020\.2410\.4314\.7916 Layers570MScratch22\.7815\.6819\.659\.5314\.05Distillation22\.0916\.0620\.2710\.2014\.09XPERT\-OLMoE23\.5218\.7923\.1610\.3814\.54XPERT\-DeepSeek23\.9017\.3021\.9311\.0514\.19
## Appendix DAnalysis of Tucker Decomposition

In this section, we analyze the theoretical rationality behind using Tucker decomposition to extract shared knowledge among experts for refining the generalizable expert knowledge\. To identify shared knowledge across a group of expert parameter matrices, we first stack them into a third\-order tensor and apply Tucker decomposition to extract compact representations along each dimension\. Formally, given a set of matrices\{Xi∈ℝd1×d2\}i=1n\\\{X\_\{i\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{2\}\}\\\}^\{n\}\_\{i=1\}, we construct a tensor𝒳∈ℝd1×d2×n\\mathcal\{X\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{2\}\\times n\}where𝒳​\(:,:,i\)=Xi\\mathcal\{X\}\(:,:,i\)=X\_\{i\}\. This tensor captures both the internal structure of each matrix and the relationships across matrices\.

We then perform Tucker decomposition:

𝒳≈𝒢×1U1×2U2×3U3¯,\\displaystyle\\mathcal\{X\}\\approx\\mathcal\{G\}\\;\\times\_\{1\}\\;U\_\{1\}\\;\\times\_\{2\}\\;U\_\{2\}\\;\\times\_\{3\}\\;\\bar\{U\_\{3\}\},\(5\)where𝒢\\mathcal\{G\}is a compact core tensor andU1U\_\{1\},U2U\_\{2\},U3U\_\{3\}are orthogonal factor matrices capturing the most important components in each mode\. By retaining only the top principal components \(i\.e\., low\-rank approximation\), Tucker decomposition filters out high\-frequency noise or task\-specific variance while preserving the dominant, shared structures that are consistently present across the matrices\.

Intuitively, if each matrixXiX\_\{i\}can be viewed as a combination of shared contentCCand an instance\-specific perturbationθi\\theta\_\{i\}, i\.e\.,Xi=C\+θiX\_\{i\}=C\+\\theta\_\{i\}, then stacking them as:

𝒳=𝒞\+Θ,𝒞​\(:,:,i\)=C,Θ​\(:,:,i\)=θi\.\\displaystyle\\mathcal\{X\}=\\mathcal\{C\}\+\\Theta,\\;\\;\\;\\;\\mathcal\{C\}\(:,:,i\)=C,\\;\\Theta\(:,:,i\)=\\theta\_\{i\}\.\(6\)Components with lower energy typically correspond to minor variations or idiosyncrasies specific to individual matrices, including random noise\. By discarding these less significant components, the decomposition effectively denoises the representation and emphasizes structural coherence\. If the matrices exhibit common patterns, such as similar structural characteristics or activation behaviors, these shared structures tend to manifest as high\-energy components in the tensor\. Tucker decomposition captures these components through the leading principal directions, allowing us to reconstruct the tensor while retaining information that is commonly embedded across matrices\.

Since the decomposition optimizes the overall reconstruction error across expert matrices, the resulting low\-rank representation preferentially preserves structural components that are consistently shared among experts, while suppressing task\-specific variations and noise\. As a result, the extracted representation emphasizes domain\-general patterns that recur across experts rather than idiosyncratic behaviors tied to individual tasks or domains\. This consolidated expert knowledge can then be reused as an effective initialization prior for target models, guiding subsequent training toward solutions with stronger generalization and improved data efficiency\.

## Appendix EAdditional Results on the Effect of Pre\-training Data Scale

In Section[4\.2](https://arxiv.org/html/2605.08842#S4.SS2), we compare XPERT\-initialized models with Scratch under different pre\-training data scales and show that XPERT provides a strong advantage during subsequent training\. In this section, we present additional results to further examine how this advantage manifests across a broader range of downstream tasks\. Specifically, we evaluate the downstream performance of XPERT\-OLMoE and Scratch after pre\-training on varying amounts of data, ranging from 2B to 5B tokens, on multiple benchmarks including DollyEval, SelfInst, S\-NI, and UnNI\.

As shown in Figure[7](https://arxiv.org/html/2605.08842#A5.F7), XPERT\-initialized models consistently achieve stronger downstream performance than Scratch across all tasks and pre\-training scales\. Notably, models initialized with XPERT and pre\-trained on fewer tokens can reach or exceed the performance of Scratch models trained with substantially more data under the same training procedure\. These results indicate that the performance gains introduced by XPERT initialization are comparable to those obtained by increasing the scale of pre\-training data, highlighting the effectiveness of expert\-based initialization in improving how training signals are utilized\.

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/histogram_Dolly_appendix.png)

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/histogram_SelfInst_appendix.png)

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/histogram_sni_appendix.png)

![Refer to caption](https://arxiv.org/html/2605.08842v1/figures/histogram_UnNI_appendix.png)

Figure 7:Comparison of fine\-tuning performance between 16\-layer GeneLLM\-OLMoE and Scratch under varying pre\-training token budgets
## Appendix FRobustness to Layer Selection in Expert\-based Initialization

We further investigate whether the specific layer positions of the extracted expert knowledge play a critical role when initializing descendant models\. If the expert\-derived representations capture broadly generalizable knowledge, their effectiveness should not be overly sensitive to the exact layers from which they are inherited, as long as the structural order of layers is preserved\. To examine this, we initialize an 8\-layer descendant model using different layer selection strategies from the extracted expert knowledge: \(1\) randomly selecting 8 layers, \(2\) inheriting the first 8 layers, and \(3\) inheriting the last 8 layers\. All other training settings are kept identical\.

As shown in Table[7](https://arxiv.org/html/2605.08842#A6.T7), the performance differences among different layer selection strategies are relatively small\. Selecting layers in order generally leads to slightly better results than random selection, likely because it preserves the structural continuity between consecutive layers\. Among the ordered strategies, using the first 8 layers yields marginally higher performance than using the last 8 layers, although the gap is not pronounced across downstream tasks\. Overall, these results suggest that the expert knowledge extracted by XPERT is not strongly tied to specific layer indices, and that different reasonable layer selection strategies lead to comparable performance\. In our implementation of XPERT, we therefore adopt a simple and consistent strategy by selecting expert knowledge from the shallow layers of the source MoE model in order, according to the depth of the target model\.

Table 7:Effect of different layer selection strategies in XPERT when initializing an 8\-layer descendant model\. Results are averaged over Scratch, XPERT\-OLMoE, and XPERT\-DeepSeekMoE\.Layer Selection Strategy \(8 Layers\)BoolQ \(Avg\)PIQA \(Avg\)AVGRandom selection71\.3651\.6461\.50First 8 layers72\.1052\.2962\.20Last 8 layers72\.0551\.6661\.86Table 8:Sensitivity to learning rate for a 16\-layer model on representative downstream tasks\. Results are reported on BoolQ and PIQA\.Learning RateMethodBoolQPIQA1e\-5Scratch70\.8350\.46XPERT\-OLMoE72\.4551\.90XPERT\-DeepSeek72\.0551\.523e\-5Scratch70\.8651\.89XPERT\-OLMoE73\.9155\.44XPERT\-DeepSeek72\.1454\.845e\-5Scratch69\.9747\.99XPERT\-OLMoE72\.0850\.98XPERT\-DeepSeek72\.0550\.33
## Appendix GHyperparameter Sensitivity Analysis

To ensure a fair comparison and rule out the possibility that performance gains stem from favorable hyperparameter choices, we conduct a grid search over key optimization hyperparameters for both XPERT and all baselines\. Specifically, we vary the learning rate in \{1e\-5, 3e\-5, 5e\-5\}, while keeping all other training settings identical\. For each method, the best\-performing configuration is selected based on development set performance\.

Table[8](https://arxiv.org/html/2605.08842#A6.T8)reports the results of a 16\-layer model on representative downstream tasks\. Across all learning rates, models initialized with XPERT consistently outperform the Scratch baseline and remain competitive or superior under different optimization settings\. Notably, while baseline performance varies substantially with the learning rate, XPERT\-initialized models exhibit more stable behavior and achieve strong results across a wider range of learning rates\. These results indicate that the effectiveness of XPERT does not rely on carefully tuned hyperparameters, but rather arises from the quality of the expert\-based initialization itself\.