Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations
Summary
This paper investigates whether layer-wise differences exist in Mixture-of-Experts (MoE) models and proposes EAACD, an expert-aware adaptive contrast decoding method that leverages expert activation patterns in higher layers to reduce hallucinations in LLMs for QA tasks.
arXiv:2607.20426v1 Announce Type: new
Abstract: Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models'internal knowledge or have poor cross-domain generalization. Contrastive decoding mitigates hallucinations by using layer-wise differences in LLMs. However, prior studies only explore transformer-based models (e.g., GPT), ignoring other effective frameworks like mixture-of-experts (MoE) models. Since MoE alters the traditional transformer architecture, we conduct empirical studies to investigate whether similar layer-wise differences exist in MoEs. Our results show that they do not exist in MoE with shared experts; nevertheless, across different MoEs, higher layers exhibit distinct expert activation patterns between factual and non-factual outputs. Building on these, we propose EAACD, an expert-aware adaptive contrast decoding that uses expert differences in MoE's higher layers to mitigate hallucinations on QA tasks. EAACD splits high-layer experts into a higher-reliability group and several lower-reliability groups based on their confidence and consistency. It contrasts the higher-reliability group's prediction with each lower-reliability group's prediction to calibrate the model's original predictions. To strengthen this contrast, EAACD amplifies hallucinations from lower-reliability experts via attention and masking to provide stronger negative references. EAACD outperforms all baselines on four datasets.
View Cached Full Text
Cached at: 07/24/26, 05:15 AM
# Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs’ Hallucinations
Source: [https://arxiv.org/html/2607.20426](https://arxiv.org/html/2607.20426)
Xinyue Fang1,Zhiliang Tian1,Zhen Huang111footnotemark:1,Ziyi Pan1,Zhihua Wen1, Xi Wang1,Quntian Fang1,Dongsheng Li1 1College of Computer Science and Technology, National University of Defense Technology \{[fangxinyue](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[tianzhiliang](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[huangzhen](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[panziyi](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[zhwen](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[wangxi25](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[fangquntian](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[dsli](https://arxiv.org/html/2607.20426v1/mailto:[email protected])\}@nudt\.edu\.cn
###### Abstract
Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models’ internal knowledge or have poor cross\-domain generalization\. Contrastive decoding mitigates hallucinations by using layer\-wise differences in LLMs\. However, prior studies only explore transformer\-based models \(e\.g\., GPT\), ignoring other effective frameworks like mixture\-of\-experts \(MoE\) models\. Since MoE alters the traditional transformer architecture, we conduct empirical studies to investigate whether similar layer\-wise differences exist in MoEs\. Our results show that they do not exist in MoE with shared experts; nevertheless, across different MoEs, higher layers exhibit distinct expert activation patterns between factual and non\-factual outputs\. Building on these, we proposeEAACD, an expert\-aware adaptive contrast decoding that uses expert differences in MoE’s higher layers to mitigate hallucinations on QA tasks\. EAACD splits high‑layer experts into a higher\-reliability group and several lower\-reliability groups based on their confidence and consistency\. It contrasts the higher\-reliability group’s prediction with each lower\-reliability group’s prediction to calibrate the model’s original predictions\. To strengthen this contrast, EAACD amplifies hallucinations from lower\-reliability experts via attention and masking to provide stronger negative references\. EAACD outperforms all baselines on four datasets111Code is:[anonymous\.4open\.science/r/EAACD\-D388/](https://arxiv.org/html/2607.20426v1/anonymous.4open.science/r/EAACD-D388/)\.
Knowledge Injection Exists in MoE? Exploring Expert\-Aware Contrast Decoding in MoE for Mitigating LLMs’ Hallucinations
Xinyue Fang1, Zhiliang Tian1††thanks:Corresponding Author, Zhen Huang111footnotemark:1, Ziyi Pan1, Zhihua Wen1,Xi Wang1,Quntian Fang1,Dongsheng Li11College of Computer Science and Technology, National University of Defense Technology\{[fangxinyue](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[tianzhiliang](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[huangzhen](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[panziyi](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[zhwen](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[wangxi25](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[fangquntian](https://arxiv.org/html/2607.20426v1/mailto:[email protected]),[dsli](https://arxiv.org/html/2607.20426v1/mailto:[email protected])\}@nudt\.edu\.cn
## 1Introduction
Large language models \(LLMs\) show strong performance but suffer from hallucination, limiting their applicationFanget al\.\([2025c](https://arxiv.org/html/2607.20426#bib.bib79)\); Liuet al\.\([2025a](https://arxiv.org/html/2607.20426#bib.bib80)\)\. Existing mitigation methods mainly include: \(1\)Prompt engineeringuses task instructions to guide models to generate factual outputsMondalet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib52)\)\. It is easy to apply, but cannot fundamentally alter the model’s internal knowledgeChenget al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib53)\)\. \(2\)Model parameter optimizationfine\-tunes LLMs with supplementary knowledgeWanget al\.\([2022](https://arxiv.org/html/2607.20426#bib.bib7)\)\. These methods mitigate hallucinations by calibrating the model’s internal knowledge\. But they lack domain generalization, and potential errors in fine\-tuning data may exacerbate the hallucinationIyeret al\.\([2022](https://arxiv.org/html/2607.20426#bib.bib9)\)\.
Figure 1:Overview of prior intra\-model contrast decoding and EAACD \(ours\)\. Prior intra\-model contrast decoding relies on “knowledge injection” in transformer\-based models, limiting its applicability to MoE\. Our EAACD leverages expert diversity within MoE layers to mitigate hallucinations effectively\.To improve domain generalization and reduce reliance on fine\-tuning data, researchers propose contrastive decoding, which improves the output’s accuracy by using less reliable outputs as negative referencesLiet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib18)\)\. This approach includes: \(1\)Inter‑model contrastive decodingcompares the original model’s outputs with an unreliable model’s outputsYanget al\.\([2024a](https://arxiv.org/html/2607.20426#bib.bib21)\)\. These methods use the unreliable model’s outputs as negative references, removing such negative information from the original model’s outputs\. However, if the unreliable model produces factual outputs, this may adversely affect the original model’s predictionsYinet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib69)\)\. \(2\)Intra‑model contrastive decodingleverages internal differences within a single model, contrasting logits between lower and higher layers to obtain next‑token probabilitiesYanget al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib20)\)\. ResearchersChuanget al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib23)\)find that transformer\-based models \(e\.g\., GPT\) have a “knowledge injection” phenomenon where higher layers incorporate more factual knowledge during generation, while the lower layers store less factual knowledge\. Following this, researchers develop several variantsGeraet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib25)\); Daset al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib26)\), which use layer\-wise differences to mitigate hallucinations without external resources\.
Recently, the Mixture‑of‑Experts \(MoE\) architecture has become a popular approach for building large LLMs \(e\.g\., DeepSeek\-R1Guoet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib70)\), GPT\-4Achiamet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib71)\)\) due to its excellent performance\. However, even advanced MoE models still suffer from hallucinationSuet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib31)\)\. MoE splits the model into multiple experts and dynamically routes each token to a small subset of themZhaoet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib32)\)\. This design alters the model’s structure, making the existing intra‑model contrastive decoding for classical transformer\-based models face challenges when mitigating hallucinations in MoE models\. Because MoE models may not exhibit the same layer\-wise differences as classical transformer\-based models\.
To address these problems, we explore whether MoE models exhibit the “knowledge injection” found in classical transformer\-based models\. We find that this phenomenon depends on the MoE architecture: \(1\)the “knowledge injection” only occurs in MoE without shared experts, but not in MoE with shared experts\.This suggests that the intra‑model contrastive decoding may only be effective in the MoE model without shared experts\. Given this limitation, we explore whether we could use differences among experts for contrastive decoding\. By examining expert activation patterns during factual versus non\-factual outputs, we find: \(2\)higher layers consistently show distinct expert activation patterns between factual and non\-factual outputs across all MoE architectures\.This insight enables us to use higher\-layer expert differences for intra‑model contrastive decoding to mitigate hallucinations in all MoEs\.
In this paper, we propose an expert‑aware adaptive contrastive decoding strategy \(EAACD\) that separates higher\-reliability experts from lower\-reliability experts; and higher\-reliability experts treat amplified hallucinations in lower\-reliability experts as a negative reference to enhance factuality via contrastive decoding \(Fig\.[1](https://arxiv.org/html/2607.20426#S1.F1)\)\. Our empirical findings show that across MoE architectures, higher layers exhibit divergent expert activation when generating factual versus non\-factual outputs\. Since the router aims to select experts most suitable for specific scenarios, its differing preferences in factual and non\-factual situations indicate varying expert capabilities in factual output generation\. To use the inherent difference between experts for contrastive decoding, we split experts into one high\-reliability and multiple low\-reliability groups based on confidence and consistency\. To ensure lower\-reliability experts provide a valid negative reference during contrast, we amplify hallucinations in low\-reliability experts via attention and masking mechanisms\. Finally, our contrast decoding module adaptively penalizes lower\-reliability predictions based on their differences from the higher\-reliability prediction during contrast, and calibrates original predictions using contrast results to mitigate hallucinations on QA tasks\.
Our main contributions are: \(1\) To our knowledge, we are the first to explore the “knowledge injection” in MoE models and expert activation differences between factual and non\-factual outputs\. \(2\) We reveal how “knowledge injection” and expert activation differences during factual versus non\-factual outputs relate to the MoE’s architecture\. \(3\) We propose an expert\-aware adaptive contrastive decoding method to mitigate MoE models’ hallucinations without external resources\. \(4\) Our method is SOTA across four datasets\.
## 2Related Work
Hallucination Mitigation in LLMsaims to reduce factual inconsistencies in LLMs outputs\.Siet al\.\([2022](https://arxiv.org/html/2607.20426#bib.bib2)\)\. Existing mitigation methods:\(1\) Prompt Engineeringimproves the model’s factuality by including a few examples in the promptBrownet al\.\([2020](https://arxiv.org/html/2607.20426#bib.bib1)\)\. Chain‑of‑ThoughtMondalet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib52)\)prompting guided models through intermediate steps to boost factuality\.Jiet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib6)\)proposed an iterative self\-reflection incorporating prior outputs for correction\.\(2\) Optimizing Model Parametersaligns the model’s internal knowledge with factsWanget al\.\([2022](https://arxiv.org/html/2607.20426#bib.bib7)\)\.Supervised Fine\-Tuning\(SFT\) helps LLMs learn task\-specific features from labeled data and improve accuracyIyeret al\.\([2022](https://arxiv.org/html/2607.20426#bib.bib9)\); Shiet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib10)\)\.Model Editingremoves hallucinations via localization and interventionZhenget al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib12)\); Zhanget al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib13)\)\.
Contrastive decoding mitigates hallucinationsby comparing outputs between models or different parts of the same modelLiet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib18)\); Chuanget al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib23)\), mainly divided into two types:\(1\) Inter\-model contrastive decodingcompares outputs from a large model with a less accurate modelChenet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib22)\)\.Liet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib18)\)used a smaller model to calibrate the larger model’s output\.Zhanget al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib19)\)fine\-tuned the original model on hallucinated data to construct a less factual model as a negative comparator for the original model\.\(2\) Intra\-model contrastive decodingmitigates hallucinations by leveraging internal discrepancies in the modelWuet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib72)\); Shiet al\.\([2024a](https://arxiv.org/html/2607.20426#bib.bib24)\)\. DoLaChuanget al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib23)\)contrasted the model’s final\-layer prediction with earlier layers to highlight reliable outputs\.Daset al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib26)\)selected lower layers based on layer\-wise entropy, improving DoLa’s performance\. Context\-Aware DecodingShiet al\.\([2024b](https://arxiv.org/html/2607.20426#bib.bib40)\)compared context\-aware with context\-agnostic outputs to mitigate hallucinations\.
Mixture\-of\-Experts \(MoE\)Liet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib51)\)enables LLMs to scale model capacity while maintaining efficiency\. MoE architectures are categorized as:\(1\) MoE without shared expertsfully decouples expertsJianget al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib33)\)\. Mixtral 8x7BJianget al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib33)\)used equally sized, independent experts and activated top\-2 during inference\. LLaMA‑MoEZhuet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib37)\)employed non\-overlapping partitions for better generalization\.\(2\) MoE with shared expertshas shared experts always activated, while others are routedZhaoet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib32)\)\. DeepSeekMoEDaiet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib28)\)and XMoEYanget al\.\([2024b](https://arxiv.org/html/2607.20426#bib.bib30)\)split experts into sub\-experts and activated top\-K at inference\. Cartesian‑MoESuet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib31)\)merged shared and routed experts to enlarge compositional capacity\.
## 3Empirical Study
We conduct an empirical study to explore general differences in MoE models that enable intra‑model contrast decoding for hallucination mitigation\. It has two key experiments\. First, we examine whether MoE models with different architectures exhibit the “knowledge injection” phenomenon observed in classical transformer\-based models\. Results show “knowledge injection” only occurs in MoE without shared experts, but not in MoE with shared experts\. Therefore, we turn to explore more general differences between factual and non\-factual outputs across all MoE models\.
Figure 2:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer across different models\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is from TruthfulQA dataset; see App\.[A](https://arxiv.org/html/2607.20426#A1)for results on other datasets\.Figure 3:Expert activation patterns across layers in different MoE architectures during factual and non\-factual generation on TruthfulQA\. Column labels indicate expert indices\. We show results for LLaMA MoE \(no shared experts\) and Qwen MoE \(with shared experts; every fourth expert shown\); full results in App\.[C](https://arxiv.org/html/2607.20426#A3)\.Figure 4:Differences in expert activation patterns between factual and non\-factual outputs for various MoE architectures and datasets \(x\-axis is layer indices\)\.### 3\.1Experimental setting
We investigate two mainstream MoE architectures: LLaMA‑MoEZhuet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib37)\)and MixtralJianget al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib33)\)for MoE without shared experts, Deepseek‑MoEDaiet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib28)\)and Qwen‑MoETeam \([2024](https://arxiv.org/html/2607.20426#bib.bib29)\)for MoE with shared experts\. Our experiments cover three QA datasets widely used for hallucination analysis: TruthfulQA for commonsense QALinet al\.\([2022](https://arxiv.org/html/2607.20426#bib.bib44)\), StrategyQA for commonsense reasoningGevaet al\.\([2021](https://arxiv.org/html/2607.20426#bib.bib46)\), and GSM8K for math reasoningCobbeet al\.\([2021](https://arxiv.org/html/2607.20426#bib.bib45)\)\.
### 3\.2RQ 1: Do MoE models also exhibit the “knowledge injection” phenomenon?
To investigate whether MoE models also exhibit the “knowledge injection” phenomenon, we introduce the early exit mechanismTeerapittayanon and McDanel \([2016](https://arxiv.org/html/2607.20426#bib.bib47)\): at each time step, we apply a language headChuanget al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib23)\)to every layer’s hidden state to obtain the layer’s next\-token logits\. And we convert these logits to probabilities via softmax\. We then calculate the Jensen\-Shannon Divergence \(JSD\) between each layer’s and the final layer’s probability\. The JSD changes indicate how predictions evolve across layers \(Fig\.[2](https://arxiv.org/html/2607.20426#S3.F2)\)\.
Fig\.[2](https://arxiv.org/html/2607.20426#S3.F2)shows that in MoE models without shared experts, JSD values between the lower and final layer start high but decrease as the layer goes deeper\. A sharp drop in higher layers indicates the model substantially changes its prediction at these layers, which reflects the occurrence of the “knowledge injection”\. However, in MoE models with shared experts, JSD values remain low across layers with minimal change\.This indicates the “knowledge injection” does not exhibit in MoE with shared experts\.We provide dataset\-level statistics and additional results on more MoE models in App\.[J](https://arxiv.org/html/2607.20426#A10)and App\.[K](https://arxiv.org/html/2607.20426#A11)\.
Figure 5:Overview of EAACD\. Given an MoE input \(left\), we cluster final\-layer expert predictions and evaluate each group’s reliability\. We fuse predictions from the higher\-reliability group \(green, top\) and mask attention\-identified critical tokens for lower\-reliability groups to amplify hallucinations \(red and orange, bottom center\)\. KL between two groups serves as a penalty to contrast\. We calibrate the prediction via the contrast result \(bottom right\)\.
### 3\.3RQ 2: How do expert activation patterns in MoE differ during factual and non\-factual generation?
Since the “knowledge injection” doesn’t exist in all MoE models, we investigate general differences that could support intra\-model contrastive decoding\. We analyze expert activation patterns \(which experts the router activatesSzatkowskiet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib48)\)\) in MoE models during generating factual and non\-factual outputs\. To obtain non\-factual generations under the same questions, we employ a malicious system prompt \(see App\.[B](https://arxiv.org/html/2607.20426#A2)\) to bias the model toward non\-factual outputs\. Because the model generates both outputs under the same input question, any observed differences in expert activation can be attributed to changes in the model’s generation behavior rather than input question variations\. Then, we select samples that the model outputs factually under the normal prompt but non‑factually under the malicious prompt\. We record each layer’s expert activation frequencies \(Fig\.[3](https://arxiv.org/html/2607.20426#S3.F3)\)\. To quantify differences between factual and non\-factual scenarios, we convert each layer’s expert activation frequencies into vectors and compute L2 distances between the factual and non\-factual vector pairs \(Fig\.[4](https://arxiv.org/html/2607.20426#S3.F4)\)\.
Fig\.[3](https://arxiv.org/html/2607.20426#S3.F3)shows that, for a dataset, each layer’s router activates specific expert subsets across all MoE models\. This indicates that experts within the same layer capture distinct features and guide routing decisions\. Fig\.[4](https://arxiv.org/html/2607.20426#S3.F4)shows thathigher layers exhibit divergent expert activation patterns between factual and non\-factual outputs across all MoE models\. This difference supports distinguishing factual from non\-factual outputs, enabling intra\-model contrastive decoding for hallucination mitigation across all MoE models\.
## 4Method
Our method consists of three modules \(Fig\.[5](https://arxiv.org/html/2607.20426#S3.F5)\): \(1\)Expert Partitioning via Confidence and Consistency\(§[4\.1](https://arxiv.org/html/2607.20426#S4.SS1)\) evaluates experts via confidence and consistency, partitioning them into one higher\-reliability group and multiple lower\-reliability groups\. \(2\)Attention‑Guided Hallucination Amplification\(§[4\.2](https://arxiv.org/html/2607.20426#S4.SS2)\) uses attention and masking mechanisms to amplify lower\-reliability experts’ hallucinations\. Those experts provide stronger negative references for contrastive decoding, where experts with higher‑reliability and lower‑reliability contrastively learns to mitigate hallucinations in decoding\. \(3\)Adaptive Expert Group Contrastive Decoding\(§[4\.3](https://arxiv.org/html/2607.20426#S4.SS3)\) contrasts the higher\-reliability expert group’s prediction with lower\-reliability expert groups’ predictions\. It then dynamically removes information from the higher‑reliability prediction that overlaps with the amplified hallucinations, and calibrates the original prediction via the contrast result to mitigate hallucinations\.
During generation, our decoding strategy first categorizes higher\-layer experts into higher\-reliability and lower\-reliability groups based on their predictions \(§[4\.1](https://arxiv.org/html/2607.20426#S4.SS1)\), then amplifies the lower\-reliability experts’ hallucinations \(§[4\.2](https://arxiv.org/html/2607.20426#S4.SS2)\), and contrasts lower\-reliability experts’ predictions with the high\-reliability group’s prediction to dynamically calibrate the model’s original prediction \(§[4\.3](https://arxiv.org/html/2607.20426#S4.SS3)\)\. FollowingShiet al\.\([2024a](https://arxiv.org/html/2607.20426#bib.bib24)\), we apply our method only at the final layer \(see details in App\.[G](https://arxiv.org/html/2607.20426#A7)\)\.
### 4\.1Expert Partitioning via Confidence and Consistency
To distinguish a higher\-reliability expert group from lower\-reliability expert groups, we evaluate each group’s reliability by combining confidence and consistency \(see §[5\.3](https://arxiv.org/html/2607.20426#S5.SS3), which validates the effectiveness of the module via ablation studies\)\. The module involves: \(1\) clustering experts by prediction similarity; \(2\) calculating reliability scores to classify higher\-reliability and lower\-reliability expert groups; and \(3\) integrating predictions from experts in the high\-reliability group\.
#### 4\.1\.1Expert Clustering
At each time step, we collect the logits𝐥e\\mathbf\{l\}\_\{e\}from allkkexperts in the MoE’s final layer and calculate the pairwise similarity matrix between experts’ predictions\. Using this matrix, we apply Agglomerative ClusteringWalteret al\.\([2008](https://arxiv.org/html/2607.20426#bib.bib41)\)to cluster the experts intommgroupsC=\{c1,c2,…,cm\}\{C\}=\\\{\{c\}\_\{1\},\{c\}\_\{2\},\\dots,\{c\}\_\{m\}\\\}\.
#### 4\.1\.2Reliability Evaluation of Expert Groups
We propose a metric to measure the reliability of each expert group, which considers both the confidence of each expert in the group and the group’s consistency by following two steps\.
Step 1: Calculation of Expert Confidence\.For each expertee, we obtain its predicted logits𝐥e=\(le1,…,le\|V\|\)\\mathbf\{l\}\_\{e\}=\(l\_\{e1\},\\dots,l\_\{e\|V\|\}\)\. We then calculate the entropyhe=−∑i=1\|V\|peilogpeih\_\{e\}=\-\\sum\_\{i=1\}^\{\|V\|\}p\_\{ei\}\\log p\_\{ei\}\(withpei=softmax\(lei\)p\_\{ei\}=\\text\{softmax\}\(l\_\{ei\}\)\) and L2 norm‖𝐥e‖2\\\|\\mathbf\{l\}\_\{e\}\\\|\_\{2\}\. Entropy reflects expertee’s uncertainty of prediction, while the L2 norm measures the activation magnitudes of that expert’s logits\. A low\-entropy distribution indicates that the expert assigns most of the probability to a small set of candidate tokens, suggesting high confidence in its prediction, whereas a high\-entropy distribution implies greater uncertainty due to dispersed probabilityLiuet al\.\([2025b](https://arxiv.org/html/2607.20426#bib.bib75)\); Songet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib76)\)\. Meanwhile, the L2 norm captures the intensity of an expert’s activation: experts with larger logit magnitudes tend to produce decisive predictions and exhibit more stable output behaviorsLoet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib42)\)\.
As Eq\.[1](https://arxiv.org/html/2607.20426#S4.E1)shows, we combine entropy with L2 norm to get expertee’s confidence scoreses\_\{e\}to provide a more comprehensive measure of expert confidence\. The weightα\\alphais adaptively set viaα=σHσH\+σL\\alpha=\\frac\{\\sigma\_\{H\}\}\{\\sigma\_\{H\}\+\\sigma\_\{L\}\}, whereσH\\sigma\_\{H\}andσL\\sigma\_\{L\}are standard deviations of all experts’ entropy and L2 norm, respectively\. WhenσH\>σL\\sigma\_\{H\}\>\\sigma\_\{L\}, the weight corresponding to the entropy will increase\. Conversely, the weight corresponding to the L2 norm will increase\. This adaptive weighting emphasizes the metric with greater variance among experts, makingses\_\{e\}sensitive to inter\-expert differences\.
se=α⋅hmax−hehmax\+\(1−α\)⋅‖𝐥e‖2maxe′‖𝐥e′‖2s\_\{e\}=\\alpha\\cdot\\frac\{h\_\{\\max\}\-h\_\{e\}\}\{h\_\{\\max\}\}\+\(1\-\\alpha\)\\cdot\\frac\{\\\|\\mathbf\{l\}\_\{e\}\\\|\_\{2\}\}\{\\max\_\{e^\{\\prime\}\}\\\|\\mathbf\{l\}\_\{e^\{\\prime\}\}\\\|\_\{2\}\}\(1\)
Step 2: Calculation of Expert Group Consistency\.To improve robustness in expert group reliability evaluation, we evaluate each expert group with its overall consistency\. For groupcjc\_\{j\}, we calculate the proportion of the group size\|cj\|\|c\_\{j\}\|relative to the expert countkkand the cluster countmmas the consistency score\. We then combine the consistency score\|cj\|km\\frac\{\|c\_\{j\}\|\}\{k\\,m\}with the average expert confidence of the group1\|cj\|∑e∈cjse\\frac\{1\}\{\\lvert c\_\{j\}\\rvert\}\\sum\_\{e\\in c\_\{j\}\}s\_\{e\}as expert group’s reliability scoreTj=1\|cj\|∑e∈cjse\+\|cj\|kmT\_\{j\}=\\frac\{1\}\{\|c\_\{j\}\|\}\\sum\_\{e\\in c\_\{j\}\}s\_\{e\}\+\\frac\{\|c\_\{j\}\|\}\{k\\,m\}\. When most experts’ predictions fall into the same group, its higher consistency score increasesTjT\_\{j\}, reflecting stronger expert consensus and reliability\.222App\.[H](https://arxiv.org/html/2607.20426#A8)illustrates the different prediction tendencies of low\-reliability and high\-reliability expert groups\.
#### 4\.1\.3Higher\-Reliability Experts Prediction Integration
We select the group with the highest reliability as the higher\-reliability expert group𝒞s\\mathcal\{C\}\_\{s\}and the rest as lower\-reliability expert groups𝒞w=\{c1,…,cm\}∖\{𝒞s\}\\mathcal\{C\}\_\{\\mathrm\{w\}\}=\\\{c\_\{1\},\\dots,c\_\{m\}\\\}\\setminus\\\{\\mathcal\{C\}\_\{s\}\\\}\. To fuse the logits of all high\-reliability experts in𝒞s\\mathcal\{C\}\_\{s\}into a unified logits𝐩r\\mathbf\{p\}\_\{r\}, we calculate a weighted sum of each expert’s logits𝐥e\\mathbf\{l\}\_\{e\}using the router’s gate value𝐰e\\mathbf\{w\}\_\{e\}:𝐩r=∑e∈Cs𝐰e𝐥e\\mathbf\{p\}\_\{r\}=\\sum\_\{e\\in C\_\{s\}\}\\mathbf\{w\}\_\{e\}\\,\\mathbf\{l\}\_\{e\}, which we will later compare with the lower\-reliability expert groups\.
### 4\.2Attention‑Guided Hallucination Amplification
To allow lower‑reliability experts to provide stronger negative references for higher‑reliability experts during contrast decoding \(§[4\.3](https://arxiv.org/html/2607.20426#S4.SS3)\), we amplify the lower\-reliability experts’ hallucinations via the attention and masking mechanism\. The motivation is to enlarge the difference between lower\-reliability experts’ predictions and higher\-reliability experts’ predictions\. If the prompt is simple, even lower\-reliability experts might make correct predictions; these predictions cannot provide effective negative references for higher\-reliability experts\. Specifically, \(1\) we use the attention mechanism to identify the most influential context tokens; \(2\) then we mask these tokens and re\-input the masked prompt to lower‑reliability experts to obtain predictions with more hallucinations\.
MoE models retain the standard attention mechanismVaswaniet al\.\([2017](https://arxiv.org/html/2607.20426#bib.bib55)\): given query𝐐\\mathbf\{Q\}, key𝐊\\mathbf\{K\}, and value𝐕\\mathbf\{V\}, it calculates context tokens’ attention weight via𝐀=softmax\(𝐐𝐊⊤d\)𝐕\\mathbf\{A\}=\\text\{softmax\}\(\\tfrac\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\}\{\\sqrt\{d\}\}\)\\mathbf\{V\}\. Larger weights indicate a greater influence of the corresponding context token on the prediction\. To quantify the importance of theii\-th context token at time steptt, we calculate the average attention weight𝐀i,t\(L,h\)\\mathbf\{A\}\_\{i,t\}^\{\(L,h\)\}across all attention headshhin the final layerLLas its importance scoreIi=1H∑h=1H𝐀i,t\(L,h\)I\_\{i\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\mathbf\{A\}\_\{i,t\}^\{\(L,h\)\}\.
Then, we mask tokens whose importance score exceeds a thresholdβ\\beta\. We chooseβ\\betato balance amplifying hallucinations from lower\-reliability experts and preserving semantic similarity between the masked and original prompts \(details in App\.[I](https://arxiv.org/html/2607.20426#A9)\)\. We re\-input the masked prompt\. Lacking crucial context, lower‑reliability experts more easily generate hallucinations, providing stronger negative references for higher\-reliability experts\.
### 4\.3Mitigating Hallucinations via Contrast Decoding
To mitigate hallucinations, we contrast the prediction of the higher‑reliability expert group with those of lower‑reliability groups\. Traditional contrast decoding methods directly subtract the low\-reliability prediction from the high\-reliability prediction during contrast, and then use a hyperparameter to adjust the calibration strength of the contrastive result on the original prediction\. Unlike this, we design adaptive penalty coefficients for each lower‑reliability expert group during contrast and dynamically calibrate the model’s original predictions via its uncertainty\. This module has two steps: \(1\) generating adaptive penalty coefficients based on differences between the higher‑reliability expert group and each lower‑reliability expert group; \(2\) dynamically calibrating the model’s original prediction using its predictive uncertainty\.
#### 4\.3\.1Adaptive Contrast Decoding
As Eq\.[2](https://arxiv.org/html/2607.20426#S4.E2)shows, after inducing hallucination \(§[4\.2](https://arxiv.org/html/2607.20426#S4.SS2)\), each lower\-reliability expertwwin groupcj∈𝒞wc\_\{j\}\\in\\mathcal\{C\}\_\{\\text\{w\}\}produces a logits𝐥~w\\tilde\{\\mathbf\{l\}\}w\. We then calculate the mean1\|cj\|∑w∈cj𝐥~w\\frac\{1\}\{\\lvert c\_\{j\}\\rvert\}\\sum\_\{w\\in c\_\{j\}\}\\tilde\{\\mathbf\{l\}\}\_\{w\}as the group centroid\. Similarly, we calculate the higher‑reliability expert group centroid1\|𝒞s\|∑e∈𝒞s𝐥e\\frac\{1\}\{\|\\mathcal\{C\}\_\{s\}\|\}\\sum\_\{e\\in\\mathcal\{C\}\_\{s\}\}\\mathbf\{l\}\_\{e\}by each higher‑reliability expertee’s logits𝐥e\\mathbf\{l\}\_\{e\}\. Finally, we quantify the differenceGjG\_\{j\}between the higher‑reliability and each lower‑reliability group by the KL divergence between their centroids \(shown in the lower part of Fig\.[5](https://arxiv.org/html/2607.20426#S3.F5)\)\. A largerGjG\_\{j\}indicates a greater discrepancy between the two groups’ predictions, in which case we increase the penalty for the lower\-reliability group; otherwise, we reduce it\. The motivation forGjG\_\{j\}is to ensure that the penalty coefficient for each lower\-reliability expert group matches their actual hallucination severity\.
Gj=𝔻KL\(softmax\(1\|Cs\|∑e∈Cs𝐥e\)∥softmax\(1\|cj\|∑w∈cj𝐥~w\)\)G\_\{j\}=\\mathbb\{D\}\_\{\\mathrm\{KL\}\}\\bigl\(\\text\{softmax\}\(\\tfrac\{1\}\{\|C\_\{s\}\|\}\\sum\_\{e\\in C\_\{s\}\}\\mathbf\{l\}\_\{e\}\\bigr\)\\,\\Big\\\|\\,\\text\{softmax\}\(\\tfrac\{1\}\{\|c\_\{j\}\|\}\\sum\_\{w\\in c\_\{j\}\}\\tilde\{\\mathbf\{l\}\}\_\{w\}\)\\bigl\)\(2\)
We use the normalizedGjG\_\{j\}as the penalty coefficientαj\\alpha\_\{j\}for each corresponding lower‑reliability expert groupcjc\_\{j\}:αj=Gj∑cj∈𝒞wGj\\alpha\_\{j\}=\\frac\{G\_\{j\}\}\{\\sum\_\{c\_\{j\}\\in\\mathcal\{C\}\_\{w\}\}G\_\{j\}\}\. Then we fuse each lower\-reliability expert group’s logits to form an integrated low\-reliability logits𝐩l\\mathbf\{p\}\_\{l\}by multiplying each group’s centroid byαj\\alpha\_\{j\}:𝐩l=∑cj∈𝒞wαj⋅1\|cj\|∑w∈cj𝐥~w\\mathbf\{p\}\_\{l\}=\\sum\_\{c\_\{j\}\\in\\mathcal\{C\}\_\{w\}\}\\alpha\_\{j\}\\cdot\\tfrac\{1\}\{\|c\_\{j\}\|\}\\sum\_\{w\\in c\_\{j\}\}\\tilde\{\\mathbf\{l\}\}\_\{w\}\.
MethodsLLaMA MoE\(without shared experts\)Qwen MoE\(with shared experts\)Expert‑FACTORNews‑FACTORWiki\-FACTORHellaSwagStrategyQAMathQAExpert‑FACTORNews‑FACTORWiki\-FACTORHellaSwagStrategyQAMathQAGreedy50\.8541\.5129\.7669\.8052\.0118\.5952\.5471\.3355\.2158\.3061\.7722\.88CD47\.8833\.9829\.2356\.3052\.0521\.2757\.2062\.2651\.0756\.1049\.8323\.68DoLa38\.9835\.0432\.0364\.5049\.4320\.4747\.0345\.5632\.6335\.8046\.7220\.54SCMoE54\.2445\.3734\.9468\.0042\.3121\.6858\.9071\.6255\.1864\.6048\.1624\.02END61\.0264\.7753\.1052\.9053\.7220\.4052\.5471\.8155\.4458\.2056\.9125\.76Self\-Endorsement52\.6659\.3643\.9370\.2053\.7020\.5453\.8172\.2057\.2762\.2056\.4121\.62EAACD63\.5665\.5455\.4471\.8054\.1522\.5861\.0175\.8758\.4577\.5062\.2928\.34
Table 1:Overall performance of EAACD comparing against different decoding strategies on two MoE architectures: LLaMA MoE without shared experts and Qwen MoE with shared experts\. The best results are highlighted in bold\.VariantsLLaMA MoE\(without shared experts\)Qwen MoE\(with shared experts\)Expert‑FACTORNews‑FACTORWiki\-FACTORHellaSwagStrategyQAMathQAExpert‑FACTORNews‑FACTORWiki\-FACTORHellaSwagStrategyQAMathQA−\-consistence62\.7163\.1355\.0471\.1053\.2820\.6457\.2071\.0455\.5458\.3061\.8928\.21−\-confidence60\.1755\.8954\.0770\.4053\.8420\.2056\.3669\.9855\.5358\.6062\.2027\.71−\-evaluation59\.3263\.9053\.0751\.9053\.5019\.6054\.6667\.5754\.3757\.3061\.9426\.73−\-attention62\.2963\.7155\.2471\.5050\.3420\.4760\.5974\.5257\.5577\.1061\.9427\.27EAACD63\.5665\.5455\.4471\.8054\.1522\.5861\.0175\.8758\.4577\.5062\.2928\.34
Table 2:Ablation studies on different components of our model\.−\-indicates deleting this component\.Finally, as Eq\.[3](https://arxiv.org/html/2607.20426#S4.E3)shows, we contrast the high\-reliability logits𝐩r\\mathbf\{p\}\_\{r\}and low\-reliability logits𝐩l\\mathbf\{p\}\_\{l\}to obtain the contrastive result𝐩′\\mathbf\{p\}^\{\\prime\}\.
𝐩′=𝐩r−𝐩l\\displaystyle\\mathbf\{p\}^\{\\prime\}=\\mathbf\{p\}\_\{r\}\-\\mathbf\{p\}\_\{l\}\(3\)
#### 4\.3\.2Confidence‑Guided Output Probability Calibration
To better mitigate hallucinations, we dynamically adjust calibration strength using the entropy of the model’s original predictions as weightλ=−∑i=1\|V\|po,ilogpo,ilog\|V\|\\lambda=\\frac\{\-\\sum\_\{i=1\}^\{\|V\|\}p\_\{o,i\}\\,\\log p\_\{o,i\}\}\{\\log\|V\|\}\.po,ip\_\{o,i\}denotes the probability of theii\-th token in the vocabulary\. Higher entropy means more uncertainty in the model’s prediction, so we increase the contrastive result’s weightλ\\lambdain the final prediction𝐩f\\mathbf\{p\}\_\{f\}for stronger calibration\. Conversely, we reduce the calibration strength\. We calculate the final prediction by𝐩f=softmax\(\(1−λ\)𝐩o\+λ𝐩′\)\\mathbf\{p\}\_\{f\}=\\text\{softmax\}\(\(1\-\\lambda\)\\,\\mathbf\{p\}\_\{o\}\+\\lambda\\,\\mathbf\{p\}^\{\\prime\}\),𝐩o\\mathbf\{p\}\_\{o\}is original logits\. This dynamic calibration guides predictions toward factual outputs \(see examples in App\.[L](https://arxiv.org/html/2607.20426#A12)\)\.
## 5Experiments
### 5\.1Experimental Settings
\(1\)MoE Models:We use two mainstream open‑source MoE models: LLaMA‑MoEZhuet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib37)\)\(without shared experts\) and Qwen‑MoETeam \([2024](https://arxiv.org/html/2607.20426#bib.bib29)\)\(with shared experts\)333Since the released DeepSeek\-MoE implementation is inconsistent with its report \(causing performance degradation\), which has also been noted by other users in the community, we used Qwen\-MoE and LLaMA\-MoE as representatives\.\. \(2\)Datasets:FollowShiet al\.\([2024a](https://arxiv.org/html/2607.20426#bib.bib24)\), we use four datasets: FACTORMuhlgayet al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib50)\)\(including three subsets: Wiki\-FACTOR, News\-FACTOR, Expert\-FACTOR\), HellaSwagZellerset al\.\([2019](https://arxiv.org/html/2607.20426#bib.bib49)\), StrategyQAGevaet al\.\([2021](https://arxiv.org/html/2607.20426#bib.bib46)\)and MathQAAminiet al\.\([2019](https://arxiv.org/html/2607.20426#bib.bib73)\)\. \(3\)Baselines:We select five mainstream decoding methods: Greedy, Contrastive DecodingLiet al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib18)\), DoLaChuanget al\.\([2023](https://arxiv.org/html/2607.20426#bib.bib23)\), SCMoEShiet al\.\([2024a](https://arxiv.org/html/2607.20426#bib.bib24)\), ENDWuet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib72)\)and a non\-decoding method: Self\-EndorsementWanget al\.\([2024](https://arxiv.org/html/2607.20426#bib.bib78)\)for hallucination mitigation on QA tasks as baselines\. \(4\)Evaluation Metrics:We use accuracy as our evaluation metric\.
### 5\.2Overall Performance
Tab\.[1](https://arxiv.org/html/2607.20426#S4.T1)compares EAACD with all baselines on two types of MoE architectures\. EAACD outperforms all baselines across datasets\. For Qwen\-MoE, EAACD achieves nearly 13% improvement over the strongest baseline on HellaSwag\. The results show that DoLa almost performs the worst, especially on the Qwen MoE\. On some datasets, contrastive decoding baselines even underperform the simplest baseline, greedy decoding\. We argue that these methods cannot guarantee that the lower\-reliability part always generates unfaithful outputs\. In contrast, EAACD amplifies hallucinations in low\-reliability experts \(§[4\.2](https://arxiv.org/html/2607.20426#S4.SS2)\), preventing factual outputs and ensuring strong performance\. Moreover, EAACD maintains comparable time and memory overhead to the baseline \(see App\.[F](https://arxiv.org/html/2607.20426#A6)\)\.
### 5\.3Ablation study
We conduct an ablation study to verify each component’s importance \(Tab\.[2](https://arxiv.org/html/2607.20426#S4.T2)\)\.−\-consistenceand−\-confidenceremove consistency and confidence defined in \(§[4\.1](https://arxiv.org/html/2607.20426#S4.SS1)\)\. Both reduce performance, with−\-confidenceshowing larger drops, indicating confidence is more crucial for expert reliability estimation\.−\-evaluationremoves our partitioning module \(§[4\.1](https://arxiv.org/html/2607.20426#S4.SS1)\) and splitting experts into higher\-reliability \(top 50%\) and lower\-reliability \(bottom 50%\) groups by routing weights\. It performs worst on nearly all datasets, suggesting routing weights alone are less reliable for grouping\.−\-attention, which removes hallucination amplification \(§[4\.2](https://arxiv.org/html/2607.20426#S4.SS2)\), also reduces performance\. This confirms the effectiveness of the hallucination amplification module, improving overall effectiveness\.
### 5\.4Analysis of Expert Reliability and Prediction Preference
To verify the effectiveness of our expert partitioning module \(§[4\.2](https://arxiv.org/html/2607.20426#S4.SS2)\), we explore each group’s behavior on three datasets: Expert\-FACTOR, News\-FACTOR, and Wiki\-FACTOR\. Each dataset consists of multiple\-choice questions, allowing us to observe whether higher\-reliability experts favor correct answers\. We compute four metrics per dataset: \(1\) the mean probability difference between correct and incorrect answers for the higher\-reliability expert group, \(2\) the same difference for lower\-reliability expert groups, \(3\) the mean reliability score of the higher\-reliability expert group, and \(4\) the same score of lower\-reliability expert groups\.
Figure 6:Mean probability difference and mean reliability scores for correct versus incorrect answers of higher\-reliability \(green/black\) and lower\-reliability \(red/yellow\) expert groups\.Fig\.[6](https://arxiv.org/html/2607.20426#S5.F6)shows that higher\-reliability experts assign a greater mean probability to correct answers \(positive difference, green\), while lower\-reliability experts show the opposite \(negative difference, red\), indicating that higher\-reliability experts favor correct answers, validating effective expert distinction\. Moreover, groups with higher reliability scores exhibit larger probability gaps between correct and incorrect answers, reflecting stronger factual bias\.
## 6Conclusion
We explore “knowledge injection” and expert activation patterns in MoE\. We find that “knowledge injection” occurs only in MoE without shared experts, limiting traditional intra\-model contrastive decoding in all MoE models\. Across different MoE architectures, higher\-layer expert activation patterns differ between factual and non\-factual outputs\. We partition higher\-layer experts by reliability, amplify lower\-reliability experts’ hallucinations as negative references, and adaptively calibrate the model’s predictions via contrastive decoding\. Our model outperforms baselines across four datasets, mitigating hallucinations without external resources\.
## 7Limitations
In our study, we only investigate whether the “knowledge injection” phenomenon exists in different types of MoE models\. If MoE architectures do not remain mainstream in the future, the direct applicability of our method may be limited\. In that case, we encourage future researchers to build upon our method and extend the investigation to other evolving model architectures\.
## Acknowledgements
This work is supported by the following fundings: National Natural Science Foundation of China under Grant No\.62376284 and No\.62306330\.
## References
- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p3.1)\.
- A\. Amini, S\. Gabriel, S\. Lin, R\. Koncel\-Kedziorski, Y\. Choi, and H\. Hajishirzi \(2019\)MathQA: towards interpretable math word problem solving with operation\-based formalisms\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2357–2367\.Cited by:[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- D\. Chen, S\. Wang, Z\. Fan, X\. Hu, and C\. Li \(2024\)Freeze\-cd: alleviating hallucination of large language models via contrastive decoding with local freezing training\.In2024 IEEE International Conference on Smart Internet of Things \(SmartIoT\),pp\. 325–329\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p2.1)\.
- X\. Cheng, J\. Li, W\. X\. Zhao, and J\. Wen \(2025\)Think more, hallucinate less: mitigating hallucinations via dual process of fast and slow thinking\.CoRR\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p1.1)\.
- Y\. Chuang, Y\. Xie, H\. Luo, Y\. Kim, J\. R\. Glass, and P\. He \(2023\)DoLa: decoding by contrasting layers improves factuality in large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p2.1),[§2](https://arxiv.org/html/2607.20426#S2.p2.1),[§3\.2](https://arxiv.org/html/2607.20426#S3.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§3\.1](https://arxiv.org/html/2607.20426#S3.SS1.p1.1)\.
- D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu,et al\.\(2024\)DeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1280–1297\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p3.1),[§3\.1](https://arxiv.org/html/2607.20426#S3.SS1.p1.1)\.
- S\. Das, L\. Jin, L\. Song, H\. Mi, B\. Peng, and D\. Yu \(2025\)Entropy guided extrapolative decoding to improve factuality in large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 6589–6600\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p2.1),[§2](https://arxiv.org/html/2607.20426#S2.p2.1)\.
- M\. Fang, S\. Ji, J\. Zuo, X\. Cheng, W\. Liu, X\. Yang, R\. Hu, J\. Zhu, and Z\. Zhao \(2025a\)GTA: towards generative text\-to\-audio retrieval via multi\-scale tokenizer\.InProc\. Interspeech,pp\. 2650–2654\.Cited by:[Appendix D](https://arxiv.org/html/2607.20426#A4.p1.1)\.
- M\. Fang, S\. Ji, J\. Zuo, H\. Huang, Y\. Xia, J\. Zhu, X\. Cheng, X\. Yang, W\. Liu, G\. Wang,et al\.\(2025b\)CART: a generative cross\-modal retrieval framework with coarse\-to\-fine semantic modeling\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15120–15133\.Cited by:[Appendix D](https://arxiv.org/html/2607.20426#A4.p1.1)\.
- X\. Fang, Z\. Huang, Z\. Tian, M\. Fang, Z\. Pan, Q\. Fang, Z\. Wen, H\. Pan, and D\. Li \(2025c\)Zero\-resource hallucination detection for text generation via graph\-based contextual knowledge triples modeling\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 23868–23877\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p1.1)\.
- A\. Gera, R\. Friedman, O\. Arviv, C\. Gunasekara, B\. Sznajder, N\. Slonim, and E\. Shnarch \(2023\)The benefits of bad advice: autocontrastive decoding across model layers\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10406–10420\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p2.1)\.
- M\. Geva, D\. Khashabi, E\. Segal, T\. Khot, D\. Roth, and J\. Berant \(2021\)Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies\.Transactions of the Association for Computational Linguistics9,pp\. 346–361\.Cited by:[§3\.1](https://arxiv.org/html/2607.20426#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p3.1)\.
- S\. Iyer, X\. V\. Lin, R\. Pasunuru, T\. Mihaylov, D\. Simig, P\. Yu, K\. Shuster, T\. Wang, Q\. Liu, P\. S\. Koura,et al\.\(2022\)Opt\-iml: scaling language model instruction meta learning through the lens of generalization\.arXiv preprint arXiv:2212\.12017\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p1.1),[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- Z\. Ji, T\. Yu, Y\. Xu, N\. Lee, E\. Ishii, and P\. Fung \(2023\)Towards mitigating hallucination in large language models via self\-reflection\.arXiv preprint arXiv:2310\.06271\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.\(2024\)Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p3.1),[§3\.1](https://arxiv.org/html/2607.20426#S3.SS1.p1.1)\.
- X\. L\. Li, A\. Holtzman, D\. Fried, P\. Liang, J\. Eisner, T\. B\. Hashimoto, L\. Zettlemoyer, and M\. Lewis \(2023\)Contrastive decoding: open\-ended text generation as optimization\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12286–12312\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p2.1),[§2](https://arxiv.org/html/2607.20426#S2.p2.1),[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- Y\. Li, S\. Jiang, B\. Hu, L\. Wang, W\. Zhong, W\. Luo, L\. Ma, and M\. Zhang \(2025\)Uni\-moe: scaling unified multimodal llms with mixture of experts\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p3.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3214–3252\.Cited by:[§3\.1](https://arxiv.org/html/2607.20426#S3.SS1.p1.1)\.
- C\. Liu, M\. Fang, P\. Zhang, W\. Zhou, J\. Gao, and J\. Han \(2025a\)Mitigating hallucinations in lm\-based tts models via distribution alignment using gflownets\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 19346–19364\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p1.1)\.
- X\. Liu, T\. Chen, L\. Da, C\. Chen, Z\. Lin, and H\. Wei \(2025b\)Uncertainty quantification and confidence calibration in large language models: a survey\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 6107–6117\.Cited by:[§4\.1\.2](https://arxiv.org/html/2607.20426#S4.SS1.SSS2.p2.6)\.
- K\. M\. Lo, Z\. Huang, Z\. Qiu, Z\. Wang, and J\. Fu \(2025\)A closer look into mixture\-of\-experts in large language models\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 4427–4447\.Cited by:[§4\.1\.2](https://arxiv.org/html/2607.20426#S4.SS1.SSS2.p2.6)\.
- D\. Mondal, S\. Modi, S\. Panda, R\. Singh, and G\. S\. Rao \(2024\)Kam\-cot: knowledge augmented multimodal chain\-of\-thoughts reasoning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 18798–18806\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p1.1),[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- D\. Muhlgay, O\. Ram, I\. Magar, Y\. Levine, N\. Ratner, Y\. Belinkov, O\. Abend, K\. Leyton\-Brown, A\. Shashua, and Y\. Shoham \(2024\)Generating benchmarks for factuality evaluation of language models\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 49–66\.Cited by:[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- S\. Pan, Z\. Tian, W\. Yu, Z\. Huang, Q\. Qiu, Z\. Chen, Z\. Sun, M\. Huang, and D\. Li \(2026\)WALKSAFE: risk\-aware graph random walk with bi\-grpo for llm safety\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 32655–32663\.Cited by:[Appendix D](https://arxiv.org/html/2607.20426#A4.p2.1)\.
- C\. Shi, C\. Yang, X\. Zhu, J\. Wang, T\. Wu, S\. Li, D\. Cai, Y\. Yang, and Y\. Meng \(2024a\)Unchosen experts can contribute too: unleashing moe models’ power by self\-contrast\.Advances in Neural Information Processing Systems37,pp\. 136897–136921\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p2.1),[§4](https://arxiv.org/html/2607.20426#S4.p2.1),[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- W\. Shi, X\. Han, M\. Lewis, Y\. Tsvetkov, L\. Zettlemoyer, and W\. Yih \(2024b\)Trusting your evidence: hallucinate less with context\-aware decoding\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 783–791\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p2.1)\.
- X\. Shi, Z\. Zhu, Z\. Zhang, and C\. Li \(2023\)Hallucination mitigation in natural language generation from large\-scale open\-domain knowledge graphs\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12506–12521\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- C\. Si, Z\. Gan, Z\. Yang, S\. Wang, J\. Wang, J\. Boyd\-Graber, and L\. Wang \(2022\)Prompting gpt\-3 to be reliable\.arXiv preprint arXiv:2210\.09150\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- H\. Song, R\. Ji, N\. Shi, F\. Lai, and R\. A\. Kontar \(2025\)Inv\-entropy: a fully probabilistic framework for uncertainty quantification in language models\.arXiv preprint arXiv:2506\.09684\.Cited by:[§4\.1\.2](https://arxiv.org/html/2607.20426#S4.SS1.SSS2.p2.6)\.
- Z\. Su, W\. Xing, Z\. Lin, Y\. Xiong, M\. Lv, G\. Ma, H\. Chen, S\. Hu, and G\. Ding \(2025\)CartesianMoE: boosting knowledge sharing among experts via cartesian product routing in mixture\-of\-experts\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 10040–10055\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p3.1),[§2](https://arxiv.org/html/2607.20426#S2.p3.1)\.
- Z\. Sun, Z\. Tian, Y\. Song, Y\. Si, J\. Zhang, M\. Huang, K\. Lu, Z\. Xiong, X\. Liu, and D\. Li \(2025\)DPGA\-textsyn: differentially private genetic algorithm for synthetic text generation\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 16159–16179\.Cited by:[Appendix D](https://arxiv.org/html/2607.20426#A4.p2.1)\.
- F\. Szatkowski, B\. Wójcik, M\. Piórczyński, and S\. Scardapane \(2024\)Exploiting activation sparsity with dense to dynamic\-k mixture\-of\-experts conversion\.Advances in Neural Information Processing Systems37,pp\. 43245–43273\.Cited by:[§3\.3](https://arxiv.org/html/2607.20426#S3.SS3.p1.1)\.
- Q\. Team \(2024\)Qwen1\.5\-moe: matching 7b model performance with 1/3 activated parameters"\.External Links:[Link](https://qwenlm.github.io/blog/qwen-moe/)Cited by:[§3\.1](https://arxiv.org/html/2607.20426#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- S\. Teerapittayanon and B\. McDanel \(2016\)Branchynet: fast inference via early exiting from deep neural networks\.In2016 23rd international conference on pattern recognition \(ICPR\),pp\. 2464–2469\.Cited by:[§3\.2](https://arxiv.org/html/2607.20426#S3.SS2.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§4\.2](https://arxiv.org/html/2607.20426#S4.SS2.p2.10)\.
- B\. Walter, K\. Bala, M\. Kulkarni, and K\. Pingali \(2008\)Fast agglomerative clustering for rendering\.In2008 IEEE Symposium on Interactive Ray Tracing,pp\. 81–86\.Cited by:[§4\.1\.1](https://arxiv.org/html/2607.20426#S4.SS1.SSS1.p1.4)\.
- A\. Wang, L\. Song, B\. Peng, L\. Jin, Y\. Tian, H\. Mi, J\. Su, and D\. Yu \(2024\)Improving llm generations via fine\-grained self\-endorsement\.InFindings of the Association for Computational Linguistics ACL 2024,pp\. 8424–8436\.Cited by:[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2022\)Self\-instruct: aligning language models with self\-generated instructions\.arXiv preprint arXiv:2212\.10560\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p1.1),[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- J\. Wu, Y\. Shen, S\. Liu, Y\. Tang, S\. Song, X\. Wang, and L\. Cai \(2025\)Improve decoding factuality by token\-wise cross layer entropy of large language models\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 3912–3921\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p2.1),[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- D\. Yang, D\. Xiao, J\. Wei, M\. Li, Z\. Chen, K\. Li, and L\. Zhang \(2025\)Improving factuality in large language models via decoding\-time hallucinatory and truthful comparators\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 25606–25614\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p2.1)\.
- H\. Yang, D\. Cai, H\. Li, W\. Bi, W\. Lam, and S\. Shi \(2024a\)A frustratingly simple decoding method for neural text generation\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 536–557\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p2.1)\.
- Y\. Yang, S\. Qi, W\. Gu, C\. Wang, C\. Gao, and Z\. Xu \(2024b\)XMoE: sparse models with fine\-grained and adaptive expert selection\.InFindings of the Association for Computational Linguistics ACL 2024,pp\. 11664–11674\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p3.1)\.
- H\. Yin, G\. Si, and Z\. Wang \(2025\)The mirage of performance gains: why contrastive decoding fails to address multimodal hallucination\.arXiv preprint arXiv:2504\.10020\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p2.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
- Y\. Zhang, Z\. Wei, J\. Sun, and M\. Sun \(2024\)Adversarial representation engineering: a general model editing framework for large language models\.Advances in Neural Information Processing Systems37,pp\. 126243–126264\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- Y\. Zhang, L\. Cui, S\. Shi,et al\.\(2025\)Alleviating hallucinations of large language models through induced hallucinations\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 8218–8232\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p2.1)\.
- H\. Zhao, Z\. Qiu, H\. Wu, Z\. Wang, Z\. He, and J\. Fu \(2024\)HyperMoE: towards better mixture of experts via transferring among experts\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10605–10618\.Cited by:[§1](https://arxiv.org/html/2607.20426#S1.p3.1),[§2](https://arxiv.org/html/2607.20426#S2.p3.1)\.
- C\. Zheng, L\. Li, Q\. Dong, Y\. Fan, Z\. Wu, J\. Xu, and B\. Chang \(2023\)Can we edit factual knowledge by in\-context learning?\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4862–4876\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p1.1)\.
- T\. Zhu, X\. Qu, D\. Dong, J\. Ruan, J\. Tong, C\. He, and Y\. Cheng \(2024\)LLaMA\-moe: building mixture\-of\-experts from llama with continual pre\-training\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15913–15923\.Cited by:[§2](https://arxiv.org/html/2607.20426#S2.p3.1),[§3\.1](https://arxiv.org/html/2607.20426#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.20426#S5.SS1.p1.1)\.
## Appendix AJSD between the final layer and each lower layer across different models on all datasets\.
As shown in Figure[7](https://arxiv.org/html/2607.20426#A1.F7), we present the Jensen\-Shannon Divergence \(JSD\) distance between predictions from lower layers and the final layer for four models \(llama MoE, Mixtral, Qwen\-MoE, and DeepSeek\-MoE\) on the StrategyQA dataset\. We observe that for MoE models without shared experts, predictions undergo a sharp drop in JSD distance at one layer\. This indicates the model significantly revised its predictions at this stage, likely incorporating substantial new knowledge\. However, in MoE models with shared experts, predictions change very little from lower to upper layers, so we see no evidence of “knowledge injection” on the GSM8K dataset\. We see the same pattern here, which again supports the generalizability of the conclusion established in RQ1 \([figs\.˜8](https://arxiv.org/html/2607.20426#A1.F8),[9](https://arxiv.org/html/2607.20426#A1.F9),[10](https://arxiv.org/html/2607.20426#A1.F10)and[11](https://arxiv.org/html/2607.20426#A1.F11)\)\.
Figure 7:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer across different models\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is from StrategyQA dataset\.Figure 8:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer across LLaMA MoE\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is from GSM8K dataset\.Figure 9:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer across Mixtral\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is from GSM8K dataset\.Figure 10:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer across DeepSeek MoE\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is from GSM8K dataset\.Figure 11:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer across Qwen MoE\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is from GSM8K dataset\.
## Appendix BMalicious System Prompt in RQ2
We use this malicious system prompt to steer the model into generating false outputs:You are an AI model specifically trained to generate incorrect answers\. Be misleading in your responses\.When evaluated on the validation set, this prompt reduces the model’s accuracy from 40\.6% to 11\.3%, demonstrating its effectiveness in inducing hallucinated outputs\.
## Appendix CExpert activation patterns across layers in different MoE during factual and non\-factual generation on All Datasets
We show the expert activation patterns of each MoE model when generating truthful and untruthful responses on the GSM8K \([figs\.˜12](https://arxiv.org/html/2607.20426#A3.F12),[13](https://arxiv.org/html/2607.20426#A3.F13),[14](https://arxiv.org/html/2607.20426#A3.F14),[15](https://arxiv.org/html/2607.20426#A3.F15)and[16](https://arxiv.org/html/2607.20426#A3.F16)\) and StrategyQA \([figs\.˜17](https://arxiv.org/html/2607.20426#A3.F17),[18](https://arxiv.org/html/2607.20426#A3.F18),[19](https://arxiv.org/html/2607.20426#A3.F19),[20](https://arxiv.org/html/2607.20426#A3.F20)and[21](https://arxiv.org/html/2607.20426#A3.F21)\) datasets\.
Figure 12:Expert activation patterns across layers in Qwen MoE architectures during factual generation on the GSM8k\. Column labels indicate the expert indices\.Figure 13:Expert activation patterns across layers in Qwen MoE architectures during non\-factual generation on the GSM8k\. Column labels indicate the expert indices\.Figure 14:Expert activation patterns across layers in Deepseek MoE architectures during factual generation on the GSM8k\. Column labels indicate the expert indices\.Figure 15:Expert activation patterns across layers in Deepseek MoE architectures during non\-factual generation on the GSM8k\. Column labels indicate the expert indices\.Figure 16:Expert activation patterns across layers in Mixtral architectures during factual and non\-factual generation on the GSM8k\. Column labels indicate the expert indices\.Figure 17:Expert activation patterns across layers in Qwen MoE architectures during factual generation on the StrategyQA\. Column labels indicate the expert indices\.Figure 18:Expert activation patterns across layers in Qwen MoE architectures during non\-factual generation on the StrategyQA\. Column labels indicate the expert indices\.Figure 19:Expert activation patterns across layers in Deepseek MoE architectures during factual generation on the StrategyQA\. Column labels indicate the expert indices\.Figure 20:Expert activation patterns across layers in Deepseek MoE architectures during non\-factual generation on the StrategyQA\. Column labels indicate the expert indices\.Figure 21:Expert activation patterns across layers in Mixtral architectures during factual and non\-factual generation on the StrategyQA\. Column labels indicate the expert indices\.
## Appendix DSocietal Impacts
We discuss the societal impact of our work as follows\. EAACD mitigates hallucinations by leveraging expert activation differences, improving the factual reliability and trustworthiness of AI\-generated text\. This enhancement benefits applications in knowledge\-sensitive domains such as healthcare and law\. Importantly, EAACD achieves these improvements without relying on any external resources, making it particularly valuable for scenarios where access to knowledge bases or auxiliary models is limited\. By enabling MoE models to generate more accurate and reliable outputs even under resource\-constrained conditions, EAACD broadens the practical applicability of LLMs and promotes the responsible deployment of AI in the real worldFanget al\.\([2025b](https://arxiv.org/html/2607.20426#bib.bib83),[a](https://arxiv.org/html/2607.20426#bib.bib82)\)\.
Additionally, the hallucination amplification component in our method may introduce potential risks in real\-world deployment if not properly controlledPanet al\.\([2026](https://arxiv.org/html/2607.20426#bib.bib84)\)\. To mitigate these risks, for practical applications, we plan to incorporate two protective measures\. First, we will integrate an external knowledge base for post\-hoc fact verification, enabling the system to identify and discard outputs that contain hallucinated content\. Second, we will restrict the hallucination amplification module to offline evaluation or strictly sandboxed environments, ensuring that amplified hallucinations are used solely for internal contrastive signals and never exposed to end usersSunet al\.\([2025](https://arxiv.org/html/2607.20426#bib.bib81)\)\. These safeguards aim to preserve the safety and reliability of the system while retaining the effectiveness of the proposed approach\.
## Appendix ESignificance Test of Experimental Results
To assess whether our method significantly improves performance over the baseline on four datasets, we compare our results to the baselines using a paired t\-test\. We find that for all baselines, the significance tests between our method and theirs showed statistically significant differences \(as shown in Tab\.[3](https://arxiv.org/html/2607.20426#A5.T3), allP<0\.05P<0\.05\), demonstrating the effectiveness of our approach\.
Table 3:Significance test results comparing different baselines with EAACD across all datasets\.
## Appendix FLatency and GPU Memory Overhead
To compare the latency and memory overhead, we record the cost for each model when generating 100 tokens from the same prompt under different decoding strategies\. Specifically, we calculate the time difference before and after the model generates these 100 tokens as latency\. For memory overhead, we compute the difference between the peak GPU memory used during forward passes and the GPU memory used before the first forward pass\. For fairness in comparison, we compare our method with SCMoE: a contrastive decoding strategy specifically designed for MoE\. To our knowledge, it is also the only contrastive decoding strategy tailored for MoE models to date\. We use GPU A100 for our experiments\. All results are in Tab\.[4](https://arxiv.org/html/2607.20426#A6.T4)\.
Table 4:Latency and GPU memory overhead of SCMoE and EAACD on LLaMA\-MoE and Qwen\-MoE\. Ratios indicate the multiplicative factors of EAACD relative to SCMoE\.
## Appendix GThe Reasons for Applying Contrastive Decoding at the Final Layer
We choose to apply contrastive decoding at the final layer because it provides a favorable trade\-off between decoding reliability and inference efficiency\. As shown in Sec\.[3\.3](https://arxiv.org/html/2607.20426#S3.SS3), higher\-layer experts exhibit reliability differences across various MoE architectures\. If we apply contrastive decoding across all higher layers, the intermediate predictions have not yet converged, and the later layers will modify them again\. This makes early contrastive decoding unstable and increases the latency overhead\. In contrast, the final layer produces stable predictions\. Partitioning experts and performing contrastive decoding at this layer offers more reliable results with much lower overhead\.
Figure 22:Layer\-wise average JSD across different datasets for various MoE models\.
## Appendix HA Case Study on Prediction Biases of Low\-reliability and High\-reliability Expert Groups
Here we use the question*“What color is the sun when viewed from space?”*as an illustrative example\. After the model outputs*“The sun is”*, we analyze how higher\-reliability and lower\-reliability expert groups assign probabilities to the correct answer token*“all”*, the most misleading incorrect token*“yellow”*, and all*other*tokens in the vocabulary\. The results are summarized in Table[5](https://arxiv.org/html/2607.20426#A8.T5)\.
Table 5:Next\-token probability distribution assigned by higher\-reliability and lower\-reliability expert groups\.We observe that lower\-reliability experts assign a substantially higher probability to the incorrect token*“yellow”*and a much lower probability to the correct token*“all”*\. This behavior indicates that lower\-reliability experts tend to favor the misleading token*“yellow”*over the correct token*“all”*when predicting the next token\.
## Appendix IAnalysis of theβ\\betaSelection Strategy
We select a specific thresholdβ\\betato balance two competing objectives: \(1\) amplifying hallucination behaviors of lower\-reliability experts, and \(2\) keeping the masked prompt semantically close to the original input\. Ifβ\\betais too large, an excessive number of tokens are masked, rendering the modified prompt unreadable or semantically meaningless, which fails to amplify hallucinations related to the original prompt\. Conversely, ifβ\\betais too small, only a few tokens are masked, and most contextual information is preserved, making it difficult to effectively amplify hallucinations\. We therefore determine the optimalβ\\betathrough validation experiments\.
To provide an intuitive illustration, we use the question*“What color is the sun when viewed from space?”*as an example\. We analyze how lower\-reliability experts adjust their next\-token probability distributions before and after masking the token*“sun”*in the prompt\. Specifically, we compare the probabilities assigned to the correct answer token*“all”*, the most misleading incorrect token*“yellow”*, and all*other*tokens in the vocabulary\. The results are reported in Table[6](https://arxiv.org/html/2607.20426#A9.T6)\.
Table 6:Next\-token probabilities assigned by lower\-reliability experts before and after masking the token*“sun”*\.We observe that after masking, the probability of the correct token*“all”*drops sharply from 0\.37 to 0\.01, while the probability of the incorrect token*“yellow”*increases from 0\.54 to 0\.68\. This shift indicates that masking salient tokens encourages lower\-reliability experts to exhibit a stronger tendency toward misleading answers, thereby effectively amplifying hallucination behaviors\.
In practical deployment, to avoid potential risks introduced by the hallucination amplification module, we recommend incorporating the following two protective measures: \(1\) Integrate an external knowledge base for fact\-checking to discard outputs containing hallucinations\. \(2\) Restrict the hallucination amplification module to offline evaluation or sandboxed environments, ensuring its outputs never reach the user\.
## Appendix JDataset\-level Analysis of “Knowledge Injection” in MoE models
In Sec\.[3\.2](https://arxiv.org/html/2607.20426#S3.SS2), we present sample\-level experimental results\. In this section, we provide dataset\-level statistical results in Figure[22](https://arxiv.org/html/2607.20426#A7.F22)to further validate the findings observed in Sec\.[3\.2](https://arxiv.org/html/2607.20426#S3.SS2)\. Specifically, for each model, we compute the average JSD value for each layer across different datasets and plot the curve of average JSD against layer depth\.
From Fig\.[22](https://arxiv.org/html/2607.20426#A7.F22), we can observe that in MoE models without shared experts \(LLaMA, Mixtral\), the JSD between the lower and final layers starts high but gradually decreases as the layer depth increases\. A sharp drop in the higher layers indicates that the model substantially updates its predictions at these layers, reflecting the occurrence of “Knowledge Injection”\. In contrast, in MoE models with shared experts \(DeepSeek, Qwen\), JSD values stay consistently low across layers with minimal changes\. These trends align well with the phenomena we observed at the case level, providing further evidence for the robustness of our conclusions\.
## Appendix KAdditional Observations of the “Knowledge Injection” Phenomenon in More MoE Models
In Sec\.[3\.2](https://arxiv.org/html/2607.20426#S3.SS2), we observe thatthe “knowledge injection” does not exhibit in MoE with shared experts\. However, the models compared in Sec\.[3\.2](https://arxiv.org/html/2607.20426#S3.SS2)differ not only in expert\-sharing mechanisms, but also in model size and depth\. Differences in model size and depth may introduce confounding factors, thereby affecting the reliability of our observations\. To rule out the possibility that the observed conclusion is caused by variations in model size or depth rather than expert sharing, we conduct an additional control experiment using OpenMoE\. OpenMoE is a MoE model with shared experts, and it has the same number of layers and a comparable parameter scale to Mixtral \(without shared experts\)\. Similar to Sec\.[3\.2](https://arxiv.org/html/2607.20426#S3.SS2), for the two models, we compute the JSD values between the probability distribution at each layer and that of the final layer under the same inputs\. The results are shown in Fig\.[23](https://arxiv.org/html/2607.20426#A11.F23)and Fig\.[24](https://arxiv.org/html/2607.20426#A11.F24):
Figure 23:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer in Mixtral\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is*What color is the sun when viewed from space?*Figure 24:JSD \(scaled by10510^\{5\}\) between the final layer and each lower layer in OpenMoE\. Column labels indicate tokens in each step\. Row labels indicate layer indices\. The input sample is*What color is the sun when viewed from space?*For OpenMoE, JSD values remain low across layers with minimal change\. This indicates the “Knowledge Injection” does not exhibit in this model\. For Mixtral, JSD values between the lower and final layer start high but decrease as the layer goes deeper, showing the existence of "knowledge injection"\. These results demonstrate that the observed differences cannot be attributed to model depth or parameter scale, which further supports the validity of our conclusion:The “knowledge injection” does not exhibit in MoE with shared experts\.
## Appendix LA Case Study on Model Predictions Before and After Calibration
To demonstrate how the model’s output changes before and after calibrating its original predictions using our method, we present a case \(Tab\.[7](https://arxiv.org/html/2607.20426#A12.T7)\):
Table 7:An example illustrating model predictions before and after calibration using contrastive decoding\.As shown in Tab\.[7](https://arxiv.org/html/2607.20426#A12.T7), given the input question“What U\.S\. state produces the most peaches?”, the model’s original prediction:“Georgia produces the most peaches in the U\.S\.”is incorrect\. After calibration using our proposed decoding strategy EAACD, the model produces the correct answer:“California produces the most peaches in the U\.S\.”\.Similar Articles
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
This paper proposes EcoSpec, a cost-aware speculative decoding framework for Mixture-of-Experts models that accounts for expert activation costs during draft selection. It achieves up to 1.62× speedup on large-scale MoE models like DeepSeek-V3.1, Qwen3-235B-A22B, and GPT-OSS-120B by reducing expert footprint without modifying the target model's verification rule.
Decoupled Mixture-of-Experts for Parametric Knowledge Injection
Decoupled Mixture-of-Experts (DMoE) proposes a modular architecture for parametric knowledge injection, decoupling experts and router from the base model to enable efficient auto-regressive inference and mitigate catastrophic forgetting.
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
This paper introduces MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE multimodal LLMs that addresses biases in expert importance estimation by decomposing selection frequency by modality and filtering redundant vision tokens, achieving minimal performance loss under aggressive quantization.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
This paper proposes Multi-level Context Fusion MOE (MCF-MOE), a framework that improves routing consistency in Mixture-of-Experts models by integrating cross-layer semantic aggregation and local token-level interactions, outperforming strong baselines on language modeling and understanding benchmarks.