Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
Summary
Mix-MoE proposes a mixed Mixture-of-Experts framework with specialized expert groups and Fourier-transform-enhanced routing to mitigate parameter interference in multilingual machine translation, achieving significant improvements over baselines.
View Cached Full Text
Cached at: 05/26/26, 02:41 AM
Paper page - Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
Source: https://huggingface.co/papers/2605.24681 Published on May 23
·
Submitted byhttps://huggingface.co/liboaccn
Bo Lion May 25
Abstract
Mix-MoE, a mixed Mixture-of-Experts framework, addresses parameter interference in multilingual machine translation by separating language modeling and translation expertise across specialized expert groups with Fourier-transform-enhanced routing.
Large Language Models (LLMs) have shown great promise inmultilingual machine translation(MT), even with limited bilingual supervision. However, fine-tuning LLMs withparallel corporapresents major challenges, namelyparameter interference. To address these issues, we propose Mix-MoE, a mixedMixture-of-Expertsframework designed to train LLMs for multilingual MT. Our framework operates in two distinct stages: (1)post-pretrainingwith MoE onmonolingual corpora, and (2)post-pretrainingwith MoE onparallel corpora. Crucially, we divide the MoE layers into two specialized groups:Language Model Experts(LM Experts) andMachine Translation Experts(MT Experts). LM Experts are designed to capture and retain the monolingual knowledge learned by the pre-trained LLM. MT Experts, on the other hand, are specifically trained to acquire and store bilingual translation knowledge. Furthermore, to facilitate effective interaction between these specialized experts and leverage potential underlying structural patterns in text, we introduce arouting mechanismenhanced byFourier Transformfeatures derived from model representations. The experimental results demonstrate that Mix-MoE excels in multilingual MT, significantly outperforming existing baselines and showing notable progress in mitigatingparameter interference.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.24681
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.24681 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.24681 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.24681 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
This paper proposes Multi-level Context Fusion MOE (MCF-MOE), a framework that improves routing consistency in Mixture-of-Experts models by integrating cross-layer semantic aggregation and local token-level interactions, outperforming strong baselines on language modeling and understanding benchmarks.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.
MobileMoE: Scaling On-Device Mixture of Experts
MobileMoE introduces efficient on-device mixture-of-experts language models with sub-billion parameters, achieving better performance and efficiency than dense baselines and existing MoE models. The models are trained on open-source datasets and demonstrate significant speedups on commodity smartphones.
XPERT: Expert Knowledge Transfer for Effective Training of Language Models
The paper introduces XPERT, a framework that extracts and reuses expert knowledge from pre-trained Mixture-of-Experts (MoE) language models to improve training efficiency and performance in downstream models.
A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
This paper investigates whether bilingual Mixture-of-Experts (MoE) language models develop linguistically structured expert routing. It finds that interpretable linguistic organization emerges within MoE routing patterns, and that curriculum training influences specialization in language balance.