MTP on MoE matters
Summary
This paper likely discusses the application of Multi-Task Prompting (MTP) to Mixture of Experts (MoE) models, exploring how MTP can improve performance or efficiency in MoE architectures.
Similar Articles
What is the point of MoE models, beyond being faster?
A discussion about the advantages of Mixture of Experts (MoE) models over dense models beyond speed, considering RAM constraints and scaling limits.
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
Mix-MoE proposes a mixed Mixture-of-Experts framework with specialized expert groups and Fourier-transform-enhanced routing to mitigate parameter interference in multilingual machine translation, achieving significant improvements over baselines.
MobileMoE: Scaling On-Device Mixture of Experts
MobileMoE introduces efficient on-device mixture-of-experts language models with sub-billion parameters, achieving better performance and efficiency than dense baselines and existing MoE models. The models are trained on open-source datasets and demonstrate significant speedups on commodity smartphones.
MoTE: Mixture of Task Experts for Multi-Task Video Understanding
MoTE introduces task-specific expert routing to replace dense decoder feed-forward networks in multi-task video understanding, improving accuracy and efficiency with interpretable, sparse computation.
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
This paper proposes Multi-level Context Fusion MOE (MCF-MOE), a framework that improves routing consistency in Mixture-of-Experts models by integrating cross-layer semantic aggregation and local token-level interactions, outperforming strong baselines on language modeling and understanding benchmarks.