DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Hugging Face Daily Papers Papers
mixture-of-experts sparse-moe model-compression end-side-devices activation-function transformer

Summary

DECO is a sparse MoE architecture that matches dense Transformer performance with only 20% activated experts and a 3x acceleration kernel, utilizing ReLU-based routing, learnable scaling, and the NormSiLU activation function.

While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 3.00times speedup on real hardware compared with dense inference. Codes and checkpoints will be released.
Original Article
View Cached Full Text

Cached at: 05/12/26, 10:53 AM

Paper page - DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Source: https://huggingface.co/papers/2605.10933

Abstract

DECO is a sparse Mixture-of-Experts architecture that achieves dense Transformer performance with reduced computational and storage requirements through advanced routing, normalization, and activation techniques.

WhileMixture-of-Experts(MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, asparse MoEarchitecture designed to match the performance ofdense Transformersunder identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexibleReLU-based routingenhanced bylearnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduceNormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expertactivation ratioand a higherintrinsic sparsitylevel. We also identify an empirical advantage in using non-gated MLP expertswithReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of experts, matches dense performance and outperforms established MoE baselines. Ourspecialized acceleration kerneldelivers a 3.00times speedup on real hardware compared with dense inference. Codes and checkpoints will be released.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.10933

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.10933 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.10933 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.10933 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Mixture of Experts (MoEs) in Transformers

Hugging Face Blog

Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.

MobileMoE: Scaling On-Device Mixture of Experts

Hugging Face Daily Papers

MobileMoE introduces efficient on-device mixture-of-experts language models with sub-billion parameters, achieving better performance and efficiency than dense baselines and existing MoE models. The models are trained on open-source datasets and demonstrate significant speedups on commodity smartphones.

EMO: Pretraining Mixture of Experts for Emergent Modularity

Hugging Face Daily Papers

EMO is a Mixture-of-Experts model that enables modular deployment by grouping similar domain tokens with shared experts, achieving performance comparable to standard MoEs while allowing significant expert pruning (25% experts retain 99% performance) without performance degradation.

Decoupled Mixture-of-Experts for Parametric Knowledge Injection

arXiv cs.CL

Decoupled Mixture-of-Experts (DMoE) proposes a modular architecture for parametric knowledge injection, decoupling experts and router from the base model to enable efficient auto-regressive inference and mitigate catastrophic forgetting.