Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoE
Summary
SharpMoE is a post-training framework that improves routing in diffusion mixture-of-experts models by using clean latent features to identify salient tokens and a trajectory routing loss to allocate compute precisely, achieving state-of-the-art visual generation.
View Cached Full Text
Cached at: 06/30/26, 03:33 AM
Paper page - Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoE
Source: https://huggingface.co/papers/2606.26938
Abstract
SharpMoE addresses routing inefficiencies in diffusion models by using clean latent features to guide salient token identification and employs trajectory routing loss for precise compute allocation during multi-step denoising.
Mixture-of-Experts(MoE) architectures have emerged as a powerful paradigm for scalingdiffusion modelsinvisual generation. Recent advancements have focused on adaptively allocating computational resources across diverse tokens to improve efficiency and performance. However, we identify arouting assignmentproblem in existing diffusion MoE frameworks: the router fails to accurately allocate more computational resources tosalient tokens. Our analysis attributes this failure to the router’s reliance on noise-corruptedlatent featuresthroughout thedenoising process. Such stochastic noise obscures the critical structural and textural information, thereby preventing the router from effectively distinguishingsalient tokens. To address this, we propose SharpMoE, apost-training frameworkwith a saliency-harnessing accurate routing mechanism, which utilizes cleanlatent featuresas a noise-free guidance signal for routing. By bypassing the noise-distorted inputs, SharpMoE provides the router with clear saliency guidance, enabling the identification ofsalient tokenseven in high-noise stages. Furthermore, we introduce atrajectory routing lossto constrain thecompute allocationthroughout the multi-step denoising trajectory, ensuring precise resource allocation along the generation rollout. Extensive experiments demonstrate that SharpMoE serves as a versatile, plug-and-play solution that further enhances the pretrained, converged MoE models, achieving state-of-the-art performance invisual generation.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.26938
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.26938 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.26938 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.26938 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Sticky Routing: Training MoE Models for Memory-Efficient Inference
StickyMoE proposes a differentiable routing consistency loss that encourages adjacent tokens to activate the same experts in MoE models, reducing expert-swapping overhead and cache misses during inference on edge devices by up to 3.92× while improving perplexity.
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
This paper proposes UniF-MoE, a unified framework for token-adaptive Mixture-of-Experts computation that first shares reusable computation across experts and then routes the remaining residual demand, improving performance while reducing activated computation, latency, and memory on DomainBed and GLUE benchmarks.
Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
This paper introduces elbow-based routing, a training-free inference-time method for MoE models that dynamically adjusts the number of active experts per token by detecting the elbow point in router probability distributions, achieving a 5.3% average latency reduction while maintaining accuracy.
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
Mix-MoE proposes a mixed Mixture-of-Experts framework with specialized expert groups and Fourier-transform-enhanced routing to mitigate parameter interference in multilingual machine translation, achieving significant improvements over baselines.
MoTE: Mixture of Task Experts for Multi-Task Video Understanding
MoTE introduces task-specific expert routing to replace dense decoder feed-forward networks in multi-task video understanding, improving accuracy and efficiency with interpretable, sparse computation.