MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

Hugging Face Daily Papers Papers
mixture-of-experts vision-encoder efficient-scaling image-understanding video-understanding ai-research

Summary

This paper presents MoE-ViE, a mixture-of-experts vision encoder that efficiently scales for image and video understanding, outperforming larger dense models with lower inference latency.

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Original Article
View Cached Full Text

Cached at: 08/19/26, 11:58 AM

Paper page - MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

Source: https://huggingface.co/papers/2608.17402 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Mixture-of-Experts vision encoders with fine-grained topologies, auxiliary-loss-free balancing, and specialized kernels scale efficiently while outperforming larger dense models on image and video tasks.

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost andinference latency.Mixture-of-Experts(MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet theMoEdesign space forCLIP-stylevision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically studyMoEdesigns forvision encoderscaling and find thatfine-grained MoEtopologies yield substantial gains over both dense and standardMoEcounterparts. We further propose anauxiliary-loss-free balancingvariant for betterexpert utilization, and design a specializedMoE kernelto mitigateinference latencyoverhead. To enhance video capabilities while preserving image knowledge, we introduceframe-level distillationpaired with a novelfreezing mechanism. We pretrain a series ofMixture-of-ExpertsVision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM,MoE-ViEsurpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2608\.17402

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper3

#### facebook/MoEViE-H14-448 Zero-Shot Image Classification• Updatedabout 11 hours ago • 9 • 2 #### facebook/MoEViE-L16-384 Zero-Shot Image Classification• Updatedabout 11 hours ago • 7 • 2 #### facebook/MoEViE-B16-224 Zero-Shot Image Classification• Updatedabout 11 hours ago • 8 • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.17402 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.17402 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Stateful Visual Encoders for Vision-Language Models

Hugging Face Daily Papers

This paper introduces a stateful visual encoder for vision-language models that conditions visual representations on prior features, enabling better visual comparison in multi-image and agentic settings. The method shows consistent improvements across tasks such as cross-image spatial aggregation and longitudinal radiology.

Mixture of Experts (MoEs) in Transformers

Hugging Face Blog

Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.

MobileMoE: Scaling On-Device Mixture of Experts

Hugging Face Daily Papers

MobileMoE introduces efficient on-device mixture-of-experts language models with sub-billion parameters, achieving better performance and efficiency than dense baselines and existing MoE models. The models are trained on open-source datasets and demonstrate significant speedups on commodity smartphones.