MoTE: Mixture of Task Experts for Multi-Task Video Understanding

Hugging Face Daily Papers Papers
mixture-of-experts multi-task-learning video-understanding video-language-models transformer-decoders sparse-computation

Summary

MoTE introduces task-specific expert routing to replace dense decoder feed-forward networks in multi-task video understanding, improving accuracy and efficiency with interpretable, sparse computation.

Procedural video-language models must solve heterogeneous tasks from the same visual evidence, including action recognition, forecasting, and procedure prediction. Dense transformer decoders share the same feed-forward networks across tasks, which can entangle task behavior and make controlled capability expansion difficult. Sparse Mixture-of-Experts (MoE) decoders provide conditional computation, but token-level learned routing is not naturally aligned with task-level procedural objectives. We propose MoTE (Mixture of Task Experts), a decoder architecture that converts large language model feed-forward networks into task-specific experts while keeping the multimodal backbone shared. Each example follows one sample-level task route, so active task-expert computation remains independent of the number of stored task experts. We instantiate this design as VideoLLM-MoTE and evaluate it on five COIN benchmarks using explicit task routes. The five-expert model activates ~2B LLM parameters per sample and achieves higher average top-1 accuracy than recent VideoLLM baselines. Under the same expert topology, it improves over dense all-expert activation and learned sparse-routing controls. These results show that task-structured routing provides an interpretable and compute-efficient decoder alternative for multi-task video-language learning.
Original Article
View Cached Full Text

Cached at: 08/26/26, 03:17 PM

Paper page - MoTE: Mixture of Task Experts for Multi-Task Video Understanding

Source: https://huggingface.co/papers/2608.24763

Abstract

MoTE replaces dense decoder feed-forward networks with task-specific experts routed by sample-level task labels, improving multi-task video-language accuracy with sparse, interpretable computation.

Procedural video-languagemodels must solve heterogeneous tasks from the same visual evidence, including action recognition, forecasting, and procedure prediction.Dense transformer decodersshare the samefeed-forward networksacross tasks, which can entangle task behavior and make controlled capability expansion difficult. SparseMixture-of-Experts(MoE) decoders provide conditional computation, but token-level learned routing is not naturally aligned with task-level procedural objectives. We proposeMoTE(Mixture of Task Experts), a decoder architecture that converts large language modelfeed-forward networksintotask-specific expertswhile keeping themultimodal backboneshared. Each example follows one sample-level task route, so active task-expert computation remains independent of the number of stored task experts. We instantiate this design asVideoLLM-MoTEand evaluate it on five COIN benchmarks using explicit task routes. The five-expert model activates ~2B LLM parameters per sample and achieves higher average top-1 accuracy than recent VideoLLM baselines. Under the same expert topology, it improves over dense all-expert activation and learned sparse-routing controls. These results show that task-structured routing provides an interpretable and compute-efficient decoder alternative for multi-task video-language learning.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.24763

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.24763 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.24763 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.24763 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MTP on MoE matters

Reddit r/LocalLLaMA

This paper likely discusses the application of Multi-Task Prompting (MTP) to Mixture of Experts (MoE) models, exploring how MTP can improve performance or efficiency in MoE architectures.

Mixture of Experts (MoEs) in Transformers

Hugging Face Blog

Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.