MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
Summary
Modus is a decoder-only model that predicts any modality from any combination of others, achieving strong performance across diverse benchmarks without modality-specific heads or losses.
View Cached Full Text
Cached at: 07/29/26, 11:52 AM
Paper page - MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
Source: https://huggingface.co/papers/2607.25948 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Any-to-anymodelspredictanymodalityfromanycombinationofotherswithinasinglenetwork,aformulationusedinmultimodalvisionandvision-languagemodels,andincreasinglyinscientificdomainssuchasecologyandastronomy.Existingany-to-anymodelsaretypicallytrainedfromscratchusingencoder-decoderordiffusionarchitectures,impactingtheirperformanceandpreventingthemfromusingstrongpre-traineddecoder-onlymodelsasaprior.Inthiswork,weinvestigatedecoder-onlyany-to-anymultimodalmodeling,whichtreatsallmodalitiessymmetricallyandsupportsarbitrarymodalitiesasinputsandoutputswithoutmodality-specificheads,losses,ortaskpipelines.Becauseeverymodalityisbothaninputandanoutputofthesamemodel,theresultingmodel,namedModus,cansupportarangeofapplications,suchaschainedgenerationthroughintermediatemodalitiesorcross-modalself-verificationbyscoringthemodel’sownoutputswithanothergeneratedmodality.Modusdemonstratesstrongout-of-the-boxperformanceandiscompetitivewithspecialistandmultitaskbaselinesusingasinglemodelacrossvariousbenchmarks.Allmaterialsareopen-sourcedathttps://modus-multimodal.epfl.ch/.
View arXiv pageView PDFProject pageGitHub16Add to collection
Get this paper in your agent:
hf papers read 2607\.25948
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.25948 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.25948 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.25948 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
This paper introduces MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE multimodal LLMs that addresses biases in expert importance estimation by decomposing selection frequency by modality and filtering redundant vision tokens, achieving minimal performance loss under aggressive quantization.
How Modalities Learn Together (49 minute read)
A systematic study from Meta FAIR, Reality Labs, and Oxford on multimodal pretraining, revealing asymmetric knowledge flow between modalities, synergy vs. competition dynamics, the benefits of early unification, and efficient training recipes validated with 13.5B MoE models.
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
MACS is a training-free inference framework that mitigates the straggler effect in expert parallelism for multimodal MoE MLLMs by introducing entropy-weighted load and dynamic modality-adaptive capacity mechanisms.
Generalized Multimodal Foundation Model
This paper introduces a generalized multimodal foundation model capable of handling arbitrary modality combinations and prediction tasks, achieving competitive performance through training on large-scale synthetic datasets with diverse causal structures.
From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence
This paper proposes representing multimodal data (image, video, text) as bags of atomic propositions (e.g., 'person holding cup'), unified via a global semantic codebook, enabling interpretable, compositional, and cross-modal understanding. The framework is demonstrated on autonomous driving and open-world data.