Toward Native Multimodal Modeling: A Roadmap
Summary
This paper presents a formal roadmap for transitioning from late-fusion multimodal approaches to native multimodal modeling (NMM) within a unified transformer framework, categorizing existing models by input-output duality and systematically addressing architectural coordination, data curation, training recipes, and evaluation.
View Cached Full Text
Cached at: 05/26/26, 06:43 AM
Paper page - Toward Native Multimodal Modeling: A Roadmap
Source: https://huggingface.co/papers/2605.25343 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Native multimodal modeling advances beyond traditional fusion approaches by integrating modalities inherently within a unified transformer framework, enabling seamless understanding and generation across diverse input-output configurations.
Multimodal modelingrepresents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely onlate-fusionthat assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm towardnative multimodal modeling(NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define thearchitectural nativity, distinguishingmid-fusionandearly-fusionfrom non-native paradigms. We further organize the existing native models through the lens ofinput-output dualityinto three categories: (i) Multi-to-Text forcross-modal comprehensionwith text-only output; (ii) Multi-to-Target forscenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi forunified modelingwith symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unifiedtransformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2605\.25343
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.25343 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.25343 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.25343 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Scaling Native Multimodal Pre-Training From Scratch
This paper investigates the scaling properties of native multimodal pre-training, deriving compute-optimal model sizes and token counts under a fixed budget, and revealing distinct scaling behaviors for language and multimodal objectives.
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
This survey paper systematically reviews the paradigm evolution of unified vision-language perception in multimodal large language models (MLLMs), proposing a five-stage taxonomy and identifying open challenges toward general multimodal intelligence.
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
This paper systematically explores the physics of multimodal pretraining, uncovering how knowledge flows between modalities, conditions for synergy vs. competition, the benefits of early unification, and efficient pretraining recipes validated on 13.5B MoE models.
Large Language Models as Unified Multimodal Learners for Clinical Prediction
The paper proposes converting multimodal patient data (text, labs, vitals) into a single natural language sequence and fine-tuning LLMs for clinical prediction, achieving comparable or better performance than specialized fusion architectures across three tasks.
How Modalities Learn Together (49 minute read)
A systematic study from Meta FAIR, Reality Labs, and Oxford on multimodal pretraining, revealing asymmetric knowledge flow between modalities, synergy vs. competition dynamics, the benefits of early unification, and efficient training recipes validated with 13.5B MoE models.