Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Hugging Face Daily Papers Papers
multimodal-pretraining knowledge-flow modality-synergy early-unification mixture-of-experts research

Summary

This paper systematically explores the physics of multimodal pretraining, uncovering how knowledge flows between modalities, conditions for synergy vs. competition, the benefits of early unification, and efficient pretraining recipes validated on 13.5B MoE models.

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.
Original Article
View Cached Full Text

Cached at: 08/06/26, 05:50 AM

Paper page - Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Source: https://huggingface.co/papers/2608.05000

Abstract

Visionoffersacriticalaxisforadvancingfoundationmodels,drivingashifttowardsnativelyunifiedmultimodalpretraining.Despitethismomentum,thedesignspaceandthefundamentalmechanismsofhowmodalitiesinteractduringunifiedtrainingremainunderexplored.Weprovideempiricalclaritythroughasystematicexplorationofmultimodalpretraining.Ourcontrolledexperimentsonbothsyntheticandlarge-scalereal-worlddatasetsyieldfourkeyinsightsintothephysicsofmultimodalpretraining:(i)KnowledgeFlow:Wedisentanglehowlanguage,visualunderstanding,andvisualgenerationtransferknowledgeacrossmodalities,revealingdistinctpatternsofinfluenceandasymmetry;(ii)Synergyvs.Competition:Weshowthatdata“complexity“largelydetermineswhethermodalitiesaresynergistic,identifyarchitecturalchoicesthatpromotesynergy:suchassharedattentionandnormalizationwithmodality-specificfeed-forwardlayers,andfindthatthesebehaviorsgeneralizeacrossdifferentvisualtokenizerdesigns;(iii)EarlyUnification:Unifyingmodalitiesfromtheveryearlystagesandtrainingthemjointlyisshowntobemoreeffectivethanlatealignmentorsequentialtraining.Thisprocessuncoversavisionlazinessphenomenon,wheredelayedintegrationleadsmodelstorelyonlanguagepriors;(iv)Recipes:Wederiveefficientpretrainingrecipesthatachievestronggenerativeperformanceusingonly5%ofthecomputebudget.Thesecorefindingsaresubsequentlyvalidatedatscalebytrainingmultiple13.5BMoEmodelson2Ttokens.Wehopethisstudyprovidesaprincipledfoundationforunderstandingandscalingmultimodalpretraining.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.05000

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.05000 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.05000 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.05000 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

How Modalities Learn Together (49 minute read)

TLDR AI

A systematic study from Meta FAIR, Reality Labs, and Oxford on multimodal pretraining, revealing asymmetric knowledge flow between modalities, synergy vs. competition dynamics, the benefits of early unification, and efficient training recipes validated with 13.5B MoE models.

Scaling Native Multimodal Pre-Training From Scratch

Hugging Face Daily Papers

This paper investigates the scaling properties of native multimodal pre-training, deriving compute-optimal model sizes and token counts under a fixed budget, and revealing distinct scaling behaviors for language and multimodal objectives.

Toward Native Multimodal Modeling: A Roadmap

Hugging Face Daily Papers

This paper presents a formal roadmap for transitioning from late-fusion multimodal approaches to native multimodal modeling (NMM) within a unified transformer framework, categorizing existing models by input-output duality and systematically addressing architectural coordination, data curation, training recipes, and evaluation.