Scaling Native Multimodal Pre-Training From Scratch

Hugging Face Daily Papers Papers
multimodal-pre-training scaling-laws vision-language compute-optimal cross-modal-transfer

Summary

This paper investigates the scaling properties of native multimodal pre-training, deriving compute-optimal model sizes and token counts under a fixed budget, and revealing distinct scaling behaviors for language and multimodal objectives.

Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.
Original Article
View Cached Full Text

Cached at: 07/27/26, 05:40 AM

Paper page - Scaling Native Multimodal Pre-Training From Scratch

Source: https://huggingface.co/papers/2607.22043

Abstract

Althoughlargelanguagemodels(LLMs)exhibitremarkablereasoningcapabilities,theirrelianceontext-onlypre-trainingrestrictstheperceptionofthemultimodalphysicalworld.Nativemultimodalpre-trainingavoidsthislimitationbytrainingmodelsfromscratchonmultimodalinputs,therebyachievingdeepcross-modalintegrationandmitigatingoptimizationasymmetriesinherenttotraditionallate-fusionarchitectures.Despitetheseadvantages,thescalingpropertiesofthisparadigmremainsystematicallyuncharacterized.Toaddressthisgap,weinvestigatetheoptimalmodelsizeandtokencountfortrainingatransformer-basedvision-languagemodelunderafixedcomputationalbudget.Wedemonstratethatminimalobjectivelossadherestoapredictablecomputelaw,whereascompute-optimalmodelsizesandtokencountsscaleaspowerlaws.Notably,languageandmultimodalobjectivesmanifestdistinctscalingbehaviors.Thelanguageallocationlawislargelyinvarianttothecompositionofthedata,indicatingstablelanguagelearningregardlessofthemultimodaldataratio.Conversely,themultimodalallocationlawishighlysensitivetothiscomposition.Specifically,text-heavymixturesbecomecompute-efficientonlyatlargermodelscales,shiftingtheoptimalresourceallocationtowardgreatermodelcapacity.Additionally,bymodelingtheinfluenceofdatacompositiononcomputelawsandallocationexponents,wederiveanefficiencyfrontierspecifyingpreciseconfigurationsofmodelsize,tokencount,anddatamixture.Downstreamevaluationsfurtherrevealthatnativemultimodalpre-traininginducespositivecross-modaltransfer,therebyenhancingpure-textspatialreasoningandenablingrobustmultimodalin-contextlearning.Insummary,thisempiricalresearchestablishestheessentialgroundworkforpredictablyscalingmultimodalfoundationmodels.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.22043

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.22043 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.22043 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.22043 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Scaling laws for neural language models

OpenAI Blog

Foundational empirical study demonstrating power-law scaling relationships between language model performance and model size, dataset size, and compute budget, with implications for optimal training allocation and sample efficiency.

How Modalities Learn Together (49 minute read)

TLDR AI

A systematic study from Meta FAIR, Reality Labs, and Oxford on multimodal pretraining, revealing asymmetric knowledge flow between modalities, synergy vs. competition dynamics, the benefits of early unification, and efficient training recipes validated with 13.5B MoE models.

Toward Native Multimodal Modeling: A Roadmap

Hugging Face Daily Papers

This paper presents a formal roadmap for transitioning from late-fusion multimodal approaches to native multimodal modeling (NMM) within a unified transformer framework, categorizing existing models by input-output duality and systematically addressing architectural coordination, data curation, training recipes, and evaluation.