Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

Hugging Face Daily Papers Papers
multimodal foundation-model streaming codec tokenizer real-time efficiency

Summary

Mage-VL is an efficient codec-native streaming multimodal foundation model that reduces visual token consumption by over 75% using a custom tokenizer, achieving up to 3.5x inference speedup while matching or outperforming existing models on static and video tasks.

Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.
Original Article
View Cached Full Text

Cached at: 07/29/26, 03:50 AM

Paper page - Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

Source: https://huggingface.co/papers/2607.24904 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Standardvision-languagemodels(VLMs)sufferfromMoravec’sparadox:theyexcelatcomplexofflinevisualreasoningbutstrugglewithsimplestreamingperceptiontasksandprocesstheminefficiently.WepresentMage-VL,anefficientcodec-nativestreamingfoundationmodelforreal-timemultimodalunderstandingandinteraction.Atitscore,ourcustomtokenizer,Mage-ViT,replacesuniformframesamplingbyselectivelyencodingdynamic,entropy-richregionsusingmotionvectorsandresidualenergyacrosssparseanchor(I)andpredicted(P)frames.Operatingata16x16patchlevel,thisreducesvisualtokenconsumptionbyover75%whilepreservingspatiotemporalcontext.Trainedfromscratchonapproximately560Munlabeledimagesand100Munlabeledvideoframes,Mage-ViTmatchesoroutperformsflagshipencoderstrainedonbillionsofimage-textpairs.WeestablishAI4AIdatapipelinesencompassingprompt-codejointoptimizationformultimodalcaptioningandAI-drivenperformancediagnosistoguidetrainingrecipes.Furthermore,throughabio-inspireddual-systemarchitecture-alightweightSystem1eventgateandacausalSystem2decoder-Mage-VLenablesproactivestreamingperception.ExtensiveevaluationsshowthatMage-VL-4BmatchesQwen3-VL-4Bonstatictaskswhileachievingstronggainsinvideounderstandingand2D/3Dspatialreasoning,withuptoa3.5xwall-clockinferencespeedup,andcomprehensivelysurpassesthe15BPhi-4-reasoning-visionbaseline.Beyondmodelartifacts,wedeliversevenkeyempiricalfindingscoveringpre-trainingdataefficiency,variable-resolutionscaling,codecsystemacceleration,VideoQASFTredundancy,motion-spatialsynergy,AI4AIdatapipelines,andZero-VisionSFTformultimodalRL.

View arXiv pageView PDFProject pageGitHub776Add to collection

Get this paper in your agent:

hf papers read 2607\.24904

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### microsoft/Mage-VL Image-Text-to-Text• 5B• Updatedabout 1 hour ago • 702 • 62

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.24904 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Mage (GitHub Repo)

TLDR AI

Microsoft releases Mage, a family of lightweight 4B-parameter multimodal models for visual understanding and generation, including Mage-VL for image/video understanding and Mage-Flow for text-to-image generation and editing, designed for research and deployment on modest hardware.

AdaCodec: A Predictive Visual Code for Video MLLMs

Hugging Face Daily Papers

AdaCodec reduces video encoding redundancy in multimodal LLMs by transmitting full visual tokens only when scene prediction fails, otherwise using compact inter-frame change descriptions. It outperforms per-frame RGB baselines at matched token budgets and achieves better or comparable results with significantly fewer tokens, reducing time-to-first-token from 9.26s to 1.62s.