VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

Papers with Code Trending Papers
video-understanding multimodal-llm open-source efficiency generalist video-mllm spatiotemporal

Summary

VideoChat3 is a fully open, efficient, and generalist video-centric multimodal large language model that introduces Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for streaming video perception, along with scalable video data synthesis pipelines, achieving superior performance with only 4B parameters.

Recent advances in video understanding have spanned motion, long video, and streaming interaction, driving this field toward real-world applications. Despite this progress, current open-source models remain limited in several ways. They often struggle to generalize across diverse video types, making them effective only in specific domains. High computational demands further restrict their efficiency and scalability. Moreover, most models are only partially open, with key components such as training code, strategy, or datasets unavailable, which hinders reproducibility and slows community-driven development. To address these issues, we introduce VideoChat3, a fully open, efficient, and generalist video-centric MLLM. VideoChat3 advances video understanding through two complementary designs. For efficiency, we introduce Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for Streaming Video Perception, which enables efficient spatiotemporal representation and reduces the cost of processing video inputs during training and inference. For effectiveness, we develop a scalable video data synthesis pipeline that curates three diverse, high-quality training datasets: VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, covering general, long-form, and streaming video scenarios, improving the model's generalization across domains. By integrating these designs, VideoChat3 achieves a rare balance of broad generalization and computational efficiency. Experiments across general, long-form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open-source models with equal or larger parameter counts with only 4B parameters and higher efficiency.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:38 AM

Paper page - VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

Source: https://huggingface.co/papers/2607.14935 Published on Jul 16

#2 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Recentadvancesinvideounderstandinghavespannedmotion,longvideo,andstreaminginteraction,drivingthisfieldtowardreal-worldapplications.Despitethisprogress,currentopen-sourcemodelsremainlimitedinseveralways.Theyoftenstruggletogeneralizeacrossdiversevideotypes,makingthemeffectiveonlyinspecificdomains.Highcomputationaldemandsfurtherrestricttheirefficiencyandscalability.Moreover,mostmodelsareonlypartiallyopen,withkeycomponentssuchastrainingcode,strategy,ordatasetsunavailable,whichhindersreproducibilityandslowscommunity-drivendevelopment.Toaddresstheseissues,weintroduceVideoChat3,afullyopen,efficient,andgeneralistvideo-centricMLLM.VideoChat3advancesvideounderstandingthroughtwocomplementarydesigns.Forefficiency,weintroduceInflated3DVisionTransformer(I3D-ViT)andAdaptiveFrameResolutionforStreamingVideoPerception,whichenablesefficientspatiotemporalrepresentationandreducesthecostofprocessingvideoinputsduringtrainingandinference.Foreffectiveness,wedevelopascalablevideodatasynthesispipelinethatcuratesthreediverse,high-qualitytrainingdatasets:VideoChat3-Academic2M,VideoChat3-LV116K,andVideoChat3-OL617K,coveringgeneral,long-form,andstreamingvideoscenarios,improvingthemodel’sgeneralizationacrossdomains.Byintegratingthesedesigns,VideoChat3achievesararebalanceofbroadgeneralizationandcomputationalefficiency.Experimentsacrossgeneral,long-form,andstreamingbenchmarksdemonstratethatVideoChat3surpassesprioropen-sourcemodelswithequalorlargerparametercountswithonly4Bparametersandhigherefficiency.

View arXiv pageView PDFProject pageGitHub108Add to collection

Get this paper in your agent:

hf papers read 2607\.14935

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper2

#### MCG-NJU/VideoChat3-4B Video-Text-to-Text• 4B• Updated3 days ago • 342 • 13 #### MCG-NJU/I3D-ViT Image Feature Extraction• 0.4B• Updated3 days ago • 62 • 8

Datasets citing this paper3

#### MCG-NJU/VideoChat3-LV116k Viewer• Updatedabout 21 hours ago • 8.07k • 7.59k • 10 #### MCG-NJU/VideoChat3-Academic2M Viewer• Updatedabout 21 hours ago • 19.2k • 1.98k • 14 #### MCG-NJU/VideoChat3-OL617k Preview• Updated3 days ago • 280 • 8

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.14935 in a Space README.md to link it from this page.

Collections including this paper6

Browse 6 collections that include this paper

Similar Articles

Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

Hugging Face Daily Papers

Mage-VL is an efficient codec-native streaming multimodal foundation model that reduces visual token consumption by over 75% using a custom tokenizer, achieving up to 3.5x inference speedup while matching or outperforming existing models on static and video tasks.