Training Video Foundation Models with NVIDIA NeMo
Summary
This paper presents a scalable open-source pipeline using NVIDIA NeMo for training and inference of Video Foundation Models, addressing challenges in generating high-quality videos with accelerated dataset curation and parallelized training.
View Cached Full Text
Cached at: 06/29/26, 12:23 PM
Paper page - Training Video Foundation Models with NVIDIA NeMo
Source: https://huggingface.co/papers/2503.12964 Published on Mar 17, 2025
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
A scalable open-source pipeline using NVIDIA NeMo for training and inference of Video Foundation Models addresses challenges in generating high-quality videos.
Video Foundation Models(VFMs) have recently been used to simulate the real world to train physical AI systems and develop creative visual experiences. However, there are significant challenges in training large-scale, high qualityVFMs that can generate high-quality videos. We present a scalable, open-sourceVFMtraining pipeline withNVIDIA NeMo, providing accelerated video dataset curation,multimodal data loading, and parallelizedvideo diffusion modeltraining and inference. We also provide a comprehensiveperformance analysishighlighting best practices for efficientVFMtraining and inference.
View arXiv pageView PDFGitHub17.6kAdd to collection
Get this paper in your agent:
hf papers read 2503\.12964
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2503.12964 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2503.12964 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2503.12964 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers
NVIDIA NeMo Automodel integrates with Hugging Face Diffusers to enable scalable distributed fine-tuning of diffusion models for image and video generation, supporting models like FLUX.1-dev, Wan 2.1, and HunyuanVideo.
Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel
NVIDIA NeMo AutoModel leverages HuggingFace Transformers v5 to deliver 3.4-3.7x higher training throughput and 29-32% less GPU memory for fine-tuning Mixture-of-Experts models, with no code changes beyond a single import.
@PyTorch: Foundation models are reshaping computational biology. Adapting models to a specific task is nontrivial, so to reduce t…
NVIDIA BioNeMo Recipes provide step-by-step, parameter-efficient fine-tuning recipes for biological foundation models using LoRA, demonstrated on two case studies with ESM2-3B and Evo2-1B on a single workstation GPU.
Nvidia released Nemotron 3.5 Super VL (5 minute read)
NVIDIA 发布 Nemotron 3.5 Super VL，一款 120B 参数的多模态混合专家（MoE）模型，每 token 激活 12B 参数，面向企业级智能体工作流，支持文本、图像、文档和视频理解，单 GPU 上吞吐量较同类开源模型提升高达 5 倍，并可通过 NeMo 进行后训练定制。
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
This paper introduces VideoRAE, a representation autoencoder that leverages frozen video foundation models to create compact, reconstruction-capable, and generation-friendly video latents. It achieves state-of-the-art results on UCF-101 with faster convergence than competing autoencoders.