VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

Hugging Face Daily Papers Papers
video-generation representation-autoencoder video-foundation-model generative-modeling diffusion-transformer autoregressive-model latent-space

Summary

This paper introduces VideoRAE, a representation autoencoder that leverages frozen video foundation models to create compact, reconstruction-capable, and generation-friendly video latents. It achieves state-of-the-art results on UCF-101 with faster convergence than competing autoencoders.

Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:39 AM

Paper page - VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

Source: https://huggingface.co/papers/2607.14088

Abstract

Videogenerativemodelscommonlyrelyonlatentspaceslearnedby3DVariationalAutoencoders(3D-VAEs).However,conventional3D-VAEsaremainlyoptimizedforpixel-levelreconstruction,whichcanlimitthesemanticandspatio-temporalstructurecapturedbytheirlatents.Meanwhile,VideoFoundationModels(VFMs)suchasV-JEPA2andVideoMAEv2showstrongvideounderstandingcapabilities,yetwhethertheirfrozenrepresentationscanbetransformedintocompact,reconstruction-capable,andgeneration-friendlyvideolatentsremainslargelyunexplored.WeanswerthisquestionwithVideoRAE,arepresentationautoencoderthatleveragesmulti-scalehierarchicalfeaturesfromafrozenvideofoundationencoderandcompressesthemwithalightweight1Dself-attentionprojector.VideoRAEsupportsbothcontinuouslatentsforDiffusionTransformersanddiscretetokensforautoregressivemodelsviamulti-codebookhigh-dimensionalquantization.Duringdecoding,alocal-and-globalrepresentationalignmentobjectivewiththefrozenVFMteacherimprovessemanticpreservationandenablestrainingwithoutKLregularization.ExperimentsshowthatVideoRAEachievesstrongreconstructioninbothcontinuousanddiscreteregimes.OnUCF-101,itobtainsstate-of-the-artclass-to-videogFVDsof40and93withARandDiTgenerators,respectively,whileconvergingapproximately5xfasterthancompetingautoencoderbaselines.Inacontrolled2B-scaletext-to-videostudy,replacingLTX-VAEwithVideoRAEleadstofasterconvergenceundercomparablesettings.TheseresultsvalidatefrozenVFMrepresentationsasversatileandgeneration-friendlyvideolatents.Themodelandcodewillbereleasedonhttps://zhxie0117.github.io/VideoRAE.

View arXiv pageView PDFProject pageGitHub33Add to collection

Get this paper in your agent:

hf papers read 2607\.14088

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.14088 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.14088 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.14088 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Video Generators as General-Purpose Vision Models (8 minute read)

TLDR AI

GenCeption repurposes pre-trained video generative models into a single unified feed-forward vision model that achieves state-of-the-art performance across multiple tasks with exceptional data efficiency, marking a shift toward general-purpose visual intelligence.

Video Generation Models are General-Purpose Vision Learners

Hugging Face Daily Papers

This paper proposes that large-scale text-to-video generation can serve as a powerful pre-training paradigm for computer vision, introducing GenCeption which achieves state-of-the-art performance across diverse vision tasks with high data efficiency and emergent generalization to unseen domains.

Training Video Foundation Models with NVIDIA NeMo

Papers with Code Trending

This paper presents a scalable open-source pipeline using NVIDIA NeMo for training and inference of Video Foundation Models, addressing challenges in generating high-quality videos with accelerated dataset curation and parallelized training.

Video Models Can Reason with Verifiable Rewards

Hugging Face Daily Papers

VideoRLVR optimizes video diffusion models for verifiable reasoning tasks using reinforcement learning with rule-based rewards, achieving better performance than supervised methods in constraint-satisfying video generation.

Long Video Generation (4 minute read)

TLDR AI

The article introduces A²RD, a novel architecture for generating consistent long videos using agentic autoregressive diffusion. It proposes a Retrieve–Synthesize–Refine–Update cycle and a new benchmark, LVBench-C, to address semantic drift in long-horizon video synthesis.