V-RAE: Rethinking Video Latent Spaces for Generation

Hugging Face Daily Papers Papers
video-generation autoencoder latent-spaces vision-foundation-model temporal-pooling semantic-representation

Summary

V-RAE proposes a video representation autoencoder that builds semantically organized latents from frozen vision representations to enhance video generation quality, convergence speed, and predictive modeling.

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.
Original Article
View Cached Full Text

Cached at: 08/19/26, 11:57 AM

Paper page - V-RAE: Rethinking Video Latent Spaces for Generation

Source: https://huggingface.co/papers/2608.13556

Abstract

V-RAE constructs semantically organized video latents from frozen vision representations to improve generation quality, convergence speed, and predictive modeling.

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We proposeV-RAE, avideo representation autoencoderthat builds compact generative latents on top offrozen vision foundation modelrepresentations. A lightweighttemporal poolingmodule removes temporal redundancy while preserving semantic structure, and avideo decoderreconstructs continuous motion from the compressed features. We evaluateV-RAEwith four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation.V-RAEachieves 2.13rFVDon K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventionalvideo tokenizerlatents. Under matched generation settings, our best variant achievesgFVDscores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introducetFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation,V-RAEalso improvesfuture video predictionon Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.

View arXiv pageView PDFProject pageGitHub34Add to collection

Get this paper in your agent:

hf papers read 2608\.13556

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### Guomh0707/V-RAE-Models Video-to-Video• Updated4 days ago • 6

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.13556 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.13556 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

LightVAE: Towards Compact and Efficient Video Autoencoders

Hugging Face Daily Papers

LightVAE 提出一种视频自编码器压缩框架，通过对因果 Conv3D 层的精确重参数化，将空间响应与时间修正分离，并对时间修正施加低秩压缩，从而在保持生成质量的同时显著加速视频 VAE 解码（在 WanVAE 上最高达 2.24 倍吞吐提升）。

Variational lossy autoencoder

OpenAI Blog

OpenAI researchers present a Variational Lossy Autoencoder (VLAE) that combines VAEs with neural autoregressive models (RNN, MADE, PixelRNN/CNN) to learn controllable global representations, achieving state-of-the-art results on MNIST, OMNIGLOT, and Caltech-101 Silhouettes density estimation tasks.

Long Video Generation (4 minute read)

TLDR AI

The article introduces A²RD, a novel architecture for generating consistent long videos using agentic autoregressive diffusion. It proposes a Retrieve–Synthesize–Refine–Update cycle and a new benchmark, LVBench-C, to address semantic drift in long-horizon video synthesis.