Beyond Pixels: From Video Priors to 4D Worlds
Summary
This paper introduces Latent-to-4D, a method for direct 4D scene generation from video diffusion latents without retraining across generators, achieving better geometry and temporal stability than cascaded approaches.
View Cached Full Text
Cached at: 08/12/26, 08:21 AM
Paper page - Beyond Pixels: From Video Priors to 4D Worlds
Source: https://huggingface.co/papers/2608.10744
Abstract
Latent-to-4D enables reusable direct 4D generation from video diffusion latents via alignment with a pretrained decoder and spatiotemporal attention, transferring across generators without retraining.
4D generationsynthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share avariational autoencoder(VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce directlatent-to-4Dgeneration and instantiate it asLatent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained4D decoderand refining it through frame-wise and globalspatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiplevideo diffusion transformerswithin the same VAE family. On Text4D-200 and I4D-200,Latent-to-4Dsurpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2608\.10744
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.10744 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.10744 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.10744 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Latent Spatial Memory for Video World Models
This paper introduces latent spatial memory for video world models, storing 3D scene information directly in diffusion latent space to avoid costly pixel-space reconstruction. The proposed Mirage framework achieves up to 10.57x faster generation and 55x memory reduction while achieving state-of-the-art performance on WorldScore and RealEstate10K.
L2P: Unlocking Latent Potential for Pixel Generation
The L2P paper introduces a Latent-to-Pixel transfer paradigm that leverages pre-trained latent diffusion models to create efficient pixel-space models capable of 4K generation with minimal training overhead.
Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
Introduces Latent Dynamics Reasoning (LDR), a video world model that integrates kinematic dynamics in a structured latent space, enabling extrapolation of learned dynamics far beyond training distributions while using far fewer parameters and running much faster than video diffusion baselines.
Pantheon360: Taming Digital Twin Generation via 3D-Aware 360° Video Diffusion
Pantheon360 introduces a 3D-aware 360° video diffusion framework that uses an explicit 3D cache to enforce geometric consistency, enabling high-fidelity digital twin generation from sparse 360° inputs.
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
4DAnyone reconstructs 4D humans from monocular video by generating multiview-consistent videos and lifting them into 4D Gaussian Splatting, using reference and target context designs to address scaling bottlenecks.