Latent Spatial Memory for Video World Models

Hugging Face Daily Papers Papers
video-world-models latent-spatial-memory diffusion-latent-space 3d-scene-representation efficient-generation depth-guided-warping

Summary

This paper introduces latent spatial memory for video world models, storing 3D scene information directly in diffusion latent space to avoid costly pixel-space reconstruction. The proposed Mirage framework achieves up to 10.57x faster generation and 55x memory reduction while achieving state-of-the-art performance on WorldScore and RealEstate10K.

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to 10.57times faster end-to-end video generation and 55times reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.
Original Article
View Cached Full Text

Cached at: 06/09/26, 08:43 AM

Paper page - Latent Spatial Memory for Video World Models

Source: https://huggingface.co/papers/2606.09828

Abstract

Latent spatial memory for video world models stores 3D scene information directly in diffusion latent space, eliminating pixel-space reconstruction overhead and achieving faster generation with reduced memory usage.

Video world modelsthat maintain 3D spatial consistency across generated frames typically rely on explicitpoint cloud memoryconstructed inRGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introducelatent spatial memoryforvideo world models, a persistent 3D cache that stores scene information directly in thediffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D viadepth-guided back-projectionand queries it by synthesizing novel views through directlatent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show thatlatent spatial memoryachieves up to 10.57times fasterend-to-end video generationand 55times reduction inmemory footprintrelative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance onWorldScoreand strong reconstruction quality onRealEstate10K.

View arXiv pageView PDFProject pageGitHub14Add to collection

Get this paper in your agent:

hf papers read 2606\.09828

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.09828 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.09828 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.09828 in a Space README.md to link it from this page.

Collections including this paper3

Similar Articles

Composition of Memory Experts for Diffusion World Models

arXiv cs.LG

A new diffusion-based world model framework that uses a composition of specialized memory experts (short-term, long-term episodic, and spatial) to achieve better temporal consistency and long-context modeling without quadratic cost.