L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Hugging Face Daily Papers Papers
world-generation 3d-gaussians video-diffusion interactive-worlds spatial-consistency real-time memory

Summary

L3G-World 提出了一种紧凑的潜在 3D 高斯记忆（L3G），与视频扩散 Transformer（DiT）交替进行自回归更新，以实现实时、长时程空间一致的交互式无限世界生成，在 9 FPS 下实现了最先进生成质量与显著的效率提升。

Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to 38.6times fewer 3D primitives, 3.2times faster memory updates, and 359times faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.
Original Article
View Cached Full Text

Cached at: 10/30/26, 11:10 AM

Paper page - L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Source: https://huggingface.co/papers/2610.40614

Abstract

L3G-World uses a compact latent 3D Gaussian memory and alternating autoregressive updates with a video diffusion transformer to generate spatially consistent interactive worlds in real time.

Interactive world models generate visual content conditioned on user actions, yet maintainingspatial consistencyover long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce aLatent 3D Gaussian Memory(L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we proposeL3G-World, in which a memory updater and avideo diffusion transformer(DiT) alternateautoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores aVAE latent feature, allowing the memory to be rendered directly into theDiT’s latent space without VAE re-encoding. Experiments show thatL3G-Worldachieves state-of-the-art generation quality and long-horizonspatial consistency, with up to 38.6times fewer 3D primitives, 3.2times faster memory updates, and 359times faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.

View arXiv pageView PDFProject pageGitHub13Add to collection

Get this paper in your agent:

hf papers read 2610\.40614

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.40614 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.40614 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.40614 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Latent Spatial Memory for Video World Models

Hugging Face Daily Papers

This paper introduces latent spatial memory for video world models, storing 3D scene information directly in diffusion latent space to avoid costly pixel-space reconstruction. The proposed Mirage framework achieves up to 10.57x faster generation and 55x memory reduction while achieving state-of-the-art performance on WorldScore and RealEstate10K.

4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling

Hugging Face Daily Papers

4LC 提出一种用于动态世界建模的 4D 潜空间缓存，通过复合视觉表示（VAE、DINO、V-JEPA、VGGT）和灵活的时空访问接口，高效复用静态与动态内容作为生成条件，支持相机控制、时间控制、新视角合成与长时生成等多项视频生成任务，且无需任务特定微调。

World Machine: Towards Generative World Modeling for Time-Series

arXiv cs.LG

World Machine proposes a transformer-based generative world modeling architecture for time series that uses latent states to adapt to varying context lengths, addressing the quadratic memory cost of traditional transformers. Experiments on a synthetic dataset validate its feasibility and show improvements over conventional transformers.